Apache Pinot

Built by engineers at LinkedIn and Uber, Apache Pinot is a real-time distributed OLAP datastore for analytics that end users see directly, not only dashboards an analyst opens now and then. At LinkedIn it backs more than fifty user-facing products, ingesting millions of events per second and serving over a hundred thousand queries per second at millisecond latency.

Data arrives from batch sources such as Hadoop HDFS, Amazon S3, Azure ADLS and Google Cloud Storage, and from streams such as Apache Kafka, Apache Pulsar and AWS Kinesis. Batch and streaming sources combine into one table for querying, and upserts during real-time ingestion keep records consistent as they change.

The engine is column oriented, with the pieces interactive queries depend on.

• Query latency: filters and aggregations over petabyte datasets return with P90 latencies in the tens of milliseconds.
• Concurrency: hundreds of thousands of concurrent queries per second when applications query it directly.
• Pluggable indexing: timestamp, inverted, StarTree, Bloom filter, range, text, JSON and geospatial index types.
• Versatile joins: arbitrary fact to dimension and fact to fact joins across petabyte datasets.
• Multitenancy: isolated logical namespaces keep data separated and resources manageable.

Access is standard SQL, reachable from a built-in query editor and a REST API. Deployment is horizontally scalable and fault tolerant, with a Helm chart for clustered installs on Kubernetes. Pinot fits best where fast aggregations run over largely immutable data with many dimensions and metrics, and it handles time series of that shape particularly well.