Apache Pinot is an open-source analytics platform that makes use of an OLAP database to supply low-latency insights into giant quantities of knowledge.
OLAP stands for On-line Analytical Processing and is a technique during which knowledge from a number of sources can be utilized collectively, permitting corporations to group knowledge from web sites, purposes, inside programs, and extra collectively for evaluation.
“For instance, a retailer shops knowledge about all of the merchandise it sells, resembling shade, dimension, price, and placement. The retailer additionally collects buyer buy knowledge, such because the identify of the gadgets ordered and whole gross sales worth, in a special system. OLAP combines the datasets to reply questions resembling which shade merchandise are extra common or how product placement impacts gross sales,” AWS wrote in a put up explaining OLAP.
Key options of Apache Pinot embrace low-latency queries, the flexibility to deal with lots of of hundreds of concurrent queries per second, batch and streaming ingestion, versatile joins, wealthy indexing choices, and extra.
It was first created at LinkedIn in 2013 as a result of the corporate needed to supply its customers interactive analytics, however with the quantity of knowledge LinkedIn had already amassed at the moment, it was struggling to seek out one thing that might scale on the stage it wanted.
“Pinot was born as a solution to our issues, a web-scale real-time analytics engine designed and constructed at LinkedIn. Pinot allows us to slice, cube and scan by massively giant portions of knowledge in real-time throughout all kinds of merchandise,” stated Praveen Neppalli Naga, engineering supervisor at LinkedIn on the time, wrote in a weblog put up when the undertaking was first introduced.
It powers 25 of LinkedIn’s user-facing options resembling Who Considered My Profile, Firm Observe Analytics, Jobs Analytics, and extra, in addition to over 30 of the corporate’s inside instruments, resembling its A/B testing platform.
In 2018, Apache Pinot joined the Apache Software program Basis as an incubator undertaking and have become a top-level undertaking in 2021.
Since its creation it has been adopted by numerous main corporations, together with Robinhood, Slack, Stripe, Goal, Uber, and Walmart.
The newest launch is 1.1, which got here out in March, including options resembling vector index help and multi-stage question engine enhancements.
Trying ahead, among the issues the undertaking maintainers are engaged on in 2024 embrace making V2 on-by-default, enabling column null storing by default, full PostgreSQL compliance, pagination, and persevering with ease-of-use updates resembling improved documentation, extra person pleasant error messages, and extra.