Downloaded 2,795 times
Uploaded byKishore Gopalakrishna
Pinot: Realtime Distributed OLAP datastore
The document provides an overview of Pinot's architecture, operations, and future direction at LinkedIn. Key features include a SQL-like interface, real-time data loading, and columnar storage, while discussing query execution processes and fault tolerance. A comparison with Druid highlights differences in architecture and operational trade-offs.
More Related Content
![[NDC 2018] Spark, Flintrock, Airflow 로 구현하는 탄력적이고 유연한 데이터 분산처리 자동화 인프라 구축](https://cdn.slidesharecdn.com/ss_thumbnails/jparktemp-180424105624-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Similar to Pinot: Realtime Distributed OLAP datastore
![UiPath Automation Suite Installation (Hands-On) [2/3]](https://cdn.slidesharecdn.com/ss_thumbnails/automationsuitecommunitysession2-251015095633-a6d862f1-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Pinot: Realtime Distributed OLAP datastore
- 1.
- 2. Agenda • Pinot @LinkedIn - Current • Pinot - Architecture • Pinot Operations • Pinot @ LinkedIn - Future Tuesday, August 18, 15
- 3.
- 4. Slice and DiceMetrics Tuesday, August 18, 15
- 5. Pinot @ LinkedIn CustomersMembers Internal tools Tuesday, August 18, 15
- 6. • 100B documents •1B documents ingested per day • 100M queries per day • 10’s of ms latency • 30 tables in prod, 250 * 3 std app nodes Pinot @ LinkedIn Tuesday, August 18, 15
- 7. Key features SQL-like interface Columnar storage and indexing Real-time dataload Tuesday, August 18, 15
- 8. (S)QL: Filters andAggs SELECT count(*) FROM companyFollowHistoricalEvents WHERE entityId = 121011 AND 'day' >= 15949 AND 'day' <= 15963 AND paid = 'y’ AND action = 'stop' Tuesday, August 18, 15
- 9. (S)QL: Group By SELECTcount(*) FROM companyFollowHistoricalEvents WHERE entityId = 121011 AND 'day' >= 15949 AND 'day' <= 15963 AND paid = 'y’ GROUP BY action Tuesday, August 18, 15
- 10. (S)QL: ORDER BYand LIMIT SELECT * FROM companyFollowHistoricalEvents WHERE entityId = 121011 AND entityId = 1000 AND action = 'start' ORDER BY creationTime DESC LIMIT 1 Tuesday, August 18, 15
- 11. Whats not supported •JOIN: unpredictable performance • NOT A SOURCE OF TRUTH • Mutation Tuesday, August 18, 15
- 12. Pinot • Data flow •Query Execution • How to use/operate • Pinot @ LinkedIn - Future Tuesday, August 18, 15
- 13.
- 14. Pinot • Pinot segments Tuesday,August 18, 15
- 15. Pinot Segment layout:Columnar storage Tuesday, August 18, 15
- 16. Pinot Segment layout:Sorted Forward Index Tuesday, August 18, 15
- 17. Pinot Segment layout:Other techniques • Indexes: Inverted index, Bitmap, RoaringBitmap • Compression: Dictionary Encoding, P4Delta • Multi Valued columns, skip lists, • Hyperloglog for unique • T-digest for Percentile, Quantile Tuesday, August 18, 15
- 18. Data aware pre-computation Star treeIndex Tuesday, August 18, 15
- 19. Pinot • Query Execution Tuesday,August 18, 15
- 20. Pinot Query Execution:Distributed Servers S1 S3 S2 S1 S3 S2 Helix Brokers Tuesday, August 18, 15
- 21. Pinot Query Execution:Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix Brokers Tuesday, August 18, 15
- 22. Pinot Query Execution:Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix 2. Fetch routing table from HelixBrokers Tuesday, August 18, 15
- 23. Pinot Query Execution:Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix 2. Fetch routing table from HelixBrokers 3. Scatter Request Tuesday, August 18, 15
- 24. Pinot Query Execution:Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix 2. Fetch routing table from HelixBrokers 3. Scatter Request 4. Process Request & send response Tuesday, August 18, 15
- 25. Pinot Query Execution:Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix 2. Fetch routing table from HelixBrokers 3. Scatter Request 4. Process Request & send response 5. Gather Response Tuesday, August 18, 15
- 26. Pinot Query Execution:Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix 2. Fetch routing table from HelixBrokers 3. Scatter Request 4. Process Request & send response 5. Gather Response 6. Return Response Tuesday, August 18, 15
- 27. Pinot Query Execution:Single Node Architecture EXECUTION ENGINE INVERTED INDEX BITMAP INDEX COLUMN FORMAT PLANNER Tuesday, August 18, 15
- 28. Pinot Query Execution:Single Node Architecture SELECT campaignId, sum(clicks) FROM Table A WHERE accountId = 121011 AND 'day' >= 15949 GROUP BY campaignId account Id daycampaign Id click Filter Operator Projection Operator Aggregation Group by Operator Combine Operator Pinot Segments Data sources Matching doc ids campaignId,Click tuple Tuesday, August 18, 15
- 29.
- 30. Cluster Management: Deployment Helix Brokers Servers •Brokers and Servers register themselves in Helix • All servers start with no use case specific configuration Controller Tuesday, August 18, 15
- 31. On boarding newuse case Helix Brokers Servers XLNT XLNT XLNT Create Table command Controller XLNT XLNTTag Servers TableName Brokers 3 XLNT_T1 1 Tuesday, August 18, 15
- 32. Segment Assignment Servers S3 S2 S1 Upload SegmentS2 S1 S3 S2 S1 S3 Helix Brokers Copies TableName 2 XLNT_T1 Controller Tuesday, August 18, 15
- 33. • AUTO recoverymode: Automatically redistribute segments on failure/addition of new nodes • Custom mode: Run in degraded mode until node is restarted/replaced. Pinot - Fault tolerance/Elasticity Tuesday, August 18, 15
- 34. Pinot vs Druid DruidPinot Architecture Realtime + Offline, Realtime only Realtime + Offline Realtime only -> consistency is hard and schema evolution/Bootstrap is hard Inverted Index Always On all columns, Fixed Configurable on per column basis Allows trade off between scanning v/s inverted index + scanning. More data can be fit in given memory size Data organization N/A Sorts data Organizing data provides speed/better compression and removes the need for inverted index Smart pre- materialization N/A star-tree Allows trade off between latency and space Query Execution Layer Fixed Plan Split into Planning and execution Smart choices can be made at runtime based on metadata/query. Tuesday, August 18, 15
- 35. • Documentation &tooling • In progress - consistency among real time replicas. • Improve cost to serve - leverage SSD, partial pre materialization • ThirdEye - Business Metrics Monitoring Pinot - Future Tuesday, August 18, 15
- 36.