Downloaded 1,123 times
![Scan
Project
Aggregate
Sort
Map { // age → heights
20 → [154, 174, 175]
21 → [167, 168, 181]
22 → [155, 166, 188]
23 → [160, 168, 178, 183]
}](https://image.slidesharecdn.com/spark-memory-sparksummit-2016-160608234716/85/Deep-Dive-Memory-Management-in-Apache-Spark-36-320.jpg)
![Scan
Project
Aggregate
Sort
Map { // age → heights
20 → [154, 174, 175]
21 → [167, 168, 181]
22 → [155, 166, 188]
23 → [160, 168, 178, 183]
}](https://image.slidesharecdn.com/spark-memory-sparksummit-2016-160608234716/75/Deep-Dive-Memory-Management-in-Apache-Spark-36-2048.jpg)
Uploaded byDatabricks
Deep Dive: Memory Management in Apache Spark
The document discusses memory management challenges in Apache Spark, focusing on efficient use of execution and storage memory. It outlines strategies for handling memory contention across various scenarios, including static and dynamic memory assignment for tasks and operators. Additionally, it introduces features from Spark 1.6 onwards, emphasizing unified memory management and the benefits of tungsten's optimized data representation.
More Related Content
Similar to Deep Dive: Memory Management in Apache Spark
Deep Dive: Memory Management in Apache Spark
- 1. Deep Dive: Memory Managementin Apache Andrew Or June 8th, 2016 @andrewor14
- 2.
- 3. 3 Efficient memory useis critical to good performance
- 5. Memory contention posesthree challenges for Apache Spark 5 How to arbitrate memory between execution and storage? How to arbitrate memory across tasks running in parallel? How to arbitrate memory across operators running within the same task?
- 6. Two usages ofmemory in Apache Spark 6 Execution Memory used for shuffles, joins, sorts and aggregations Storage Memory used to cache data that will be reused later
- 7. Iterator 4, 3, 5,1, 6, 2 Sort 4 3 5 1 6 2 Iterator 1, 2, 3, 4, 5, 6 1 2 3 4 5 6 Execution memory Take(3) What if I want the sorted values again?
- 8. Iterator 4, 3, 5,1, 6, 2 Sort 4 3 5 1 6 2 Iterator 1, 2, 3, 4, 5, 6 1 2 3 4 5 6 Take(3) Iterator 4, 3, 5, 1, 6, 2 Sort 4 3 5 1 6 2 Iterator 1, 2, 3, 4, 5, 6 1 2 3 4 5 6 Take(4) ...
- 9. Sort Iterator 4, 3, 5,1, 6, 2 4 3 5 1 6 2 Iterator 1, 2, 3, 4, 5, 6 1 2 3 4 5 6 Cache 4 3 5 1 6 21 2 3 4 5 6 Execution memory Storage memory Take(5)Take(4)Take(3) ...
- 10. Challenge #1 How toarbitrate memory between execution and storage?
- 11. Easy, static assignment! 11 Totalavailable memory Execution Storage Spark 1.0 May 2014
- 12. Easy, static assignment! 12 ExecutionStorage Spill to disk Spark 1.0 May 2014
- 13. Easy, static assignment! 13 ExecutionStorage Spark 1.0 May 2014
- 14. Easy, static assignment! 14 ExecutionStorage Evict LRU block to disk Spark 1.0 May 2014
- 15. 15 Inefficient memory useleads to bad performance
- 16. Easy, static assignment! 16 Executioncan only use a fraction of the memory, even when there is no storage! Execution Storage Spark 1.0May 2014
- 17. Storage Easy, static assignment! 17 Efficientuse of memory required user tuning Execution Spark 1.0May 2014
- 18. 18 Fast forward to2016… How could we have done better?
- 19.
- 20. 20 Unified memory management Spark1.6+ Jan 2016 What happens if there is already storage? Execution Storage
- 21. 21 Unified memory management Spark1.6+ Jan 2016 Evict LRU block to disk Execution Storage
- 22. 22 Unified memory management Spark1.6+ Jan 2016 What about the other way round? Execution Storage
- 23. 23 Unified memory management Spark1.6+ Jan 2016 Evict LRU block to disk Execution Storage
- 24. Design considerations 24 Why evictstorage, not execution? Spilled execution data will always be read back from disk, whereas cached data may not. What if the application relies on caching? Allow the user to specify a minimum unevictable amount of cached data (not a reservation!). Spark 1.6+ Jan 2016
- 25. Challenge #2 How toarbitrate memory across tasks running in parallel?
- 26. Easy, static assignment! Workermachine has 4 cores Each task gets 1/4 of the total memory Slot 1 Slot 2 Slot 3 Slot 4
- 27. Alternative: Dynamic assignment Theshare of each task depends on number of actively running tasks (N) Task 1
- 28. Alternative: Dynamic assignment Now,another task comes along so the first task will have to spill Task 1
- 29. Alternative: Dynamic assignment Eachtask is now assigned 1/N of the memory, where N = 2 Task 1 Task 2
- 30. Alternative: Dynamic assignment Eachtask is now assigned 1/N of the memory, where N = 4 Task 1 Task 2 Task 3 Task 4
- 31. Alternative: Dynamic assignment Lastremaining task gets all the memory because N = 1 Task 3 Spark 1.0+ May 2014
- 32. Static vs dynamicassignment 32 Both are fair and starvation free Static assignment is simpler Dynamic assignment handles stragglers better
- 33. Challenge #3 How toarbitrate memory across operators running within the same task?
- 34. SELECT age, avg(height) FROMstudents GROUP BY age ORDER BY avg(height) students.groupBy("age") .avg("height") .orderBy("avg(height)") .collect() Scan Project Aggregate Sort
- 35. Worker has 6 pagesof memory Scan Project Aggregate Sort
- 36. Scan Project Aggregate Sort Map { //age → heights 20 → [154, 174, 175] 21 → [167, 168, 181] 22 → [155, 166, 188] 23 → [160, 168, 178, 183] }
- 37. Scan Project Aggregate Sort All 6 pageswere used by Aggregate, leaving no memory for Sort!
- 38. Solution #1: Reserve apage for each operator Scan Project Aggregate Sort
- 39. Solution #1: Reserve apage for each operator Scan Project Aggregate Sort Starvation free, but still not fair… What if there were more operators?
- 40.
- 41.
- 42. Scan Project Aggregate Sort Solution #2: Cooperative spilling Sortforces Aggregate to spill a page to free memory
- 43. Scan Project Aggregate Sort Solution #2: Cooperative spilling Sortneeds more memory so it forces Aggregate to spill another page (and so on)
- 44. Scan Project Aggregate Sort Solution #2: Cooperative spilling Sortfinishes with 3 pages Aggregate does not have to spill its remaining pages Spark 1.6+ Jan 2016
- 45. Recap: Three sourcesof contention 45 How to arbitrate memory … ● between execution and storage? ● across tasks running in parallel? ● across operators running within the same task? Instead of avoid statically reserving memory in advance, deal with memory contention when it arises by forcing members to spill
- 46. Project Tungsten 46 Binary in-memorydata representation Cache-aware computation Code generation (next time) Spark 1.4+ Jun 2015
- 47. “abcd” 47 • Native: 4bytes with UTF-8 encoding • Java: 48 bytes – 12 byte header – 2 bytes per character (UTF-16 internal representation) – 20 bytes of additional overhead – 8 byte hash code Java objects have large overheads
- 48. 48 Schema: (Int, String,String) Row Array String(“data”) String(“bricks”) 5+ objects, high space overhead, expensive hashCode() BoxedInteger(123) Java objects based row format
- 49. 6 “bricks” 49 0x0 12332L 48L 4 “data” (123, “data”, “bricks”) Null tracking bitmap Offset to var. length data Offset to var. length data Tungsten row format
- 50. Cache-aware computation 50 ptr keyrec ptr key rec ptr key rec Naive layout Poor cache locality ptrkey prefix rec ptrkey prefix rec ptrkey prefix rec Cache-aware layout Good cache locality E.g. sorting a list of records
- 51. Off-heap memory 51 Available forexecution since Apache Spark 1.6 Available for storage since Apache Spark 2.0 Very important for large heaps Many potential advantages: memory sharing, zero copy I/O, dynamic allocation
- 52. For more info... DeepDive into Project Tungsten: Bringing Spark Closer to Bare Metal https://www.youtube.com/watch?v=5ajs8EIPWGI Spark Performance: What’s Next https://www.youtube.com/watch?v=JX0CdOTWYX4 Unified Memory Management https://issues.apache.org/jira/browse/SPARK-10000
- 53. Databricks Community Edition Freeversion of cloud based platform in beta More than 8,000 users registered Users created over 61,000 notebooks in different languages http://www.databricks.com/try 53
- 54.