Downloaded 16 times
![SQL core in depth
o.a.s.sql.SchemaRDD
Extends RDD[Row].
Has logical plan tree.
Provides LINQ-like interfaces to construct
logical plan.
select, where, join, orderBy, …
Executes the plan.
28](https://image.slidesharecdn.com/sparksqlcatalyst-140912023144-phpapp02/85/Introduction-to-Spark-SQL-and-Catalyst-Spark-SQL-Calalyst-28-320.jpg)
![SQL core in depth
o.a.s.sql.SchemaRDD
Extends RDD[Row].
Has logical plan tree.
Provides LINQ-like interfaces to construct
logical plan.
select, where, join, orderBy, …
Executes the plan.
28](https://image.slidesharecdn.com/sparksqlcatalyst-140912023144-phpapp02/75/Introduction-to-Spark-SQL-and-Catalyst-Spark-SQL-Calalyst-28-2048.jpg)
Uploaded byscalaconfjp
Introduction to Spark SQL and Catalyst / Spark SQLおよびCalalystの紹介
SparkSQL is a Spark component that allows SQL queries to be executed on Spark. It uses Catalyst, which provides an execution planning framework for relational operations like SQL parsing, logical optimization, and physical planning. Catalyst defines logical and physical operators, expressions, data types and provides rule-based optimizations to transform query plans. The SQL core in SparkSQL builds SchemaRDDs to represent queries and allows reading/writing to Parquet and JSON formats.
More Related Content
![[ScalaMatsuri] グリー初のscalaプロダクト！チャットサービス公開までの苦労と工夫](https://cdn.slidesharecdn.com/ss_thumbnails/scalamaturiscala-140914203407-phpapp01-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Similar to Introduction to Spark SQL and Catalyst / Spark SQLおよびCalalystの紹介
Introduction to Spark SQL and Catalyst / Spark SQLおよびCalalystの紹介
- 1. Introduction to SparkSQL& Catalyst Takuya UESHIN ! ScalaMatsuri2014 2014/09/06(Sat)
- 2. Who am I? Takuya UESHIN @ueshin github.com/ueshin Nautilus Technologies, Inc. A Spark contributor 2
- 3. Agenda What isSparkSQL? Catalyst in depth SparkSQL core in depth Interesting issues How to contribute 3
- 4.
- 5. What is SparkSQL? SparkSQL is one of Spark components. Executes SQL on Spark Builds SchemaRDD like LINQ Optimizes execution plan. 5
- 6. What is SparkSQL? Catalyst provides a execution planning framework for relational operations. Including: SQL parser & analyzer Logical operators & general expressions Logical optimizer A framework to transform operator tree. 6
- 7. What is SparkSQL? To execute query needs some steps. Parse Analyze Logical Plan Optimize Physical Plan Execute 7
- 8. What is SparkSQL? To execute query needs some steps. Parse Analyze Logical Plan Optimize Physical Plan Execute 8 Catalyst
- 9. What is SparkSQL? To execute query needs some steps. Parse Analyze Logical Plan Optimize Physical Plan Execute 9 SQL core
- 10.
- 11. Catalyst in depth Provides a execution planning framework for relational operations. Row & DataType’s Trees & Rules Logical Operators Expressions Optimizations 11
- 12. Row & DataType’s o.a.s.sql.catalyst.types.DataType Long, Int, Short, Byte, Float, Double, Decimal String, Binary, Boolean, Timestamp Array, Map, Struct o.a.s.sql.catalyst.expressions.Row Represents a single row. Can contain complex types. 12
- 13. Trees & Rules o.a.s.sql.catalyst.trees.TreeNode Provides transformations of tree. foreach, map, flatMap, collect transform, transformUp, transformDown Used for operator tree, expression tree. 13
- 14. Trees & Rules o.a.s.sql.catalyst.rules.Rule Represents a tree transform rule. o.a.s.sql.catalyst.rules.RuleExecutor A framework to transform trees based on rules. 14
- 15. Logical Operators BasicOperators Project, Filter, … Binary Operators Join, Except, Intersect, Union, … Aggregate Generate, Distinct Sort, Limit InsertInto, WriteToFile 15 Project Filter Join Table Table
- 16. Expressions Literal Arithmetics UnaryMinus, Sqrt, MaxOf Add, Subtract, Multiply, … Predicates EqualTo, LessThan, LessThanOrEqual, GreaterThan, GreaterThanOrEqual Not, And, Or, In, If, CaseWhen 16 + 1 2
- 17. Expressions Cast GetItem,GetField Coalesce, IsNull, IsNotNull StringOperations Like, Upper, Lower, Contains, StartsWith, EndsWith, Substring, … 17
- 18. Optimizations ConstantFolding NullPropagation ConstantFolding BooleanSimplification SimplifyFilters FilterPushdown CombineFilters PushPredicateThroughProject PushPredicateThroughJoin ColumnPruning 18
- 19. Optimizations NullPropagation, ConstantFolding Replace expressions that can be evaluated with some literal value to the value. ex) 1 + null => null 1 + 2 => 3 Count(null) => 0 19
- 20. Optimizations BooleanSimplification Simplifiesboolean expressions that can be determined. ex) false AND $right => false true AND $right => $right true OR $right => true false OR $right => $right If(true, $then, $else) => $then 20
- 21. Optimizations SimplifyFilters Removesfilters that can be evaluated trivially. ex) Filter(true, child) => child Filter(false, child) => empty 21
- 22. Optimizations CombineFilters Mergestwo filters. ex) Filter($fc, Filter($nc, child)) => Filter(AND($fc, $nc), child) 22
- 23. Optimizations PushPredicateThroughProject PushesFilter operators through Project operator. ex) Filter(‘i === 1, Project(‘i, ‘j, child)) => Project(‘i, ‘j, Filter(‘i === 1, child)) 23
- 24. Optimizations PushPredicateThroughJoin PushesFilter operators through Join operator. ex) Project(“left.i”.attr === 1, Join(left, right) => Join(Project(‘i === 1, left), right) 24
- 25. Optimizations ColumnPruning Eliminatesthe reading of unused columns. ex) Join(left, right, LeftSemi, “left.id”.attr === “right.id”.attr) => Join(Project(‘id, left), right, LeftSemi) 25
- 26. SQL core indepth
- 27. SQL core indepth Provides: Physical operators to build RDD Conversion from Existing RDD of Product to SchemaRDD support Parquet file read/write support JSON file read support Columnar in-memory table support 27
- 28. SQL core indepth o.a.s.sql.SchemaRDD Extends RDD[Row]. Has logical plan tree. Provides LINQ-like interfaces to construct logical plan. select, where, join, orderBy, … Executes the plan. 28
- 29. SQL core indepth o.a.s.sql.execution.SparkStrategies Converts logical plan to physical. Some rules are based on statistics of the operators. 29
- 30. SQL core indepth Parquet read/write support Parquet: Columnar storage format for Hadoop Reads existing Parquet files. Converts Parquet schema to row schema. Writes new Parquet files. Currently DecimalType and TimestampType are not supported. 30
- 31. SQL core indepth JSON read support Loads a JSON file (one object per line) Infers row schema from the entire dataset. Giving the schema is experimental. Inferring the schema by sampling is also experimental. 31
- 32. SQL core indepth Columnar in-memory table support Caches table like RDD.cache, but as columnar style. Can prune unnecessary columns when read data. 32
- 33.
- 34. Interesting issues Supportthe GroupingSet/ROLLUP/CUBE https://issues.apache.org/jira/browse/SPARK-2663 Use statistics to skip partitions when reading from in-memory columnar data https://issues.apache.org/jira/browse/SPARK-2961 34
- 35. Interesting issues Pluggableinterface for shuffles https://issues.apache.org/jira/browse/SPARK-2044 Sort-merge join https://issues.apache.org/jira/browse/SPARK-2213 Cost-based join reordering https://issues.apache.org/jira/browse/SPARK-2216 35
- 36.
- 37. How to contribute See: Contributing to Spark Open an issue on JIRA Send pull-request at GitHub Communicate with committers and reviewers Congratulations! 37
- 38. Conclusion Introduced SparkSQL& Catalyst Now you know them very well! And you know how to contribute. ! Let’s contribute to Spark & SparkSQL!! 38