Mastering Apache Spark 2.x(Second Edition)
上QQ阅读APP看书,第一时间看更新

How to go from Unresolved Logical Execution Plan to Resolved Logical Execution Plan

The ULEP basically reflects the structure of an AST. So again, the AST is generated from the user's code implemented either on top of the relational API of DataFrames and Datasets or using SQL, or all three. This AST can be easily transformed into a ULEP. But, of course, a ULEP can't be executed. The first thing that is checked is if the referred relations exist in the catalog. This means all table names and fields expressed in the SQL statement or relational API have to exist. If the table (or relation) exists, the column names are verified. In addition, the column names that are referred to multiple times are given an alias in order to read them only once. This is already a first stage optimization taking place here. Finally, the data types of the columns are determined in order to check if the operations expressed on top of the columns are valid. So for example taking the sum of a string doesn't work and this error is already caught at this stage. The result of this operation is a Resolved Logical Execution Plan (LEP).