Spark SQL, Dataframe and Dataset

 

Spark: Spark SQL, Dataframe and Dataset

Spark SQL

Spark SQL is a library for structured data processing which provides SQL like API on top of spark stack it supports relational data processing and SQL literal syntax to perform operations on data.

Like an RDD, a DataFrame and DataSet is an immutable distributed collection of data. Unlike an RDD, data in DataSet is organized into named columns, like a table in a relational database.

DataFrame allows developers to impose a structure onto a distributed collection of data, allowing higher-level abstraction, it also provides a domain specific language API.

RDD vs DataFrame

RDD is the core abstraction over which DataFrame and Dataset are built upon.

1. RDD is low-level and type-safe API.
2. RDDs are mainly for semi-structured and non-structured data but is easier to write inefficient code over RDD.

RDD operation

1. DataFrame has Column and schema and structure data.
2. Dataframe provides High-Level abstraction and provides DSL and query language to manipulate and extract data.

Dataset Operation

RDD vs DataSet GroupBy

Dataframe and Dataset have inbuilt optimizer which optimizes the data flow and is much faster and more memory efficient because of tungsten encoders.

DataSets

Dataset provides typesafe API which provides you the relational and functional transformation while benefitting from other optimization which RDD doesn't provide.

In Dataset you can mix and match your functional code/lambda and relational expression.

listingDS.groupByKey(ls => ls.zipCode).agg(avg($ “price”).as[Double])

Creating a DataSet From Json

val ds = spark.read.json(“/databricks-public-datasets/data/iot/iot_devices.json”).as[DeviceIoTData]

From RDD and DataFrame

myDF.toDS /rdd.toDS

DataSet Transformation:(Typed and Untyped)

Typed Transformation:

Typed Transformation on DataSet

GroupByKey On DataSet

Calling GroupByKey on Dataset returns another type of Dataset “KeyValueGroupedDataset”.

which has a special set of aggeration operation available over them which return the DataSet again.

groupByKey on DataSet

KeyValueGroupedDataset Aggeration Operations

KeyValueGroupedDataset Aggeration Operation which Returns Dataset Back

What to pass in TypedColumn?

Various functions are available such as avg, min, max, sum which can be pass as the TypedColumn argument.

agg(avg($"columnname").as[Double]) ---> Returns Dataset 

ReduceByKey In DataSet

ReduceByKey using reduceGroups

Aggregators

Aggregator is the contract for user-defined typed Custom Aggregate function.

Aggregator Function Contract

Implementing of Aggregate for string concatenate