Thursday, March 24, 2022

Pair RDDs: Transformations and Actions

 

Pair RDDs: Transformations and Actions

RDD[(k, v)] // <------- treated specially by Spark
val rdd: RDD[WikipediaPage] = ...
val pairRdd = rdd.map( wikipediapage => (wikipediapage.title, wikipediapage.text) )
def groupByKey(): RDD[(K, Iterable[V])
def reduceByKey(func: (V, V) => V): RDD[(K, V)]
def join[W](other: RDD[(K, W)]): RDD[(K, (V, W))]

groupByKey (transformation)

def groupByKey(): RDD[(K, Iterable[V])]case class Event(Organizer: String, name: String, budget: Int)
val rdd = sc.parallelize(...)
val eventsRdd = rdd.map(event => (event.organizer, event.budget))
val groupedRDD = eventsRdd.groupByKey()
groupedRDD.collect().foreach(println)
Result :
// (Organizer1, CompactBuffer(42000))
// (Organizer2, CompactBuffer(20000, 44400, 87000))

reduceByKey (transformation)

def reduceByKey( func(V, V) => V ): RDD[(K, V)] // V corresponds to the values of Pair RDD, we only operate on the value since a pair RDD is in the form of Key Values.calculate the total budget per organizationval eventsRdd = rdd.map(event => (event.organizer, event.budget)) 
val totalBudgetsRdd = eventsRdd.reduceByKey( _ + _ )
// at this point we already have keys and values. So reduceByKey means reduce the values corresponding to the given key using the given function.

totalBudgetsRdd.collect().foreach(println)
Result:
// (Organizer1, 42000)
// (Organizer2, 151400)

mapValues (transformation)

def mapValues[U](f: V => U): RDD[(K, U)]

sortByKey() (transformation)

RDD[(K, V)] to RDD[(K, V)]val rdd1 = sc.parallelize(Seq((“India”,91),(“USA”,1),(“Brazil”,55),(“Greece”,30),(“China”,86),(“Sweden”,46),(“Turkey”,90),(“Nepal”,977))) 
val rdd2 = rdd1.sortByKey()
rdd2.collect();
output:Array[(String,Int)] = (Array(Brazil,55),(China,86),(Greece,30),(India,91),(Nepal,977),(Sweden,46),(Turkey,90),(USA,1)// may not print result in sorted order
rdd2.foreach(println)
//will print result in sorted order
rdd2.collect().foreach(println)

countByKey (action)

def countByKey(): Map[K, Long]

Map vs FlatMap (transformation)

val rdd = sc.parallelize(Seq("Roses are red", "Violets are blue"))rdd.map(_.length).collect

res1: Array[Int] = Array(13, 16)
rdd.flatMap(_.split(" ")).collect

res2: Array[String] = Array("Roses", "are", "red", "Violets", "are", "blue")

Running Spark Jobs on YARN

 

Running Spark Jobs on YARN

yarn cluster mode
Yarn Client Mode
Different Deployment Modes across the cluster

Partitioning in Apache Spark

 

Partitioning in Apache Spark

Two types of partitioning:

p=key.hashCode() %noOfPartitions
Performance gain when data was partitioned before applying the transformation.

How to partition your RDD

val pairRDD = data.map(x =>(x.key,x.value));
val partitioner = new RangePartitioner(8,pairRDD) ;
val partitionedRDD = pairRDD.partitionBy(partitioner).persist();

Why partition data?

Figure 3

Spark Architecture and Deployment Environment

 

Spark Architecture and Deployment Environment

./bin/spark-submit \
--class <main-class> \
--master <master-url> \
--deploy-mode <deploy-mode> \
--conf <key>=<value> \
... # other options
<application-jar> \
[application-arguments
  • --deploy-mode: Whether to deploy your driver on the worker nodes (cluster) or locally as an external client (client) (default: client)
Yarn Architecture
  • YARN allows you to dynamically share and centrally configure the same pool of cluster resources between all frameworks that run on YARN. You can throw your entire cluster at a MapReduce job, then use some of it on an Impala query and the rest on Spark application, without any changes in configuration.
  • Spark standalone mode requires each application to run an executor on every node in the cluster, whereas with YARN, you choose the number of executors to use.