PySpark - Revision history

Admin: /* Save As Text File Method */

2017-10-11T22:00:04Z

Save As Text File Method

@@ Line 890: / Line 890: @@
 <pre>
 '0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n'
 </pre>

Admin: /* For Each Method */

2017-10-11T21:59:51Z

For Each Method

@@ Line 845: / Line 845: @@
 <pre>
 [ ('a', 2), ('b', 2), ('c', 1), ('d', 2) ]
 </pre>

Admin: /* Count By Key Method */

2017-10-11T21:59:29Z

Count By Key Method

@@ Line 845: / Line 845: @@
 <pre>
 [ ('a', 2), ('b', 2), ('c', 1), ('d', 2) ]
 </pre>

Admin: /* Count Method */

2017-10-11T21:56:49Z

Count Method

← Older revision		Revision as of 21:56, 11 October 2017
Line 813:		Line 813:

	===Count Method===		===Count Method===

	~~Documentation: https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.count~~

	Documentation: https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.count		Documentation: https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.count

Admin: /* Actions */

2017-10-11T21:56:30Z

Actions

Show changes

Admin: /* Repartition Transformation */

2017-10-11T21:33:59Z

Repartition Transformation

← Older revision		Revision as of 21:33, 11 October 2017
Line 729:		Line 729:

	<pre>		<pre>
	rdd1 = rdd1.repartition(4)		rdd1 = rdd1.repartition(8)
	</pre>		</pre>

Admin: /* Usage */

2017-10-11T21:33:39Z

Usage

← Older revision		Revision as of 21:33, 11 October 2017
Line 308:		Line 308:
	</pre>		</pre>

	=~~Usage=~~		=RDDs=

	~~==RDD=~~=

	RDD = Resilient distributed dataset		RDD = Resilient distributed dataset
Line 318:		Line 316:
	RDDs keep track of the transformations applied to each chunk, speeding up computations. If any chunks are lost, the transformations can be re-applied.		RDDs keep track of the transformations applied to each chunk, speeding up computations. If any chunks are lost, the transformations can be re-applied.

	===Internal Workings===		==Internal Workings==

	RDDs operate in parallel - they are distributed in parallel, executed in parallel, transformed in parallel.		RDDs operate in parallel - they are distributed in parallel, executed in parallel, transformed in parallel.
Line 361:		Line 359:
	Calling collect is the user telling Spark to go into action - up until that point, the maps, filters, and reduce operations are just accumulated. When collect is called, the actions are actually carried out.		Calling collect is the user telling Spark to go into action - up until that point, the maps, filters, and reduce operations are just accumulated. When collect is called, the actions are actually carried out.

	===Creating RDDs===		==Creating RDDs==

	To create an RDD, you can use two ways:		To create an RDD, you can use two ways:
Line 397:		Line 395:
	RDDs have no schema, and can therefore be heterogeneous. For example, you can parallelize a list that consists of a tuple, and a dict, and a list, and Spark is okay with that. When you collect the results again (which returns all of the data back to the driver, or master, node), the resulting data set will function as any list containing a tuple, a dict, and a list.		RDDs have no schema, and can therefore be heterogeneous. For example, you can parallelize a list that consists of a tuple, and a dict, and a list, and Spark is okay with that. When you collect the results again (which returns all of the data back to the driver, or master, node), the resulting data set will function as any list containing a tuple, a dict, and a list.

	===Local Mode vs Cluster Mode===		==Local Mode vs Cluster Mode==

	PySpark can be run in local mode or in cluster mode.		PySpark can be run in local mode or in cluster mode.

	====Local Mode====		===Local Mode===

	In local mode, things work like your "standard" Python script - there is one namespace, and if any process (on a worker or master node) changes data, those changes are visible to every worker. This creates a lot of communication overhead, so it isn't sustainable for a large cluster, but it might be useful if you have, say, two or three nodes. This reduces the cognitive overhead of switching your entire script to work for a cluster.		In local mode, things work like your "standard" Python script - there is one namespace, and if any process (on a worker or master node) changes data, those changes are visible to every worker. This creates a lot of communication overhead, so it isn't sustainable for a large cluster, but it might be useful if you have, say, two or three nodes. This reduces the cognitive overhead of switching your entire script to work for a cluster.

	====Cluster Mode====		===Cluster Mode===

	In cluster mode, the user submits a job for execution to the driver (master) node. The master creates a directed acyclic graph (see [[Graphs/DAGs]]) to decide which workers will perform the task. The master prepares each task's "closure" (materials, data, variables, methods to make sure the worker node actually has ''everything required to finish'' the task). It then copies that information over to the worker.		In cluster mode, the user submits a job for execution to the driver (master) node. The master creates a directed acyclic graph (see [[Graphs/DAGs]]) to decide which workers will perform the task. The master prepares each task's "closure" (materials, data, variables, methods to make sure the worker node actually has ''everything required to finish'' the task). It then copies that information over to the worker.
Line 411:		Line 409:
	In cluster mode, the data is static - that's what we mean when we say copies are distributed to each worker. Each worker node can modify the data without affecting other workers' copies or the master node's copy.		In cluster mode, the data is static - that's what we mean when we say copies are distributed to each worker. Each worker node can modify the data without affecting other workers' copies or the master node's copy.

	====Example====		===Example===

	An example of a calculation in cluster mode, to illustrate:		An example of a calculation in cluster mode, to illustrate:
Line 419:		Line 417:
	By contrast, if we were running this in local mode, we would have to proceed in serial - starting at the beginning of the interval, and proceeding to the end of the interval. (Or, alternatively, we would need multiple delta_x, start_x, end_x, and running_sum variables.)		By contrast, if we were running this in local mode, we would have to proceed in serial - starting at the beginning of the interval, and proceeding to the end of the interval. (Or, alternatively, we would need multiple delta_x, start_x, end_x, and running_sum variables.)

	===Transformations===		==Transformations==

	There are a LOT of transformations available. Full list available in PySpark documentation:		There are a LOT of transformations available. Full list available in PySpark documentation:
	* https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD		* https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD

	====Lists====		===Lists===

	A list of the most important/common transforms:		A list of the most important/common transforms:
Line 528:		Line 526:
	\|}		\|}

	====Map Transform====		===Map Transform===

	The map transform is probably the most common; it applies a function to each element of the RDD.		The map transform is probably the most common; it applies a function to each element of the RDD.
Line 561:		Line 559:
	</pre>		</pre>

	====Filter Transformation====		===Filter Transformation===

	The filter transformation is a way of filtering out data according to boolean criteria.		The filter transformation is a way of filtering out data according to boolean criteria.
Line 577:		Line 575:
	</pre>		</pre>

	====FlatMap Transformation====		===FlatMap Transformation===

	Flat map is like map, in that it applies a function to each element of an RDD. However, it flattens anything that results. So if a function returns, say, a list of tuples, or a list of lists, flatMap will flatten those to a single, flat, shallow list. Compare:		Flat map is like map, in that it applies a function to each element of an RDD. However, it flattens anything that results. So if a function returns, say, a list of tuples, or a list of lists, flatMap will flatten those to a single, flat, shallow list. Compare:
Line 597:		Line 595:
	</pre>		</pre>

	====Distinct Transformation====		===Distinct Transformation===

	The distinct transformation is like the Numpy unique() function, it returns all of the unique values found in an RDD.		The distinct transformation is like the Numpy unique() function, it returns all of the unique values found in an RDD.
Line 618:		Line 616:
	NOTE: the distinct method is computationally expensive, as it parses all of the data.		NOTE: the distinct method is computationally expensive, as it parses all of the data.

	====Sample Transformation====		===Sample Transformation===

	This transformation samples the full data set and returns the sample. Think of it as picking out M representatives from a population of N things.		This transformation samples the full data set and returns the sample. Think of it as picking out M representatives from a population of N things.
Line 645:		Line 643:
	</pre>		</pre>

	====LeftOuterJoin Transformation====		===LeftOuterJoin Transformation===

	Operation inspired by SQL, left outer join joins two data sets on an index based on keys. Outer join means it will keep any keys that are only in one or the other data set. Left outer join means it will keep any keys that are only in the left data set and not in the right data set.		Operation inspired by SQL, left outer join joins two data sets on an index based on keys. Outer join means it will keep any keys that are only in one or the other data set. Left outer join means it will keep any keys that are only in the left data set and not in the right data set.
Line 682:		Line 680:


	====Join Transformation====		===Join Transformation===

	The join transformation, like SQL joins, discards any keys that are not in both data sets. If we use the same example RDDs as before:		The join transformation, like SQL joins, discards any keys that are not in both data sets. If we use the same example RDDs as before:
Line 710:		Line 708:
	The tuples are not returned in order. The two b keys result in two tuples, as mentioned. The left-only key still returns a tuple, but uses null (None) for the value from the right.		The tuples are not returned in order. The two b keys result in two tuples, as mentioned. The left-only key still returns a tuple, but uses null (None) for the value from the right.

	====Intersection Transformation====		===Intersection Transformation===

	Intersection performs a set intersection of two RDDs:		Intersection performs a set intersection of two RDDs:
Line 726:		Line 724:
	</pre>		</pre>

	====Repartition Transformation====		===Repartition Transformation===

	This function will change the number of partitions into which the data set is distributed.		This function will change the number of partitions into which the data set is distributed.
Line 734:		Line 732:
	</pre>		</pre>

	===Actions===		==Actions==



	=Flags=		=Flags=

	[[Category:Data Engineering]]		[[Category:Data Engineering]]

Admin: /* LeftOuterJoin Transformation */

2017-10-11T21:31:55Z

LeftOuterJoin Transformation

@@ Line 647: / Line 647: @@
 ====LeftOuterJoin Transformation====
-Operation inspired by SQl, left outer join joins two data sets on an index based on keys. Outer join means it will keep any keys that are only in one or the other data set. Left outer join means it will keep any keys that are only in the left data set and not in the right data set.
+Operation inspired by SQL, left outer join joins two data sets on an index based on keys. Outer join means it will keep any keys that are only in one or the other data set. Left outer join means it will keep any keys that are only in the left data set and not in the right data set.
 <pre>
@@ Line 680: / Line 680: @@
 The tuples are not returned in order. The two b keys result in two tuples, as mentioned. The left-only key still returns a tuple, but uses null (None) for the value from the right.
 ===Actions===

Admin: /* Sample Transformation */

2017-10-11T21:16:07Z

Sample Transformation

@@ Line 644: / Line 644: @@
 print("Population count: %d / Sample count: %d" % (file_data.count(), data_sample.count()) )
 </pre>
 ===Actions===

Admin: /* Sample Transformation */

2017-10-11T21:07:24Z

Sample Transformation

← Older revision		Revision as of 21:07, 11 October 2017
Line 642:		Line 642:

	<pre>		<pre>
	print("Population count: %d / Sample count: %d" % (file_data.count(), data_sample.count()) )		print("Population count: %d / Sample count: %d" % (file_data.count(), data_sample.count()) )
	</pre>		</pre>