Docker/Jupyter PySpark - Revision history

Admin: /* Persistence and getting data in */

2017-09-27T02:22:45Z

Persistence and getting data in

@@ Line 61: / Line 61: @@
 Here's how we do that:
 Create the local directory where your notebooks will live:
@@ Line 68: / Line 76: @@
 $ mkdir spark-stuff/
 </pre>
 =Flags=
 {{DockerFlag}}

Admin: /* Test it out */

2017-09-27T02:16:32Z

Test it out

@@ Line 54: / Line 54: @@
 rdd = sc.parallelize(range(1000))
 rdd.takeSample(False, 5)
 </pre>

Admin: /* Testing it out */

2017-09-27T02:09:12Z

Testing it out

← Older revision		Revision as of 02:09, 27 September 2017
Line 25:		Line 25:
	</pre>		</pre>

	==~~Testing~~ it ~~out~~==		==Fire it up==

	Fire up the Docker container with the command above:		Fire up the Docker container with the command above:
Line 41:		Line 41:
	--NotebookApp.keyfile=/etc/ssl/notebook/notebook.key		--NotebookApp.keyfile=/etc/ssl/notebook/notebook.key
	--NotebookApp.certfile=/etc/ssl/notebook/notebook.crt		--NotebookApp.certfile=/etc/ssl/notebook/notebook.crt
			</pre>

			==Test it out==

			Once you have your notebook open, execute the following Python code to ensure it works ok:

			<pre>
			import pyspark
			sc = pyspark.SparkContext('local[*]')

			# do something to prove it works
			rdd = sc.parallelize(range(1000))
			rdd.takeSample(False, 5)
	</pre>		</pre>

Admin: Created page with "=Install= To run PySpark in a Jypyter notebook using Docker, we use a Docker image curated by the Jupyter project: jupyter/docker-stacks. Link to jupyter/pyspark-notebook on..."

2017-09-27T02:08:37Z

Created page with "=Install= To run PySpark in a Jypyter notebook using Docker, we use a Docker image curated by the Jupyter project: jupyter/docker-stacks. Link to jupyter/pyspark-notebook on..."

New page

=Install=

To run PySpark in a Jypyter notebook using Docker, we use a Docker image curated by the Jupyter project: jupyter/docker-stacks.

Link to jupyter/pyspark-notebook on Dockerhub: https://hub.docker.com/r/jupyter/pyspark-notebook/

Link to jupyter/pyspark-notebook on Github: https://github.com/jupyter/docker-stacks/tree/master/pyspark-notebook

Link to jupyter/docker-stacks on Github: https://github.com/jupyter/docker-stacks

==Get the docker container==

The short version: get the docker image using docker pull:

<pre>
$ docker pull jupyter/pyspark-notebook
</pre>

That's it. There is no long version.

To run it, we need to pass traffic from port 8888 on our machine into port 8888 on the Docker image:

<pre>
$ docker run -it --rm -p 8888:8888 jupyter/pyspark-notebook
</pre>

==Testing it out==

Fire up the Docker container with the command above:

<pre>
$ docker run -it --rm -p 8888:8888 jupyter/pyspark-notebook
</pre>

This will print out the URL for the Jupyter notebook. There are also ways to pass in a custom certificate, if you want to allow others to access the Jupyter notebook. These are all detailed in the jupyter/pyspark-notebook README, under the section "Notebook Options": https://github.com/jupyter/docker-stacks/tree/master/pyspark-notebook#notebook-options

<pre>
docker run -d -p 8888:8888 \
-v /some/host/folder:/etc/ssl/notebook \
jupyter/pyspark-notebook start-notebook.sh \
--NotebookApp.keyfile=/etc/ssl/notebook/notebook.key
--NotebookApp.certfile=/etc/ssl/notebook/notebook.crt
</pre>

=Flags=

{{DockerFlag}}