Jupyter/MPI

Main Jupyter page: Jupyter

ipyparallel documentation: https://ipyparallel.readthedocs.io/en/latest/

Steps

Install OpenMPI

Start by installing OpenMPI:

brew install open-mpi

Install Necessary Notebook Modules

Install the mpi4py library:

pip3 install mpi4py

Install the ipyparallel notebook extension:

jupyter serverextension enable --py ipyparallel

Start MPI Cluster

Then start an MPI cluster using ipcluster:

ipcluster start --n=4

The output should look like this:

$ ipcluster start --n=4
2017-11-27 05:25:47.988 [IPClusterStart] Starting ipcluster with [daemon=False]
2017-11-27 05:25:47.989 [IPClusterStart] Creating pid file: /Users/charles/.ipython/profile_default/pid/ipcluster.pid
2017-11-27 05:25:47.989 [IPClusterStart] Starting Controller with LocalControllerLauncher
2017-11-27 05:25:48.998 [IPClusterStart] Starting 4 Engines with LocalEngineSetLauncher
2017-11-27 05:26:19.332 [IPClusterStart] Engines appear to have started successfully

If you do pass an --engines flag, though, it could be problematic:

$ ipcluster start --engines=MPI --n=4
2017-11-27 05:24:09.772 [IPClusterStart] Starting ipcluster with [daemon=False]
2017-11-27 05:24:09.773 [IPClusterStart] Creating pid file: /Users/charles/.ipython/profile_default/pid/ipcluster.pid
2017-11-27 05:24:09.773 [IPClusterStart] Starting Controller with LocalControllerLauncher
2017-11-27 05:24:10.777 [IPClusterStart] Starting 4 Engines with MPI
2017-11-27 05:24:10.813 [IPClusterStart] ERROR |
            Engines shutdown early, they probably failed to connect.

            Check the engine log files for output.

            If your controller and engines are not on the same machine, you probably
            have to instruct the controller to listen on an interface other than localhost.

            You can set this by adding "--ip='*'" to your ControllerLauncher.controller_args.

            Be sure to read our security docs before instructing your controller to listen on
            a public interface.

2017-11-27 05:24:10.813 [IPClusterStart] ERROR | IPython cluster: stopping
2017-11-27 05:24:13.816 [IPClusterStart] Removing pid file: /Users/charles/.ipython/profile_default/pid/ipcluster.pid

Problems

Problems sharing a variable using px magic

Ideally, we want something like this to work:

rc = ipp.Client()

then:

%px foo = 5

print(rc[:]['foo'])
rc[:]['bar'] = lambda : 2 * foo
rc[:].apply_sync(parallel.Reference('bar'))

However, this fails.

Documentation:

• https://ipyparallel.readthedocs.io/en/latest/api/ipyparallel.html#classes
• Suggests answer may be push/pull?
• Gives px example with variable assignment: https://github.com/ipython/ipyparallel/blob/527dfc6c7b7702fb159751588a5d5a11d8dd2c4f/docs/source/magics.rst

# import numpy here and everywhere
In [25]: with rc[:].sync_imports():
   ....:    import numpy
importing numpy on engine(s)

In [27]: %px a = numpy.random.rand(2,2)
Parallel execution on engines: [0, 1, 2, 3]

In [28]: %px numpy.linalg.eigvals(a)
Parallel execution on engines: [0, 1, 2, 3]
Out [0:68]: array([ 0.77120707, -0.19448286])
Out [1:68]: array([ 1.10815921,  0.05110369])
Out [2:68]: array([ 0.74625527, -0.37475081])
Out [3:68]: array([ 0.72931905,  0.07159743])

In [29]: %px print 'hi'
Parallel execution on engine(s): all
[stdout:0] hi
[stdout:1] hi
[stdout:2] hi
[stdout:3] hi

More hints, but nothing solid: https://github.com/ipython/ipyparallel/blob/1cc0f67ba12a4c18be74384800aa906bc89d4dd3/docs/source/direct.rst

Original notebook: https://github.com/charlesreid1/ipython-in-depth/blob/master/examples/Parallel%20Computing/Using%20Dill.ipynb

ipython parallel built in magic (mentions px magic, but no examples):

• Cell magic: https://ipython.readthedocs.io/en/stable/interactive/magics.html#cell-magics
• All magic: https://ipyparallel.readthedocs.io/en/latest/magics.html