OpenMPI

Installing

Prerequisites

OpenMPI creates Fortran bindings. If you need these, you'll have o install a Fortran compiler (I use Gfortran on Mac, the Gfortran page gives details about installing gfortran on Mac).

Since Fortran is a mess on Mac OS X, and I never use Fortran anyway, I didn't want to do this. So, I used the --disable-mpi-f77 configure option.

Mac Leopard (OS X 10.5)

For 10.5, I installed LAM MPI. It has since been dropped in favor of OpenMPI.

Mac Snow Leopard (OS X 10.6)

The following is my configure line to configure OpenMPI:

#!/bin/sh

./configure \
    --prefix=${HOME}/pkg/openmpi/1.4.3 \
    --disable-mpi-f77
    \
    CC="/usr/bin/gcc" \
    CXX="/usr/bin/g++" \
    \
    CXXFLAGS="-arch x86_64" \
    CCFLAGS="-arch x86_64" \

Note that OpenMPI is picky about compilers.

First, you have to specify exactly which compiler you want to use: you shouldn't say gcc, you should say /usr/bin/gcc.

Second, you have to specify architecture flags for all of the compilers. This is important, because otherwise the compilers will have problems by either producing 32-bit binaries, or maybe one compiler will produce a 64-bit binary and another will produce a 32-bit binary, etc.

NOTE: If you want to build both 32-bit and 64-bit binaries, see the #Building Fat Binaries section below.

Building Fat Binaries

A fat binary is a binary that has been built to run on multiple architectures. In the case of Mac Snow Leopard, a fat binary would be a 32-bit or a 64-bit binary.

After struggling with trying to build a fat binary (see #Bad register name below), I found this page on OpenMPI's website: [1]

According to the above page, it's impossible to build a one-pass fat binary, and so you actually have to run the configure/make/make install once for each architecture.

To do this, use the script at /path/to/openmpi_source/contrib/dist/macosx-pkg/buildpackage.sh

The magic of the script consists of two parts:

• the lipo -create command, which combines the binaries and libraries created for each architecture into a single universal binary/library
• some text manipulation to create a universal header file

The script is run like this:

/path/to/openmpi_source/contrib/dist/macosx-pkg/buildpackage.sh openmpi-x.y.z.tar.gz /path/to/openmpi/prefix

This then creates a zipped .dmg (disk image) file in /tmp, which you can then proceed to install.

Errors

C and Fortran 77 Compilers Not Link Compatible

When I first installed OpenMPI on Snow Leopard, I saw an error like this:

checking if C and Fortran 77 are link compatible... no
**********************************************************************
It appears that your Fortran 77 compiler is unable to link against
object files created by your C compiler.  This typically indicates
one of a few possibilities:

  - A conflict between CFLAGS and FFLAGS
  - A problem with your compiler installation(s)
  - Different default build options between compilers (e.g., C
    building for 32 bit and Fortran building for 64 bit)
  - Incompatible compilers

Such problems can usually be solved by picking compatible compilers
and/or CFLAGS and FFLAGS.  More information (including exactly what
command was given to the compilers and what error resulted when the
commands were executed) is available in the config.log file in this
directory.
**********************************************************************
configure: error: C and Fortran 77 compilers are not link compatible.  Can not continue.

Digging into config.log showed this error:

onfigure:35860: checking if C and Fortran 77 are link compatible
configure:35910: /usr/bin/gcc -c -O3 -DNDEBUG -finline-functions -fno-strict-aliasing  conft
est_c.c
configure:35917: $? = 0
configure:35944: /usr/local/bin/gfortran -o conftest   conftest.f conftest_c.o  >&5
ld: warning: in conftest_c.o, file was built for unsupported file format which is not the architecture being linked (i386)

The problem was twofold: first, I needed to compile 64-bit binaries with the C compiler, which wasn't happening, because it was trying to create an i386 binary. Second, the Fortran compiler was producing an "unsupported file format".

I fixed the problem by adding the "-arch x86_64" flag to both the C and Fortran compilers.

Bad register name

Initially, I tried to create MPI compiler wrappers that would create fat binaries that would work with either 32-bit or 64-bit platforms. I used a configure line that specified -arch for both 32 and 64 bit, like this:

#!/bin/sh

./configure \
    --prefix=${HOME}/pkg/openmpi/1.4.3 \
    CC="/usr/bin/gcc -arch x86_64 -arch i386" \
    CXX="/usr/bin/g++ -arch x86_64 -arch i386" \
    F77="/usr/local/bin/gfortran -arch x86_64 -arch i386" \
    \
    CPP="/usr/bin/gcc -E" \
    CXXCPP="/usr/bin/g++ -E"

and configure ran fine but when I got to make I saw this:

atomic-asm.S:5:bad register name `%rbp'
atomic-asm.S:6:bad register name `%rsp'
atomic-asm.S:13:bad register name `%rbp'
atomic-asm.S:14:bad register name `%rsp'
atomic-asm.S:21:bad register name `%rbp'
atomic-asm.S:22:bad register name `%rsp'
atomic-asm.S:30:bad register name `%rdi)'
atomic-asm.S:38:bad register name `%rsi'
atomic-asm.S:39:bad register name `%rdx'
atomic-asm.S:48:bad register name `%rdx'
atomic-asm.S:50:bad register name `%rdx'
lipo: can't open input file: /var/folders/fk/fkDjOi33E4Og4oM1PTL3kk+++TI/-Tmp-//ccEEm4zh.out (No such file or directory)
make[2]: *** [atomic-asm.lo] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1

Googling "bad register name rbp" or any of the other "bad register name" lines above turns up a lot of information about problems when building a 32-bit binary with a compiler in 64-bit mode, or vice-versa. The problem turned out to be a problem with the build system (GNU Autotools) that OpenMPI uses - in some cases it's just not possible to build fat binaries in one-pass. See the #Building Fat Binaries section above for details on how to resolve the issue and build fat binaries.

NOTE: Normally, you can use the techniques specified on the Compiling_Software#Compiler_Specifications page to build fat binaries, but in this case it didn't work.