Here is a MPI rank file generator script written in bash for use on Blue Biou. The idea here is to assign tasks (individual processes in an MPI program) intelligently on a Power7 machine with four eight-core, quad-thread processors.

If you run <= 32 tasks per node, it will skip every four threads ensuring you run one task per core.

If you run <= 64 tasks per node, it will assign threads 1 and 2 of each core, skipping 3 and 4.

The script takes as input the filename of a PBS generated node file. It is safe to pass the environment variable $PBS_NODEFILE if using this script within a job submission script.

#!/bin/bash
# Given a PBS nodes file, generate a rank file for openmpi
#
# Chandler Wilkerson
# Fri Mar 12 2010

function usage() {
    echo "Usage: $0 PBS_NODEFILE"
    echo
    exit 1
}

test $# -lt 1 && usage

test -s $1 || usage

NODES=$(uniq $1)
NNODES=$(uniq $1 | wc -l)

NTASKS=$(cat $1 | wc -l)

PPN=$(($NTASKS / $NNODES))

FIRSTNODE=$(head -n 1 $1)

# Sanity checks
#

if [ $NTASKS -ne $(($PPN * $NNODES)) ]; then
    echo "Number of tasks per node is not even?"
    exit 1
fi

# This script needs to run on the first node.

if [ $HOSTNAME != $FIRSTNODE ]; then
    echo "This script must be run on the first node"
    echo " (If this is the case, this script is getting confused)"
    exit 1
fi

# Make it so:

if [ $PPN -le "32" ]; then
    stride=4
    threads=0
    nthreads=1
elif [ $PPN -le "64" ]; then
    stride=4
    threads="0 1"
    nthreads=2
elif [ $PPN -le "96" ]; then
    stride=4
    threads="0 1 2"
    nthreads=3
else
    stride=1
    threads=0
    nthreads=1
fi

CURRTASK=0

for node in $NODES; do
    for task in $(seq 0 $(($PPN/$nthreads - 1)) ); do
        for thread in $threads; do
            CURRSLOT=$(($task*$stride + $thread))
            echo "rank ${CURRTASK}=$node slot=$CURRSLOT"
            CURRTASK=$(($CURRTASK+1))
        done
    done
done

Here’s an example of how to use the script:

PATH_TO_SCRIPT/build_rankfile.sh $PBS_NODEFILE > $TMPDIR/rankfile.txt
...
mpiexec -rf $TMPDIR/rankfile.txt ...