Portals4 Runtime Execution Options

Execution of the compiled program with Portals4 support requires the Portals 4 Reference Implementation Library. Both the Portals4 shared library and yod job launcher are required to successfully run the GNU UPC program compiled for Portals4.

By default the Portals 4 Reference Implementation Library installs in the /usr/local directory. For most of the systems /usr/local/bin and /usr/local/lib are already added by the system to the user’s execution and library paths. However, if the Portals4 library is installed in a different place (e.g. /usr/local/gupc-p4) access to the shared libraries and yod job launcher must be provided. There are two recommended methods for identifying the location of the Portals4 library, prior to running a linked UPC program:

  1. Add the location of the Portals4 library to the LD_LIBRARY_PATHenvironment variable. For example,
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/gupc-p4/lib"
    export LD_LIBRARY_PATH
  2. As system administrator add an entry into the system’s shared library configuration directory. For example (Fedora Core x86_64):
    su root
    echo '/usr/local/gupc-p4/lib' > /etc/ld.so.conf.d/portals4-x86_64.conf
    chmod a-w /etc/ld.so.conf.d/portals4-x86_64.conf
    ldconfig
    exit

Also, make sure that the yod job launcher is on your PATH. For example if your default shell is bash:

export PATH="/usr/local/gupc-p4/bin:$PATH"

Running the program with yod

Hydra Program Manager must be set up to allow for yod -n N executable, where N is number of threads to spawn, command to properly launch the executable over the InfiniBand network.

More information on Hydra PM can be found at anl.gov web site.

SSH Launcher

Add the following environment variables to use Hydra’s SSH launcher:

export HYDRA_HOST_FILE=/path/to/nodes/host/file
export HYDRA_LAUNCHER=ssh

In the example above, slurm launcher can be used instead of ssh if properly set.

export HYDRA_HOST_FILE=/path/to/nodes/hostsfile
export HYDRA_LAUNCHER=ssh

The host file defined by the HYDRA_HOST_FILE defines the compute nodes (hosts) used for the program execution. For example:

% cat /path/to/nodes/hostsfile
thor1
thor2
thor3
thor4

A simple invocation of a UPC program is shown below.

yod -n N upc_program

where N is the number of UPC threads (i. e., the value of THREADS) to instantiate.

A file containing the compute nodes list can also be specified on the yod command line:

yod -f hostsfile -n N upc_program

The compute nodes can also be specified on the yod command line:

yod -hosts thor1,thor2 -n N upc_program

SLURM Launcher

As yod auto detects the SLUM resource manager, a UPC program can be executed in the SLURM environment. For example, salloc can be used to allocate resources for the UPC program:

salloc -n 8 yod upc_program

By using yod, a UPC program can also be executed in the SLURM batch scripts.

Above, yod's option for number of threads is not needed as it is acquired from the SLURM allocation.

When executing in the SLURM environment, HYDRA_HOST_FILE environment variable must not be set. Also, there is no need for the HYDRA_LAUNCHER=slurm environment variable either.

Program Exit Code

The exit code from the UPC application program is provided to the user as a result of invoking the yod job launcher.

Program Arguments

Additional application program arguments can be specified on the yod command line right after the name of the program. For example:

yod -n 16 upc_program arg1 arg2 ...

YOD Options

The yod job launcher provides the following options:

-n
Specify the number of threads to run. Note that number of specified yod threads must match the number of statically compiled UPC threads.
-hosts
Specify the list of compute nodes to execute on.
-f hostfile
Specify the file containing the list of compute nodes.

To get more information on other yod options use the following command:

yod --help

Environment Variables

The following environment variables will affect UPC program execution.

UPC_SHARED_HEAP_SIZE
UPC_SHARED_HEAP_SIZE sets the maximum amount of shared heap (per UPC thread) for the program. The default is 256MB per UPC thread. The provided heap size value is optionally multiplied by a scaling factor. Valid scaling factor suffixes are: K (Kilobytes), M (Megabytes), G (Gigabytes), and T(Terabytes). For example, to allocate the heap size of one (1) Gigabyte:
bash
  export UPC_SHARED_HEAP_SIZE=1G
csh
  setenv UPC_SHARED_HEAP_SIZE 1G
TMP | TMPDIR
A path to use for file based mmap-ed node local memory access optimization. By default /tmp is used.
UPC_NODE_LOCAL_MEM
Disable node local memory access optimization by setting this environment variable to OFF. Useful for debugging purposes only.
UPC_FORCETOUCH
Disable startup page by page access of the local shared memory by setting this environment variable to _0_. Page by page memory touch ensures the correct memory affinity among threads running on the same node. Useful for faster startup time on systems with only one thread per node.
UPC_BACKTRACE
Enable backtrace generation if a fatal error occurs in the UPC program. Set this environment variable to 1 to enable backtrace. [default: disabled]
UPC_BACKTRACEFILE
Template for the backtrace files if explicitly requested by the user. [default: stderr]
UPC_BACKTRACE_GDB
The file path of the GDB debugger that will be used to generate a backtrace. [default: gdb]

Node Local Memory Access Optimization

The GUPC for Portals4 runtime supports node local memory access optimization. Access to shared memory of threads on the same node is performed via direct memory access instead of Portals4 PUT/GET routines. Portals4 library is used to determine which threads reside on the same node.

The node local memory access supports the following options:

POSIX
POSIX shared memory is used to map and access other threads shared memories. POSIX shared objects are named as upc-mem-THREADID-PID. This is the default configuration.
MMAP
File based mmap-ed memory is used to map and access other threads shared memories. To activate this option specify --with-upc-node-local-mem=mmap as the GUPC configuration option. By default files are created under /tmp directory. This can be changed in the execution time by specifying the desired path with TMP or TMPDIR environment variables. Files are named in a similar fashion as POSIX shared objects.

Node local memory access optimization can be disabled in the configuration time by specifying --disable-upc-node-local-mem option or by setting the environment variable UPC_NODE_LOCAL_MEM=OFF in the execution time.

Go to top