Portals4 Runtime Execution Options
Execution of the compiled program with Portals4 support requires the Portals 4 Reference Implementation Library. Both the Portals4 shared library and yod job launcher are required to successfully run the GNU UPC program compiled for Portals4.
By default the Portals 4 Reference Implementation Library installs in the /usr/local directory. For most of the systems /usr/local/bin and /usr/local/lib are already added by the system to the user’s execution and library paths. However, if the Portals4 library is installed in a different place (e.g. /usr/local/gupc-p4) access to the shared libraries and yod job launcher must be provided. There are two recommended methods for identifying the location of the Portals4 library, prior to running a linked UPC program:
- Add the location of the Portals4 library to the LD_LIBRARY_PATHenvironment variable. For example,
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/gupc-p4/lib" export LD_LIBRARY_PATH
- As system administrator add an entry into the system’s shared library configuration directory. For example (Fedora Core x86_64):
su root echo '/usr/local/gupc-p4/lib' > /etc/ld.so.conf.d/portals4-x86_64.conf chmod a-w /etc/ld.so.conf.d/portals4-x86_64.conf ldconfig exit
Also, make sure that the yod job launcher is on your PATH. For example if your default shell is bash:
Running the program with yod
Hydra Program Manager must be set up to allow for yod -n N executable, where N is number of threads to spawn, command to properly launch the executable over the InfiniBand network.
More information on Hydra PM can be found at anl.gov web site.
Add the following environment variables to use Hydra’s SSH launcher:
In the example above, slurm launcher can be used instead of ssh if properly set.
export HYDRA_HOST_FILE=/path/to/nodes/hostsfile export HYDRA_LAUNCHER=ssh
The host file defined by the HYDRA_HOST_FILE defines the compute nodes (hosts) used for the program execution. For example:
% cat /path/to/nodes/hostsfile thor1 thor2 thor3 thor4
A simple invocation of a UPC program is shown below.
yod -n N upc_program
where N is the number of UPC threads (i. e., the value of THREADS) to instantiate.
A file containing the compute nodes list can also be specified on the yod command line:
yod -f hostsfile -n N upc_program
The compute nodes can also be specified on the yod command line:
yod -hosts thor1,thor2 -n N upc_program
As yod auto detects the SLUM resource manager, a UPC program can be executed in the SLURM environment. For example, salloc can be used to allocate resources for the UPC program:
salloc -n 8 yod upc_program
By using yod, a UPC program can also be executed in the SLURM batch scripts.
Above, yod's option for number of threads is not needed as it is acquired from the SLURM allocation.
When executing in the SLURM environment, HYDRA_HOST_FILE environment variable must not be set. Also, there is no need for the HYDRA_LAUNCHER=slurm environment variable either.
Program Exit Code
The exit code from the UPC application program is provided to the user as a result of invoking the yod job launcher.
Additional application program arguments can be specified on the yod command line right after the name of the program. For example:
yod -n 16 upc_program arg1 arg2 ...
The yod job launcher provides the following options:
- Specify the number of threads to run. Note that number of specified yod threads must match the number of statically compiled UPC threads.
- Specify the list of compute nodes to execute on.
- -f hostfile
- Specify the file containing the list of compute nodes.
To get more information on other yod options use the following command:
The following environment variables will affect UPC program execution.
- UPC_SHARED_HEAP_SIZE sets the maximum amount of shared heap (per UPC thread) for the program. The default is 256MB per UPC thread. The provided heap size value is optionally multiplied by a scaling factor. Valid scaling factor suffixes are: K (Kilobytes), M (Megabytes), G (Gigabytes), and T(Terabytes). For example, to allocate the heap size of one (1) Gigabyte:
bash export UPC_SHARED_HEAP_SIZE=1G
csh setenv UPC_SHARED_HEAP_SIZE 1G
- TMP | TMPDIR
- A path to use for file based mmap-ed node local memory access optimization. By default /tmp is used.
- Disable node local memory access optimization by setting this environment variable to OFF. Useful for debugging purposes only.
- Disable startup page by page access of the local shared memory by setting this environment variable to _0_. Page by page memory touch ensures the correct memory affinity among threads running on the same node. Useful for faster startup time on systems with only one thread per node.
- Enable backtrace generation if a fatal error occurs in the UPC program. Set this environment variable to 1 to enable backtrace. [default: disabled]
- Template for the backtrace files if explicitly requested by the user. [default: stderr]
- The file path of the GDB debugger that will be used to generate a backtrace. [default: gdb]
Node Local Memory Access Optimization
The GUPC for Portals4 runtime supports node local memory access optimization. Access to shared memory of threads on the same node is performed via direct memory access instead of Portals4 PUT/GET routines. Portals4 library is used to determine which threads reside on the same node.
The node local memory access supports the following options:
- POSIX shared memory is used to map and access other threads shared memories. POSIX shared objects are named as upc-mem-THREADID-PID. This is the default configuration.
- File based mmap-ed memory is used to map and access other threads shared memories. To activate this option specify --with-upc-node-local-mem=mmap as the GUPC configuration option. By default files are created under /tmp directory. This can be changed in the execution time by specifying the desired path with TMP or TMPDIR environment variables. Files are named in a similar fashion as POSIX shared objects.
Node local memory access optimization can be disabled in the configuration time by specifying --disable-upc-node-local-mem option or by setting the environment variable UPC_NODE_LOCAL_MEM=OFF in the execution time.