The GUPC runtime supports UPC program backtrace (also called stack backtrace or stack traceback). It is used during interactive and/or post-mortem debugging and can be used to determine the sequence of nested functions called up to the point where the backtrace is generated.

The GUPC backtrace is generated in the following situations:

  1. On catastrophic events when the GUPC runtime aborts the running thread. By default, the GUPC backtrace uses GDB to provide detailed information on the thread’s stack frames. Only the first thread reaching the abort statement generates a backtrace log.
  2. On a specific request from the user. Sending a pre-configured signal to the operating system process associated with a specific UPC thread will cause a backtrace to be generated for that UPC thread. By default, SIGUSR1 is used to signal a backtrace.

By default, backtrace on catastrophic events is disabled, but it can be enabled by setting the "UPC_BACKTRACE" environment variable.

Backtrace via a user request can be redirected to a file or to stderr. By default stderr is used.

Backtrace Logs

There are three kinds of backtrace logs depending on the configuration and the capabilities of the underlying operating system.

DETAILED
GDB’s 'bt' command is used to produce the backtrace with nested procedures and their arguments shown. To get full benefit of this back trace, an executable must be compiled with debugging turned on (e.g. -g -O0). Detailed backtrace logs are produced only on catastrophic events.
SIMPLE
The GLIBC backtrace capability (http://www.gnu.org/software/libc/manual/html_node/Backtraces.html) and and addr2line program are used to produce the source file and line number for each stack frame.  addr2line (part of binutils) must be installed when GUPC is configured..
RAW
The GLIBC 'backtrace' capability is used to produce procedure names and addresses for each stack frame. Depending on the compiler option '-rdynamic' (instructs the ELF linker to create a special section with additional symbols) backtrace will generate differing output.
With the '-rdynamic' option
Procedure name, addresses, and offsets from the beginning of the procedure is displayed for each frame.
Without the '-rdynamic' option
Raw addresses are displayed. The UPC runtime library checks if the '-rdynamic' option is supported and adds the appropriate options on the linker command line, if needed.

Backtrace Events

The following events create backtrace logs (depending on the GUPC configuration and environment variable settings):

  1. Fatal signal (SIGSEGV, SIGBUS, SIGFPE) or GUPC runtime failure
  2. User defined signal received by the UPC thread. The user can request a backtrace log by sending a predefined signal to the process that has a UPC thread mapped to it.

By sending a signal to a UPC thread (a process associated with the UPC thread) causes the process to dump backtrace information. The following is an example backtrace output if the addr2line program is present on the system:

[   3][0] upc_barrier.upc:295   __upc_wait()
[   3]       BARRIER ID: 0
[   3][1] upc_barrier.upc:340   __upc_barrier()
[   3][2] bt-example.upc:38     proc0()
[   3][3] bt-example.upc:61     upc_main()

Note that upon detecting a "__upc_wait()" stack trace, the backtrace code prints the barrier ID on the next output line.

As mentioned above, gdb is used to generate a backtrace when catastrophic events are detected. By default a segmentation violation in a UPC program produces the following output:

% ./segv-backtrace
./segv-backtrace: UPC error: Segmentation fault.
Aborted (core dumped)

However, with backtrace enabled the following output is generated:

% UPC_BACKTRACE=1 ./segv-backtrace
./segv-backtrace: UPC error: Segmentation fault.
Thread 0 GDB backtrace:
0x0000003af1eac2ce in __libc_waitpid (pid=<optimized out>, stat_loc=0x0,
options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:32
32            return INLINE_SYSCALL (wait4, 4, pid, stat_loc, options, NULL);
#0  0x0000003af1eac2ce in __libc_waitpid (pid=<optimized out>, stat_loc=0x0,
options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:32
#1  0x000000000040e430 in __upc_fatal_backtrace ()
#2  0x000000000040846d in __upc_fatal ()
#3  0x000000000040e5fc in __upc_fault_handler ()
#4  <signal handler called>
#5  0x0000000000406a65 in proc2 (a=5) at segv-backtrace.upc:5
#6  0x0000000000406a7e in proc1 (a=5) at segv-backtrace.upc:9
#7  0x0000000000406a95 in proc0 (a=5) at segv-backtrace.upc:13
#8  0x0000000000406aa5 in upc_main () at segv-backtrace.upc:21
#9  0x0000000000407d8c in __upc_run_this_thread ()
#10 0x0000000000407e1c in __upc_run_threads ()
#11 0x0000000000408604 in main ()
Aborted (core dumped)

The SMP-based UPC runtime has a monitor thread which creates the processes that are mapped to UPC threads and then monitors those processes. Depending on the configuration, sending a backtrace signal to the monitor thread causes the following depending on the value of the UPC_BACKTRACEFILE environment variable.

UPC_BACKTRACEFILE is NOT set
The UPC monitor thread shows the mapping between UPC thread numbers and their system process IDs (pid).
UPC_BACKTRACEFILE is set
The UPC monitor thread sends the signal to all UPC threads to dump their backtrace files. The location of the trace files depends on the value of the UPC_BACKTRACEFILE environment variable. By default, files are dumped in the current directory with file names in the form of "backtrace.THREAD-ID".

Backtrace Configuration

The following configuration options are provided to control the backtrace behavior in the GUPC runtime:

--enable-upc-backtrace
Enable/disable backtrace. [default=enabled]
--enable-upc-backtrace-gdb
Enable/disable usage of GDB for backtrace on catastrophic events. [default=enabled]
--with-upc-backtrace-gdb=[path-to-gdb]
Specify the GDB program to use to generate a catastrophic backtrace report. [default: gdb]
--enable-upc-backtrace-signal
Enable backtrace via user initiated signal. [default=enabled]
--with-upc-backtrace-signal=[SIGNAL]
Use the specified 'SIGNAL' for the backtrace requests. [default=SIGUSR1]

Note

Use of GLIBC backtrace capability is disabled if GLIBC does not support backtrace.

Backtrace Environment Variables

UPC_BACKTRACE
Enable backtrace for runtime fatal events. By default backtrace logging on fatal events is disabled (event though it may be configured).
UPC_BACKTRACE_GDB=[path-to-gdb]
Override the configured GDB for backtrace logging (e.g. UPC_BACKTRACE_GDB=/usr/local/bin/gdb).
UPC_BACKTRACEFILE="file-prefix"
If specified, backtrace lines are written to the files with the specified prefix. These files have names with the following form: file-prefix.THREAD-ID. By default (if no file-prefix is specified) the string "backtrace" is used.

Backtrace support for high-end tools (e.g. STAT)

By defining the UPC_BACKTRACEFILE environment variable, the user can redirect backtrace logs to a file instead of the screen (stderr). If an empty UPC_BACKTRACEFILE value is given, the backtrace log file names will in the form of "backtrace.MYTHREAD", Otherwise, UPC_BACKTRACEFILE is used as a file prefix to direct all files to the specified directory (e.g. if set to "/tmp/trace", backtrace log files will be in the form of "/tmp/trace-PID.MYTHREAD").

Backtrace log files can be used by the stat-merge python script developed by the GUPC and STAT developers.

The following image represents the result of the backtrace results created by the STAT tool.

STAT backtrace merge

Instrumentation

The GUPC compiler and runtime support instrumentation of UPC shared accesses and UPC runtime library calls, as well as instrumentation of all user-specified function calls.

Instrumentation is enabled by providing the "-fupc-instrument" and "-fupc-instrument-functions" options on the UPC program command line. Both of these options imply -fno-upc-inline-lib.

MPIR debugging support

The GUPC compiler runtime supports the MPIR parallel debugging protocol as specified by the MPI Forum Working Group on Tools: http://www.mpi-forum.org/docs/mpir-specification-10-11-2010.pdf

The following MPIR debugging options are supported:

  1. Start the UPC program with a parallel debugger tool
  2. Attach to an already running UPC program

The following MPIR variables/functions are present in the GUPC runtime:

VOLATILE int MPIR_being_debugged
MPIR_being_debugged is an integer variable that is set or cleared by the tool to notify the starter process that a tool is present.
MPIR_PROCDESC *MPIR_proctable
MPIR_proctable is a pointer variable set by the starter process that points to an array of MPIR_PROCDESC structures containing MPIR_proctable_size elements. This array of structures is the process descriptor table.
int MPIR_proctable_size
MPIR_proctable_size is an integer variable set by the starter process that specifies the number of elements in the procedure descriptor table pointed to by the MPIR_proctable variable.
VOLATILE int MPIR_debug_state
MPIR_debug_state is an integer value set in the starter process that specifies the state of the MPI job at the point where the starter process calls the MPIR_Breakpoint function.
VOLATILE int MPIR_debug_gate
MPIR_debug_gate is an integer variable that is set to 1 by the tool to notify the MPI processes that the debugger has attached.
void MPIR_Breakpoint() {}
MPIR_Breakpoint is the subroutine called by the starter process to notify the tool that an MPIR event has occurred.
int MPIR_partial_attach_ok
MPIR_partial_attach_ok is a symbol of any type (preferably int) that informs the tool that the MPI implementation supports attaching to a subset of the MPI processes.

Portals4 Debug Logging

GNU UPC configured for Portals4 runtime provides support for logging of specific runtime/system events (e.g. accesses to the shared memory). Logging is enabled through a set of environment variables that are set to a list of "facilities" that have debugging output logged.

Logging Environment Variables

The following environment variables control the logging capabilities of the Portals4 GNU UPC runtime:

UPC_DEBUG
If set, specifies a list of "facilities" that will have debugging output logged.
UPC_DEBUGFILE
Path of log file where UPC runtime debug logs are written.
UPC_LOG
Specifies a list of "facilities" that will be logged.
UPC_LOGFILE
Path of log file where UPC runtime logs are written.
UPC_NO_WARN
The UPC_NO_WARN variable causes startup warnings (such as those displayed when debugging or tracing is enabled) to be omitted.
UPC_QUIET
UPC_QUIET causes all non-application-generated output to be omitted (including both warnings and the initial display of UPC thread layout).
UPC_POLITE
Yield the processor frequently while spin-locking.
UPC_STATS
Specifies a list of "facilities" for will be logged.
UPC_STATSFILE
Path of log file where UPC runtime statistics are written.
UPC_TRACE
If set, specifies a list of "facilities" that will be traced.
UPC_TRACEFILE
Path of log file where UPC trace logs are written.

For all environment variables above that set a filename path, each appearance of a single % will be substituted with the process pid. Two % signs together escape a single %. Non-existent intermediate directories will be created. As a special case, if the filename is "stdout" or "stderr", then output will be directed to the specified file descriptor. A filename with no % indicates that the file will be shared across all processes.

Logging Facilities

The following logging facilities are provided:

ADDR
UPC casts to local and access to PTS’s.
ALLOC
UPC dynamic memory allocation
BARRIER
UPC barrier/notify/wait operations
BROADCAST
UPC runtime internal broadcast operations
COLL
UPC collectives
INFO
General information, program info.
LOCKS
UPC lock operations
MEM
UPC shared memory accesses
MISC
Miscellaneous functions
PORTALS
Portals operations
SYSTEM
System calls

For convenience, a facility "ALL" is provided to enable logging on all facilities.

ALL
Enable logging for all facilities.
Go to top