8 Output of informational messages of Run-Time System and error messages

8.1 Functions of informational and error messages output

Each message, reported by Run-Time System, is either informational or error message.

Informational messages are output by the functions rtl_printf, rtl_mprintf (see section 1.3), and by the function pprint with parameter prefix, equal to 0 or 1 (see section 7.3).

Moreover, for informational messages output Run-Time System provides the function rtl_iprintf that is syntactically equivalent to the function printf and returns the number of output symbols:

int rtl_iprintf (char *format, . . . );

Any trace information, output to informational message stream by tprintf and dvm_trace functions (see section 7.3) is informational message too.

Error message output is performed by eprintf and epprintf functions, and also by the function pprint with prefix parameter, equal to 2 or 3 (see section 7.3).

8.2 Controlling standard output streams and streams of informational and error message output

The direction and redirection of the streams, intended for output of information and error messages, is performed by the parameter current.par and syspar.* files.

The file current.par is the parameter file of current startup of Run-Time System and is created by Run-Time System startup program in the current working directory. The file consists of the following line sequence:

CurrentPar[0] = <parameter>;
CurrentPar[1] = <parameter>;
. . . . . . . . . . . . . . . . . . . . . . . .
CurrentPar[n] = <parameter>;

The parameters, placed from the right of "=" and determining information output, can be:

1. +o[<regime>][<path>[.<extension>]]

The parameter instructs Run-Time system to redirect stdout stream to the file (files with name (names).

[<path>][<processor number>].<extension> .

If <extension> option is not specified, the extension of the file (files) is "sto".

The <regime> option defines a way to redirect stdout stream and can be "*" or "+". In the first case the redirection is performed at all the processors to its own for every processor file with the name above. In the second case the redirection is also performed at all the processors but to the common for all the processors file with the name.

<path>.<extension> .

At that if <path> option (together with <extension> option) is not specified, the redirection is performed to the file with "stdout" name (without extension).

If <regime> option is omitted, the redirection of stdout stream is performed only at the processor with zero internal number to the file <path>.<extension> (or to "stdout" file, if <path> option is not specified).

2. +e[<regime>][<path>[.<extension>]]

The parameter specifies the redirection of stderr stream to file (files) with a name (names)

[<path>][<processor number>].<extension> .

The redirection of stderr stream is simularly to stdout stream redirection, considered above but extension of the file (files) is "ste" by default, and the file name is "stderr" if <path> option is omitted.

3. +i[<symbols, determining streams>]

This parameter instructs Run-Time System to redirect informational messages to the streams, determined by specified symbols.

The symbols, determining the streams, can be:
"o" - stdout stream;
"e" - stderr stream;
"f" - file.

If both symbols "o" and "e" are specified and stdout and stderr streams are not redirected to the files or redirected to the same file, informational messages will be output to stdout stream only (informational messages are not duplicated). When informational messages are directed to the file (symbol "f" is specified) the file name (path) is defined by SysInfoFileName parameter in parameter syspar.* files (see below). If no symbols are specified informational messages will not be output.

Redirection of stdout and stderr streams to the files can be done using following parameters, specified in syspar.* files:

When parameters are output and processed the first specification of standard output stream redirection to the file has the highest priority. Partially stdout and stderr streams redirection using "+o" and "+e" parameters of current.par file suppresses the stream redirection by listed above parameters of syspar.* files (the file current.par is output and processed earlier than syspar.* file).

If the instruction "+i" is omitted in the file current.par the informational message output is determined by the parameters, specified in syspar.* file.

The parameters are:

If value of the parameters SysInfoStdOut and SysInfoStdErr is equal to 1 then informational messages are also redirected to the file and are output to the screen if stdout is redirected to the file (redirection of a user program output to the file must be specified by "+o" parameter of current.par file). Controlling values of the parameters any combination of SCREEN - STDOUT, REDIRECTED TO THE FILE can be obtained. If SysInfoStdOut and SysInfoStdErr parameters are equal to one, and stdout and stderr streams are not redirected or redirected to the same file informational messages will be output to stdout stream only (informational messages are not duplicated).

Note, that the redirection of informational messages using the current.par parameter "+i" has priority under the redirection, specified by the parameters SysInfoStdOut, SysInfoStdErr and SysInfoFile of syspar.* file.

Output of informational messages to the file (SysInfoFile = 1) is also controlled by next parameters of syspar.* file:

If FatInfoNoOpen parameter is equal to zero opening failure of informational message file causes warning output to the error message stream and then the program will run without output of informational messages to the file.

If SysInfoSepFile is equal to 0, each processor outputs informational messages directly to the file, specified by SysInfoFileName parameter. If SysInfoSepFile is equal to 1, each processor accumulates informational messages in its own temporary file, having name <internal processor number>.out. After work completion all temporary files are jointed in the one file (specified by parameter SysInfoFileName), and then are deleted.

All error messages are:

• output to stderr stream, if stdout and stderr streams are not redirected to the same file;
• directed to the stdout stream, if user program output (or stderr stream) was redirected to the file;
• output to the file, specified by the parameter SysInfoFileName, if SysInfoFile parameter is equal to 1 (to the file, informational messages are output to).

Therefore error messages always:

• accompany output of the user program;
• are output to the screen (if stdout and stderr are not simultaneously redirected to the files).

Note 1. Output of informational messages before input of SysInfoPrint parameter value is determined by state of common flag of informational messages output at program startup (before Run-Time System initialization). The flag is assigned by non-zero value in the case if Run-Time System was compiled with predefined compilation variable _INIT_INFO_. Otherwise informational messages are not output up to input of SysInfoPrint parameter.

Note 2. Usually output of informational messages is performed as follows. Up to input of parameters informational messages are directed to the stdout stream. After SysInfoStdOut and SysInfoStdErr parameters input if the fist parameter is equal to zero but the second one is not, informational messages will be redirected to stderr stream. If both parameters are not equal to zero, and stdout and stderr streams are redirected to the screen or to the same file, informational message will be output to the stdout stream only.
Informational message output with stderr stream priority is provided as alternative. For this purpose Run-Time System should be compiled with defined compilation variable _INFO_STDERR_. As result stderr stream will be play the same role as stdout stream in standard scheme considered above.

Note 3. If informational message output is set to the file and to stdout (stderr) stream, but stdout (stderr) stream is also redirected to this file, output of informational messages to the file is locked (SysInfoFile parameter is set to zero).

8.3 Classes of informational messages and their output controlling

All informational messages are divided on non-crossed classes; each of them is corresponded to functioning of certain component of Run-Time System. Output of each class messages is controlled independently by listed below parameters from syspar.*, statist.*, systrace.* and usrdebug.* files. The parameter is considered to be specified in syspar.* files by default.

• VersStartPrint – flag to output Run-Time System version number before starting the program.
• VersFullStartPrint – flag to output Run-Time System version number and version numbers of all parameter files before starting the program.
• VersFinishPrint – flag to output Run-Time System version number after the program termination.
• VersFullFinishPrint – flag to output Run-Time System version number and version numbers of all parameter files after the program termination.
• SysParPrint – flag to output Run-Time System initialization parameters before starting the program.
• ProcListPrint – flag to output processor matrix before starting the program. Internal and external numbers and identifier are reported for each processor.
• WeightListPrint – flag to output coordinate processor weights before starting the program.
• ParamRunPrint – flag to output information about initial parameters and starting modes of Run-Time System before starting the program (standard or non-standard startup, base message passing system type and so on).
• EndProgMemoryPrint – flag to output information about non-freed memory when terminating work.
• EndProgObjectPrint – flag to output information about not deleted static DVM-objects when terminating work.
• EndProgCheckSumPrint – flag to output information about state of checked memory areas when terminating work.
• InputParPrint – flag to output messages about input beginning for each parameter file.
• EndReadParPrint – flag to output messages about input completion for each parameter file.
• TimeExpendPrint - mode to output time spent by the program. If the parameter is equal to zero, time consumption information is not reported. The parameter, equal to 1, specifies brief output mode of time consumption information. The parameter with value, more than 1 (2,3,4,5) specifies detailed mode. The parameter is set in files with statist name.
• CallCountPrint – flag to output statistics of Run-Time System function calls, acceptable for the user program (for each function the count of function calls is reported). The parameter is set in the files with statist name. If CalìCountPrint value is equal to one the information about functions, invoked at least one time, is output only and if the value is equal to 2, information about all functions is output.
• TraceClosePrint – flag to output messages about completion of uploading each buffer with accumulated system trace into a file (when tracing in memory buffer). The parameter is set in the files with name systrace.
• DbgInfoPrint – flag to output informational messages of dynamic control subsystem and execution trace. The parameter is set in files with usrdebug name.

Messages of any class are output only in the case if common flag of informational message output is set (using SysInfoPrint parameter, considered in section 8.2).

9 Measuring time of user program interval execution

In addition to main tools of performance analysis and gathering information, required for it, Run-Time System provides subsidiary mechanism of measuring execution time of user program intervals. When this mechanism is used the results of measuring are output to system trace streams. To mark beginning and end of measured interval macrocalls MeasureStart and MeasureFinish are used. Measured intervals can be nested:

The macrocall MeasureStart is equivalent to the function dvm_trace call (see section 7.3) with event number 1 (the event name is MeasureStart), and macrocall MeasureFinish is equivalent to dvm_trace function call with event number 2 (the event name is MeasureFinish). As additional information for the macrocall MeasureStart the enclosure level of measured interval (1, 2, … , N in the scheme above) is output and for the macrocall MeasureFinish the enclosure level and program execution time, passed from MeasureStart macrocall moment with the same enclosure level, are output to the opened streams.

Note that trace events with numbers 1 and 2 can be disabled as any other events. That is alternative to exclusion of MeasureStart and MeasureFinish macrocalls from the program.

10 Requesting global and local sizes of specified dimension of specified object

The function GetObjectSize returns size of the object specified by reference ObjRef for the dimension Axis (global size or the size of non-distributed object).

The object can be:

• distributed array (then a reference - it is the first word of its header);
• abstract machine representation;
• processor system;
• parallel loop.

The size of parallel loop dimension is equal to

ceil ( (LastIndex - InitIndex + 1)/Step )

where:

If Axis equal to 0, the size of whole object (product of sizes of all dimensions) is returned.

If ObjRef is not reference to any object above, the function returns zero.

The function GetObjectLocSize is similarly to GetObjectSize function considered above but returns local size of dimension Axis (or local size of the whole object) that is the size of the object part, mapped on the current processor. If the object is processor system, then the values, returned by GetObjectSize and GetObjectLocSize functions are equal.

Note that the local size of whole object having no local part on the current processor is equal to zero.

The functions GetObjectSize and GetObjectLocSize are not traced and not processed by statistics accumulation program, therefore they are faster, then getsiz_ and locsiz_ functions, considered in document rtsIDe.doc.

11 Tools to support Run-Time System built-in program of statistic information accumulation

11.1 Matrix of execution characteristics of the user program current interval

11.1.1 Primary execution characteristics of user program current interval

All Run-Time System functions, the user program can invoke, are subdivided on the following not crossed groups:

The group names are defined as C named number constants in system.def file of Run-Time System source files. A number of function groups is defined by named constant StatGrpCount. The correspondence <function name> Þ <group name> is defined in source file intergrp.c.

Execution characteristics of functions of every group are gathered in the current interval time consumption matrix IM, created when new current interval is created (interval matrix of characteristics, matrix of the current interval characteristics). When matrix IM is created, its pointer must be assigned to the global variable

s_GRPTIMES (* CurrInterPtr)[StatGrpCount]; .

After leaving the current interval statistics accumulation program must process contained in matrix IM data and restore in CurrInterPtr variable the pointer to the matrix of the interval, completed interval is nested in.

The matrix IM is squire matrix, which i-th row elements contain execution characteristics of the functions of i-th group, directly called by the user program. Each function of i-th group can invoke (directly or indirectly) the functions of any j-th group (i and j can be equal). Run-Time System accumulates characteristics of such calls in IM[i,j] element. So, i-th row of matrix IM contains decomposition of execution characteristics of i-th group functions, directly called by user program, by direct or indirect function calls of all groups.

IM[i,j] element is a structure of the type

where:

When IM matrix is created, zero value must be assigned to all listed above characteristic in all elements of the matrix.

CallCount characteristic is increased by 1/n every time after exit from j-th group function (n is number of processors, executing current branch of the parallel program).

ProductTime and LostTime characteristics are increased by time/n and (n- 1)*time/n correspondently after exit from j-th group function (time is execution time of j-th group function not taking into account the time of the calls of other group functions, n is a number of processors, executing current branch).

On completion of the function (of i-th group), called directly by the user program, ProductTime and LostTime characteristics, contained in IM[i][i] element, and CallCount characteristic, contained in IM[UserGrp][i] element, are increased (about row with UserGrp number see below). It allows, first, to have total elapsed time of i-th group functions, called by the user program, in one row of IM matrix, and, second, to have information about amount of each group function calls, performed directly by the user program.

The meaning of characteristics, contained in elements of the row with UserGrp number, differs from the meaning of characteristics in other rows of matrix IM. ProductTime and LostTime characteristics of IM[UserGrp][UserGrp] element is productive and lost times of the user program itself (not taking into account the time of Run-Time System function calls). ProductTime and LostTime characteristics in all other elements of UserGrp row are equal to zero. CallCount characteristic of [UserGrp][i] element is zero, and for all other elements IM[UserGrp][j] (j ¹ UserGrp) contains average number of j-th group function calls from the user program per processor.

11.1.2 About characteristics of passing message function group

Besides the function group UserGrp, considered in section 11.1, program execution characteristics of the function group MsgPasGrp (message passing) also differ from the same named characteristics of other groups.

All execution time of any message passing function is considered as lost time, that is accumulated in LostTime characteristic in "pure" form, that is without correction, taking into account a number of processors, executing the current branch (without multiplication by (n-1)/n coefficient) The number of message passing function calls are also summarized in CallCount characteristic without division by n. In ProductTime characteristic time of real (actual) dissynchronization of the program is accumulated instead of average productive time per processor.

Real dissynchronization time is integral measure of productive time loses because of not simultaneous performing of message receiving by one processor and invoking the function, sending this message, by the other processor. To calculate this time, Run-Time System keeps astronomical time of message passing function call as auxiliary information, accompanying the message (origin astronomical time at all the processors is the same, see time synchronization in section 6). Depending on used message passing scheme (see section 5), the time of real dissynchronization is calculated at receiving side in the following way.

1. Synchronous message exchange.

Real dissynchronization time is equal to absolute value of difference between the time of the function rtl_Recv call by receiving side and the time of the function rtl_Send call by sending side.

2. Asynchronous message exchange.

Real dissynchronization time is equal to difference between time of the function rtl_SendA call by sending side and time of the function rtl_RecvA call by receiving side. If the difference is negative, real dissynchronization time is zero.

3. Message exchange in "NO WAIT" mode.

Real dissynchronization time is equal to difference between time of the function rtl_Sendnowait call by sending side and time of the function rtl_Waitrequest call by receiving side. If the difference is negative, real dissynchronization time is zero.

Total time of real dissynchronization is a sum of real dissynchronization times of separate interconnection actions <send-receive>.

11.1.3 Calculation of summary characteristics of program execution using interval matrix of characteristics

Denote the set of function group numbers as GNS:

GNS = { MsgPasGrp, StartRedGrp, WaitRedGrp, RedGrp, StartShdGrp, WaitShdGrp, ShdGrp, DistrGrp, ReDistrGrp, MapPLGrp, DoPLGrp, PrgBlockGrp, IOGrp, RemAccessGrp, UserDebGrp, StatistGrp, SystemGrp }

1. Characteristic of own productive time of the program execution itself (not taking into account Run-Time System calls):

OUPT = IM[UserGrp][UserGrp].ProductTime

2. Characteristic of own lost time of program execution itself (not taking into account Run-Time System calls):

OULT = IM[UserGrp][UserGrp].LostTime

3. Characteristic of productive execution time of i-th group functions, directly called by the user program (i ¹ MsgPasGrp):

4. Characteristic of productive execution time of Run-Time System functions, directly called by the user program:

5. Characteristic of lost execution time of i-th group functions, directly called by the user program:

6. Characteristic of lost execution time of Run-Time System functions, directly called by the user program:

7. Characteristic of productive time of user program execution:

UPT = OUPT + SPT

8. Characteristic of lost time of the user program execution:

ULT = OULT + SLT

9. Characteristic of the number of Run-Time System function calls, performed directly by user program:

10. Characteristic of real dissynchronization time when executing i-th group functions, directly called by user program:

RMT_i = IM[i][MsgPasGrp].ProductTime

11. Characteristic of real dissynchronization time of user program execution:

12. Characteristic of own productive time of j-th group function execution (not taking into account other group function calls) (j ¹ MsgPasGrp):

13. Characteristic of own time of j-th group function execution (not taking into account other group function calls):

11.1.4 Auxiliary matrix of user program characteristics

To control and debug information accumulation and visualization for performance analysis Run-Time System has auxiliary matrix of user program execution characteristics:

s_GRPTIMES TaskInter[StatGrpCount][StatGrpCount]; .

The matrix TaskInter corresponds to initial interval of the user program execution. The information accumulation is performed independently on main mechanism of statistics accumulation. Program execution characteristics, contained in TaskInter matrix elements, are not corrected by the number of processors, executing the current branch.

On program termination specified part of TaskInter matrix is output to Run-Time System informational message stream.

The parameters TimeExpendPrint and StatGrpName of statist.* files control information accumulation in TaskInter matrix. If TimeExpendPrint is equal to zero, the information accumulation in TaskInter matrix is not done. Non-zero value of the parameter defines accumulation mode and structure and form of output information.

11.1.5 Supporting characteristic accumulation of input/output DVM-operations implemented in Fortran

Usually when the user program calls any Run-Time System function, the function group number is saved in global variable

int CurrOper; .

During the called function execution and on its completion the value of CurrOper variable is used as matrix IM row number to keep measured characteristics.

When DVM input/output operations in Fortran are executed, Run-Time System functions that are not input/output ones are called. To correct accumulation of characteristics of Fortran input/output operations it is necessary:

• to assign CurrOper variable IOGrp value (input/output function is executed);
• to save unchanged the value of CurrOper variable up to input/output operation completion when any Run-Time System function is called by the user program;
• to allow CurrOper variable updating after input/output operation completion.

To implement these requirements Run-Time System provides the functions

void biof_(void);

and

void eiof_(void); .

The user program at the beginning of input/output operation calls the function biof_. The function biof_ assigns IOGrp value to CurrOper variable and 1 to global variable

unsigned char CurrOperFix; .

If CurrOperFix variable has non-zero value, CurrOper value is not changed when the user program invokes Run-Time System functions.

The function eiof_ is intended to inform Run-Time System about input/output operation completion. Its execution consists of the assignment to CurrOperFix zero value, allowing updating CurrOper value.

11.2 Tools of external controlling statistics accumulation program

The external control of statistics accumulation program functioning is implemented by the following parameters, specified in parameter files statist.*:

11.3 Initializing statistics accumulation program

When statistics accumulation program is initialized, the values are assigned to the following global variables (corresponding to the external parameters, considered in section 11.2):

After initialization of mentioned above global variables the function

void stat_init (void); (statist.c file)

is called to more specific statistics accumulation tuning.

In particular, stat_init function must give to Run-Time System initial interval characteristic matrix (must initialize CurrInterPtr pointer, see section 11.1.1).

After stat_init function return Run-Time System considers that statistics accumulation is activated.

11.4 Processing calls of Run-Time System functions by statistics accumulation program

After exiting stat_init function, considered in section 11.3, Run-Time System begins to accumulate characteristics of user program execution in interval matrix of characteristics.

The characteristics are counted in IM matrix on entry in and exit from any Run-Time System function, accessible to the user program. Entry in and exit from some Run-Time System functions require additional processing. Such functions are the functions, which call and exit times are required to calculate user program potential dissynchronization and time variation. When entering and/or leaving such function Run-Time System calls the function

void stat_event(int EventNumber); (statist.c file),

where EventNumber is occurred trace event number. The parameter EventNumber exactly defines as processed function as occurred event type (entry in processed function or exit from it).

The stat_event function is called by EventNumber event only if EventNumber-th element of global array

unsigned char IsStat[];

is non-zero. Non-zero (1-255) value of IsStat[EventNumber] byte defines the processing way of the event with EventNumber number by stat_event function. The array IsStat is initialized in Run-Time System module statevnt.c, where for entry in and exit from each function, accessible for user program, zero or non-zero value of corresponding element of IsStat array is set.

Moreover, stat_event function can be invoked only if the following requirements, common for all processed functions, are satisfied:

• real dissynchronization time and time variation counting is enabled (Run-Time System startup parameter IsTimeVariation is non-zero);
• processed function is called directly by user program or Run-Time System function dopl_ for starting or waiting for shadow edge renewing;
• processed function is not a function, called directly by user program in the time of input/output DVM-operation execution in Fortran (CurrOperFix is zero, see section 11.1.5).

Before the function stat_event call Run-Time System assigns to the global variable

ObjectRef StatObjectRef;

reference to the current processed object. Such object dependently on occurred event, can be:

• processor system;
• abstract machine representation;
• distributed array;
• distributed array map;
• parallel loop;
• reduction variable;
• reduction group;
• shadow edges group of distributed array.

For a distributed array the zero-numbered word of its header is assigned to the variable StatObjectRef.

The reference to the current processed object kept in variable StatObjectRef, can be used by statistics accumulation program to obtain inquired information about object by Run-Time System functions.

Note, that prior to stat_event function call Run-Time System keeps current astronomical time in global variable

double Curr_dvm_time; .

So during stat_event function execution the current time request by dvm_time function is not needed.

11.5 Terminating statistics accumulation program

When terminating the user program Run-Time System invokes the function

void stat_done (void); (statist.c file)

for final processing gathered information by statistics accumulation program (if such processing is required). After exit from this function Run-Time System writes statistics accumulation buffers of all the processors into bit file with the name, defined by the StatFileName parameter of external control (see section 11.2). The processor buffers are placed in this file sequentially in the order of increasing internal numbers of the processors. Then it is considered statistics accumulation is disabled.

11.6 Access to some system information

Some characteristics of Run-Time System and its hardware-software environment may be useful and are contained in the following global variables.