<= Debugging DVM-program (beginning)

Debugging DVM-program (continuation)
* February, 2001*

- last edited 07.10.02 -


11 Parameters of DVM-system

Base set of parameters, controlling Run-Time System operating, is provided with DVM-system and located in dvm_sys/par subdirectory. File names reflect purpose of the parameters, placed in the files:

checksum - checksums of the parameter files, listed below (they are checked only when parameter files with extension .rel are started; and they are not corrected);
statist - parameters, specifying data accumulation modes for DVM-program performance analyzer;
sysdebug - parameters of build-in debugging tools of Run-Time System;
syspar - parameters, controlling Run-Time System functioning;
systrace - parameters, specifying system trace modes;
trcevent - list and parameters of traced events;
usrdebug - parameters of dynamic control and user trace.

File name extensions reflects a way of applying given parameter subset:

.rel - parameter set, provided maximal speed of program execution;
.deb - parameter set for debugging a user program and Run-Time System; built-in Run-Time System tools of internal self-checking, system trace accumulation and informational messages output are enabled;
.par - sets of correcting parameters.

The correcting parameter sets are the files with the names:

deb_err, deb_trc, deb_red, deb_dif, deb_size, deb_ptrc - parameters used for dvm-command execution according to methods of program debugging (see section 10);
out - parameters of Run-Time System informational messages output, acting if environment variable dvmout=on (see sections 7 and 11.3);
outoff - parameters of Run-Time System informational messages output, acting if environment variable dvmout=off (see sections 7 and 11.3).

A user chooses the base set, more suitable for his purposes by specifying corresponding extension in environment variable dvmpar. He can also have own parameter sets (in any his working directory and with any file names and extensions), correcting the base set. The user specifies in these files only the parameters (contained in any of files with .rel or .deb extensions), that he wishes to correct. These directories and file names must be listed in environment variable usrpar of the dvm-command startup file (see section 7).

All parameters of the base set have the commentaries. Full lists of parameters of dynamic control and the user trace are presented in sections 11.1 and 11.2. Parameters of Run-Time System information messages output and main parameters, controlling system trace (as more frequently used when debugging a program) are considered in sections 11.3 and 11.4.

11.1 The parameters of dynamic control

The parameters of dynamic control, listed below, are set in files with name usrdebug.

11.2 Parameters of trace accumulation and comparison

The parameters, listed below, are set in files with the name usrdebug.

0 - generating trace configuration file. Only processor with the number TraceOptions.WrtHeaderProc writes to this file.
1 - trace accumulating and writing into the file <processor number>.<TraceOptions.Ext>.
2 - joining of the two previous modes.
3 - comparing the execution trace with the reference one, given in file TraceOptions.TraceFile.
0 - trace is not accumulated;
1 - accumulating loops and iterations trace only;
2 - the previous level plus accumulating trace of variable modifications;
3 - accumulating full trace.

11.3 Parameters of standard data streams redirection and Run-Time System informational messages output control

The listed below parameters are set in files with syspar, statist, systrace, trcevent, sysdebug and usrdebug names. A parameter is set in files with name syspar by default.

11.4 Parameters controlling system tracing

Controlling trace mode and trace event set is implemented by the parameters, kept in files systrace.* and trcevent.*. Let consider main from them. A parameter is kept in files systrace.* by default.

11.4.1 Enabling and disabling tracing

11.4.2 Specifying opened (enabled) trace streams

11.4.3 Specifying trace modes

11.4.4 Controlling form of accumulated information

11.4.5 Controlling internal self-checking Run-Time System mechanisms, functioning during trace accumulation

In the process of trace information accumulation Run-Time System can check certain memory areas. In the case of such area modification it terminates the program and outputs corresponding error message. Such areas are:

The control parts of dynamically allocated memory blocks are created by special instruction in parameter files sysdebug.* and are filled by the code, also specified in these files. Their contents can be checked each time, when memory is allocated or freed, and also in the process of tracing.

Explicitly specified memory area is checked by comparison its reference checksum with the current one. The reference checksum is calculated when Run-Time System is initialized.

The initial and final addresses of code memory are specified by the program, starting Run-Time System by means of writing these addresses to the Run-Time System executable module. The same program calculates and writes to Run-Time System executable module reference checksum of code memory. During its operating (in particular, when trace events occurred) Run-Time System checks code memory, comparing its current checksum with reference one.

Controlling mechanisms of memory area checking is implemented by the following parameters of systrace.* files.

11.4.6 Controlling output of additional information when tracing some Run-Time System functions in extended mode

0 - additional information is not reported;
1 - to output of imprint initial and last values of index variables of parallel loop for current processor;
2 - to output loop map additionally.

12 Diagnostics messages of dynamic debugger

The common format of error messages of dynamic debugger:

(<processor number>)<context> File: <file>, Line: <line> (<count> times) <error message>

where:

<processor number> - Number of processor, where error occurred. It is reported only if a program is executed on several processors.
<context> - Context, where the error occurred. It can be one of the following forms:
  sequential branch - the error occurred in sequential part of the program;
  Loop( No(N1), Iter(I1,I2,…)), …, Loop( No(Nm), Iter(I1,I2,…)) - the error occurred when m-dimensional loop was executed.
<file> - Name of file, where the error occurred.
<line> - Line number.
<count> - A number of given error repetitions in the given context. It is output when all detected errors are reported.
<error message> - Error description message.

12.1 Dynamic control

Error message

Description

Writing to read-only variable <var> Writing to read-only variable is detected.
Using non-initialized private variable <var> Access to non-initialized variable is detected.
Using non-initialized element <elem> Access to non-initialized distributed array element is detected.
Using variable <var> before asynchronous reduction competed Access to reduction variable before reduction operation completion.
Access to non-local element <elem> Access to non-local element of distributed array.
Writing to shadow element <elem> of array Writing to shadow element of array.
Shadow element <elem> was not updated Access to shadow elements before completion of shadow edge renewing operation.
Data dependence in loop due to access to element <elem> Data dependence in parallel loop is detected.
Using shadow element <elem> before asynchronous shadow renew competed Usage of shadow element <elem> of distributed array during shadow edge renewing operation execution.
Writing to remote data buffer <var> Writing to remote data buffer <var>.
Write to remote element <elem> in sequential branch Writing to distributed array element <elem> in sequential branch of the program without own computation specification.
Reading remote element <elem> in sequential branch Using not local element <elem> of distributed array in sequensial branch of the program.
WAIT for reduction without START Waiting for reduction completion is issued without the reduction start.
Using an element outside of array limits: <elem> Access to element of array beyond its limits.
START for reduction without WAIT Absence of waiting for asynchronous reduction completion operation for corresponding operation of asynchronous reduction start.
Reduction operation was not started Reduction variable is specified, but corresponding computation of reduction operation was never started.

12.2 Trace accumulation and comparison

Error message

Description

Bad file structure Trace file structure is incorrect.
Undefined keyword Unknown keyword appears in trace file.
Bad command syntax Wrong structure of trace record.
Can't open a file <file name> Specified file can’t be opened.
Trace file <file name> is empty Specified trace file is empty.
Bad trace structure (missing current program construct) Trace file structure is incorrect. Record of executable construct beginning is missed.
No current program construct Record of executable construct beginning is missed.
Unexpected task or iteration of loop There is no record about given iteration or task execution in reference trace.
Double execution of task or iteration, No = <iter no> Repeated execution of the same iteration or task.
Unexpected execution of program construct There is no record about given loop or task execution in reference trace.
Abnormal loop exit Loop end doesn't correspond to the record in reference trace.
Unexpected use of variable There is no given variable usage record in reference trace.
Unexpected trace record There is no given event performance record in reference trace.
Different <type> values: <standard value> != <current value> Variable value differ from the variable value in reference traced.
Different <type> values of reduction variable: <standard value> != <current value> Result value of reduction computation differ from the value in reference trace.

13 Structure of trace configuration file

# Trace size = <size of full trace file in bytes>
# String count = <a number of lines of trace file>
SL, PL or

TR

<construct number> (<number of surrounding construct>) [<construct rank>] {<file name>, <line number>} = <trace accumulation level>, (<dimension>:<first iteration>, < last iteration >, <iteration step>)
  # Trace size = <construct trace size in bytes for specified trace level>
  # String count = <number of construct trace lines for specified trace level>
  # Count of traced iterations = <number of traced loop iterations or tasks>

EL: <construct number>
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

SL, PL or TR <construct number> (<number of surrounding construct>) [<construct rank>] {<file name>, <line number>} = <trace accumulation level>, (<dimension>:<first iteration>, < last iteration >, <iteration step>)
  # Trace size = <construct trace size in bytes for specified trace level>
  # String count = <number of construct trace lines for specified trace level>
  # Count of traced iterations = <number of traced loop iterations or tasks>

EL: <construct number>

14 Execution trace structure

When executions are traced, accumulated trace information consists of two parts:

The header exists in trace even if the trace accumulation is disabled for whole program. Its structure looks like the structure of trace configuration loop, but without calculated values of trace size for whole program and for loops:

MODE = <accumulation trace level for whole program>,

SL, PL or TR <construct number> (<number of surrounding construct>) [<construct rank>] {<file name>, <line number>} = <trace accumulation level>, (<dimension>:<first iteration>, < last iteration >, <iteration step>)

EL: <construct number>
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

SL, PL or TR <construct number> (<number of surrounding construct>) [<construct rank>] {<file name>, <line number>} = <trace accumulation level>, (<dimension>:<first iteration>, < last iteration >, <iteration step>)

EL: <construct number>

Trace body is absent, when trace accumulation is disabled for whole program. Otherwise trace body consists of a lot of records of the following types:

15 Error messages of Run-Time System

The error messages, occurring in process of Run-Time System operating, can be divided (by error level) on warnings and error messages, breaking the program execution. The errors, breaking the program execution, can be usual (a user errors most likely) and fatal, requiring, possibly, a participation of program builder.

The error messages have one of the following forms:

*** RTS warning <number of error class> . <error number in class> : <error message text>
*** RTS err <number of error class> . <error number in class> : <error message text>
*** RTS fatal err <number of error class> . <error number in class> : <error message text>

The error class is defined by functional and module structure of Run-Time System.

The file name and line number (as user program, as Run-Time System), where the error occurred are reported before the error message:

USRFILE = <file name of user program>; USRLINE = <line number>;
SYSFILE = <file name of Run-Time System>; SYSLINE = <line number>;

The errors may be divided on two groups according to their occurrence way. The first group is the errors, occurring on all the processors at once. Input/output processor outputs such error messages. The second group is the errors that can occur not on all the processors. Each processor outputs such error messages itself, adding to the message its internal and external numbers. When working on one processor, the error messages are output without its numbers.

Run-Time System provides possibility to output all error messages of the first group by all the processors and all errors of the second group only by input/output processor. The first possibility is implemented if the parameter MultiProcErrReg is set equal to 2 (parameter files with name syspar). To output error messages of the second group only by input/output processor this parameter must be equal to zero (this possibility is used only in experimental purposes). Standard value of parameter MultiProcErrReg is equal to 1.

All the error messages, divided by subjects and ordered by class numbers and error numbers in the class are described in document "DVM-program Run-Time System. Detailed design".

The numbers of the messages, classified by possible causes, are listed below. The numbers of "fatal error" messages are not listed. The errors, which can't occur in the programs, produced by C-DVM and F-DVM compilers (in the case of correct functioning of the compilers), but can occur only when programming in terms of Run-Time System functions (however, in the case of memory corruption by user program any error are possible in any program) are also not considered.

15.1 Start and completion errors

These errors are indicated by the messages with 01x.xxx and 02x.xxx numbers.

The following messages belong to this type:

Moreover on the program termination the messages with 022.002 and 022.003 numbers can appear. These messages indicate wrong checksum of code and data memory, i.e. modifying the memory, which can't be modified ("memory corruption").

During the program execution the messages with 230.000, 230.001, 230.002, 230.003, 230.004, 250.000, 250.001 numbers can also indicate "memory corruption".

15.2 Errors of the type "... is not a DVM object"

"The object, passed to the function, is not DVM-object".

Possible reason is that the object is used before creation/initialization or after deletion.

The message numbers are:

030.000, 030.010, 030.020, 032.000, 032.010, 032.015, 032.020, 032.030, 032.050, 032.070, 032.080, 032.085, 036.000, 036.010, 036.060, 036.066, 037.000, 037.010, 037.020, 038.000, 038.010, 038.012, 038.014, 038.032, 038.038, 038.040, 038.050, 044.000, 044.040, 046.011, 046.014, 046.050, 046.054, 046.090, 060.010, 060.065, 060.075, 060.115, 065.000, 065.003, 065.020, 069.020, 070.000, 070.002, 070.010, 070.016, 070.030, 070.040, 070.050, 070.060, 070.070, 070.080, 080.000, 080.020, 080.040, 080.050, 080.060, 080.070, 080.080, 080.150, 080.160, 080.200, 080.250, 120.006, 130.065, 132.010, 132.030, 132.040, 132.050, 144.010, 144.030, 144.040, 144.050, 219.000.

15.3 Errors of the type "[the object] is not a/the …"

"The object, passed to the function, is not the object of required type".

As a rule the correspondence between the object type and executed operation is checked by C-DVM and F-DVM compilers. This error is caused more likely by the object [descriptor] "corruption".

The message numbers are:

030.001, 030.011, 030.021, 032.001, 032.011, 032.016, 032.021, 032.031, 032.051, 032.071, 032.081, 032.086, 036.001, 036.011, 036.061, 036.067, 037.001, 037.011, 037.021, 038.001, 038.011, 038.013, 038.015, 038.033, 038.039, 038.041, 038.051, 040.005, 040.008, 040.010, 044.001, 044.002, 044.041, 044.042, 044.060, 046.000, 046.010, 046.012, 046.015, 046.051, 046.052, 046.055, 046.091, 060.011, 060.012, 060.013, 060.050, 060.051, 060.060, 060.061, 060.066, 060.070, 060.071, 060.076, 060.080, 060.081, 060.090, 060.091, 060.100, 060.116, 060.200, 065.001, 065.004, 065.021, 069.021, 070.001, 070.003, 070.008, 070.014, 070.031, 070.041, 070.051, 070.061, 070.071, 070.081, 080.001, 080.002, 080.021, 080.022, 080.041, 080.051, 080.061, 080.071, 080.081, 080.151, 080.161, 080.201, 080.202, 080.251, 080.252, 090.000, 090.001, 090.010, 090.011, 090.020, 090.030, 090.040, 090.050, 090.051, 090.060, 092.000, 092.001, 092.010, 092.011, 092.020, 094.000, 096.000, 110.000, 110.003, 110.009, 110.012, 120.000, 120.001, 120.002, 120.003, 120.005, 130.001, 130.004, 130.005, 130.031, 130.035, 130.061, 130.066, 131.000, 131.001, 131.020, 131.022, 131.040, 131.042, 132.011, 132.014, 132.016, 132.031, 132.041, 132.051, 133.001, 133.004, 133.005, 140.001, 140.020, 140.021, 140.030, 141.001, 141.030, 143.000, 143.001, 143.020, 143.022, 143.040, 143.042, 144.011, 144.014, 144.016, 144.031, 144.041, 144.051, 219.001, 219.010, 219.011, 219.020, 219.021.

15.4 Errors of the type "… is not a subsystem of the current/parental PS"

"Processor system, passed to the function, is not current/parental processor system or its direct or indirect subsystem".

The messages indicate a violation of subsystem hierarchy. For example, the subtask attempts to use/change the object, belonging to higher hierarchy level.

The message numbers are:

032.012, 032.013, 032.032, 032.052, 032.082, 032.083, 036.012, 036.013, 036.068, 036.069, 038.025, 038.026, 038.042, 038.043, 044.003, 044.044, 044.045, 046.017, 046.058, 046.059, 070.013, 070.027, 080.006, 080.025, 080.047, 080.057, 080.067, 080.157, 080.167, 080.206, 080.255, 090.004, 090.005, 090.014, 090.015, 090.023, 090.024, 092.004, 092.005, 092.014, 092.015, 092.023, 092.024, 094.006, 094.007, 096.006, 096.007, 101.001, 101.011, 110.002, 110.005, 110.011, 110.014, 130.003, 130.033, 130.037, 130.063, 130.067, 131.004, 131.005, 132.032, 132.033, 133.003, 140.003, 140.033, 141.003, 141.033, 143.004, 143.005, 144.032, 144.033, 219.002.

15.5 Errors of the type "…was not created by the current subtask" and "…was not started by the current subtask"

"The object passed to the function was not created in the current subtask" and "Asynchronous operation, passed to the function, was not started by the current subtask".

The messages indicate a violation of subsystem hierarchy. For example, the subtask attempts to use/change the object, belonging to other subtask. Or the subtask attempts to wait for asynchronous operation (implemented by the pair of START- WAIT statements), started by other subtask.

The message numbers are:

060.067, 060.077, 060.117, 065.006, 069.022, 070.005, 070.006, 070.032, 070.042, 070.052, 070.062, 070.072, 070.082, 080.003, 080.023, 080.056, 080.066, 080.073, 080.082, 080.156, 080.166, 080.203, 080.253, 120.008, 131.023, 131.043, 132.012, 132.017, 132.042, 132.052, 143.023, 143.043, 144.012, 144.017, 144.042, 144.052.

15.6 Errors of the type "…has not been aligned/mapped" and "…has already been aligned/mapped"; "…does not exist" and "…already exists"

Possible reason of such errors is wrong order of the array and other DVM-object creation/usage. For example, the array A, aligned with the array B, can be created (by malloc statement) only if the array B already exists (was created). Similarly, parallel loop can be aligned with the array, only if the array exists. Or vice versa repeated creation of the object is performed. Note that repeated creation of the array (malloc) in sequential program may be not detected, and (repeated) creation of the template or shadow edge group is invisible for sequential program.

The message numbers are:

032.017, 032.088, 036.002, 036.003, 036.014, 036.051, 038.002, 038.020, 038.021, 038.024, 038.031, 040.000, 040.006, 044.005, 044.006, 044.047, 046.001, 046.019, 046.020, 046.061, 060.014, 060.015, 060.052, 060.062, 060.072, 060.082, 060.101, 060.201, 065.007, 065.009, 065.022, 070.009, 070.015, 080.005, 080.024, 080.205, 080.254, 090.002, 090.003, 090.012, 090.013, 090.021, 090.022, 090.041, 090.031, 090.052, 090.053, 090.061, 092.002, 092.003, 092.012, 092.013, 092.021, 092.022, 094.004, 094.005, 096.004, 096.005, 101.000, 101.010, 110.001, 110.004, 110.010, 110.013, 130.000, 130.002, 130.006, 130.030, 130.032, 130.036, 130.060, 130.062, 133.000, 133.002, 133.006, 140.000, 140.002, 140.022, 140.032, 141.000, 141.002, 141.032, 219.012, 219.022.

15.7 Errors of the type "… has already been started/inserted …", "… has not been started/completed" and "… the reduction group is empty"

Such errors are a result of wrong sequence of the actions with DVM-objects: reduction operation and shadow edge groups, remote access buffers and so on. For example, new start of the operation before the completion of started one, start of the operation with empty group, waiting for completion of not started operation, including variables (arrays) in the group during execution of operation with this group, deleting object during the operation execution and so on.

The message numbers are:

060.120, 060.140, 069.023, 070.004, 070.033, 070.034, 070.044, 070.053, 070.054, 070.063, 070.073, 070.083, 080.007, 080.008, 080.009, 080.010, 080.011, 080.026, 080.027, 080.028, 080.029, 080.030, 080.042, 080.043, 080.044, 080.045, 080.046, 080.052, 080.053, 080.054, 080.062, 080.063, 080.064, 080.072, 080.083, 080.152, 080.153, 080.154, 080.162, 080.163, 080.164, 080.207, 080.208, 080.209, 080.210, 080.211, 080.256, 080.257, 080.258, 080.259, 080.260, 131.003, 131.024, 131.044, 131.045, 132.013, 132.019, 132.034, 132.043, 132.053, 143.003, 143.024, 143.044, 143.045, 144.013, 144.019, 144.034, 144.043, 144.053.

15.8 Index and value errors

The errors of this type occurs in the case of a boundary violation of DVM-object dimension (for example, when aligning array or describing parallel loop), indexing of not-existing dimension, wrong ratio between indexes (for example, initial index is more than last one), wrong DVM-object rank and so on. Incorrect values can be passed to Run-Time System, for example, in GENBLOCK vector or as shadow edge widths (or as result of "memory clearing"). Run-Time System messages contain the expression (<equality or inequality>) - incorrect ratio. Description of the values, contained in each message, can be found in document "DVM-program Run-Time System. Detailed design" (rtsDDe.doc file).

The message numbers are:

030.022, 030.023, 032.019, 032.022, 032.035, 032.036, 032.037, 032.038, 032.039, 032.055, 032.056, 032.057, 032.090, 032.091, 032.092, 036.020. 036.021, 036.022, 036.023, 036.024, 036.025, 036.026, 037.002, 037.003, 037.012, 037.022, 038.016, 038.017, 038.044, 038.045, 040.001, 040.002, 040.003, 044.019, 044.020, 044.021, 044.022, 044.023, 044.024, 044.025, 044.026, 044.027, 044.028, 046.013, 046.016, 046.030, 046.031, 046.056, 046.057, 060.000, 060.001, 060.019, 060.020, 060.021, 060.022, 060.023, 060.024, 060.025, 060.026, 060.027, 060.028, 060.029, 060.030, 069.000, 069.001, 069.002, 069.003, 069.024, 080.014, 080.015, 080.031, 080.032, 080.033, 080.034, 080.035, 080.214, 080.215, 080.216, 080.220, 080.225, 080.226, 080.227, 080.230, 080.235, 080.236, 080.237, 080.261, 080.264, 080.265, 080.266, 080.270, 080.275, 080.276, 080.277, 080.280, 080.285, 080.286, 080.287, 080.290, 080.291, 094.001, 094.002, 094.003, 096.001, 096.002, 096.003, 110.006, 110.015, 120.004, 120.200, 130.010, 130.011, 130.012, 130.013, 130.014, 130.015, 130.016, 130.017, 130.018, 130.019, 130.040, 130.041, 130.042, 130.043, 130.044, 130.045, 130.046, 130.047, 130.070, 140.010, 140.011, 140.023, 140.031, 140.036, 140.037, 140.038, 140.041, 140.042, 140.043, 140.044, 141.010, 141.011, 141.031, 141.036, 141.037, 141.038, 219.015, 219.016, 219.017, 219.025, 219.026, 219.027.

15.9 Other semantic errors

The array or template redistribution and remapping, which is impossible or was not allowed when the array or template was created.

The message numbers are:

030.002, 036.050, 036.065, 038.030, 038.037, 044.043, 046.053, 065.005, 065.010.

Attempt to execute parallel loop, in which the order of iterations is changed using ACROSS scheme (scheme of execution of the loops with data dependence between iterations).

The message numbers are: 060.110, 060.111.

Attempt to extend a set of local iterations of parallel loop with changed order of iteration execution.

The message numbers are: 060.210, 060.211.

Including reduction variable simultaneously in two different groups.

The message number is 070.007.

The distribution of arrays in SHADOW_START or SHADOW_WAIT sub-directive is differ from the loop distribution.

The message number is 060.040.

Incorrectly distributed objects in non-regular access (for example, a number of distributed dimensions of remote array is not equal to 1).

The message numbers are:

140.004, 140.034, 140.035, 140.040, 140.050, 140.051, 141.004, 141.034, 141.035.

15.10 Memory allocation and the number of objects errors

The errors of this type occur in the case of lack of the memory because of the errors in the sizes of created arrays. Allocation memory errors can also occur because of the memory "corruption" by the user program or its "looping".

The message numbers are:

200.000, 200.001, 200.002, 200.003, 200.004, 200.005, 200.006, 200.007, 200.008.

The following messages are output when too many objects (distributed arrays, remote access buffers) were created. In C-DVM these errors can be caused by imbalance of malloc and free statements for distributed arrays or by the usage of local in the function (or in the block) "static" distributed arrays (the arrays with constant dimensions).

The message numbers are:

040.004, 040.007, 130.029, 140.060.

15.11 Errors of low level message passing

Message passing errors are the result of incorrect interaction of Run-Time System with message passing system.

The following reasons are possible:

It is recommended to developer or its representative.

The message numbers are:

210.000, 210.001, 210.002, 210.003, 210.004, 210.005, 212.000, 212.001, 212.002, 212.003, 213.000, 213.001, 213.002, 214.000, 214.001, 219.003.

16 Structure of system trace file

The system trace file has the following structure:

The trace of each event cab be enabled or disabled by its number (see section 6 and 11.4).

Structure of accumulated trace information is defined by trace mode. In brief mode the following is outputted:

In detailed trace mode saving information is extended and depends on event type. If the event is function call, then its call parameters are outputted. When returning function its result parameters are output. For each event the detailed trace mode can be specified independently of common mode.

When tracing accumulated information also contains all informational messages and messages about Run-Time System errors.

To make representation of function calls belonging to different nested levels clear, desirable indentation may be specified in parameters, controlling system trace (see section 11.4).