14 Regular access to remote data

If distributed array elements are located not on all the processors, executing current branch of the parallel program, where they are required, such array will be called remote, and its elements will be called remote ones. An access to remote distributed array elements is implemented by creating on each processor, where at least one element is required and absent, a special buffer (local buffer of remote elements) and putting in it required remote data.

14.1 Creating remote element buffer of distributed array

If parallel program branch, required remote array elements is parallel loop iteration, the remote data, required for all its iterations will be considered as a single whole.

In this case required elements of remote distributed array are defined by linear sampling rules (regular access), specified for each its dimension as

A_i*V_k(i) + B_i , where:

Local parts of remote elements of each processor, executing parallel loop iterations, are considered as local parts of distributed array, which will be called global buffer of remote elements (or remote element buffer). The remote element buffer is mapped just as the parallel loop, but its i-th dimension is matched with k(i)-th dimension of the parallel loop (in the case, when linear sampling rule with non-zero A_i is specified for i-th dimension of the remote array).

By virtue of sampling rules considered above required remote elements are represented as stretched block, but put to the buffer in compact (not stretched) mode.

Dealing with remote element buffer is similar to dealing with distributed array (see section 13).

For the purpose of optimization the remote element buffers can be combined the groups and loading all buffers of the group is executed by one operation (see sections 14.6-14.10).

The function crtrbl_ creates buffer to locate elements of remote distributed array with RemArrayHeader header, required to execute parallel loop, defined by *LoopRefPtr reference. Remote array must be mapped on the processor system, which each element belongs to the current processor system. When the function is invoked, the loop must be current and mapped.

Created buffer is the distributed array with rank, lesser than remote array rank by a number of constant linear rules of remote element sampling. Its header BufferHeader is an array of 2*r+2 elements of "long" type, where r is the buffer rank (extended header of distributed array, see section 13.4). The buffer header allocation in memory (static or dynamic) is perfumed by a user program, and the buffer initialization is performed by Run-Time System when executing the function crtrbl_.

Any i-th element of AxisArray array can contain parallel loop dimension number or 0 or -1. In the first case CoeffArray[i] must be non-zero (not constant sampling rule). In the second case CoeffArray[i] must be equal to zero (constant sampling rule). The third case defines free (i+1)-th dimension of the remote array, i.e. the dimension, not matched with any parallel loop dimension ("presents everywhere").

So, remote element buffer rank is equal to a number of non-zero elements of AxisArray array. A size of each j-th dimension of the buffer is defined by j-th non-zero element of AxisArray array (let i be this element index). If AxisArray[i] > 0 (linear sampling rule), the size of j-th dimension of the buffer is equal to the size of AxisArray[i])-th dimension of specified parallel loop (in this case the loop dimension size is equal to absolute difference between initial and last values of the dimension index variable). If AxisArray[i] = -1 (free dimension of the distributed array), the size of the buffer j-th dimension is equal to (i+1)-th dimension of remote array.

The array AxisArray can't contain two same parallel loop dimension numbers.

The sizes of AxisArray, CoeffArray and ConstArray arrays must be equal to remote distributed array rank.

Correct sampling rules must not violate remote distributed array boundaries.

The base pointer BasePtr must refer to any variable of the same type as the type of the remote distributed array elements. The variable value is irrelevant, because the base pointer is used only for access to remote elements of the distributed array, allocated in the created buffer.

Dealing with the header of remote elements buffer and base pointer is considered in detail in section 14.5.

If the flag *StaticSignPtr of static buffer is not equal to zero, then the created buffer will be not deleted automatically, when the control exits the program block (see section 8). Such buffer can be deleted only explicitly using the function delrb_ considered below.

The function returns non-zero, if created buffer has a local part on the current processor, and zero in other case.

Note 1. If all sampling rules are constant ones, created buffer rank must be zero by virtue of writing above. In this case Run-Time System creates one-dimensional distributed array-buffer, containing single element.

Note 2. Let when parallel loop is mapped by mappl_ function for some dimension initial value of index variable is more than its last value for positive step or its last value of index variable is more than its initial value for negative step (empty loop). Then these loop usage as crtrbl_ function parameter will be resulted in creation of dummy remote element buffer. 0-th word of the buffer header (distributed array reference) will be assigned by null value. Dummy remote element buffer can be used in all remote access functions, but it is not allowed to use the buffer header as parameter of the functions, dealing with distributed arrays. An access to local part of distributed array by dummy buffer header is not allowed also.

The buffer of remote elements, required for parallel loop and for program branches, that are not the loop iterations, can be also created by crtrba_ and crtrbp_ functions considered below.

The function crtrba_ creates the buffer to locate elements of remote distributed array with RemArrayHeader header, which are required to deal with distributed (local) array, specified by LocArrayHeader. Remote and local distributed arrays must be mapped on the processor systems (perhaps, each one in its own), all elements of which belong to the current processor system.

Required elements of remote distributed array are specified for each its not-free dimension by linear sampling rules as:

A_i*V_k(i) + B_i , where:

The remote element buffer, created by crtrba_ function is mapped just as the local distributed array, but its i-th dimension is matched with k(i)-th dimension of the local array (in the case, when linear sampling rule with non-zero A_i is specified for i-th dimension of the remote array).

All parameters of crtrba_ function (except LocArrayHeader) are similar to same named parameters of crtrbl_ function, considered above.

The function returns non-zero, if created buffer has a local part on the current processor, and zero in other case.

The function crtrbp_ creates the buffer to locate elements of remote distributed array with RemArrayHeader header, which must be present on each processor of the system, specified by *PSRefPtr reference.

Remote distributed array must be mapped on the processor system, all elements of which must belong to the current processor system. *PSRefPtr reference defines the current processor system or its direct or indirect subsystem. NULL value of PSRefPtr pointer or zero value of *PSRefPtr reference specify the current processor system.

Required elements of remote distributed array for (i+1)-th dimension are defined by i-th word of CoordArray array: non-negative value defines remote array element coordinate, negative one specifies whole dimension of the remote array (analogue of free dimension of the remote array in crtrbl_ and crtrba_ function description). A number of elements in CoordArray array must be equal to remote distributed array rank.

The rank of remote element buffer, created by crtrbp_ function, is equal to a number of negative elements of CoordArray array. The size of j-th buffer dimension is equal to j-th negative element of CoordArray array. If all elements of CoordArray array are non-negative (exactly one remote element is specified), Run-Time System creates one-dimensional distributed array-buffer, consisting of single element.

While executing crtrbp_ function, remote element buffer is mapped so that all required, and therefore, located in the buffer remote elements assist on all processors of the system, specified by PSRefPtr parameter. To do this, Run-Time System creates subsidiary one-dimensional representation of the current abstract machine, consisting of single element and maps it with replication on all dimensions on specified in the function call processor system. Remote element buffer is mapped on subsidiary representation with replication on its single dimension.

The function returns non-zero, if created buffer has a local part on the current processor, and zero in other case.

14.2 Initializing loading remote element buffer of distributed array

The function loadrb_ initializes specified buffer loading by the distributed array elements, for which the buffer was created by crtrbl_, crtrba_ or crtrbp_ function.

The function can be used for already loaded buffer. In this case actual reloading is performed if *RenewSignPtr is not equal to zero, and isn't performed otherwise. The buffer reloading is not allowed (independently on *RenewSignPtr parameter value), if previous loading (actual or dummy) was not completed by waitrb_ function (see section 14.3).

As loaded buffer as its remote distributed array must be mapped on the processor systems, which each element must belong to the current processor system.

The buffer loading can't be started by loadrb_ function, if it was included in any buffer group (see sections 14.6-14.10).

The function returns zero.

14.3 Waiting for completion of loading remote element buffer of distributed array

long waitrb_ (long BufferHeader[]);

BufferHeader - header of the remote element buffer.

Loading the buffer specified in the function call must be initialized by the current subtask. Waiting for completion of not initialized earlier buffer loading is not allowed.

The function returns zero.

14.4 Deleting remote element buffer of distributed array

long delrb_ (long BufferHeader[]);

BufferHeader - header of the remote element buffer to be deleted.

The remote element buffer can be deleted by delrb_ function only if it was created in the current subtask and in the current program block (or its sub-block) (see sections 8 and 10).

To delete remote element buffer the function delobj_ can also be used (see section 17.5).

The buffer, which loading is not completed by waitrb_ function, can't be deleted.

If deleted buffer was included in any buffer group, it is excluded from the group (see sections 14.6 - 14.10). In this case the buffer group must not be in loading state.

The function returns zero.

14.5 Access to distributed array elements, allocated in remote element buffer

Let r be the buffer rank (a number of remote distributed array dimensions, having non-constant linear sampling rules or being free). Then remote element buffer header, specified in crtrbl_ (crtrba_, crtrbp_) function call can be declared, for example, as

long BufferHeader[2*r+2];

Let also a base pointer, corresponding to remote distributed array element type be BasePtr. Then a remote element, defined by index vector (I₁, ... , I_m, ... , I_r), and located in the buffer local part, is referred by linear index in the following way:

For C language the base pointer can be NULL in crtrbl_, crtrba_ and crtrbp_ function calls. Then the remote element, placed in the buffer local part, can be accessed as:

where Type is the type of the element of the remote distributed array.

For remote element buffer, created by crtrbl_ function.

Let m dimension of the buffer correspons to linear sampling rule of remote elements (bounded dimension) and, therefore, to some dimension k of a parallel loop. Then when accessing the buffer element a value of I_m coordinate can be replaced by a value of the loop index variable V_k: I_m = V_k – min(V_k,init , V_k,last) (V_k,init and V_k,last are initial and last values of the loop index variable V_k). It means, that the crossing from the buffer index space to the parallel loop index space involves a transition to distributed array-buffer with non-zero low index boundaries. Therefore to support access to the buffer elements by the loop index variables, when executing crtrbl_ Run-Time System for all bounded dimensions saves in the buffer header min(V_k,init , V_k,last) values as low shadow edges of the array dimensions (see description of extended header of distributed arrays in section 13.4). Low shadow edges of free buffer dimensions are set to zero in its header.

In C language to access to the buffer elements by parallel loop index variables the following macrocomands can be used:

Each of these macros is L-value in the C language.

It is assumed, that when the buffer was created by crtrbl_ function the base pointer was equal to NULL.

14.6 Creating group of remote element buffers

The function crtbg_ creates empty buffer group (containing no buffers) and returns reference to the group.

If the flag *StaticSignPtr of static group is not equal to zero, then the created group will not be deleted, when the control exits the program block (see section 8). Such group can be deleted only explicitly using the function delbg_ considered below.

If *DelBufSignPtr is not equal to zero, then all included in the group buffers will be deleted when the group is deleted. If deleting is explicit, all included buffers are deleted explicitly , and if deleting is implicit, they are deleted implicitly.

14.7 Including remote element buffer in the group

As the buffer group as the buffer to be included in the group must be created in the current subtask and can't be in loading state. The buffer can be reincluded in the group, but the buffer can't be included in the group, if it was included in the other group.

The function returns zero.

14.8 Starting loading remote element buffers of specified group

All buffers of the group, specified in the function call, must be mapped on the processor systems, which each element must belong to the current processor system.

The function loadbg_ can be used for already loaded buffer group. In this case actual reloading is performed if *RenewSignPtr is not equal to zero, and isn't performed otherwise. The buffer group reloading is prohibited (independently on *RenewSignPtr parameter value), if its previous loading (actual or dummy) was not completed by waitbg_ function (see section 14.9).

The function returns zero.

14.9 Waiting for completion of loading remote element buffers of specified group

long waitbg_ (RegularAccessGroupRef *RegularAccessGroupRefPtr);

*RegularAccessGroupRefPtr - reference to the buffer group.

Loading the buffer group, specified in the function call, must be started by the current subtask. Waiting for completion of not initialized earlier buffer group loading is not allowed.

The function returns zero.

14.10 Deleting group of remote element buffers

long delbg_ (RegularAccessGroupRef *RegularAccessGroupRefPtr);

*RegularAccessGroupRefPtr - a reference to the buffer group to be deleted.

The buffer group can be deleted by delbg_ function only if it was created in the current subtask and in the current program block (or its sub-block) (see sections 8 and 10).

To delete the buffer group the function delobj_ can also be used (see section 17.5).

The buffer group, which loading was not completed by waitbg_ function, can't be deleted.

If the buffer group was created by crtbg_ function with non-zero value of *DelBufSignPtr parameter, then when the group is deleted all remote elements buffers, included in the group, will be also deleted.

The function returns zero.

14.11 Requesting type of distributed array element access

The function rmkind_ defines type of access to elements of distributed array with ArrayHeader header, which is required for execution of parallel loop, specified by *LoopRefPtr reference.

The array elements, required for the loop execution, are specified similarly to specifying remote array elements, required for parallel loop execution (see section 14.1). BufferHeader, BasePtr, StaticSignPtr, AxisArray, CoeffArray and ConstArray parameters are similarly to same named parameters of crtrbl_ function.

Distributed array must be mapped on processor system, which each element belongs to the current processor system. At the moment of the function call parallel loop must be current and mapped.

The function returns the following values, specifying access type:

1 - local access;
2 - "SHADOW" type access;
3 - "FULL SHADOW" type access;
4 - "REMOTE" type access (remote access).

Access type is defined in the following way. Let PS be the processor system, specified parallel loop is mapped on. Then

1. The access is local one, if at each processor of PS system all distributed array elements, required for the loop execution, belong to the array local part;
2. The access is "SHADOW" type access if at each processor of PS system all required elements of distributed array belong to the array local part, extended by shadow edges;
3. The access is "FULL SHADOW" type access if at each processor of PS system all required elements of the array belong to the array local part, extended by full shadow edge;
4. The access is remote ("REMOTE") if at least one processor of PS at least one required element of the array doesn't belong to the array local part, extended by full shadow edge;

If access has "SHADOW" or "FULL SHADOW" type, distributed array shadow edge widths, specified when the array was created, are written to LowShdWidthArray and HiShdWidthArray arrays.

In the case of remote access rmkind_ function calls crtrbl_ function to create remote element buffer with BufferHeader header.

15 Non-regular access to remote data

A scheme of non-regular access to the elements of remote distributed array is similar to scheme of the regular access, considered in section 14. But its application area is restricted by the arrays, having the only distributed dimension (all other dimensions are replicated). The distributed array elements, required for parallel loop execution and placed to the buffer are defined by the set of index vectors of the form

where:

Index matrix ME is "integer" two-dimensional distributed array, its first dimension is distributed and the second one is replicated. Here and below distributed array is considered to be replicated along i-th dimension if the value of multi-value function F( (I₁, …, I_i, …, I_n) ) of the array mapping doesn't depend on I_i value (see section 7.1). The size MESize₁ of the first dimension of the matrix ME is usually (but not obligatory) equal to the size of distributed (k-th) dimension of remote array. The size MESize₂ of the second dimension of the matrix ME is equal to

MAX(ME[V_k,0]) + 1 .
0 = V_k < MESize₁

The meaning of the index matrix elements is:

15.1 Creating remote element buffer of non-regular access

The function crtibl_ creates buffer to locate elements of remote distributed array with RemArrayHeader header, required to execute parallel loop, defined by *LoopRefPtr reference.

Remote array must be mapped on the processor system, which each element belongs to the current processor system. It must have single distributed dimension (others must be replicated).

When the function is invoked, the parallel loop must be current and mapped. Its index variable is considered above variable V_k of coordinate sampling rule of distributed dimension of remote array. Initial and last values of V_k variable (V_k,init and V_k,last) can't be negative or more than size of first dimension of index matrix minus 1.

Created buffer is two-dimensional distributed array, its first dimension is distributed in the same way as the dimension (single) of given parallel loop, and second one is replicated. The size of the first dimension of the buffer is equal to absolute value of difference between last and initial values of index variable V_k of the parallel loop plus 1, the size of the second dimension is equal to size of the second dimension of index matrix minus 1.

The header BufferHeader of the created buffer is the array of 6 elements of "long" type (extended header of distributed array, see section 13.4). The buffer header allocation in memory (static or dynamic) is performed by a user program, but its initialization is performed by Run-Time System when executing the function crtibl_.

Dealing with remote element buffer is similar to dealing with distributed array (see section 13).

For the purpose of optimization the remote element buffers can be combined the groups and loading all buffers of the group is executed by one operation (see sections 15.6 - 15.10).

The base pointer BasePtr must refer to any variable of the same type as the type of the remote distributed array elements (the variable value is irrelevant).

In detail dealing with the header of remote elements buffer and the base pointer is considered in section 15.5.

The index matrix header MEHeader is the array of 3 or 6 elements of "long" type (standard or extended header of distributed array, see section 13.4). When crtibl_ function is invoked, index matrix must be mapped on processor system, each element of which must belong to the current processor system. First dimension of index matrix must be distributed in the same way as the dimension of given parallel loop (to within coefficient of linear mapping rule), and second one must be replicated. Widths of low and high shadow edges, specified when index matrix was created, must be zero.

The sizes of ConstArray array must be equal to remote distributed array rank. A value of ConstArray[k-1] is irrelevant (k is a number of the distributed dimension of the remote array).

If the flag *StaticSignPtr is not equal to zero, then the created buffer will not be deleted, when the control exits the a program block (see section 8). Such buffer can be deleted only explicitly using the function delib_ considered below.

The function returns non-zero, if created buffer has a local part on the current processor, and zero in other case.

Note. Let when the parallel loop is mapped by mappl_ function initial value of V_k index variable is more than its last value for positive step or its last value of V_k index variable is more than its initial value for negative step (empty loop). Then this loop usage as crtibl_ function parameter will be resulted in creation of dummy remote element buffer. 0-th word of the buffer header (distributed array reference) will be assigned by null value. Dummy remote element buffer can be used in all remote access functions, but it is not allowed to use the buffer header as parameter of the functions, dealing with distributed arrays. An access to local part of distributed array by dummy buffer header is not allowed also.

The remote element buffer of non-regular access can be also created by the function

The distinction of crtib_ function from crtibl_ function is lack of LoopRefPtr parameter, specifying the reference to parallel loop. Created buffer will be distributed along both dimensions in the same way as given index matrix The size of its first dimension will be equal to the size of first dimension of index matrix, and the size of second dimension will be equal to the size of second dimension of index matrix minus 1.

The function crtib_ allows to create the buffer and starts its loading (see section 15.2) before creation and mapping of parallel loop, which will use the elements of remote distributed array, located in the buffer.

15.2 Starting loading remote element buffer of non-regular access

The function loadib_ initializes specified buffer loading by the distributed array elements, for which the buffer was created by crtibl_ (or crtib_) function.

The function can be used for already loaded buffer. In this case actual reloading is performed if *RenewSignPtr is not equal to zero, and isn't performed otherwise. The buffer reloading is not allowed (independently on *RenewSignPtr parameter value), if previous loading (actual or dummy) was not completed by waitib_ function (see section 15.3).

As loaded buffer as its remote distributed array must be mapped on the processor systems, which each element must belong to the current processor system.

The buffer loading can't be started by loadib_ function, if it was included in any buffer group (see sections 15.6 - 15.10).

The function returns zero.

15.3 Waiting for completion of loading remote element buffer of non-regular access

long waitib_ (long BufferHeader[]);

BufferHeader - header of the remote element buffer of non-regular access.

Loading the buffer specified in the function call must be initialized by the current subtask. Waiting for completion of not initialized earlier buffer loading is not allowed.

The function returns zero.

15.4 Deleting remote element buffer of non-regular access

long delib_ (long BufferHeader[]);

BufferHeader - header of the remote element buffer to be deleted.

The remote element buffer can be deleted by delib_ function only if it was created in the current subtask and in the current program block (or its sub-block) (see sections 8 and 10).

To delete remote element buffer the function delobj_ can also be used (see section 17.5).

The buffer, which loading is not completed by waitib_ function, can't be deleted.

If deleted buffer was included in any buffer group, it is excluded from the group (see sections 15.6 - 15.10). In this case the buffer group must not be in loading state.

The function returns zero.

15.5 Access to remote elements, allocated in the buffer

The remote element buffer of non-regular access is two-dimensional array with extended header. Therefore its header may be declared, for example, as

long BufferHeader[6];

Let a base pointer, corresponding to remote distributed array element type be BasePtr. Then a remote element, defined by index vector (I₁, I₂), and located in the buffer local part, is referred by linear index in the following way:

BasePtr[ BufferHeader[1]*I₁ + I₂ + BufferHeader[2] ] .

For C language the value of the base pointer can be specified equal to NULL in the function crtibl_ (or crtib_) call. Then the access to the allocated in the buffer remote element can be performed as follows:

( (Type *)BufferHeader[2] )[ BufferHeader[1]*I₁ + I₂ ] ,

where Type is the type of the remote distributed array elements.

Access to the buffer elements by index variable of parallel loop.

First dimension of remote element buffer corresponds to first (and single) dimension of parallel loop, which needs in located in the buffer (see section 15.1). So, the value of I₁ coordinate can be changed by the value of the loop index variable V_k (k is distributed dimension number of remote array): I₁ = V_k – min(V_k,init , V_k,last) (V_k,init and V_k,last are initial and last values of the loop index variable V_k). It means, that transition from the buffer index space to the parallel loop index space causes transition to distributed array-buffer with non-zero low edge of index for the first dimension. So to support access to the buffer elements via the loop index variable Run-Time system, executing crtibl_ function keeps min(V_k,init , V_k,last) value in BufferHeader[4] as low edge of first dimension of the array (see description of extended header of distributed arrays in section 13.4).

If the buffer is created by crtib_ function, zero value is assigned to the header element BufferHeader[4].

Low edge os second (replicated) dimension of the buffer is set to 1 in its header (in BufferHeader[5]). It means, that J-1 coordinate of second dimension of the buffer will correspond to J coordinate of second dimension of index matrix.

In C language to access to the buffer elements via index variable of parallel loop macro command RBElm2 (see section 14.5) can be used. In RBElm2 macro command call the value of index variable of second dimension of index matrix is specified as Index₂.

15.6 Creating group of remote element buffers of non-regular access

The function crtig_ creates an empty buffer group (not containing buffers) and returns a reference to the group.

If the flag *StaticSignPtr of static group is not equal to zero, then the created group will not be deleted, when the control exits the program block (see section 8). Such group can be deleted only explicitly using the function delig_, considered below.

If *DelBufSignPtr is not equal to zero, then all included in the group buffers will be deleted when the group is deleted. If deleting is explicit, all included buffers are deleted explicitly , and if deleting is implicit, the ones are deleted implicitly.

15.7 Including remote element buffer of non-regular access in the group

As the buffer group as the buffer to be included in the group must be created in the current subtask and can't be in loading state. The buffer can be reincluded in the group, but the buffer can't be included in the group, if it was included in the other group.

The function returns zero.

15.8 Starting loading remote element buffers of specified group

All buffers of the group, specified in the function call, must be mapped on the processor systems, which each element must belong to the current processor system.

The function loadig_ can be used for already loaded buffer group. In this case actual reloading is performed if *RenewSignPtr is not equal to zero, and isn't performed otherwise. The buffer group reloading is prohibited (independently on *RenewSignPtr parameter value), if its previous loading (actual or dummy) was not completed by waitig_ function (see section 15.9).

The function returns zero.

15.9 Waiting for completion of loading remote element buffers of specified group

long waitig_ (IndirectAccessGroupRef *IndirectAccessGroupRefPtr);

*IndirectAccessGroupRefPtr - reference to the buffer group.

Loading the buffer group, specified in the function call, must be started by the current subtask. Waiting for completion of not initialized earlier buffer group loading is not allowed.

The function returns zero.

15.10 Deleting group of remote element buffers of non-regular access

long delig_ (IndirectAccessGroupRef *IndirectAccessGroupRefPtr);

*IndirectAccessGroupRefPtr - reference to the buffer group to be deleted.

The group of remote element buffers can be deleted by delig_ function only if it was created in the current subtask and in the current program block (or its sub-block) (see sections 8 and 10).

To delete the buffer group the function delobj_ can also be used (see section 17.5).

The buffer group, which loading was not completed by waitig_ function, can't be deleted.

If the buffer group was created by crtig_ function with non-zero value of *DelBufSignPtr parameter, then when the group is deleted all remote elements buffers, included in the group, will be also deleted.

The function returns zero.