c - How to use MPI_Gatherv for collecting strings of diiferent length from different processor including master node?

Question

I am trying to collect different strings of different length from all processors (including the master node) into a single string (array of characters) at the master node. Here is the prototype for MPI_Gatherv:

int MPI_Gatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
            void *recvbuf, const int *recvcounts, const int *displs,
            MPI_Datatype recvtype, int root, MPI_Comm comm)**.

I am unable to define some parameters like recvbuf,recvcounts and displs. Could any one provide source code example in C for this?

Answer 1

As has been pointed out, there are lots of examples of using MPI_Gatherv, including here on stack overflow; an answer which starts off describing how scatter and gather work, and then how the scatterv/gatherv variants extend that, can be found here.

Crucially, for the simpler Gather operation, where every chunk is of the same size, the MPI library can easily precompute where each chunk should go in the final compiled array; in the more general gatherv operation, where this is less clear, you have the option - in fact, the requirement - to spell out exactly where each item should start.

The only extra complication here is that you're dealing with strings, so you probably don't want everything shoved right together; you'll want extra padding for spaces, and of course a null terminator at the end.

So let's say you have five processes wanting to send strings:

Rank 0: "Hello"    (len=5)
Rank 1: "world!"   (len=6)
Rank 2: "Bonjour"  (len=7)
Rank 3: "le"       (len=2)
Rank 4: "monde!"   (len=6)

You'll want this to be assembled into a global string:

Hello world! Bonjour le monde!
          111111111122222222223
0123456789012345678901234567890

recvcounts={5,6,7,2,6};  /* just the lengths */
displs = {0,6,13,21,24}; /* cumulative sum of len+1 for padding */

You can see that displacement 0 is 0, and displacement i is equal to the sum of (recvcounts[j]+1) for j=0..i-1:

   i    count[i]   count[i]+1   displ[i]   displ[i]-displ[i-1]
   ------------------------------------------------------------
   0       5          6           0    
   1       6          7           6                 6
   2       7          8          13                 7
   3       2          3          21                 8
   4       6          7          24                 3

And that's straightforwardly implemented:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include "mpi.h"

#define nstrings 5
const char *const strings[nstrings] = {"Hello","world!","Bonjour","le","monde!"};

int main(int argc, char **argv) {

    MPI_Init(&argc, &argv); 

    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    /* Everyone gets a string */    

    int myStringNum = rank % nstrings;
    char *mystring = (char *)strings[myStringNum];
    int mylen = strlen(mystring);

    printf("Rank %d: %s
", rank, mystring);

    /*
     * Now, we Gather the string lengths to the root process, 
     * so we can create the buffer into which we'll receive the strings
     */

    const int root = 0;
    int *recvcounts = NULL;

    /* Only root has the received data */
    if (rank == root)
        recvcounts = malloc( size * sizeof(int)) ;

    MPI_Gather(&mylen, 1, MPI_INT,
               recvcounts, 1, MPI_INT,
               root, MPI_COMM_WORLD);

    /*
     * Figure out the total length of string, 
     * and displacements for each rank 
     */

    int totlen = 0;
    int *displs = NULL;
    char *totalstring = NULL;

    if (rank == root) {
        displs = malloc( size * sizeof(int) );

        displs[0] = 0;
        totlen += recvcounts[0]+1;

        for (int i=1; i<size; i++) {
           totlen += recvcounts[i]+1;   /* plus one for space or  after words */
           displs[i] = displs[i-1] + recvcounts[i-1] + 1;
        }

        /* allocate string, pre-fill with spaces and null terminator */
        totalstring = malloc(totlen * sizeof(char));            
        for (int i=0; i<totlen-1; i++)
            totalstring[i] = ' ';
        totalstring[totlen-1] = '';
    }

    /* 
     * Now we have the receive buffer, counts, and displacements, and 
     * can gather the strings 
     */

    MPI_Gatherv(mystring, mylen, MPI_CHAR,
                totalstring, recvcounts, displs, MPI_CHAR,
                root, MPI_COMM_WORLD);


    if (rank == root) {
        printf("%d: <%s>
", rank, totalstring);
        free(totalstring);
        free(displs);
        free(recvcounts);
    }

    MPI_Finalize();
    return 0;
}

Running gives:

$ mpicc -o gatherstring gatherstring.c -Wall -std=c99
$ mpirun -np 5 ./gatherstring
Rank 0: Hello
Rank 3: le
Rank 4: monde!
Rank 1: world!
Rank 2: Bonjour
0: <Hello world! Bonjour le monde!>