c++ - Modifying boost array in function with MPI leads to assertion fail

Question

I am trying to modify a boost array in a function. The code runs fine in serial and produces the expected results, whereas with MPI I get an error. Here is a minimal reproducible example:

#include <stdio.h>
#include <iostream>
#include "boost/multi_array.hpp"


static void myfunc(boost::multi_array<double, 3>& foo){
            std::cout<<foo[0][0][0]<<std::endl;
        }

int main(){
    typedef boost::multi_array<double, 3> array_type;
    typedef array_type::index index;
    array_type foo(boost::extents[1][1][1]);
    myfunc(foo);
  return 0;
}

The error I get is:

.../src/boost/boost/multi_array/base.hpp:135: Reference boost::detail::multi_array
::value_accessor_n<T, NumDims>::access(boost::type<Reference>, boost::detail::multi_array::value_accessor_n<T, NumDims>::index, TPtr, const size_type
*, const index*, const index*) const [with Reference = boost::detail::multi_array::sub_array<double, 2>; TPtr = double*; T = double; long unsigned in
t NumDims = 3; boost::detail::multi_array::value_accessor_n<T, NumDims>::index = long int; boost::detail::multi_array::multi_array_base::size_type = 
long unsigned int]: Assertion `idx - index_bases[0] >= 0' failed.

The root rank prints out foo[0][0][0], but the other ranks do not, which is when the error occurs. My naive interpretation of this behavior would be that the other ranks do not carry foo[0][0][0], therefore I get something like an "index out of bound error". Any ideas on how I can fix this?

Edit: When I print out the size of the boost array each rank prints a size of (initial size of array)/(number of ranks).

question from:https://stackoverflow.com/questions/66066736/modifying-boost-array-in-function-with-mpi-leads-to-assertion-fail

Answer 1

Indeed, the problem is that not all ranks carry all indices. In particular, if the total number of ranks is N_ranks, each individual rank is rank and the dimensions of the boost array are NxNxN, each rank carries [rank * N / N_ranks:(rank + 1) * N / N_ranks, 0:N, 0:N]. Therefore, if one wants to access the foo array, it should be indexed in a manner dependent on the number of ranks.