parallel processing - doParallel error in R: Error in serialize(data, node$con) : error writing to connection

Question

This is my code. The stuff inside the loop makes sense.

        library(foreach)
        library(doParallel)
        cl <- makeCluster(7)
        registerDoParallel(cl) 

        elasticitylist = foreach(i=1:nhousehold) %dopar% {

            pricedraws = out$betadraw[i,12,] 
            elasticitydraws[,,i]= probarray[,,i] %*% diag(pricedraws)
            elasticitydraws[,,i] = elasticitydraws[,,i] * as.vector(medianpricemat)

        } 

I keep getting this error:

Error in serialize(data, node$con) : error writing to connection

I know I have enough cores (there are 20). Can anyone help with this? It seems the answer is nowhere to be found in docs!

When I run ps -ef| grep user on my unix server, I get:

/apps/R.3.1.2/lib64/R/bin/exec/R --slave --no-restore -e parallel:::.slaveRSOCK() --args MASTER=localhost PORT=11025 OUT=/dev/null TIMEOUT=2592000 METHODS=TRUE XDR=TRUE

Answer 1

The functions serialize and unserialize are called by the master process to communicate with the workers when using a socket cluster. If you get an error from either of those functions, it usually means that at least one of the workers has died. On a Linux machine, it might have died because the machine was almost out of memory, so the out-of-memory killer decided to kill it, but there are many other possibilities.

I suggest that you use the makeCluster outfile="" option when creating the cluster object so that output from the workers is displayed. If you're lucky, you'll get an error message from a worker before it dies that will help you to solve the problem.