32bit 64bit - How to get a big sparse matrix in R? (> 2^31-1)

Question

I use some C++ code to take a text file from a database and create a dgcMatrix type sparse matrix from the Matrix package. For the first time, I'm trying to build a matrix that has more than 2^31-1 non-sparse members, which means that the index vector in the sparse matrix object must also be longer than that limit. Unfortunately, vectors seem to use 32-bit integer indices, as do NumericVectors in Rcpp.

Short of writing an entire new data type from the ground up, does R provide any facility for this? I don't think I can use too exotic a solution as I need glmnet to recognize the resultant object.

Answer 1

The sparse matrix algebra R package spam with its spam64 extension supports sparse matrices with more than 2^31-1 non-zero elements.

A simple example (requires ~50 Gb memory and takes ~5 mins to run):

## -- a regular 32-bit spam matrix
library(spam) # version 2.2-2
s <- spam(1:2^30)
summary(s) 
## Matrix object of class 'spam' of dimension 1073741824x1,
##     with 1073741824 (row-wise) nonzero elements.
##     Density of the matrix is 100%.
## Class 'spam'

## -- a 64-bit spam matrix with 2^31 non-zero entries
library(spam64)
s <- cbind(s, s) 
summary(s) 
## Matrix object of class 'spam' of dimension 1073741824x2,
##     with 2147483648 (row-wise) nonzero elements.
##     Density of the matrix is 100%.
## Class 'spam'

## -- add zeros to make the dimension 2^31 x 2^31
pad(s) <- c(2^31, 2^31) 
summary(s) 
## Matrix object of class 'spam' of dimension 2147483648x2147483648,
##     with 2147483648 (row-wise) nonzero elements.
##     Density of the matrix is 4.66e-08%.
## Class 'spam'

The implementation is based on the .C64() R interface to compiled code available in dotCall64.

Note: Not all function of spam support 64-bit matrices (yet).

Some links:

• https://cran.r-project.org/package=spam
• https://cran.r-project.org/package=spam64
• https://cran.r-project.org/package=dotCall64
• https://doi.org/10.1016/j.cageo.2016.11.015
• https://doi.org/10.1016/j.softx.2018.06.002

I am one of the authors of dotCall64 and spam.