r - How to get summary statistics for multiple variables by multiple groups?

Question

I know that there are many answers provided in this forum on how to get summary statistics (e.g. mean, se, N) for multiple groups using options like aggregate , ddply or data.table. I'm not sure, however, how to apply these functions over multiple columns at once.

More specifically, I would like to know how to extend the following ddply command over multiple columns (dv1, dv2, dv3) without re-typing the code with different variable name each time.

library(reshape2)
library(plyr)

group1 <- c(rep(LETTERS[1:4], c(4,6,6,8)))
group2 <- c(rep(LETTERS[5:8], c(6,4,8,6)))
group3 <- c(rep(LETTERS[9:10], c(12,12)))
my.dat <- data.frame(group1, group2, group3, dv1=rnorm(24),dv2=rnorm(24),dv3=rnorm(24))
my.dat

data1 <- ddply(my.dat, c("group1", "group2","group3"), summarise,
               N    = length(dv1),
               mean = mean(dv1,na.rm=T),
               sd   = sd(dv1,na.rm=T),
               se   = sd / sqrt(N)
)
data1

How can I apply this ddply function over multiple columns such that the outcome will be data1, data2, data3... for each outcome variable? I thought this could be the solution:

dfm <- melt(my.dat, id.vars = c("group1", "group2","group3"))
lapply(list(.(group1, variable), .(group2, variable),.(group3, variable)), 
   ddply, .data = dfm, .fun = summarize, 
   mean = mean(value), 
   sd = sd(value),
   N=length(value),
   se=sd/sqrt(N))

Looks like it's in the right direction but not exactly what I need. This solution provides the statistics by each group separately. What I need an outcome as in data1 (e.g. first aggregated group is people who are at A, E and I; the second is those who are at group B, E and I etc...)

Answer 1

Here's an illustration of reshaping your data first. I've written a custom function to improve readability:

mysummary <- function(x,na.rm=F){
  res <- list(mean=mean(x, na.rm=na.rm),
              sd=sd(x,na.rm=na.rm),
              N=length(x))
  res$se <- res$sd/sqrt(res$N)
  res
}

library(data.table)

res <- melt(setDT(my.dat),id.vars=c("group1","group2","group3"))[,mysummary(value),
    by=.(group1,group2,group3,variable)]

> head(res)
   group1 group2 group3 variable  mean        sd N       se
1:      A      E      I      dv1  9.75  6.994045 4 3.497023
2:      B      E      I      dv1  9.50  7.778175 2 5.500000
3:      B      F      I      dv1 16.00  4.082483 4 2.041241
4:      C      G      I      dv1 14.50 10.606602 2 7.500000
5:      C      G      J      dv1 10.75 10.372239 4 5.186119
6:      D      G      J      dv1 13.00  4.242641 2 3.000000

Or without the custom function, thanks to @Jaap

melt(setDT(my.dat),
     id=c("group1","group2","group3"))[, .(mean = mean(value),
                                           sd = sd(value),
                                           n = .N,
                                           se = sd(value)/sqrt(.N)),
                                       .(group1, group2, group3, variable)]