All Leave-One-Out Models (ALOOM)

Example of using aloom and some interesting results

We use publicly available mutagenicity dataset from Kazius et al. (2005)⁽⁶⁾. It contains 4335 compounds, 2400 categorised as “mutagen” and the remaining 1935 compounds as “nonmutagen”. The dataset is available from the QSARdata⁽⁷⁾ R package and each compound comes with 1579 descriptors. Half of the dataset is used for training and the remaining half for testing.

1. Create train and test datasets

library(QSARdata)
library(caret)

data(Mutagen)
x           <- as.matrix(Mutagen_Dragon)
rownames(x) <- rownames(Mutagen_Dragon)
colnames(x) <- colnames(Mutagen_Dragon)
y           <- Mutagen_Outcome

set.seed(1)
lvFolds <- createFolds(y,k=2)

train.x <- x[-lvFolds[[1]],]
train.y <- y[-lvFolds[[1]]]
test.x  <- x[lvFolds[[1]],]
test.y  <- y[lvFolds[[1]]]
            

The train dataset consists of 2168 samples, while test has 2167. Here they are available as csv files: train_x.csv, train_y.csv, test_x.csv, test_y.csv.

2. Create aloom object with randomForest⁽⁹⁾

NOTE 1: On machine with 48 CPUs AMD Opteron 6168 with 15 Gb RAM when using all 48 CPUs this takes 23 hours
NOTE 2: On machine with 24 CPUs Intel Xeon 6240@2.60GHz with 4 Gb RAM when using all 24 CPUs this takes 8 hours

library(aloom)
library(parallel)
library(randomForest)

ntree     <- 1000
num.cores <- detectCores()

fit <- aloom(train.x, train.y, test.x, method="rf",list(ntree=ntree),mc.cores=num.cores)
            

2. Create aloom object with glmnet⁽¹⁰⁾

NOTE 1: On machine with 48 CPUs AMD Opteron 6168 with 15 Gb RAM when using all 48 CPUs this takes 4 hours
NOTE 2: On machine with 24 CPUs Intel Xeon 6240@2.60GHz with 4 Gb RAM when using all 24 CPUs this takes 2 hours

Prior to calling aloom() we execute cv.glmnet() to find optimal lambda.

library(aloom)
library(parallel)
library(glmnet)

cv.fit          <- cv.glmnet(train.x,train.y,family="binomial",type.measure="auc")
selected.lambda <- cv.fit$lambda.1se
lambda          <- cv.fit$lambda
model.params    <- list(lambda=lambda, alpha=1, selected.lambda=selected.lambda)

num.cores <- detectCores()

fit <- aloom(train.x, train.y, test.x, method="glmnet",model.params,mc.cores=num.cores)
            

3. Examine aloom object

All Leave-One-Out Models, as well as the original model, are created during the execution of aloom(). Their predictions of test samples are the return results.
An aloom object is a list containing:

• predicted.y - predicted categories of test samples produced by the original model
• predicted.prob.y - predicted probabilities of test samples produced by the original model
• aloom.probs - ALOOM probabilities of test samples. It is a matrix with rownames equal to names of test samples and colnames equal to names of training samples.

predicted.y      <- fit$predicted.y
predicted.prob.y <- fit$predicted.prob.y
aloom.probs      <- fit$aloom.probs
            

Calculate original's misclassification error, ALOOM's proportion of NA and ALOOM's misclassification error.

original.misclassification <- sum(predicted.y!=test.y)/length(test.y)

find.na       <- function(x){if ((min(x) < 0.5) & (max(x) > 0.5)) TRUE else FALSE}
predicted.na  <- apply(aloom.probs,1,find.na)
aloom.proportion.na <- sum(predicted.na)/length(predicted.na)

aloom.misclassification <-  sum(predicted.y[!predicted.na]!=test.y[!predicted.na])/length(test.y[!predicted.na])
            

4. Calculate ALOOM individual prediction intervals, examine their width and show the sample with the maximum width

min.aloom <- apply(aloom.probs,1,min)
max.aloom <- apply(aloom.probs,1,max)
            

Calculate width of every ALOOM individual prediction interval and examine its distrubition.

width <- max.aloom - min.aloom
summary(width)
            

ALOOM individual prediction interval width stats are:

id.with.max.width <- rownames(test.x)[which.max(width)]
max.width.range   <- c(min.aloom[which.max(width)],max.aloom[which.max(width)])