Integrated analysis and comparison of prediction strategies

Practical 23: Genomics, systems biology and multi-omics

Learning objectives

Compare early and late integration on identical held-out animals; keep transformations in training; interpret paired errors and operational limits.

Biological question and data

Does integration add predictive information beyond either hypothetical molecular layer alone?

The split, penalty and integration weights are supplied. No test outcome chooses a feature, penalty or mixture weight. The fixture is a small illustrative simulation; all responses use arbitrary units.

No biological download is required. Numerical inputs are hypothetical, not observed animal data. The runnable examples use base R. Count and abundance units are specified where used.

1. Specify the competing strategies

Compare separate protein/metabolite ridge models, weighted early concatenation and a fixed average of separate predictions.

helper_dir <- if (dir.exists("R")) "R" else "../R"
stopifnot(file.exists(file.path(helper_dir,"teaching_data.R")))
source(file.path(helper_dir,"teaching_data.R"))
d <- teaching_data()
source(file.path(helper_dir,"prediction_helpers.R"))
P <- t(d$protein);M <- t(d$metabolite);X <- cbind(P,M)
y <- d$metadata$response;train <- 1:32;test <- 33:48
stopifnot(identical(rownames(P),rownames(M)))
models <- list(protein=fit_teaching_ridge(P[train,],y[train]),
 metabolite=fit_teaching_ridge(M[train,],y[train]),
 early=fit_teaching_ridge(X[train,],y[train],weights=c(rep(1/sqrt(ncol(P)),ncol(P)),
 rep(1/sqrt(ncol(M)),ncol(M)))))
stopifnot(max(abs(models$protein$center-colMeans(P[train,])))<1e-12)

Weights are applied after training standardization in the helper. Re-standardizing the weighted columns would cancel these intended block weights.

2. Predict without using test outcomes

The late strategy uses a prespecified equal average. It is not a test-optimized ensemble.

prediction <- cbind(protein=predict_teaching_ridge(models$protein,P[test,]),
 metabolite=predict_teaching_ridge(models$metabolite,M[test,]),
 early=predict_teaching_ridge(models$early,X[test,]))
prediction <- cbind(prediction,late=rowMeans(prediction[,c("protein","metabolite")]),
 baseline=mean(y[train]))
error <- sweep(prediction,1,y[test],"-")
data.frame(strategy=colnames(prediction),test_RMSE=sqrt(colMeans(error^2)))

             strategy test_RMSE
protein       protein  1.319743
metabolite metabolite  1.244327
early           early  1.215457
late             late  1.242978
baseline     baseline  1.348718

stopifnot(nrow(prediction)==length(test),all(is.finite(prediction)))

All strategies face the same animals and loss. The comparison can show whether the selected strategy helps in this realization, rather than assuming more layers must improve performance.

3. Inspect a paired comparison

Subtract squared errors animal by animal. Inspect variability, not just a ranking.

improvement <- error[,"protein"]^2-error[,"early"]^2
c(mean_paired_MSE_improvement=mean(improvement),
 animals_improved=sum(improvement>0),animals_evaluated=length(improvement))

mean_paired_MSE_improvement            animals_improved 
                   0.264384                    8.000000 
          animals_evaluated 
                  16.000000 

plot(y[test],prediction[,"early"],pch=19,xlab="Observed response",ylab="Early-integration prediction")
abline(0,1,col="#d95f02")

Positive improvement means smaller squared error for early integration than the protein model. The sign can differ across animals and seeds.

4. Quantify conditional paired uncertainty

Resample the held-out animal error differences, keeping the strategy comparison paired. This evaluates variability of mean improvement for the supplied fitted predictions; it does not retrain models.

set.seed(2302)
paired_boot <- replicate(500,mean(sample(improvement,replace=TRUE)))
c(estimated_improvement=mean(improvement),
 quantile(paired_boot,c(.025,.975)))

estimated_improvement                  2.5%                 97.5% 
           0.26438400           -0.07542548            0.60786647 

hist(paired_boot,xlab='Mean paired MSE improvement',
 main='Conditional animal-resampling illustration')
abline(v=0,lty=2)

5. Separate this uncertainty from deployment uncertainty

The split changes farm composition, and these test animals are not a universal population. Correlated evaluation units may require grouped resampling. Retraining and tuning variation are also omitted from the displayed bootstrap.

Evaluate calibration, availability and assay cost alongside predictive loss. A strategy using an unavailable molecular layer cannot support the same decision as an available baseline. Late weights chosen from these test outcomes would consume the evaluation set; legitimate selection must be internal to training.

Interpretation challenge

Can an interval favoring early integration establish improvement on every future farm?

Suggested reasoning

No. It is conditional on this sample, split, models and resampling assumptions. External transfer, fitting uncertainty and operational feasibility need separate evidence.

Interpretation and limits

The source contains a restricted educational ridge helper, not a general modelling package. One farm-composition split cannot qualify an operational predictor. Nested tuning, independent cohorts, calibration, measurement availability and costs belong in a real comparison.

Exercises

1. Why does the fixed late average avoid one kind of leakage?
2. How would you choose mixture weights legitimately?
3. Why might adding a layer worsen prediction?

Suggested answers

1. Test outcomes do not choose the weight.
2. Estimate/tune them in training-only validation, then freeze the pipeline before evaluation.
3. Noise, small samples, redundant features, shift or unsuitable scaling/regularization can outweigh added information.

Continue learning

B12 Statistics, B15 Integration and B18 Synthesis.

Continue with integrated interpretation. Return to the practical index or the unified map.