Network stability and biomarker evaluation

Practical 24: Genomics, systems biology and multi-omics

Learning objectives

Assess resampling stability of a descriptive network; separate selection stability from causal or predictive validation; define a biomarker’s intended use.

Biological question and data

How stable is a correlation edge when a small set of fictional animals is resampled?

Animals are resampled as paired observations across proteins. The complete protein panel has a supplied common latent driver. The correlation threshold is fixed at .55. Bootstrap inclusion frequencies describe this estimator on this dataset, not posterior probabilities of true interactions.

No biological download is required. Numerical inputs are hypothetical, not observed animal data. The runnable examples use base R. Count and abundance units are specified where used.

1. Construct the reference network

Use an explicit symmetric correlation rule and exclude self-edges.

Show R code 
helper_dir <- if (dir.exists("R")) "R" else "../R"
stopifnot(file.exists(file.path(helper_dir,"teaching_data.R")))
source(file.path(helper_dir,"teaching_data.R"))
d <- teaching_data()
X <- t(d$protein);threshold <- .55
reference <- abs(cor(X))>=threshold;diag(reference) <- FALSE
stopifnot(isSymmetric(reference))
which(reference & upper.tri(reference),arr.ind=TRUE)
          row col
protein03   3   4
protein03   3   5
protein04   4   5
protein03   3   6
protein04   4   6
protein02   2   7
protein03   3   7
protein04   4   7
protein05   5   7
protein06   6   7
protein02   2   8
protein03   3   8
protein04   4   8
protein05   5   8
protein06   6   8
protein07   7   8

Edges reflect pooled abundance correlation. Diet or other common causes can contribute to this pattern.

2. Resample animals and estimate inclusion frequency

Keep all features of a sampled animal together. One hundred resamples demonstrate the mechanism with modest computation.

Show R code 
set.seed(2401)
frequency <- matrix(0,ncol(X),ncol(X),dimnames=list(colnames(X),colnames(X)))
for(b in 1:100) {
 index <- sample(seq_len(nrow(X)),replace=TRUE)
 A <- abs(cor(X[index,,drop=FALSE]))>=threshold
 diag(A) <- FALSE
 frequency <- frequency+A/100
}
stopifnot(isSymmetric(frequency),all(frequency>=0),all(frequency<=1+1e-12))
pairs <- which(upper.tri(frequency),arr.ind=TRUE)
out <- data.frame(a=colnames(X)[pairs[,1]],b=colnames(X)[pairs[,2]],
 inclusion=frequency[pairs])
head(out[order(out$inclusion,decreasing=TRUE),],10)
           a         b inclusion
13 protein03 protein06      1.00
21 protein06 protein07      1.00
27 protein06 protein08      1.00
28 protein07 protein08      1.00
19 protein04 protein07      0.98
24 protein03 protein08      0.97
25 protein04 protein08      0.96
26 protein05 protein08      0.96
18 protein03 protein07      0.93
6  protein03 protein04      0.89

An edge can be frequent because a shared technical or biological driver is stable. Frequency alone does not establish a direct interaction.

3. Define the biomarker use before selection

Specify the decision, population and validation required for an abundance feature.

Show R code 
use <- data.frame(task=c("predict later performance","monitor current status","causal intervention target"),
 evidence=c("held-out animals and decision-time availability","calibrated assay and clinical/production relevance",
 "causal design and targeted perturbation evidence"))
use
                        task                                           evidence
1  predict later performance    held-out animals and decision-time availability
2     monitor current status calibrated assay and clinical/production relevance
3 causal intervention target   causal design and targeted perturbation evidence

These tasks have different evidentiary requirements. A feature discovered using all outcomes has not received an independent prediction test.

4. Distinguish discrimination from a decision threshold

For a fictional status marker, calculate sensitivity and specificity at several supplied cutoffs. A threshold changes the action implied by the same measurements.

Show R code 
status <- c(0,0,0,0,1,1,1,1)
marker <- c(.1,.3,.4,.7,.2,.6,.8,.9)
threshold_result <- t(sapply(c(.3,.5,.7),function(cut) {
 positive <- marker>=cut
 c(sensitivity=mean(positive[status==1]),
 specificity=mean(!positive[status==0]))
}))
data.frame(cutoff=c(.3,.5,.7),threshold_result)
  cutoff sensitivity specificity
1    0.3        0.75        0.25
2    0.5        0.75        0.75
3    0.7        0.50        0.75

5. Validate the intended biomarker use

A threshold for screening favors different errors from one used for a costly intervention. Predictive values also depend on prevalence in the target population. Choosing a threshold and estimating its performance on the same observations is a selection procedure, not independent validation.

Network stability, assay reproducibility, predictive discrimination and decision utility are different properties. A stable edge does not validate a biomarker assay or a causal intervention target. Specify the task, collection timing, target population and required follow-up before declaring a candidate useful.

Interpretation challenge

Why might a marker that discriminates well in a case-enriched study have different positive predictive value in routine screening?

The target prevalence and sampling differ. Positive predictive value depends on prevalence as well as sensitivity and specificity, so the screening context needs evaluation.

Interpretation and limits

Bootstrap stability is conditional on the current sample and analysis rule. It does not address all external shifts or verify identification. Predictive markers need complete training-only selection and independent evaluation; causal targets require stronger evidence.

Exercises

  1. Why resample animals rather than each protein independently?
  2. Is an inclusion frequency of .9 a .9 probability of a real interaction?
  3. What if threshold is tuned to create a desired module?
  1. Paired feature relationships must be retained within the sampling unit.
  2. No; it is the observed bootstrap inclusion frequency under the procedure.
  3. The choice is exploratory and needs honest reporting and separate validation.

Continue learning

B14 Networks, B12 Statistics and B18 Synthesis.

Continue with reporting. Return to the practical index or the unified map.