Standalone models and YAML examples

Input bindings, a complete ssSNPBLUP model, solver settings, covariance estimation and truth evaluation.

Quick start · Installation · Results and diagnostics

One YAML file specifies input files, column bindings, the statistical model, natural covariance matrices, fitting controls, threads and requested outputs. The interface uses schema: gsuite.model/v1-draft. Unknown keys and unsupported combinations are rejected. The YAML specification gives the full syntax and supported combinations.

Input and model sections

Section	Meaning
traits	Ordered response/channel names, also used for environments
inputs	All CSV/TSV, PLINK, covariance, assignment and prediction-grid files
relationships	Named pedigree, single-step, APY or supplied relationship models
bases	Optional power, Legendre or Chebyshev regression bases
covariances	Natural variances/covariances or FA loadings and specific variances
model	Observation input, fixed formula, named random effects and residual structure
analysis	task: solve for supplied covariances, or task: reml for estimation
execution	Formulation, solver, factorization, threads and linear-solve tolerances
output	Destination, selected coefficient tables, predictions and optional uncertainty
evaluation	Optional comparison with supplied true genetic values after fitting

Matrices are covariances, never inverses. Inline matrices use one YAML list item per row, for example - [1.0, 0.2]. Their order follows the generated channels; labelled inline or external matrices must match that order exactly. Matrices must be finite, symmetric and positive definite. Under REML they are starting values; otherwise they are fixed supplied values.

Column bindings use neutral logical names such as individual, parent1, parent2, mother, time or environment. IDs remain strings, including leading zeros. Wide phenotypes map trait names to response columns. Long phenotypes use layout: long and responses: {trait: trait_column, value: value_column}. One record ID is one residual occasion: ID/trait pairs must be unique and mapped covariates must agree within an ID. Distinct visits need distinct record IDs. Absent trait rows are missing responses, without imputation. Independent prediction grids remain wide.

Complete ssSNPBLUP example

Save this as model.yml beside the supplied example’s phenotypes.csv, pedigree.csv and genotypes.bed/.bim/.fam. The physical columns in this synthetic example use animal, sire and dam; the logical bindings remain individual, parent1 and parent2 for plant, human and animal datasets. Edit the physical bindings for another dataset. The matrices below are covariances on their natural scale, in the declared [milk, fat] trait order. The example requests eight threads for marker work and one for other phases; request at least eight Slurm CPUs, or edit the YAML to match your allocation.

The included example contains:

example/
  model.yml
  pedigree.csv
  phenotypes.csv
  genotypes.bed
  genotypes.bim
  genotypes.fam
  true_breeding_values.csv

phenotypes.csv has columns record_id,animal,milk,fat,herd,age; pedigree.csv has animal,sire,dam, with 0 for unknown parents. PLINK FAM IIDs match the pedigree individual IDs; BED is SNP-major and BIM lists markers. Repeated observations use distinct record_id values. The example uses NA for one missing response. The truth CSV is optional reference material and is deliberately absent from the model’s inputs.

Save the following complete specification as model.yml beside those inputs. All covariance matrices are supplied values; analysis.task: solve holds them fixed rather than estimating them.

# Tiny synthetic example: covariance matrices follow [milk, fat].
schema: gsuite.model/v1-draft
name: single_step_example
traits: [milk, fat]
inputs:
  observations:
    type: table
    file: phenotypes.csv
    id: record_id
    columns: {individual: animal, site: herd, age: age}
    responses: {milk: milk, fat: fat}
  pedigree:
    type: pedigree
    file: pedigree.csv
    columns: {individual: animal, parent1: sire, parent2: dam}
  genotypes: {type: plink, prefix: genotypes, id: IID}
relationships:
  A: {type: pedigree, input: pedigree}
  H:
    type: single_step
    pedigree: A
    genotypes: genotypes
    genomic: {adjustment: none, polygenic_weight: 0.0, independent_ridge: 0.05}
covariances:
  genetic:
    - [1.0, 0.1]
    - [0.1, 0.04]
  individual_environment:
    - [0.4, 0.02]
    - [0.02, 0.02]
  residual:
    - [2.0, 0.08]
    - [0.08, 0.05]
model:
  data: observations
  fixed: "factor(site) + age"
  random:
    genetic: {group: individual, relationship: H, covariance: genetic}
    individual_environment: {group: individual, covariance: individual_environment}
  residual: residual
analysis: {task: solve}
execution:
  formulation: ss_snp
  solver: pcg
  sparse_backend: pardiso
  preconditioner: qnn_reuse
  pedigree_factorization: ancestral_sweeps
  convergence: {relative_tolerance: 1.0e-10, maximum_iterations: 4000}
  threads: {default: 1, marker_products: 8, marker_setup: 8}
output:
  directory: ../results/example
  tables: [fixed_effects, random_effects, diagnostics]
  predictions:
    genetic_values: {effects: [genetic], levels: all}

From that directory, validate, inspect and fit:

gsuite validate model.yml
gsuite describe model.yml
gsuite run model.yml

The selected outputs are genetic_values.csv, fixed_effects.csv, random_effects.csv and diagnostics.json under ../results/example/. run.json records numerical acceptance, residual checks and native timings; execution.json records whole-workflow timings, result collection and sampled process-tree memory. Check their acceptance fields before using predictions. Repeating the fit requires a fresh output directory, for example gsuite run model.yml --output ../results/example-repeat.

This supplied-covariance model selects packed PCG, exact ancestral pedigree sweeps and Qnn reuse. Change model/task declarations in YAML rather than choosing a different executable. The model specification gives the full syntax for other effects and covariance families.

Solver and threads

For fast packed ssSNPBLUP:

execution:
  formulation: ss_snp
  solver: pcg
  sparse_backend: pardiso
  pedigree_factorization: ancestral_sweeps
  preconditioner: qnn_reuse
  threads: 16

A scalar sets every applicable thread request. Override particular operations with threads: {default: 16, decode: 4}. Packed controls are marker_products, marker_setup, decode, pedigree, preconditioner and vectors. Compiled models use factor and relationship; raw-genotype APY also uses decode and genomic_products. A mapping without default uses one for unspecified operations. Request matching Slurm CPUs. Thread settings are phase-specific requests: Python preparation and dependency-ordered pedigree work can be serial.

ancestral_sweeps preserves the full statistical model with a smaller retained pedigree factor and exact peripheral sweeps, including supplied metafounders. full_qnn is available for comparison and is the default. qnn_reuse avoids a second global pedigree factor; shifted_pedigree is the alternative packed preconditioner. Linear PCG tolerances belong in execution.convergence. Ordinary compiled models support direct or PCG solving. Exact/APY ssGBLUP uses compiled PCG and formulation: ss_gblup.

Covariance estimation

analysis.estimate selects free covariance blocks; omitted blocks stay fixed. Sparse-direct compiled REML supports unstructured and FA covariance and scalar or separable multivariate serial parameters. Natural starts are transformed internally. analysis.constraints can fix named optimizer coordinates; describe lists them. A fixed Cholesky coordinate is not a fixed covariance entry. Direct optimizer controls include maximum_step and maximum_line_search_reductions.

Packed EM-REML estimates unstructured genetic and independent residual covariance for ordinary pedigrees, with one genetic channel per trait and fixed additional identity-effect covariances. Its update control is optimizer.step_fraction in (0, 1]. Score traces have their own explicit solve and precision budgets under analysis.traces. Exact traces are small references; stochastic traces require explicit probes, seed, total solve cap and maximum score MCSE. Stochastic convergence also requires an independent final score check. See packed REML controls and limits.

Omit analysis.likelihood to skip global determinant work in packed REML. Optional method: exact requires maximum_dimension; method: slq requires maximum_operator_products, with optional probe/depth/precision controls. It reports the likelihood change from the start to the final covariance state, with separate sampling MCSE and Lanczos-depth sensitivity. It does not control EM updates or certify a global optimum. Unqualified optional likelihood does not reject an otherwise qualified point fit. Compiled direct REML retains its absolute likelihood and line-search acceptance.

Model capabilities and limits

The complete examples cover pedigree, packed ssSNPBLUP, exact/APY ssGBLUP, direct/maternal and weighted incidence, regression, external relationships, groups/metafounders, FA, serial residuals and estimation. The specification gives exact syntax and route-specific supported combinations.

Supplied FA uses natural loadings and specific variances with an identified triangular leading loading block; it reduces covariance parameters, not genetic coefficient count. Serial models support scalar or separable trait-by-time AR1 and exponential covariance with actual gaps and missing traits. Compiled models also accept general finite trait-by-visit covariance blocks.

Fixed/random groups, explicit phantom group-parent absorption, combined bases and supplied/estimated Gamma are distinct statistical declarations. Packed ordinary random groups use implicit sweeps; packed fixed/combined groups prepare ordinary inheritance columns once, requiring O(individuals × groups) storage. Metafounders retain every correlated latent base. Gamma pseudo-EM requires an explicit absolute marker base and is separate from joint REML.

Sparse external natural covariance uses a retained sparse inverse-action factor; fill and exact precision-diagonal setup can be costly. Raw PLINK APY uses an explicit ordered core and bounded tiles/file-backed cross products. Industrial APY preparation remains unqualified. Packed REML currently excludes FA, serial/grouped residuals, groups/metafounders and joint PEV requests.

Existing selected exact/stochastic PEV options remain conditional on supplied covariances, with explicit work budgets and separate qualification. Their precision is not improved by this interface cleanup. The model-only validator skips uncertainty cases.

Compare predictions with supplied truth

An optional top-level evaluation block makes gsuite run model.yml fit and score in one invocation with evaluation settings in the same YAML. Truth is used only after an accepted fit.

For the 20K gsim dataset, add this alongside model, execution and output:

evaluation:
  predictions: genetic_values
  truth:
    file: true_breeding_values.csv
    columns:
      individual: animal
      trait: trait
      value: true_breeding_value
  subset: heldout
  individual: individual
  genotypes: genotypes

output:
  directory: results/prediction20k-evaluated
  predictions:
    genetic_values: {effects: [genetic], levels: all}

Replace the existing output block rather than declaring it twice. The predictions key references a named output.predictions request with individual levels. truth.columns maps the long truth CSV’s individual, trait and numeric value columns. Its file resolves beside the YAML. individual references a mapped observation column, not the source CSV header. Optional genotypes references a PLINK input; its existing FID/IID-to-individual mapping is reused. No second genotype path or training table is needed.

Run in a compute-node allocation with the model’s requested CPUs:

gsuite run /path/to/dataset/model.yml

Fitting and evaluation use managed $TMPDIR scratch and are collected together at output.directory. Use a fresh result directory. No shell scoring or copy step is needed. execution.threads: 8 still controls the native fit.

subset is heldout, phenotyped or all. An individual is phenotyped if it has any observed response in model.data; repeated rows, long-format rows and fully missing records use that same definition. Held-out individuals have no observed training response in any trait. This option does not mask training phenotypes: the input must already encode the intended training/held-out split. Partially observed traits do not make an individual held out. Truth targets must match the requested prediction levels and traits. Duplicate/nonfinite truth, empty subsets and missing predictions fail explicitly.

evaluation/summary.csv reports Pearson accuracy, Spearman rank correlation, RMSE, mean absolute error, mean prediction-minus-truth bias, and the intercept and slope of truth regressed on prediction. Ideal mean bias/intercept are zero and slope is one. When genotypes is supplied, results are grouped by all, genotyped and non-genotyped targets. evaluation/matched.csv retains every scored pair; evaluation/assessment.json records definitions, hashes and cohort settings. evaluation-inputs/ preserves the frozen evaluation-only targets. No truth centering, rescaling or model tuning is performed.

execution.json includes separate preparation, fitting and evaluation times and workflow memory. Overall acceptance requires a successful requested evaluation as well as fitting. A scoring failure preserves the accepted native fit and its diagnostics; native numerical acceptance and evaluation acceptance remain separate. Acceptance checks a complete labelled comparison, not an accuracy threshold. One simulated dataset does not establish population-level bias or sampling uncertainty.