Standalone commands, results and diagnostics

Command behavior, output files, managed scratch, acceptance, timings and memory.

Quick start · Installation · Model guide

Commands and paths

gsuite validate model.yml
gsuite describe model.yml
gsuite run model.yml

validate checks the model and input tables without calling the native driver; it is not a numerical reference campaign. describe additionally prints expanded dimensions, channel order, parameter coordinates and execution settings without fitting. Both commands read inputs. run chooses the equations and task from the YAML, including supplied-covariance solving or REML.

Input paths and output.directory resolve beside the original YAML, even when launched from another directory. Source inputs are never rewritten. The final result directory must be fresh. To repeat a fit with another destination:

gsuite run model.yml --output /path/to/results/a-new-run

A source bundle installs its launcher under install/bin/. From that bundle, add "$PWD/install/bin" to PATH. Developers running from a checkout use python tools/build/hpc_single_step/gsuite.py with an explicit --exe /path/to/gsuite_model_driver.

Scratch and result collection

The runner creates a private working directory under $TMPDIR, or the system temporary directory when $TMPDIR is unset. --scratch-directory /existing/path selects another parent. SQLite snapshots use this scratch by default; an explicit YAML execution.preparation.directory remains honored. Tables, long-row grouping and prediction grids are spooled to temporary SQLite storage. Pedigree/effect indexes, genotype metadata, supplied matrices and native factors still require memory. There is no application RAM ceiling, admission guard or run timeout.

Prepared inputs, native work and labelled output are written in scratch, then collected into output.directory (or --output). The runner reserves the destination exclusively, so another invocation cannot overwrite it. Successful collection removes only this invocation’s scratch. Failed fits also collect diagnostics and exit nonzero. If copying fails, scratch is retained and its recovery path is printed; the final directory may contain a partial copy.

Output files and acceptance

Every fit retains the requested prediction CSVs, native diagnostics, checksums and logs. Common files are:

File	Contents
run.json	Native acceptance, iteration counts, residual checks and phase timings
execution.json	Workflow acceptance, execution environment, timings and sampled process-tree memory
fixed_effects.csv, random_effects.csv	Requested coefficient tables
model.yml, inputs/	Original YAML snapshot and prepared inputs
console.log, worker.json	Command output and worker diagnostics

The YAML snapshot preserves its original syntax and relative paths; execution.json.model_path identifies their original resolution base. Copied inputs and native reports retain recorded execution paths; they are not a restart contract for prepared fits.

REML adds variance_components.csv, parameters.csv, reml_history.csv, convergence.json and likelihood.json. The latter reports scale, method, status and the value/error/work diagnostics available for that route. Read acceptance and convergence flags before using estimates. Failed covariance fits retain diagnostic states/history but publish no qualified prediction tables.

Optional truth evaluation adds evaluation/summary.csv, matched predictions and an assessment record. Overall acceptance requires the requested comparison to succeed as well as fitting. A scoring failure preserves the accepted native fit and its diagnostics; native numerical acceptance and evaluation acceptance remain separate.

Timings and memory

execution.json reports preparation, fitting, native preparation/evaluation, Python output formatting, result collection and total elapsed time. Native phase timings, iteration counts and residual gates remain in run.json. Native peak RSS excludes Python preparation.

Workflow memory is sampled aggregate RSS of the coordinator, Python worker and native descendants. It can count shared pages more than once and miss brief peaks between samples; it is not Slurm/cgroup accounting. Unsupported sampling is reported as unavailable rather than zero. YAML thread counts are requests for particular phases; some preparation and dependency-ordered work is serial.

Installation validation

For source bundles, bash hpc_tasks/validate.sh runs the full campaign described in installation and maintenance. Receipts record actual case counts and hashes. These checks do not establish industrial runtime, memory or stochastic precision. No remote job is dispatched automatically. Historical acceptance is recorded separately in the GenomeDK validation record.