Architecture

How the gsuite R packages, native libraries and shared genomic resources fit together.

gsuite is organised by user task. Shared data preparation lives in gbase; analysis methods live in independently installable R packages; numerical and statistical computation remains in the corresponding native libraries. The root gsuite development integration currently provides mixed-model and solver R interfaces. The ecosystem documentation is maintained here independently of that integration. A root R package release is not currently planned; greml/gsolve R packaging is deferred, with separate or combined packaging undecided.

The standalone command is gsuite run model.yml for both solving and REML. It calls the native gsuite_model_driver executable. The greml and gsolve library names identify their mixed-model and numerical-solver responsibilities; they are not separate user commands. Existing R interfaces are unchanged.

Native capabilities and R availability

Native greml’s conditional marker operator has an opt-in ancestral pedigree factor with exact sweeps for the remaining individuals. Greml owns the inheritance reduction and coefficient maps; installed gsolve owns the smaller sparse factor. Its reuse preconditioner shares the complete solve. The YAML frontend exposes execution.pedigree_factorization: ancestral_sweeps and checks native capability support; existing R interfaces remain unchanged. See the factor-reduction study.

The optional standalone HPC prediction frontend accepts CSV pedigree and phenotypes, PLINK genotypes and YAML covariance matrices. Its source-only pedigree-blend adapter composes public greml pedigree providers with the native conditional ssSNPBLUP PCG operator; numerical computation remains in greml, gsolve and genotype owners. The standalone model/task driver lives in greml apps/model_driver/; its input preparation, compiled-model tasks, packed-model tasks and augmented prediction routes are separate translation units. Tasks select supplied-value solving or REML, with optional uncertainty where supported by the model route. The driver consumes installed public interfaces; gsolve retains numerical solver ownership. Optional YAML truth evaluation is owned by the standalone frontend, outside the native model inputs. model_evaluation.py validates labelled targets and freezes cohorts from observed records; prediction_metrics.py is shared by the runner and the developer CSV comparison command. The worker evaluates only after an accepted native fit and collects targets, metrics and failure diagnostics with separate numerical/evaluation acceptance. Truth never enters record assembly, covariance estimation or prediction equations. See the greml/apps/model_driver/ organization. This standalone interface does not add R controls. See the representative workload and qualification.

The experimental compact YAML interface binds generic input columns, named effects, polynomial bases and natural covariance matrices. Its Python layer validates and expands declarations and labels results. Ordinary pedigree and joint random-regression solves use greml’s public compiled-MME facade; ssSNPBLUP uses the existing packed PCG route. Numerical model evaluation remains native. This standalone interface exposes supplied-covariance solving and deterministic sparse-direct REML for unstructured or factor-analytic genetic covariance and scalar/separable serial residual parameters; R interfaces remain unchanged. Whole blocks or named optimizer coordinates can be held fixed, with native free-score convergence and likelihood acceptance. Wide phenotype rows and long ID/trait/value rows lower to the same residual occasions. The frontend groups long rows by explicit record ID, validates shared mapped covariates and orders observed scalars by declared trait order. Native sparse incidence, covariance submatrices and solvers are unchanged; source tables and normalized observation rows are streamed through temporary SQLite storage, while pedigree/effect-level indexes remain in memory. Explicit correlated channels can bind different individual-ID columns while sharing one relationship and joint channel covariance. Python lowers these bindings, including weighted multiple-individual contributions per channel, to existing sparse record references; the packed native adapter uses all-record prediction preparation and retains its bounded record cache. Owner-filtered marker-weight training remains a separate strict contract. Selected-channel outputs distinguish individual genetic coefficients from record contributions evaluated at each channel’s mapped individual. Selected ordinary packed genetic uncertainty lowers explicit individual/trait requests to native target solves. greml reuses the fitted conditional PCG operator/preconditioner and its full or ancestral factor, and composes existing marker/H prior covariance actions. Python validates requests and labels results; it does not invert mixed-model equations. Every target retains transformed and original-equation checks. Stochastic requests compose gsolve’s existing variance-reduced diagonal estimator with that PCG action, using explicit sketch/probe/solve budgets and independent prior/posterior streams. Requests can select either posterior-PEV or information-covariance probing. The latter subtracts the same posterior action from the reported prior, then reconstructs PEV. Genotyped-only targets use G directly with batched trait products, avoiding extra pedigree solves; mixed targets use the existing H action. Diagnostics separate posterior query time from additional information-prior actions. Predictions and uncertainty have separate acceptance fields: finite imprecise candidates remain labelled unqualified, while numerical failures publish no uncertainty estimates. Regression, group and metafounder uncertainty remain outside this YAML adapter. Uncertainty CSV validation and relabelling stream rows directly to output, avoiding additional whole-population Python row/label copies. Stochastic PEV can isolate requested traits in residual probes. Greml maps reporting coordinates to colors and reuses the fitted PCG action; gsolve owns the generic masked estimator. All masked queries count toward the solve cap, while MCSE uses independent complete trait groups. The sketch and statistical model are unchanged. YAML exposes this opt-in choice as probing: trait. The compiled adapter also accepts labelled external relationship covariances, fixed interactions, composite identity-effect groups, Legendre/Chebyshev bases, grouped trait residual covariances and scalar/separable multivariate AR1/exponential residuals. It lowers to installed public greml relationships and residual components; Python does not fit the model or implement a new numerical solver. Dense relationship preparation and long serial blocks have separate scalability limits. Native compact format 7 is checked before result creation. Typed random-effect covariance declarations can supply FA loadings and specific variances. The frontend validates shape/order and materializes the small implied channel covariance for input inspection. Compiled prediction uses greml’s public FA random-group/natural-state APIs. Packed preparation validates the native FA layout and the implied covariance/used genetic precision, then reuses its existing channel matrix and PCG solver. Environment labels can select channels through existing long-format response bindings. No population incidence or new solver is introduced; FA does not reduce the number of genetic coefficients. Native identification and derivative work retain their scaling limits. FA estimation and uncertainty remain outside this YAML prediction integration. Univariate packed serial residuals instead lower group/position metadata and natural variance/correlation or rate to greml’s exact Markov-innovation record adapter. It transforms complete sparse design rows and responses once, with actual observed gaps and O(records) metadata, retaining scalar prepared blocks and the existing all-record prediction/cache path. It does not construct dense series matrices, alter the genetic model or add a PEV precision method. Multivariate separable serial covariance uses greml’s q-state conditional innovations preparation, including intermittent missing traits. Public gsolve eigensystems whiten the small observed-trait covariance; response and complete sparse incidence are transformed together and cached with identity residual precision. Compiled prediction consumes the same native cache. Metadata scales with observations and traits; sparse filter design history can grow with distinct coefficient columns in an incomplete series. This introduces no dense population/time covariance or fixed application RAM ceiling. Serial covariance estimation and industrial throughput qualification remain separate. Supplied-genomic ssGBLUP composes public greml genomic covariance/APY constructors with SingleStepPrecisionOperator and compiled PCG. The adapter passes explicit backend/thread and unlimited diagonal-workspace allowances, and retains the native exact marginal pedigree treatment. APY accepts supplied blocks or raw PLINK genotypes with ordered core IDs. The latter reuses native tiled BED preparation, a retained core factor and paired file-backed covariance/precision actions. Implicit H accepts those operators without constructing genomic precision CSC. Python computes means one marker at a time; genomic preparation RAM is O(core^2) plus bounded tiles and linear metadata, while the cross block occupies O(noncore*core) disk. No application RAM ceiling is imposed. Compiled ssGBLUP prediction omits unused H covariance metadata by default, preserving exact precision diagonals, determinants and PCG preconditioning. Complete metadata is an explicit setup option; native constructors retain complete metadata by default. Industrial APY preparation, precision-diagonal and prediction scaling remain unqualified. Compiled YAML models can supply metafounder Gamma and unknown-parent assignments through public native Pedigree. Explicit random genetic-group means reuse PedigreeGroupProportions; the example adapter materializes F once and expands compiled incidence, then reports individual F-gamma contributions. These are distinct models with independent dense-reference checks. The packed adapter composes GeneticGroupDataOperator with cached original records and an independent repeated channel covariance; inheritance and reporting remain implicit pedigree sweeps. It prepares exact group diagonals once with operator probes, without an n-by-g matrix or expanded record incidence. Supplied metafounders use full Qnn or an exact ancestral factor retaining all latent bases and their supplied correlated precision block. All models use the original precision actions; metafounder factors additionally apply an exact diagonal congruence with consistent solve maps for numerical balance. Group models also support ordinary ancestral sweeps. In metafounder models, only ordinary non-ancestral individuals enter the peripheral sweeps; all latent coefficients remain available. Combined base/group models are rejected. Compiled group incidence setup still has O(n times groups) storage. No new numerical solver, R control or fixed RAM ceiling is introduced. This schema is the sole standalone model interface; simulator output uses the same input/effect/covariance declarations and execution settings. The frontend streams CSV/TSV rows into a disposable SQLite snapshot, including long-input occasion grouping, residual bindings and record prediction grids. Prepared prediction targets are checksum-protected JSONL files read one row at a time during reporting. CLI preparation closes and removes its own temporary snapshot; optional execution.preparation.directory selects existing scratch storage. Pedigree and effect-level indexes, PLINK metadata and supplied dense matrices remain in memory. This reduces retained Python row storage without changing native equations or imposing a fixed application RAM ceiling.

Generic matrix-free reuse is owned by greml: adaptive trace-prefix extension and opt-in bounded warm/cache controls, with experimental solution recycling and residual trace estimation. gsolve owns the corresponding RHS-relative independent-PCG, supplied-product coarse correction and orthogonal projection primitives. These native options add no R control names; packed routes retain their existing caches.

Native library	Capability	Immediate boundary	Standalone R package
gbits	Packed genotype storage, PLINK BED access, selections, transformations and genotype products	Does not own genomic resource metadata, general solvers or statistical models	No; used through gbase and root gsuite
gmat	Ordered genotype panels, matrix and sparse-LD resources, LD algebra and LD scores	Delegates packed BED decoding to gbits and does not fit downstream statistical models	No; task-neutral access is through gbase
gsolve	General operators, factorizations, eigensystems and iterative or direct numerical methods	Contains no genotype, likelihood or mixed-model semantics	Deferred; current solver R interfaces use the development integration, with regional eigensystems also used by gbase
greml	Mixed-model construction, ML/REML inference, prediction and uncertainty	Uses gsolve numerical primitives and does not own genotype formats	Deferred; current mixed-model R interfaces use the development integration
glma	Linear, logistic and LOCO association plus LD-aware marker-set evidence	Association evidence belongs here; enrichment belongs to gsea	Yes: glma
gbayes	Bayesian regression, fixed-variance single-effect summary inference, parameter learning and posterior calculations	Does not own fine-mapping interpretation or score application	Yes: gbayes
gscore	Scoring-specific transformations and efficient score application	Model fitting remains with association or Bayesian model owners	Yes: gscore
gmap	Regional fine-mapping, posterior-set interpretation and colocalisation	Reuses gbayes for Bayesian fitting	Yes: gmap
gcorr	LD-score and regional correlation methods, covariance models and MR	The gcorr(Glist, stat, method=...) workflow supplies common setup for ordinary LDSC, GNOVA and an illustrative SumHer baseline; HESS reuses stored eigen-region definitions and reference samples, projects reference-oriented Z/N arrays region by region, defaults to no sample overlap and returns compact moments. The ldsc() workflow accepts direct named Z/N arrays or checked single- or multi-trait statistics and Glist-owned LD, with stratified or grouped set models. LDSC, partitioned LDSC, GNOVA and SumHer retain marker identity in R and use numeric native arrays with integer jackknife blocks and compact results. Set names are resolved once to integer rows; disjoint grouped scores retain one membership and one value per marker through native fitting; compact multi-trait BED projections use physical marker/sample indices without chromosome-wide metadata copies for joint HESS without persistent LD/eigenvectors. The R adapter selects an automatic rank budget, records explicit genomic-control scaling and reports inference blockers. Native gcorr owns joint correction, the original HESS software variance systems and genome correlation jackknife, and the signed moment/strict conditional alternatives; compact R fits default to software uncertainty. Colocalisation belongs to gmap	Yes: gcorr
gsea	Gene-set enrichment, genomic-feature regression and chromosome-held-out gene ranking	Reuses glma marker-set evidence and optional gbayes pathway models	Yes: gsea

gbase is the standalone shared R package for task-neutral genotype, summary statistics and LD preparation. Its pure R getChromosomeSets() helper extracts disjoint chromosome marker sets from Glist metadata without opening genotype or LD files. gsim is the standalone R package for simulation. The root gsuite package contains the mixed-model and solver integration and the suite-level version and documentation interfaces.

The capability overview lists the analyses available through each package. The package-boundary guide records the dependency rules.

Scalar individual-level Bayesian scheduling and realized genetic-variance decomposition remain owned by native gbayes. The standalone R adapter exposes the existing controls, variance traces and scheduler continuation state; gact owns the worked simulation/prediction example. See the Bayesian guide. The scalar BED adapter summarizes VG, VLE, VLD and draw-wise heritability in fit$posterior using native posterior-summary routines and original chain draws.

Summary preparation accepts ordinary data frames, marker-by-trait matrices and gact Z/N lists. checkStat(..., ldcheck=TRUE, impute=TRUE) harmonizes alleles, checks reference QC and LD consistency, masks rejected scores, then imputes supported missing Z-scores. impute() runs prediction alone. Gbase owns the R contract and thin native orchestration: gbits counts packed genotypes, gmat reads complete regional standardized dosage, and gsolve supplies dense products, Cholesky factors and verified eigensystems. The adapter retains bounded local LD and compact score diagnostics; it saves no genotype matrices or LD files. Ancestry references are disjoint integer sample selections on the same Glist. Explicit covariance mixtures cover independent standardized meta-analysis scores with constant coefficients; they do not recover unknown study LD.

Both standalone LOCO methods use glma’s joint full-panel controller and gbayes’s batched prediction engine. infinitesimal_loco fixes one Gaussian/ridge prior with equal variance on empirically standardized markers; prior_selected_loco compares ridge/elastic priors. Blocked RHE and a shared Monte Carlo grid estimate global variances, which remain fixed for chromosome refits. glma owns the residual-offset scan and structure calibration, gbayes owns prediction fitting and Gram reuse, and gsolve owns matrix products. The standalone bridge no longer calls the slow chromosome-specific variance-ratio searches. See the variance procedure.

Named disjoint marker sets define separate genetic components in either standalone LOCO method. glma validates the partition, adds the unassigned background group and composes greml’s existing multi-component RHE estimator with the joint engine. RHE supplies a variance profile; the scalar Monte Carlo grid refines its total. Global component normalizations and parameters remain fixed during chromosome exclusions. No native gbayes or gbase interface change is required, and calls without sets retain their existing behavior.

Shared genomic resources

Glist is the serializable contract connecting data preparation with analysis. gbase::gprep() validates a PLINK panel and can attach sparse LD computed by gmat from gbits-backed genotypes. The same Glist can then be passed to analysis packages without copying the genotype or LD data into R.

gprep(task = "eigen", sets = regionSets) also prepares full regional LD eigenvalues directly from BED. gmat supplies centred unit-norm columns through gbits, gsolve supplies dense products and eigenvalues-only decomposition, and gbase attaches the named spectra and sample identities to Glist$eigen. The smaller sample-space Gram matrix is used when markers outnumber samples. This route is independent of sparse-LD cutoffs and does not fit a gene test. Analysis packages still require an explicit consumer interface for this resource.

flowchart LR
    bed["PLINK BED / BIM / FAM"] --> gprep["gbase::gprep()"]
    gprep --> glist["Glist"]
    glist --> glma[glma]
    glist --> gbayes[gbayes]
    glist --> gscore[gscore]
    glist --> gmap[gmap]
    glist --> gcorr[gcorr]
    glist --> gsea[gsea]

The Glist fields ldfiles, idsLD and rsidsLD identify the stored LD resource, its reference samples and its marker order. validate_Glist() checks the lightweight descriptor used by ordinary calls. verify_ld() performs the full sparse-LD integrity scan when a resource crosses a trust boundary.

Native consumers open one gmat LD reader per high-level call and reuse it across regions, traits, chains or marker sets. Regional matrices, independent blocks and eigensystems remain tied to the same Glist identity. This avoids repeated payload checks and unnecessary dense copies.

Analysis flow

Each R package validates task-specific inputs, calls its native implementation and returns labelled R results. Shared preparation does not blur statistical ownership:

• glma fits association models and produces marker- or set-level evidence;
• gbayes fits Bayesian regression models and produces posterior quantities;
• gscore applies prepared weights, including Glist-resolved marker-set scores, while model fitting remains with the model owner;
• gmap performs regional fine-mapping, posterior-set and Bayesian-FDR interpretation, and colocalisation;
• gcorr fits summary-statistic correlation, covariance and MR models; and
• gsea performs enrichment or PoPS-style gene ranking after gene or feature evidence has been prepared.

A connected association-to-enrichment workflow therefore follows:

flowchart LR
    stats["Checked association statistics"] --> vegas["glma::vegas()"]
    ld["Glist sparse LD or regional BED spectra"] --> vegas
    vegas --> genes["Gene-level evidence"]
    genes --> enrichment["gsea::gsea()"]

The VEGAS step processes the supplied marker sets directly. One sparse-LD reader prepares eigenvalue spectra in bounded parallel batches; the resulting ordinary R object can be reused across traits and sessions without reopening LD or repeating decompositions. Cross-gene covariance, when required by a downstream enrichment model, is a separate explicit input. Alternatively, glma reuses the full within-region spectra stored by gbase::gprep(task="eigen"); constant weights require no new decomposition. Nonuniform weights continue to use the sparse-LD route. gbase owns preparation and identity validation; glma owns the statistic and tail evaluation. VEGAS selects checked summary rows by marker ID and verifies their alleles; aligned database statistics need no reference reconstruction or manual sorting.

Method-specific controls, outputs and qualification limits belong in the method catalogue and the linked workflow guides, rather than on this architecture page.

Mixed models and numerical methods

The root gsuite package exposes the mixed-model integration. R formulas or explicit covariance components are prepared for greml, which owns model semantics, estimation, prediction and uncertainty. greml delegates general factorization and solving to gsolve. gbits supplies packed genotype operations, and gmat owns reusable genotype-panel and matrix resources.

Native greml also provides opt-in generic PCG identities and a separate common-scale profiled REML entry point for V = g K + e I. These remain native C++ controls; the existing R dispatch and optimizer defaults are unchanged. The changed native layouts require matching full ABI47 and core/adapter SOVERSION3 consumers. Focused numerical verification does not establish large-panel throughput or population calibration.

flowchart LR
    input["Formula or covariance components"] --> gsuite["gsuite R interface"]
    gsuite --> greml["greml: model and inference"]
    greml --> gsolve["gsolve: numerical methods"]
    gbits["gbits: packed genotypes"] --> gsuite
    gmat["gmat: genomic resources"] --> gsuite
    greml --> result["Labelled fit, prediction and diagnostics"]

The mixed-model guide explains this boundary, and the model catalogue records the supported model and backend combinations.

Dependency rules

• Task packages may depend on gbase; gbase does not depend on a task package.
• Task packages do not import the root gsuite package.
• A task package may use another task package only for an explicit scientific workflow, such as the gsea re-export of glma::vegas().
• Packed genotype decoding and algebra belong to gbits; panel identity, matrices and LD resources belong to gmat.
• General numerical methods belong to gsolve; mixed-model semantics belong to greml; each other statistical method belongs to its task library.
• New dependencies must follow this direction and must not introduce cycles.

gsim is separate from the analysis dependency graph. It creates ordinary R data that can be passed into examples and analysis packages.

Regional dense MCMC follows the same ownership boundaries: gbase maps marker sets and exposes full BED LD through installed gmat decoding and R’s BLAS; native gmap validates and binds the region, then reuses native gbayes sampling. Familiar argument adapters remain in the gmap and gbayes R packages. Native gmap owns qgg-compatible marginal-PIP grouping; it does not require qgg at runtime. gact imports gbase for shared marker-set mapping and explicitly installs the standalone analysis packages through install_gsuite(). Familiar QC, LD, MCMC and scoring arguments remain R adapters over their existing native owners. Bayesian PoPS splits chromosome folds and assembles predictions in the gsea R package; native gsea/gbayes owns nuisance projection, cross-products and sampling. Fold results retain posterior diagnostics without repeated training matrices. Standalone gbayes binds verified Glist LD blocks to native factorized BayesC/BayesR for mcmc-eigen, independently of the annotated SBayesRC model. Native gbayes owns retained joint-effect genetic variance and chain diagnostics; the R task packages arrange these into familiar post and conv tables. When the genetic reference is the fitted operator, accumulation reuses the cached residual quadratic. Both task R interfaces pass optional scalar-summary LD-swap controls to the existing native gbayes sampler; no sampler is duplicated in the R layer or native gmap. Native gbayes prepares the LD-neighbour graph once per fit and shares it across chains. Sparse-LD operators scan existing stored correlations; factorized operators derive neighbours from their fitted cross-products. The R packages expose pooled and per-chain proposal counts. Probability defaults to zero, preserving the existing sampler and timing evidence.

Standalone gbayes::gbayes(data=..., Glist=..., method="bayesc") and BayesR accept automatic phenotype-scaled BED priors with parameter learning and retained diagnostics. Scalar BED responses are centred on their original units; see the defaults and controls for initialization, explicit-prior compatibility and convergence limits.

Fixed-covariance compiled prediction selects value-only numerical assembly: base MME values, RHS and residual precision caches are unchanged, while global first/second covariance-derivative matrices and RHS arrays are absent. The full derivative assembly remains the default for estimation. This avoids O(parameter_count^2 * coefficient_pattern_nonzeros) storage that prediction previously allocated without using it; block-Jacobi inputs are unchanged.

The extended YAML adapters preserve native ownership. Fixed inherited group contrasts and independent group means compose public pedigree proportion operators with compiled incidence; combined metafounder/group models retain a separate ordinary inheritance base and metafounder covariance base. Explicit phantom group-parent codes are absorbed into missing-parent assignments, not biological parent nodes. Packed Gamma pseudo-EM runs through the existing native estimator on an explicitly declared absolute marker base; the verified prepared estimate rebuilds the prediction pedigree. No Gamma uncertainty is implied. PEV precision development has resumed with bounded independent-pilot allocation.

Sparse external natural covariance is streamed from an ID-labelled lower triangle. Greml retains its gsolve sparse factor as an inverse-action operator, with exact precision diagonal metadata; no dense inverse is formed. Supplied nonseparable trait-by-visit residual covariance lowers to the existing native patterned residual family. Within-unit grids remain dense and intended for moderate sizes. Format-8 transport is checked before fitting. Packed fixed and combined group models prepare ordinary inheritance proportions once, including reference-coded fixed contrasts as record columns. Their O(ordinary individuals × groups) proportion arrays are explicit preparation resources. No repeated pedigree factorization or dense genomic matrix is added. Independent-pilot PEV allocation chooses a fixed final sample count inside the posterior solve cap; pilot values are discarded. Industrial qualification remains separate.

Packed ssSNPBLUP REML now lowers to a native greml EM session with refreshed record precision/RHS and preconditioners, sharing the exact pedigree factor and packed genotype provider across covariance states. Natural genetic/residual starts, whole-block fixing, exact or bounded stochastic traces, score MCSE and dual equation-residual gates are explicit. Optional start-to-final restricted likelihood uses bounded exact references or paired matrix-free SLQ with separate MCSE/depth-sensitivity status. Default fitting does no global determinant work. The YAML frontend separates strict shared validation, analysis/thread controls, model lowering, native execution and labelled estimation reports into modules; the generic gsuite run model.yml coordinator reuses those same modules for every supported route/task. Preparation and fitting run in a private temporary worker, with native/CSV results and failed-run diagnostics collected to the YAML output directory. Process-tree RSS sampling is separate from native peak RSS; execution and collection timings do not change numerical qualification. validate and describe inspect model inputs without native fitting. Existing R interfaces and the YAML schema remain unchanged. The optional GenomeDK binary producer collects that already built native closure and relocates copied library paths. Its installer uses a fresh minimal Conda runtime and admits the artifact only after exact references and backend agreement; it changes deployment, not model compilation or numerical ownership. Native sources/R/build tools are excluded from the recipient payload. This private binary workflow requires actual Linux qualification and retains the existing redistribution limits. The standalone user guide owns installation and execution instructions; the GenomeDK validation record reports the accepted binary installation and its evidence limits. There are no legacy syntax aliases. See packed REML for the initial model scope and qualification limits.