Runs one scan unit: builds the unit's regions and filter(s) with
build_scan_regions, runs glow_scan_chunk (once for
gene/window; once per category for coding), and writes
the per-unit flat results table plus the policy-gated sidecars, emitting log
lines through logger. A scan "unit" is one chromosome (gene/coding) or
one genome chunk (window). The thin per-type entry scripts resolve the
prepared inputs and call this with region_type set.
run_scan_unit(
gds_path,
null_model,
pi_models,
pi_features,
reference_medians,
b_func = NULL,
b_model = NULL,
region_type,
config,
results_path,
output_dir,
output_policy,
chr = NULL,
chunk = NULL,
unit_id = NULL,
staar_context = NULL,
use_spa = NULL,
z_scale = 1,
max_regions = NA_integer_,
progress_every = 0L,
logger = NULL,
n_cores = NULL,
verbose = 0,
debug_errors = FALSE
)Character. Path to the unit's GDS/aGDS file.
The resolved
prepared inputs (from prepare_scan_run), passed to the scan engine.
reference_medians is this unit's chromosome's annotation-median cache.
The B source (exactly one non-NULL), passed to the engine.
Character: "gene", "window",
"coding" or "noncoding" (the gene-centric non-coding masks,
see the section below).
An environment or named list of resolved config values. Used for
the region/filter build (see build_scan_regions) and the engine
knobs ld_threshold, mac_threshold, collapse_method,
output_format.
Character. Path the flat results table is written to.
Character. Directory under which sidecar folders
(evidence/, staar_detail/, result_objects/) are created.
Integer or character chromosome (required for
gene/coding; for window it is taken from chunk).
Default NULL.
A single-row genome-chunk data.frame (required for
window). Default NULL.
Character or integer used in the sidecar filenames and log
lines to identify the unit (e.g. the chr number or chunk id). Default
derived from chr / chunk$chunk_id.
A STAAR context or NULL (STAAR off). Default NULL.
Threaded to the engine (see
glow_scan_chunk). Defaults NULL / 1.
Integer or NA. If not NA, cap the regions to the first
max_regions (smoke-test; the --max-genes/--max-windows
parallel). Default NA.
Integer threaded to the engine. Default 0L.
A function(msg) for log lines, or NULL (silent). Default
NULL.
Integer or NULL (default). Forked workers for this unit. NULL
reads config$n_cores and falls back to 1. With more than one, the
regions are grouped into blocks by size (see the section "Scheduling over
the workers"), each block runs glow_scan_chunk in a forked
worker with its own GDS handle (parallel::mclapply, dynamic
dispatch), and the block outputs are merged and restored to the regions
table's order by label, so the written table equals the single-core one.
The progress callback is off inside the workers; the parent logs one
scheduler line per regions table and one line per finished block. Forking
is unavailable on Windows (a value above 1 warns and runs serially).
Memory scales with the workers, since each holds its own genotype and LD
matrices.
Integer threaded to the engine. Default 0.
Logical (default FALSE). Threaded to the engine and to
GLOWr::glow_region_test(): when TRUE, the message of an error row carries the
failing call and the stack of function names. Results unchanged.
Invisibly, a named list with results (the written flat table),
summary (a 1-row tally; the engine summary for gene/window, or a
per-category-stacked summary for coding), errors (a data.frame), and
results_path.
Compute order (coding). Categories are looped outer — one
glow_scan_chunk call per category over the gene table, with the
category's filter as the only difference — so each call is exactly a
gene-style call and coding reuses the scan engine unchanged. Each call's result
table is tagged with a category column and the tables are row-bound;
the combined table leads with label, chr, category (gene-first column
order) and is sorted (start, category) so the on-disk ROW order is
gene-first (each gene's categories together at its genomic position),
matching the gene-first aggregation output.
Region summary. region_summary is enabled for
region_type %in% c("gene") (gene and custom-gene-list scans both
dispatch as "gene") and disabled for window/coding, so the
gene flat table carries the 7 region-summary columns and the other schemas are
unchanged.
Writes. The flat results table is written to results_path (CSV
by default, or fst when config$output_format == "fst"). Policy-gated
sidecars (evidence, staar_detail, result_objects) are
written under output_dir into their own on-demand folders, only when
both the policy flag is on and the scan engine
produced the artifact (so no orphan/empty folders).
For region_type = "noncoding" the unit is a chromosome and
build_scan_regions returns one regions table per mask of
config$noncoding_categories (default the eight masks of
GLOWr::noncoding_masks()), built once from the chromosome's
annotations by GLOWr::define_regions_noncoding. Each mask's table is
run like a coding category (the label check, max_regions and the
scheduler apply per table; region_summary is off), the rows are
tagged with category, and the combined table leads with
label, chr, category and is sorted gene-first by the gene anchor of
noncoding_gene_anchor (a gene's masks together, because their
candidate spans differ), then by mask. A row's start, end and
midpoint describe the candidate span, which is wide for an enhancer
far from its gene. Two sidecars are written beside the flat table on every
completed unit, for the aggregation's completeness table: the error table as
<results_path stem>_errors.csv (zero rows when no cell errored, so a
rerun never leaves an earlier run's errors beside a clean table) and the
per-mask tallies (cells attempted, tested, skipped for an empty set or a
single rare variant, errored) as <results_path stem>_summary.csv.
The log carries one line per mask with the genes, the selected records and
the links.
For region_type = "gene" the unit's variant index is passed to
build_scan_regions, which tiles every gene with more GDS
records than config$gene_max_records into segments of
config$gene_segment_bp (see that function). The log states how many
genes were segmented and into how many segments.
With n_cores > 1 the regions of a table are not run in genomic order.
Heavy genes sit next to each other on the genome, and a block of consecutive
regions that holds several of them keeps one worker busy long after the
others have finished. Instead, every region's GDS record count \(n_i\) is
read from the unit's variant index (two binary searches per region, no
genotype read; the gene segmentation's n_records is reused where
present) and its cost is taken as \(c_i = n_i^2\), a heuristic that lets a
few heavy regions dominate the sums. With \(C = \sum_i c_i\) and
\(\tau = C / (8\,\mathrm{n\_cores})\), a region with \(c_i \ge \tau\)
forms a block of its own; the remaining regions, in decreasing cost, are
packed into blocks that close at 25 regions or when the next region would
push the summed cost above \(\tau\); and the blocks are dispatched in
decreasing summed cost, so the heaviest work starts first and the light
blocks fill the tail. When \(C = 0\) the regions keep their order in blocks
of 25. Each region is computed independently of its block, so the merged
rows are the serial ones; the merge puts the flat rows, the error rows and
the sidecar entries back in the order of the labels in the regions table.
This needs unique labels, which the function enforces at every
n_cores: a repeated label stops the run with the duplicates named,
because the sidecars and the error table are keyed by label and a duplicate
already loses information silently in the serial path. For the coding type
each category's pass over the gene table is scheduled this way, one category
after another. The log's scheduler line gives the number of blocks, how many
are a heavy region alone, the heaviest region and \(\tau\); each block line
gives the block's region count, its heaviest record count, its tally and
its wall time, so the reader sees where the unit's time went.
if (FALSE) { # \dontrun{
bundle <- prepare_scan_run(env, "runs/myrun/outputs", chroms = 22L)
run_scan_unit(
gds_path = "data/chr22.gds",
null_model = bundle$null_model, pi_models = bundle$pi_models,
pi_features = bundle$pi_features, reference_medians = meds22,
b_func = bundle$b_func, b_model = bundle$b_model,
region_type = "gene", chr = 22, config = env,
results_path = "runs/myrun/outputs/results/glow_chr22.csv",
output_dir = "runs/myrun/outputs",
output_policy = glow_output_policy(), logger = message)
} # }