R/get_B_data_loader.R
prepare_B_training_data.RdStandardizes and validates user-provided GWAS summary statistics for use
in B estimation via get_B. This enhanced version includes
smart data conversions, study/PMID filtering, position-based duplicate
removal, and comprehensive processing to produce publication-ready datasets.
prepare_B_training_data(
data,
format = "auto",
trait_type = NULL,
column_mapping = NULL,
filter_include = NULL,
filter_exclude = NULL,
filter_pattern = TRUE,
qc_filters = list(min_sample_size = 500, max_pvalue = 1, min_maf = 0, remove_duplicates
= TRUE, duplicate_priority = "N", remove_na = TRUE, remove_na_cols = c("MAF", "P",
"N")),
convert_or_to_beta = TRUE,
extract_text_values = TRUE,
filter_autosomes = TRUE,
verbose = 1
)Either a file path (CSV/TSV/Excel) or a data.frame containing GWAS summary statistics. Supported file formats: .csv, .tsv, .txt, .xlsx, .xls
Character string specifying the input format: "auto" (default), "gwas_catalog", "csv", "excel", or "dataframe". When "auto", the function auto-detects the format from file extension
Character string specifying trait type: "binary", "continuous", or NULL (default). When NULL, the function performs per-row auto-detection based on OR/BETA columns. When specified, serves as a default that can be overridden by row-level indicators. A TRAIT_TYPE column will be added to the output data with per-variant trait type assignments.
Named list for custom column name mapping
Named list for inclusion filters. Each element should be
a column name with a vector of values to include. Only rows matching at least
one value per specified column will be kept. Takes precedence over filter_exclude.
Example: list(STUDY = c("Study1", "Study2"), CHR = c(1, 2, 3))
Named list for exclusion filters. Each element should be
a column name with a vector of values to exclude. Rows matching any value in
any specified column will be removed. Ignored for columns specified in filter_include.
Example: list(`Reported trait` = c("trait1", "trait2"), CHR = "X")
Logical. If TRUE (default), uses pattern matching (grepl) for character columns. If FALSE, uses exact matching.
List of quality control parameters:
min_sample_size: Minimum N (default: 500)
max_pvalue: Maximum P-value (default: 1)
min_maf: Minimum MAF (default: 0)
remove_duplicates: Remove duplicates (default: TRUE)
duplicate_priority: Column for duplicate resolution (default: "N")
remove_na: Remove rows with NA in required columns (default: TRUE)
remove_na_cols: Columns that must have non-NA values when remove_na=TRUE (default: c("MAF", "P", "N")). Users can add additional required columns as needed, e.g., c("MAF", "P", "N", "BETA")
Logical. If TRUE (default), converts OR to log(OR) for binary traits when BETA is not available
Logical. If TRUE (default), attempts to extract numeric values from text fields (P-values, sample sizes). WARNING: Text extraction for sample sizes uses heuristics to detect when multiple groups should be summed (e.g., "men and women", "cases and controls"). While robust, this is NOT 100\ the original study descriptions, especially for publication-quality analyses.
Logical. If TRUE (default), converts CHR to numeric and filters to keep only autosomal chromosomes (CHR 1-22), removing sex chromosomes (X, Y), mitochondrial chromosome (MT), and variants with invalid CHR values. If FALSE, leaves CHR column completely unchanged (no conversion, no filtering), retaining all variants regardless of chromosome. Set to FALSE to preserve original CHR values unchanged.
Integer controlling verbosity: 0=silent, 1=messages (default), 2=detailed messages
An S3 object of class glow_training_data containing:
data: data.frame with standardized columns in standard order: rsID, CHR, POS, MAF, P, N, BETA, TRAIT_TYPE, TRAIT, STUDY, PMID, FIRST_AUTHOR, ...
metadata: List with comprehensive processing information including conversions applied, filters used, QC summary, column mappings, and mapping sources (user-specified vs auto-detected)
Enhanced Processing Pipeline:
The function performs the following steps:
Load data from file or data.frame
Standardize column names
Convert columns to numeric types
Smart Conversions:
Extract sample size (N) from text descriptions
Extract P-values from text (handles "5 x 10^-8", etc.)
Preserve PVALUE_MLOG as P_mlog10 (not converted to P)
Convert OR to log(OR) for binary traits
MAF harmonization (MAF > 0.5 becomes 1 - MAF)
Add TRAIT_TYPE column
Filter to autosomes (CHR 1-22) when filter_autosomes=TRUE; leave CHR unchanged when FALSE
Filter by studies and/or PMIDs (optional)
Apply QC filters (enhanced with position-based duplicate removal)
Reorder columns to standard format
Validate data
CRITICAL WARNINGS - Smart Data Processing:
This function includes several "smart" automatic data processing features for convenience. While robust, these are heuristic approaches with limitations. Users are responsible for verifying that automatic operations are appropriate for their specific data.
1. Sample Size Extraction (from text):
What it does: Extracts numeric sample sizes from text descriptions like "76,067 women, 66,420 men" and intelligently sums or takes max
Limitations: Pattern-based heuristic; may misinterpret complex text
Failure scenarios: Non-standard formats, ambiguous descriptions, multiple cohorts with unclear relationships
User action: ALWAYS verify extracted N values against source data; for critical analyses, manually prepare clean N column
2. P-value Text Extraction:
What it does: Parses p-values from text like "5 x 10^-8", "< 0.001"
Limitations: Pattern-based; handles common formats but not all variations
Failure scenarios: Non-standard notation, unusual characters, ambiguous inequality symbols
User action: Check for NA values after extraction; verify converted p-values are reasonable; for non-standard formats, prepare numeric P column
3. PVALUE_MLOG Preservation:
What it does: Preserves PVALUE_MLOG column as P_mlog10 (does NOT convert to P)
Purpose: Retains -log10(p) values for better numerical stability with extreme p-values
User action: If you need regular p-values, use get_B() with training_P_mlog10 parameter which handles conversion internally with proper numerical precision
4. OR to BETA Conversion:
What it does: Converts odds ratios to log odds ratios: BETA = log(OR)
Assumptions: Column truly contains OR (not RR, HR, etc.); correct allele direction; consistent units
Failure scenarios: Column contains beta values mislabeled as OR; percentage increases stored incorrectly; mixed effect measures
User action: VERIFY OR column contains true odds ratios; inspect values (typically 0.5-2.0); check effect directions are consistent
5. CHR Handling (when filter_autosomes = TRUE):
What it does: Converts chromosome names to numeric and filters to autosomes (1-22)
When filter_autosomes = FALSE: Leaves CHR column completely unchanged (no conversion, no filtering)
Limitations: Non-standard naming may not be recognized; assumes human chromosomes; sex/MT chromosomes removed when filter_autosomes=TRUE
User action: Set filter_autosomes=FALSE to retain all chromosomes unchanged; for non-human data, manually prepare CHR column before processing
6. MAF Harmonization:
What it does: Automatically converts MAF > 0.5 to 1 - MAF
Assumptions: Column represents minor allele frequency (not major allele frequency); MAF definition is standard
Failure scenarios: Column is actually effect allele frequency (not necessarily minor); allele frequency for specific allele regardless of whether it's minor
User action: VERIFY that automatic conversion is appropriate; check if your data uses MAF or EAF convention; ensure consistency
General Recommendations for Critical Analyses:
Exploratory work: Use automatic features with verbose=2 to see warnings
Publication-quality: Manually prepare clean columns (N, P, BETA, CHR, MAF)
Non-standard data: Pre-process data manually before calling this function
Always: Inspect result$metadata to see what conversions were applied
Always: Check result$data to verify conversions are sensible
Document: Record which automatic features were used and verified
CRITICAL WARNING - Sample Size Extraction:
When extract_text_values = TRUE (default), the function attempts to
extract sample sizes from text descriptions. This uses intelligent heuristics
to detect when multiple groups should be summed:
Summed: "76,067 women, 66,420 men" -> 142,487
Summed: "5,000 cases + 5,000 controls" -> 10,000
Single: "394,929 individuals" -> 394,929
USER RESPONSIBILITY:
This extraction is heuristic and may not be 100\
ALWAYS verify extracted sample sizes against original study descriptions
Check warnings produced during extraction
For critical analyses, manually specify sample sizes in a clean N column
The function will warn you when it makes decisions about summing
To inspect extracted values, check result$data$N after processing and
compare with original text in your input data.
TRAIT_TYPE Column (Sophisticated Per-Row Detection):
The output always includes a TRAIT_TYPE column with per-variant trait type assignment using multiple evidence sources. The function uses a conservative approach, defaulting to "unknown" when confidence is low.
Detection Signals (in decreasing confidence order):
OR column with value (excluding ambiguous names like "OR.or.BETA") -> binary
Case Size + Control Size both present -> binary
Disease name in trait column (e.g., "sclerosis", "diabetes") -> binary
Quantitative trait name (e.g., "BMI", "bone mineral density") -> continuous
"Susceptibility" keyword in annotations -> binary
BETA present with no binary indicators -> continuous (low confidence)
Important Notes:
This detection is robust but NOT error-proof. Manual verification is recommended for critical analyses.
The function prioritizes avoiding false classifications - when uncertain, it assigns "unknown" rather than guessing.
Trait type is less critical when using P-value-based (vs. BETA-based) methods for B estimation.
Users can override auto-detection by explicitly specifying trait_type.
This enables correct handling of mixed-trait datasets (e.g., GWAS Catalog) where different variants come from different trait types.
Duplicate Removal:
Duplicates are removed by both rsID and genomic position (CHR:POS). When duplicates exist, the variant with the largest value in the priority column (default: N = sample size) is kept.
Flexible Filtering System:
The filtering system allows you to filter on ANY column in your data:
filter_include: Keep only rows matching specified values
filter_exclude: Remove rows matching specified values
filter_pattern: Use pattern matching (TRUE) or exact matching (FALSE)
qc_filters$remove_na_cols: Specify columns that must have non-NA values
NA filtering is controlled by qc_filters$remove_na and qc_filters$remove_na_cols.
By default, only MAF, P, and N are required to be non-NA. You can add additional
required columns (e.g., BETA) by specifying them in qc_filters$remove_na_cols.
This eliminates the need for pre-filtering and works with any data structure. See Examples for common use cases (ALS, BMD, excluding overlapping studies).
get_B for B estimation using prepared data
if (FALSE) { # \dontrun{
# Example 1: Standard analysis (default - autosomes only)
result <- prepare_B_training_data(
data = "gwas.csv",
filter_autosomes = TRUE # Default
)
# Example 2: Keep all chromosomes (legacy behavior)
result <- prepare_B_training_data(
data = "gwas.csv",
filter_autosomes = FALSE # Retain X, Y, MT, invalid CHR
)
# Check which columns were renamed
print(result$metadata$column_mappings)
print(result$metadata$mapping_sources)
# Example 3: ALS filtering (exclude specific traits and annotations)
als_data <- prepare_B_training_data(
data = "ALS-known-SNPs.xlsx",
trait_type = "binary",
column_mapping = list(BETA = "Beta_numeric"),
filter_exclude = list(
`Reported trait` = c(
"Amyotrophic lateral sclerosis (C9orf72 mutation interaction)",
"Amyotrophic lateral sclerosis (age of onset)",
"Survival in sporadic amyotrophic lateral sclerosis"
),
`P-value annotation` = c("(age of onset)", "(survival)", "(site of onset)"),
chr = "X"
),
qc_filters = list(
remove_na = TRUE,
remove_na_cols = c("MAF", "P", "N", "BETA") # BETA also required for ALS
),
verbose = 2
)
# Example 4: BMD filtering (include specific studies only)
bmd_data <- prepare_B_training_data(
data = "bmd_gwas.xlsx",
trait_type = "continuous",
filter_include = list(
STUDY = c("Atlas", "153 new loci", "613 new loci")
),
filter_autosomes = FALSE, # Keep all chromosomes (legacy behavior)
qc_filters = list(
remove_na = TRUE,
remove_na_cols = c("MAF", "P", "N", "BETA") # BETA also required for BMD
),
verbose = 2
)
# Example 5: Flexible filtering on any column
gwas_data <- prepare_B_training_data(
data = "my_gwas.csv",
trait_type = "binary",
filter_exclude = list(
PMID = c("12345678", "87654321"), # Exclude overlapping studies
Population = "European" # Exclude specific population
),
filter_include = list(
CHR = 1:22 # Autosomes only
),
filter_pattern = TRUE, # Use pattern matching
verbose = 2
)
} # }