File structure checks - #35
Conversation
Added ability to filter BV-BRC genomes based on genome stat metadata, including CheckM completeness and CheckM contamination as defaults. Also enabled optional checking of genome length, CDS count, and GC content based on deviations from median distribution per bug. Updated genome downloader functions to use ftpes_download_one instead of old ftp_download_one, and moved a dot function helper to inside the ftpes_download_one loop to avoid issues with BiocParallel workers not understanding what that helper is.
Uncommenting a helper that was still needed for several calls. Fixing stuff I broke myself!
Updating to remove BiocParallel implementation that caused bugs. Patching back to prior Future implementation instead. CLI version of genome downloading appears to have issues that may be separate and requires further troubleshooting.
data_processing.R runDataProcessing() missing(ref_file_path) check ignores if the default path is used. Updated to accept the default path if a user doesn't specify rather than erroring at the end of the processing steps.
Updated default workers to 8 for CLI and FTP downloads, made two-try FTP download option the default, added a helper to purge broken file sets (i.e., if .faa is missing for an AccID but .fna and .gff download successfully).
ftpes --> ftps is_complete --> .is_complete
… consolidate on furrr
Cleaning up parallel implementation in data_curation.R, standardizing with furrr and dropping old duplicated within-loop helper function that was needed for earlier BiocParallel implementation. Standardized values for contamination and completeness across functions. Added a parameter in prepareGenomes to allow users to set number of workers. Reduced console clutter for final "out" print.
…to qc_filtering
…to qc_filtering
In data_curation.R, adding ability to export genome stats and QC metadata into human readable CSVs. `load_tables` allows you to run "example <- exportTables("data/Staphylococcus_aureus/Sau.duckdb", load_tables=TRUE)" and import the tables into R so you can play around with them directly. By default, it just saves the tables as exported CSVs. In data_processing.R, added a similar exporter that allows you to save out all the feature data and other DuckDB-connected tables, with the ability to output sequences too, and append AST phenotype data to the feature tables. Tables can be saved out as csv (default), tsv, Parquet, and xlsx. Added `writexl` to Suggests in DESCRIPTION. Fixed some reversions where I broke things in the PR this branch is based on. Will add those same fixes to that PR momentarily to keep things aligned.
Ran data processing workflow for Campylobacter coli. This can be unzipped and used as the test data for exportProcessedData() rather than having to run it all beforehand. Data curation step still requires that prepareGenomes() be run for a test bug, and Staphylococcus argenteus is a good use case for this.
Clarified a few names based on code review feedback.
An addendum to the problem.Positron:
RStudio:
|
My initial testing with Claude looking for the source of the inconsistency issueTesting 
The point here is that the taxon/genome counts are not reproducible run-to-run or machine-to-machine unless you force a full cache refresh first. Two fresh sessions, same species query, no code changes — and one returns 872 targets/3 post-filter genomes, the other 926/27, with no warning that the local BV-BRC cache is partially stale. Only after This is a distinct, arguably more fundamental bug than the Panaroo hang itself: whatever cache-staleness logic decides how many taxon IDs/genomes to pull is silently non-deterministic depending on what's already sitting in the local cache, and there's no signal to the user that their result set is incomplete. It also matters for the Panaroo investigation directly — the "39 genomes" from the original PR description were almost certainly generated against a stale/partial cache, so the actual offending-genome subset may not be stable across your own re-runs either. |
Fix unreachable BV-BRC cache staleness check
.ensure_bvbrc_cache() only called .updateBVBRCdata() when the cache file/table was completely absent, so its max_age_days staleness check never ran once a cache existed. bvbrc_bac_data is one global table covering all of Bacteria, so this meant results silently depended on whenever *any* prior species query happened to build the cache, with no reproducibility across machines/sessions regardless of age.
Root-caused this from divergent prepareGenomes("Staphylococcus epidermidis") genome counts between two environments (872 vs 926 targets) that only converged (2211/2212) after a manual .updateBVBRCdata(max_age_days = 0).
Now .ensure_bvbrc_cache() always delegates to .updateBVBRCdata(),
which already has the correct fresh-vs-stale logic. max_age_days is
now a parameter on .ensure_bvbrc_cache() (default 30, unchanged) for
future tuning; left as-is since this is an internal function.
4 participants
The problem:
It has been a known issue that sometimes, pangenomes will fail to complete through
data_processing.R. This has prevented Klebsiella from running in prior attempts, and Abhirupa isolated the issue to a subset when the pangenome generation was run in batches and smaller pangenomes merged. Some genomes in one particular subset appeared to cause a failure, but we had no further diagnostic information. Because we have also seen inflation in feature matrix sizes as more genomes have been added, I originally sought to implement QC filtering to exclude contaminated or incomplete genomes. However, in my current testing using these QC filters, I have found another, smaller subset of genomes that do not complete in Panaroo.Details:
prepareGenomes(user_bacs = "Staphylococcus epidermidis", max_checkm_contam = 0)returns 39 genomes with AST data, a very small set. Panaroo fails to generate a pangenome after 1 hour of running. Logs stop printing when invalid genes are being removed from GFFs, a necessary step for non-Prokka GFFs. Despite this freeze, CPUs are locked at 400% use, indicating 4 cores are fully occupied during this entire time. Disk I/O is static, with no read and a constant small write, but no output is observed. RAM use is static and low. No hardware approaches its limits during the freeze. This occurs with our older Panaroo 1.5.3 container, and a much more recent 1.7.0 container, and after purging all Docker images and caches and rebuilding everything freshly, and when purging the package and installing everything freshly. Pangenomes run and complete in 5-10min for other bugs with 400 genomes or fewer.Approach:
In this PR, I am planning on subsetting this batch of 39 genomes down to identify an even smaller set that does not complete. Once I identify a few candidate GFFs that may be responsible, I will try to find atypical patterns in the way their gene headers are formatted that might lead to a CPU deadlock.