Export metadata and feature data tables by epbrenner · Pull Request #34 · JRaviLab/amRdata

epbrenner · 2026-07-28T22:22:02Z

Testing:

For this PR, I have included bundled test data in /inst/extdata/Campy_testdata.zip. Extract this into a single directory of your choosing. This bundled data is not intended to the ultimate example data for final package release.

For data_curation.R:
Run
prepareGenomes("Staphylococcus argenteus")
then
exportTables("data/Staphylococcus_argenteus/Sar.duckdb")

For data_processing.R:
Unzip the Campy test data first, then run
exportProcessedData("inst/extdata/Campy_testdata/Cco_parquet.duckdb")

There are several thrilling parameters to fiddle around with.

What this does:

Adds the ability to export basic stats into human readable CSVs from data_curation.R workflow.
Adds the ability to export all the processed feature data into a variety of formats from data_processing.R workflow.

What's the point?

Makes amRdata more independently useful to people even if they don't run the full amR suite. For example, you could export feature matrices for custom modeling workflows outside of ours, or just get an idea of what distributions of different QC and AST labels are across different taxa.

Added ability to filter BV-BRC genomes based on genome stat metadata, including CheckM completeness and CheckM contamination as defaults. Also enabled optional checking of genome length, CDS count, and GC content based on deviations from median distribution per bug. Updated genome downloader functions to use ftpes_download_one instead of old ftp_download_one, and moved a dot function helper to inside the ftpes_download_one loop to avoid issues with BiocParallel workers not understanding what that helper is.

Uncommenting a helper that was still needed for several calls. Fixing stuff I broke myself!

Updating to remove BiocParallel implementation that caused bugs. Patching back to prior Future implementation instead. CLI version of genome downloading appears to have issues that may be separate and requires further troubleshooting.

data_processing.R runDataProcessing() missing(ref_file_path) check ignores if the default path is used. Updated to accept the default path if a user doesn't specify rather than erroring at the end of the processing steps.

Updated default workers to 8 for CLI and FTP downloads, made two-try FTP download option the default, added a helper to purge broken file sets (i.e., if .faa is missing for an AccID but .fna and .gff download successfully).

ftpes --> ftps is_complete --> .is_complete

… consolidate on furrr

Cleaning up parallel implementation in data_curation.R, standardizing with furrr and dropping old duplicated within-loop helper function that was needed for earlier BiocParallel implementation. Standardized values for contamination and completeness across functions. Added a parameter in prepareGenomes to allow users to set number of workers. Reduced console clutter for final "out" print.

…to qc_filtering

In data_curation.R, adding ability to export genome stats and QC metadata into human readable CSVs. `load_tables` allows you to run "example <- exportTables("data/Staphylococcus_aureus/Sau.duckdb", load_tables=TRUE)" and import the tables into R so you can play around with them directly. By default, it just saves the tables as exported CSVs. In data_processing.R, added a similar exporter that allows you to save out all the feature data and other DuckDB-connected tables, with the ability to output sequences too, and append AST phenotype data to the feature tables. Tables can be saved out as csv (default), tsv, Parquet, and xlsx. Added `writexl` to Suggests in DESCRIPTION. Fixed some reversions where I broke things in the PR this branch is based on. Will add those same fixes to that PR momentarily to keep things aligned.

Ran data processing workflow for Campylobacter coli. This can be unzipped and used as the test data for exportProcessedData() rather than having to run it all beforehand. Data curation step still requires that prepareGenomes() be run for a test bug, and Staphylococcus argenteus is a good use case for this.

jananiravi

Few quick comments. Will test it out tomorrow.

jananiravi · 2026-07-29T04:37:31Z

-    )
-    return(all_complete)
+    if (isTRUE(verbose))
+      message("All genomes already complete.")


Not complete status but completed downloading? Is something missing here?

We check the length of the diff of already (already downloaded) and ids (the full set needed), and if it's 0, then none require download. All genomes complete message is triggered.

Makes sense. Download of all genomes complete.
Or something like that indicates what's complete?

Clarified a few names based on code review feedback.

jananiravi · 2026-07-29T20:16:49Z

Few quick comments. Will test it out tomorrow.

@epbrenner prepareGenomes + exportTables --> they run locally with Sar.

jananiravi · 2026-07-29T20:21:50Z

After running the campy --> metadata.csv

what's TET doing in resistant_classes? genome_drug.resistance_phenotype marks them as susceptible. @epbrenner @AbhirupaGhosh

AbhirupaGhosh · 2026-07-29T20:27:42Z

After running the campy --> metadata.csv
what's TET doing in `resistant_classes`? genome_drug.resistance_phenotype marks them as susceptible. @epbrenner @AbhirupaGhosh

resistant_classes is a column that adds all the drug classes the genome is resistant to. If there are more than 1, say TET and FLQ, it will show TET_FLQ and it shows susceptible to the genomes that are resistant to none.

jananiravi

is this going to be merged to qc_filtering and then qc_filtering --> main? that might be helpful to avoid duplicate reviews.

jananiravi · 2026-07-29T20:34:25Z

+    gene_count = list(source = "gene_count", stem = "gene_count", appendable = TRUE),
+    protein_count = list(source = "protein_count", stem = "protein_count", appendable = TRUE),
+    domain_count = list(source = "domain_count", stem = "domain_count", appendable = TRUE),
+    struct = list(source = "struct", stem = "struct", appendable = TRUE),


struct or gene_struct?

jananiravi · 2026-07-29T20:38:59Z

+      count_if_present("metadata"),
+      count_if_present("genome_data"),
+      count_if_present("amr_phenotype"),
+      count_if_present("metadata_qc"),
+      count_if_present("metadata_qc_rejections"),
+      count_if_present("filtered"),
+      count_if_present("files"),
+      count_if_present("bac_data")


how do these checks tie back to basic_tables? is the latter not exported? so should "files" be added to basic_tables?

jananiravi · 2026-07-29T20:39:21Z

+    "genome_data",
+    "amr_phenotype",
+    "metadata_qc",
+    "metadata_qc_rejections"


see below. add files?

eboyer221

When I tested this, it looks like genome IDs such as 263.1, 263.10, and 263.100 are being written to the CSV as plain numbers rather than text, since they happen to look numeric. When I read that export back (with readr::read_csv(), pandas, or Excel's defaults) all three collapsed to the same value, since trailing zeros after a decimal don't count (263.1 and 263.10 parse as numerically identical). So the three distinct genomes merged under one ID, and it'll happen for any taxon with more than ~9 genomes. I also tried forcing quotes around every value (quote = "all") to see if that would prevent it, but the collision still happened on read. Quoting only protects against commas breaking the file format, it doesn't tell the reader "treat this as text."

This affects every genome_id-keyed table both export functions produce (genome_data, metadata, amr_phenotype_wide, gene_count, etc.).
Two options I see: (1) add a clear warning telling users to force col_types/dtype=str on ID columns when reading CSV/TSV back, or (2) recommend Parquet as the default/safe export format for programmatic reuse, since arrow preserves the real column type - I verified this did not occur with Parquet.

eboyer221 · 2026-07-29T20:51:10Z

+#'
+#' @return Invisibly returns a list with exported file paths and, if requested,
+#'   in-memory tables.
+#' @keywords internal


Suggested change

#' @keywords internal

#' @export

Without @export this stays internal-only after a normal install. library(amRdata); exportTables(...) fails even though that's exactly what the PR description tells testers to run.

After applying this change, run devtools::document() and commit the regenerated NAMESPACE/man/*.Rd. exportProcessedData() has the same problem (has @export in source but isn't in NAMESPACE since docs were never regenerated on this branch).

And that's why it didn't run the first time for me. Good catch, Emily!!

epbrenner

Oh I have to submit my own review to my own PR to actually post these comments, huh? Oops.

epbrenner · 2026-07-29T15:58:42Z

-    )
-    return(all_complete)
+    if (isTRUE(verbose))
+      message("All genomes already complete.")


We check the length of the diff of already (already downloaded) and ids (the full set needed), and if it's 0, then none require download. All genomes complete message is triggered.

jananiravi · 2026-07-29T21:29:20Z

When I tested this, it looks like genome IDs such as 263.1, 263.10, and 263.100 are being written to the CSV as plain numbers rather than text, since they happen to look numeric. When I read that export back (with readr::read_csv(), pandas, or Excel's defaults) all three collapsed to the same value, since trailing zeros after a decimal don't count (263.1 and 263.10 parse as numerically identical). So the three distinct genomes merged under one ID, and it'll happen for any taxon with more than ~9 genomes. I also tried forcing quotes around every value (quote = "all") to see if that would prevent it, but the collision still happened on read. Quoting only protects against commas breaking the file format, it doesn't tell the reader "treat this as text."

This affects every genome_id-keyed table both export functions produce (genome_data, metadata, amr_phenotype_wide, gene_count, etc.).
Two options I see: (1) add a clear warning telling users to force col_types/dtype=str on ID columns when reading CSV/TSV back, or (2) recommend Parquet as the default/safe export format for programmatic reuse, since arrow preserves the real column type - I verified this did not occur with Parquet.

@epbrenner @AbhirupaGhosh didnt we have this fix in already?

epbrenner and others added 16 commits July 22, 2026 11:17

Style code (GHA)

1c4e216

Fixing helper function bug

5761630

Uncommenting a helper that was still needed for several calls. Fixing stuff I broke myself!

Style code (GHA)

f00c12a

Swapping BiocParallel back to Future

b0de152

Updating to remove BiocParallel implementation that caused bugs. Patching back to prior Future implementation instead. CLI version of genome downloading appears to have issues that may be separate and requires further troubleshooting.

Patching ref_file_path parameter check

eec360f

data_processing.R runDataProcessing() missing(ref_file_path) check ignores if the default path is used. Updated to accept the default path if a user doesn't specify rather than erroring at the end of the processing steps.

Genome download updates in data_curation.R

ee3e0e4

Updated default workers to 8 for CLI and FTP downloads, made two-try FTP download option the default, added a helper to purge broken file sets (i.e., if .faa is missing for an AccID but .fna and .gff download successfully).

Style code (GHA)

02d0c9c

rename functions

ec6209f

ftpes --> ftps is_complete --> .is_complete

Add future/furrr to DESCRIPTION; fix .with_future_plan() scoping bug;…

e2b88bd

… consolidate on furrr

Document future::multisession testing gotcha in README

e9b1d39

Merge branch 'qc_filtering' of https://github.com/JRaviLab/amRdata in…

fd7a9aa

…to qc_filtering

Merge branch 'qc_filtering' of https://github.com/JRaviLab/amRdata in…

0d1b8f2

…to qc_filtering

epbrenner requested review from AbhirupaGhosh, amcim, eboyer221 and jananiravi July 28, 2026 22:22

epbrenner self-assigned this Jul 28, 2026

epbrenner added the enhancement New feature or request label Jul 28, 2026

jananiravi reviewed Jul 29, 2026

View reviewed changes

Updated variable/function names

5a944a9

Clarified a few names based on code review feedback.

jananiravi previously approved these changes Jul 29, 2026

View reviewed changes

Merge branch 'main' into export_tables

6ecae92

jananiravi dismissed their stale review via 6ecae92 July 29, 2026 20:17

jananiravi reviewed Jul 29, 2026

View reviewed changes

eboyer221 requested changes Jul 29, 2026

View reviewed changes

epbrenner commented Jul 29, 2026

View reviewed changes

Uh oh!

Conversation

epbrenner commented Jul 28, 2026

Testing:

What this does:

What's the point?

Uh oh!

jananiravi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jananiravi commented Jul 29, 2026

Uh oh!

jananiravi commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AbhirupaGhosh commented Jul 29, 2026

Uh oh!

jananiravi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eboyer221 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

epbrenner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jananiravi commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jananiravi commented Jul 29, 2026 •

edited

Loading