kg_microbe.transform_utils.mediadive package

Submodules

kg_microbe.transform_utils.mediadive.mediadive module

MediaDive KG.

Example script to transform downloaded data into a graph format that KGX can ingest directly, in either TSV or JSON format: https://github.com/NCATS-Tangerine/kgx/blob/master/data-preparation.md

Input: any file in data/raw/ (that was downloaded by placing a URL in incoming.txt/yaml and running run.py download.

Output: transformed data in data/raw/MediaDive:

Output these two files: - nodes.tsv - edges.tsv

kg_microbe.transform_utils.mediadive.mediadive.HTTP_CACHE_FILENAME = 'mediadive_transform_cache.sqlite'

HTTP response cache for the API fallback, kept beside the bulk JSONs. The old name is the file requests_cache.install_cache(“mediadive_cache”) left in the working directory; it is adopted on first use so nothing re-downloads.

class kg_microbe.transform_utils.mediadive.mediadive.MediaDiveTransform(input_dir=None, output_dir=None)

Bases: Transform

Template for how the transform class would be designed.

DATA_INPUTS: tuple = ('mappings/kgmicrobe_unified_entity_mappings.sssom.tsv.gz',)

Repo-relative curation files this transform reads, beyond its own data/raw/ download.

Declared so freshness tooling can tell that an output is stale against its data rather than only its code. Without it a mapping correction lands, every consumer keeps reporting FRESH, and a re-merge silently ships the old groundings: #778 corrected 16 isolation-source ids and #786 rewrote the unified chemical SSSOM, and the merged KG built afterwards still asserted 75 organisms isolated from a “Cell Line”, because nothing re-ran the transforms that read those files (#812).

Paths are relative to the repo root. Keep them tracked in git — the freshness check uses commit time, not mtime, because git checkout rewrites mtimes without changing content (#797).

List every curation file read, not a representative one. A partial declaration fails silently and looks identical to a complete one: ontologies_stubs declared 1 of the 11 files it read and was reported fresh after changes to the other ten (#839). Where the set comes from a constant, derive this from it rather than restating it.

REQUIRED_CONSUMED_INPUTS: tuple = ('bacdive_taxon_lookup',)

Named generated inputs that must actually be read before fresh finalization. Unlike DATA_INPUTS, these are checked after upstream producers have run.

TRANSFORM_INPUTS: tuple = ('ontologies', 'bacdive')

Reads ontologies/chebi_nodes.tsv and chebi_edges.tsv (roles/categories) via constants.py (#1035), plus BacDive’s intermediate strain-taxid TSV (#1091).

download_yaml_and_get_json(url, target_dir)

Download MetaDive data using a url.

Parameters:

url (str) – Path provided by MetaDive API.

Return type:

Dict[str, str]

get_compounds_of_solution(id)

Get ingredients of solutions via bulk data or MediaDive API.

First checks bulk downloaded data, then makes API call if needed.

Parameters:

id (str) – ID of solution.

Returns:

Dictionary of {compound_name: compound_id}.

get_json_object(fn, url_extension, target_dir)

Download YAML file if absent and return contents as a JSON object.

First checks bulk downloaded data, then YAML cache, then makes API call.

Parameters:

• fn (Union[Path, str]) – YAML file path.

• url_extension (str) – API endpoint extension (e.g., “medium/123”).

• target_dir (Path) – Directory for YAML cache.

Return type:

Dict[str, str]

Returns:

Dictionary.

run(data_file=None, show_status=True)

Run the transformation, closing the API session afterwards.

standardize_compound_id(id, compound_name=None)

Get standardized IDs via unified chemical mappings, bulk data, or legacy mappings.

Lookup order: 1. Unified chemical mappings by compound name (ChemicalMappingLoader) 2. Legacy MicroMediaParam mappings by compound name (fallback during transition) 3. Bulk downloaded data (embedded ChEBI/KEGG/PubChem/CAS-RN) 4. Custom ingredient prefix (last resort)

Parameters:

• id (str) – MediaDive compound ID.

• compound_name (Optional[str]) – Compound name for mapping lookup.

Returns:

Standardized ID.

Module contents

MediaDive transform.

class kg_microbe.transform_utils.mediadive.MediaDiveTransform(input_dir=None, output_dir=None)

Bases: Transform

Template for how the transform class would be designed.

DATA_INPUTS: tuple = ('mappings/kgmicrobe_unified_entity_mappings.sssom.tsv.gz',)

Repo-relative curation files this transform reads, beyond its own data/raw/ download.

Declared so freshness tooling can tell that an output is stale against its data rather than only its code. Without it a mapping correction lands, every consumer keeps reporting FRESH, and a re-merge silently ships the old groundings: #778 corrected 16 isolation-source ids and #786 rewrote the unified chemical SSSOM, and the merged KG built afterwards still asserted 75 organisms isolated from a “Cell Line”, because nothing re-ran the transforms that read those files (#812).

Paths are relative to the repo root. Keep them tracked in git — the freshness check uses commit time, not mtime, because git checkout rewrites mtimes without changing content (#797).

List every curation file read, not a representative one. A partial declaration fails silently and looks identical to a complete one: ontologies_stubs declared 1 of the 11 files it read and was reported fresh after changes to the other ten (#839). Where the set comes from a constant, derive this from it rather than restating it.

REQUIRED_CONSUMED_INPUTS: tuple = ('bacdive_taxon_lookup',)

Named generated inputs that must actually be read before fresh finalization. Unlike DATA_INPUTS, these are checked after upstream producers have run.

TRANSFORM_INPUTS: tuple = ('ontologies', 'bacdive')

Reads ontologies/chebi_nodes.tsv and chebi_edges.tsv (roles/categories) via constants.py (#1035), plus BacDive’s intermediate strain-taxid TSV (#1091).

chebi_categories: Dict[str, str]

chebi_labels: Dict[str, str]

chebi_role_edges: Dict[str, list]

chebi_roles: Dict[str, list]

download_yaml_and_get_json(url, target_dir)

Download MetaDive data using a url.

Parameters:

url (str) – Path provided by MetaDive API.

Return type:

Dict[str, str]

get_compounds_of_solution(id)

Get ingredients of solutions via bulk data or MediaDive API.

First checks bulk downloaded data, then makes API call if needed.

Parameters:

id (str) – ID of solution.

Returns:

Dictionary of {compound_name: compound_id}.

get_json_object(fn, url_extension, target_dir)

Download YAML file if absent and return contents as a JSON object.

First checks bulk downloaded data, then YAML cache, then makes API call.

Parameters:

• fn (Union[Path, str]) – YAML file path.

• url_extension (str) – API endpoint extension (e.g., “medium/123”).

• target_dir (Path) – Directory for YAML cache.

Return type:

Dict[str, str]

Returns:

Dictionary.

run(data_file=None, show_status=True)

Run the transformation, closing the API session afterwards.

standardize_compound_id(id, compound_name=None)

Get standardized IDs via unified chemical mappings, bulk data, or legacy mappings.

Lookup order: 1. Unified chemical mappings by compound name (ChemicalMappingLoader) 2. Legacy MicroMediaParam mappings by compound name (fallback during transition) 3. Bulk downloaded data (embedded ChEBI/KEGG/PubChem/CAS-RN) 4. Custom ingredient prefix (last resort)

Parameters:

• id (str) – MediaDive compound ID.

• compound_name (Optional[str]) – Compound name for mapping lookup.

Returns:

Standardized ID.