kg_microbe.transform_utils.gold package

Submodules

kg_microbe.transform_utils.gold.gold module

Transform the GOLD (Genomes OnLine Database, JGI) pre-transformed KGX TSVs.

GOLD is unusual among our sources: it arrives already in KGX shape, so there is nothing to parse. That makes a blind copy tempting and wrong. The modeling review of the shipped payload (docs/GOLD_TRANSFORM_REVIEW.md) found nine issues that a copy would carry straight into a merge — including 279,618 MaterialSample nodes with no incident edge, and 4,220 subclass_of edges that violate Biolink’s domain/range.

So this is a validating passthrough. It conforms the schema to the KG-Microbe standard, repairs what can be repaired mechanically, and reports the rest rather than silently propagating it.

What it changes, and why each is safe:

Adds the standard columns. Upstream nodes lack description, synonym, deprecated and same_as; edges lack knowledge_level and agent_type. Every other transform emits them, and they are how a consumer tells an assertion from a prediction. GOLD entries are curated submissions, so they are filled as knowledge_assertion / manual_agent.
Drops the upstream edge ``id``. The post-merge cleanup drops it anyway; not emitting it keeps the header standard.
Adds ``biolink:OntologyClass`` to ecosystem nodes. subclass_of requires OntologyClass on both ends, and the GOLD ecosystem hierarchy is exactly that — a class hierarchy. Pipe-adding the category is the same device METPO nodes already use (METPO:1001000|biolink:Procedure). Without it those 4,220 edges are a Biolink violation.
Applies the NCBITaxon trim. KG-Microbe deliberately restricts NCBITaxon to microbes (exclusion_branches.tsv removes Viruses, Viridiplantae, Metazoa and others). GOLD is a genome database covering all of life, so ingesting it unfiltered reintroduces 23,695 excluded taxa — phages, marine viruses, Picea glauca, a starling — and the 76,034 organisms typed to them, undoing the trim the ontologies transform performs. Those, and the study nodes left holding nothing but them, are dropped.
Drops samples and studies. KG-Microbe wants neither. MaterialSample was 279,670 nodes with 242 incident edges, and Study was reachable only via IndividualOrganism -related_to-> Study, a predicate asserting nothing usable (27.5% of the export). What the ingest is for — the organism to environment link, occurs_in — is kept. Measured: nodes 975,839 -> 637,286, edges 1,110,984 -> 823,250, and the pre-existing orphan warning falls from 279,618 to 5. occurs_in ends at 287,706 rather than its upstream 286,725 — it rises, because organisms the taxid remap rescues keep their environment edges.
Remaps retired NCBI taxids from merged.dmp in the taxonomy dump. GOLD’s export carries taxa NCBI has since merged, and judging them by the retired id makes the trim drop them along with the organisms typed to them. Measured against the real payload: 611 taxa and 1,473 organisms retained that would otherwise have been discarded, with 561 nodes collapsing onto an existing id. Applied before the trim, since that is the point.
Resolves uninformative ecosystems upward, and drops what resolves to nothing. 63.5% of occurs_in edges pointed at a node labelled “Unclassified”, across 1,663 distinct such nodes — not a bucket you can filter, but 1,663 empty ids. The meaning is in the hierarchy: the single largest target, carrying 39,446 organisms, sits three “Unclassified” hops below Mammals: Human. Each edge now names the nearest informative ancestor and keeps the original target in Biolink’s original_object, so the collapse is auditable and reversible. Edges resolving only to the hierarchy root — 58,340 of them, asserting an organism lives somewhere — are dropped. Measured: 229,366 occurs_in edges remain, none pointing at an “Unclassified”, with Mammals: Human (39,472), Fecal, Soil, Plants, Blood and Marine the largest targets.

What it deliberately does not change, because each needs a modelling decision rather than a patch, and is reported instead:

the 23,695 GOLD-only taxa absent from the NCBITaxon output;
the 78 taxon name disagreements with the ontologies output.
Corrects the environment predicate and bridges the vocabulary. Upstream ships occurs_in, which Biolink defines as holding “between a process and a material entity or site” — the subject here is an organism, a material entity, so located_in is the right one. Neither constrains domain/range beyond named thing, which is why KGXVal never flagged it. All 229,366 environment edges are re-predicated.

The ecosystem vocabulary is then bridged two ways: the GOLD ontology’s curated skos:exactMatch links (531 nodes), then a label join against ontologies KG-Microbe already loads (537 more, overwhelmingly UBERON host anatomy — Rumen, Blood, Nasopharynx, Urine). Together 45.7% of environment edges reach an ontology term in two hops, against 26.9% from the curated mapping alone. Label normalisation was measured and contributes one edge, so the uncovered remainder is genuinely absent rather than spelled differently.

The label join is restricted to site-shaped prefixes. Unrestricted it produced Alkaline -> PATO:0001430, Benzene -> CHEBI:16716 and Sperm -> CL:0000019; an organism is not located_in a quality, a molecule or a cell type. 18 further labels name a host taxon rather than a site and are left alone — in_taxon would assert the microbe is a plant, and the correct shape is bacdive’s inverse taxon location_of organism.

Still open: better coverage of the gold.ecosystem: vocabulary. Without it these edges remain an island — KG-Microbe models environments in ENVO everywhere else (BacDive isolation sources, Madin habitats, PREGO), so a GOLD environment and a BacDive environment for the same organism never meet. Resolution shrinks the target vocabulary from 4,226 nodes to the named ones, which is what makes a crosswalk tractable.

Still open upstream, unaffected by this transform: versioned filenames or a published checksum, and the six Saccharomyces rows that lose their hybrid x marker.

class kg_microbe.transform_utils.gold.gold.GOLDTransform(input_dir=None, output_dir=None)

Bases: Transform

Conform the pre-transformed GOLD KGX TSVs to the KG-Microbe standard.

DATA_INPUTS: tuple = ('mappings/gold_ecosystem_mesh_sites.tsv',): The MeSH site curation this transform reads. Declared so the freshness check notices an edit to it — a transform that reads a curation file without declaring it is reported fresh while its output is stale, which is #812, then #839, and would have been this transform next (#876).

TRANSFORM_INPUTS: tuple = ('ontologies', 'ontologies_stubs'): Reads this transform’s output; see Transform.TRANSFORM_INPUTS (#845).

run(data_file=None, show_status=True)

Conform the upstream TSVs and write standard nodes/edges.

Parameters:

data_file (Union[Path, None, str]) – Unused; the filenames are fixed by the download entry.
show_status (bool) – Unused; retained for the base-class signature.

Raises:

FileNotFoundError – If either upstream file is missing.

Return type:

None

kg_microbe.transform_utils.gold.gold.GOLD_NODES_FILE = 'GOLD_nodes.tsv': Upstream filenames inside data/raw/gold/.

Module contents

GOLD transform module.

class kg_microbe.transform_utils.gold.GOLDTransform(input_dir=None, output_dir=None)