kg_microbe.transform_utils.gold package

Submodules

kg_microbe.transform_utils.gold.alias_observations module

Conserve complete native observations of the finite selected-alias GOLD cohort (#1340).

class kg_microbe.transform_utils.gold.alias_observations.AliasObservations(index)

Bases: object

Retain the exact independently projected native assertions, without graph edits.

kg_microbe.transform_utils.gold.crosswalk_quarantine module

Quarantine seven unsupported label-derived identities without deleting GOLD observations (#1294).

class kg_microbe.transform_utils.gold.crosswalk_quarantine.CrosswalkQuarantine(transform, stack, nodes_in, edges_in, *, policy_path=None, curation_path=None)

Bases: object

Bind the finite source cohort before graph output and audit each withheld proposal.

consume(role, path)

Bind the original lexical locator and return the immutable parser path.

contract()

Distinguish the exact reviewed current cohort from a genuinely absent cohort.

finish()

Require conservation before recording the mandatory audit or completed native contract.

intercept(identifier, label, target, proposal)

Withhold only the seven reviewed label-derived exact-match proposals.

observe_location(native, emitted)

Track the actual emitted current-stage assertion, including descendant source targets.

verify_bindings(*, byte_verified=False)

Do not let lost/restamped live receipts erase any actual finite authority read (#1301).

kg_microbe.transform_utils.gold.crosswalk_quarantine.verify_record_owned_locations(index, directory, *, report, admission)

Rejoin pinned native contexts and old/current projections on saved admission, without edits.

kg_microbe.transform_utils.gold.crosswalk_quarantine.verify_recorded_inputs(report, report_path=None)

Require canonical evidence roles/origins, not substitute current files for missing consumption.

kg_microbe.transform_utils.gold.fold_restoration module

Audit restored source-owned observations without treating lexical equality as identity (#1316).

class kg_microbe.transform_utils.gold.fold_restoration.FoldRestoration(index, nodes, edges, ncbi)

Bases: object

Disk-backed complete native witnesses for diagnostic historical folds, never active aliases.

counts()

Summarize native evidence separately from emitted graph multiplicity.

evidence(identifier)

Rejoin each full source observation and record its old and restored projections.

verify_graph(edges, *, quoting)

Conserve all restored source observations and remove only fold-generated taxon projections.

kg_microbe.transform_utils.gold.gold module

Transform the GOLD (Genomes OnLine Database, JGI) pre-transformed KGX TSVs.

GOLD is unusual among our sources: it arrives already in KGX shape, so there is nothing to parse. That makes a blind copy tempting and wrong. The modeling review of the shipped payload (docs/GOLD_TRANSFORM_REVIEW.md) found nine issues that a copy would carry straight into a merge — including 279,618 MaterialSample nodes with no incident edge, and 4,220 subclass_of edges that violate Biolink’s domain/range.

So this is a validating passthrough. It conforms the schema to the KG-Microbe standard, repairs what can be repaired mechanically, and reports the rest rather than silently propagating it.

What it changes, and why each is safe:

• Adds the standard columns. Upstream nodes lack description, synonym, deprecated and same_as; edges lack knowledge_level and agent_type. Every other transform emits them, and they are how a consumer tells an assertion from a prediction. GOLD entries are curated submissions, so they are filled as knowledge_assertion / manual_agent.

• Drops the upstream edge ``id``. The post-merge cleanup drops it anyway; not emitting it keeps the header standard.

• Adds ``biolink:OntologyClass`` to ecosystem nodes. subclass_of requires OntologyClass on both ends, and the GOLD ecosystem hierarchy is exactly that — a class hierarchy. Pipe-adding the category is the same device METPO nodes already use (METPO:1001000|biolink:Procedure). Without it those 4,220 edges are a Biolink violation.

• Applies the NCBITaxon trim. KG-Microbe deliberately restricts NCBITaxon to microbes (exclusion_branches.tsv removes Viruses, Viridiplantae, Metazoa and others). GOLD is a genome database covering all of life, so ingesting it unfiltered reintroduces 23,695 excluded taxa — phages, marine viruses, Picea glauca, a starling — and the 76,034 organisms typed to them, undoing the trim the ontologies transform performs. Those, and the study nodes left holding nothing but them, are dropped.

• Drops samples and studies. KG-Microbe wants neither. MaterialSample was 279,670 nodes with 242 incident edges, and Study was reachable only via IndividualOrganism -related_to-> Study, a predicate asserting nothing usable (27.5% of the export). What the ingest is for — the organism to environment link, occurs_in — is kept. Measured: nodes 975,839 -> 637,286, edges 1,110,984 -> 823,250, and the pre-existing orphan warning falls from 279,618 to 5. occurs_in ends at 287,706 rather than its upstream 286,725 — it rises, because organisms the taxid remap rescues keep their environment edges.

• Remaps retired NCBI taxids from merged.dmp in the taxonomy dump. GOLD’s export carries taxa NCBI has since merged, and judging them by the retired id makes the trim drop them along with the organisms typed to them. Measured against the real payload: 611 taxa and 1,473 organisms retained that would otherwise have been discarded, with 561 nodes collapsing onto an existing id. Applied before the trim, since that is the point.

• Resolves uninformative ecosystems upward, and drops what resolves to nothing. 63.5% of occurs_in edges pointed at a node labelled “Unclassified”, across 1,663 distinct such nodes — not a bucket you can filter, but 1,663 empty ids. The meaning is in the hierarchy: the single largest target, carrying 39,446 organisms, sits three “Unclassified” hops below Mammals: Human. Each edge now names the nearest informative ancestor and keeps the original target in Biolink’s original_object, so the collapse is auditable and reversible. Edges resolving only to the hierarchy root — 58,340 of them, asserting an organism lives somewhere — are dropped. Measured: 229,366 occurs_in edges remain, none pointing at an “Unclassified”, with Mammals: Human (39,472), Fecal, Soil, Plants, Blood and Marine the largest targets.

What it deliberately does not change, because each needs a modelling decision rather than a patch, and is reported instead:

• the 23,695 GOLD-only taxa absent from the NCBITaxon output;

• the 78 taxon name disagreements with the ontologies output.

• Corrects the environment predicate and bridges the vocabulary. Upstream ships occurs_in, which Biolink defines as holding “between a process and a material entity or site” — the subject here is an organism, a material entity, so located_in is the right one. Neither constrains domain/range beyond named thing, which is why KGXVal never flagged it. All 229,366 environment edges are re-predicated.

  The ecosystem vocabulary is then bridged two ways: the GOLD ontology’s curated skos:exactMatch links (531 nodes), then a label join against ontologies KG-Microbe already loads (537 more, overwhelmingly UBERON host anatomy — Rumen, Blood, Nasopharynx, Urine). Together 45.7% of environment edges reach an ontology term in two hops, against 26.9% from the curated mapping alone. Label normalisation was measured and contributes one edge, so the uncovered remainder is genuinely absent rather than spelled differently.

  The label join is restricted to site-shaped prefixes. Unrestricted it produced Alkaline -> PATO:0001430, Benzene -> CHEBI:16716 and Sperm -> CL:0000019; an organism is not located_in a quality, a molecule or a cell type. 18 further labels name a host taxon rather than a site and are left alone — in_taxon would assert the microbe is a plant, and the correct shape is bacdive’s inverse taxon location_of organism.

Still open: better coverage of the gold.ecosystem: vocabulary. Without it these edges remain an island — KG-Microbe models environments in ENVO everywhere else (BacDive isolation sources, Madin habitats, PREGO), so a GOLD environment and a BacDive environment for the same organism never meet. Resolution shrinks the target vocabulary from 4,226 nodes to the named ones, which is what makes a crosswalk tractable.

Still open upstream, unaffected by this transform: versioned filenames or a published checksum, and the six Saccharomyces rows that lose their hybrid x marker.

class kg_microbe.transform_utils.gold.gold.GOLDTransform(input_dir=None, output_dir=None)

Bases: Transform

Conform the pre-transformed GOLD KGX TSVs to the KG-Microbe standard.

DATA_INPUTS: tuple = ('mappings/gold_ecosystem_mesh_sites.tsv', 'mappings/gold_abscess_crosswalk_policy.json', 'mappings/gold_taxon_dispositions.json', 'mappings/gold_selected_taxon_aliases.json')

The MeSH site curation this transform reads. Declared so the freshness check notices an edit to it — a transform that reads a curation file without declaring it is reported fresh while its output is stale, which is #812, then #839, and would have been this transform next (#876).

REQUIRED_AUDIT_FILES: tuple = ('gold_label_crosswalk_quarantine.tsv', 'gold_taxon_dispositions.tsv', 'gold_lexical_fold_restorations.tsv', 'gold_selected_taxon_aliases.tsv')

Producer-owned disposition reports whose emitted bytes must survive finalization/merge.

REQUIRED_CONSUMED_INPUTS: tuple = ('gold_crosswalk_policy', 'gold_mesh_site_curation', 'gold_native_nodes', 'gold_native_edges', 'gold_taxon_disposition_policy', 'gold_selected_taxon_alias_policy')

Named generated inputs that must actually be read before fresh finalization. Unlike DATA_INPUTS, these are checked after upstream producers have run.

TRANSFORM_INPUTS: tuple = ('ontologies', 'ontologies_stubs')

Reads this transform’s output; see Transform.TRANSFORM_INPUTS (#845).

finalize(*, file_prefix='', fresh_run=False)

Transition legitimate published output replacements without rebasing native inputs (#1342).

property producer_native_inputs

Retain the reviewed selection contract independently of mutable receipts.

run(data_file=None, show_status=True)

Conform the upstream TSVs and write standard nodes/edges.

Parameters:

• data_file (Union[Path, None, str]) – Unused; the filenames are fixed by the download entry.

• show_status (bool) – Unused; retained for the base-class signature.

Raises:

FileNotFoundError – If either upstream file is missing.

Return type:

None

verify_native_inputs(*, byte_verified=False, output_dir=None)

Verify lexical paths as well as the immutable parser snapshots.

classmethod verify_recorded_native_inputs(report, report_path, *, admission=None)

Validate persisted cohort/authority bindings; never edit a graph during admission.

kg_microbe.transform_utils.gold.gold.GOLD_NODES_FILE = 'GOLD_nodes.tsv'

Upstream filenames inside data/raw/gold/.

kg_microbe.transform_utils.gold.selected_taxon_aliases module

Finite native selected-taxonomy alias evidence for eleven GOLD records (#1327).

No graph is edited here. Existing taxdump behavior is retained outside the three reviewed original IDs. New or changed eligible GOLD records require re-review; selected aliases are never inferred from labels or a full-ontology-only target.

class kg_microbe.transform_utils.gold.selected_taxon_aliases.SelectedTaxonAliases(policy, cohort, merges, evidence, bindings, optional, admission, snapshots_getter)

Bases: object

Immutable finite resolution cache, mandatory audit and retained native input guards.

evidence_for(original_id)

Return copied full evidence only for an admitted original eligible identifier.

resolve(original_id)

Compose only approved finite aliases with unchanged one-step taxdump behavior.

kg_microbe.transform_utils.gold.selected_taxon_aliases.prepare(transform, stack, nodes_path, edges_path, ncbi_path, taxdump_path)

Consume the finite policy/active JSON while retaining existing original role receipts.

kg_microbe.transform_utils.gold.selected_taxon_aliases.restore_recorded(report, report_path, *, admission=None)

Reconstruct exact native evidence and audit; absence of a contract never means zero.

kg_microbe.transform_utils.gold.taxon_dispositions module

Retain native GOLD referents without unsupported taxonomic identity (#1316).

The source labels do not establish strains. This producer policy preserves native declarations and taxon associations, with complete original assertions in a mandatory audit. Lexical equality is diagnostic, never identity: source records and their native observations are retained. Existing trims and taxid remapping remain separate decisions. No graph is edited during admission.

class kg_microbe.transform_utils.gold.taxon_dispositions.GoldDeclarationGuard(proofs, admission=None, *, transform=None, authority=None, requested_ids=())

Bases: object

Retain exact proof bytes and selected authority through source/merge publication.

bind_members(node_members, edge_members)

Bind explicit admitted bundle members, excluding unrelated neighboring ontologies.

contract()

Save explicit members and proof inventory with the selected finalized authority.

declaration_for(identifier)

Never resolve an unknown ID during merge or infer a category from its prefix.

native_node_input()

Expose the exact already-admitted native source identity for reference-resolution auditing.

verify(*, output_dir=None, metadata_only=False)

Check the selected authority and originally consumed roles through terminal publication.

class kg_microbe.transform_utils.gold.taxon_dispositions.TaxonDispositions(transform, stack, crosswalk)

Bases: object

Bind source selection once, audit it, and protect completed output through finalization.

consume(role, path)

Keep actual original role receipts separate from caller-mutable snapshots.

contract()

Return an independent serialization; zero cohorts still have a real audit.

finish()

Verify exact emitted declarations/associations before publishing the mandatory audit.

verify(*, output_dir=None, metadata_only=False)

Check active role bindings and original audit even on repeat/late publication.

class kg_microbe.transform_utils.gold.taxon_dispositions.VerifiedGoldDeclaration(payload)

Bases: object

Data-bearing native witness, with copied accessors rather than mutable admitted state.

property declaration

Return an independent copy of all approved semantic fields.

property declaration_sha256

Bind the approved semantic declaration.

property evidence_sha256

Bind the complete disposition, including original and rejected claims.

property identifier

Return the exact source-owned native identifier.

matches(row, *, merged=False)

Permit only the exact native declaration; no prefix or nameless-ID waiver.

property native_taxon_assertions

Expose all original native taxon assertions, not a selected parent guess.

property original_node

Expose complete original native fields and their physical source locator.

kg_microbe.transform_utils.gold.taxon_dispositions.admitted_gold_declarations(report, report_path, requested_ids, *, admission)

Provide explicitly requested native declarations without altering an admitted graph.

kg_microbe.transform_utils.gold.taxon_dispositions.gold_reference_requests(node_paths, edge_paths, *, own_gold=False, quoting=3, row_reader=None)

Derive the exact request set from actual explicit members, never trust a receipt’s empty claim.

kg_microbe.transform_utils.gold.taxon_dispositions.load_gold_declarations(transform, requested_ids, *, admission=None)

Select only the sibling GOLD authority; genuine zero references require no authority.

kg_microbe.transform_utils.gold.taxon_dispositions.restore_gold_declarations(transform, report, report_path, *, admission=None)

Retain live role checks after fresh-instance repeat finalization.

kg_microbe.transform_utils.gold.taxon_dispositions.verify_recorded_gold_declarations(report, report_path, *, admission=None)

Rejoin explicit consumer members and the exact selected GOLD proof contract.

kg_microbe.transform_utils.gold.taxon_dispositions.verify_recorded_taxon_inputs(report, report_path, *, admission=None, requested_ids=())

Attach a nameless-only provider while checking the complete native cohort once.

Module contents

GOLD transform module.

class kg_microbe.transform_utils.gold.GOLDTransform(input_dir=None, output_dir=None)

Bases: Transform

Conform the pre-transformed GOLD KGX TSVs to the KG-Microbe standard.

DATA_INPUTS: tuple = ('mappings/gold_ecosystem_mesh_sites.tsv', 'mappings/gold_abscess_crosswalk_policy.json', 'mappings/gold_taxon_dispositions.json', 'mappings/gold_selected_taxon_aliases.json')

The MeSH site curation this transform reads. Declared so the freshness check notices an edit to it — a transform that reads a curation file without declaring it is reported fresh while its output is stale, which is #812, then #839, and would have been this transform next (#876).

REQUIRED_AUDIT_FILES: tuple = ('gold_label_crosswalk_quarantine.tsv', 'gold_taxon_dispositions.tsv', 'gold_lexical_fold_restorations.tsv', 'gold_selected_taxon_aliases.tsv')

Producer-owned disposition reports whose emitted bytes must survive finalization/merge.

REQUIRED_CONSUMED_INPUTS: tuple = ('gold_crosswalk_policy', 'gold_mesh_site_curation', 'gold_native_nodes', 'gold_native_edges', 'gold_taxon_disposition_policy', 'gold_selected_taxon_alias_policy')

Named generated inputs that must actually be read before fresh finalization. Unlike DATA_INPUTS, these are checked after upstream producers have run.

TRANSFORM_INPUTS: tuple = ('ontologies', 'ontologies_stubs')

Reads this transform’s output; see Transform.TRANSFORM_INPUTS (#845).

finalize(*, file_prefix='', fresh_run=False)

Transition legitimate published output replacements without rebasing native inputs (#1342).

property producer_native_inputs

Retain the reviewed selection contract independently of mutable receipts.

run(data_file=None, show_status=True)

Conform the upstream TSVs and write standard nodes/edges.

Parameters:

• data_file (Union[Path, None, str]) – Unused; the filenames are fixed by the download entry.

• show_status (bool) – Unused; retained for the base-class signature.

Raises:

FileNotFoundError – If either upstream file is missing.

Return type:

None

verify_native_inputs(*, byte_verified=False, output_dir=None)

Verify lexical paths as well as the immutable parser snapshots.

classmethod verify_recorded_native_inputs(report, report_path, *, admission=None)

Validate persisted cohort/authority bindings; never edit a graph during admission.