kg_microbe.transform_utils.microbedecoder package

Submodules

kg_microbe.transform_utils.microbedecoder.bacdive_join_dispositions module

Refuse six reviewed joins and retain their snapshots on unresolved source records.

Only the approved finite snapshot subjects move; literals, attribute objects, source context and independent metabolism/FAPROTAX observations remain intact.

class kg_microbe.transform_utils.microbedecoder.bacdive_join_dispositions.BacDiveJoinGuard(policy_payload, evidence_payload, raw_stream, gss_stream, *, fields, bindings, admission)

Bases: object

Bind actual source reads, six refusals, and the complete retained BacDive-link family.

contract(): Expose a copied completion contract, never a mutable cache or implicit acceptance.

dispatch(source_record, native_row, accepted_subject, token, claim): Return an audit row only for the exact would-be strain-to-LPSN assertion.

finish(transform): Seal the completed audit and finite contract after existing source deduplication.

require_complete(): Run before atomic source file handles publish any newly written members.

snapshot_node(source_record, native_row, accepted_subject): Select an admitted record subject once, after its contradicted link was audited.

verify(snapshots, contract, *, output_dir, configured_selectors=None): Keep mandatory read/audit/output guards live through staging, repeat and publication.

kg_microbe.transform_utils.microbedecoder.bacdive_join_dispositions.prepare(transform, raw_snapshot, raw_path, policy_path): Use actual immutable CSV/GSS reads before the existing source output transaction.

kg_microbe.transform_utils.microbedecoder.bacdive_join_dispositions.restore(report, report_path, *, admission=None): Independently reconstruct from mandatory actual input roles, not mutable audit claims.

kg_microbe.transform_utils.microbedecoder.bacdive_join_dispositions.verify_recorded_inputs(report, report_path, *, admission=None): Retain the native/read guard on the caller’s admission until final publication.

kg_microbe.transform_utils.microbedecoder.bacdive_join_dispositions.verify_transform(transform, *, output_dir=None): Restore a fresh object’s read-only guard on repeat; do not invent producer completion.

kg_microbe.transform_utils.microbedecoder.chemical_curation module

Apply a reviewed chemical spelling correction to one complete source record.

This is not a global chemical alias or a modification of the pinned MIM release. Schink (1984), Table 4 and the Ilyobacter tartaricus species description, link strain GraTa2 / DSM 2382 to tartrate fermentation and the reported products. The broad native tartrate class avoids inventing a counterion or stereoisomer.

class kg_microbe.transform_utils.microbedecoder.chemical_curation.RecordChemicalCuration(chebi_nodes_input)

Bases: object

Resolve one record/field while retaining native ownership of its target.

The caller must retain the supplied ChEBI input through consumed-input finalization and call resolve for the reviewed field even when its current value is empty. A changed known record requires a new review, not fallback to an old or guessed chemical identity.

resolve(row, source_key, literal)

Correct only the authenticated record; never normalize another label.

Return type:: str | None

kg_microbe.transform_utils.microbedecoder.chemical_curation.canonical_raw_row_sha256(row)

Bind every literal CSV field, independently of column order or ordinal.

Return type:: str

kg_microbe.transform_utils.microbedecoder.crosswalk_quarantine module

Quarantine only individually reviewed, snapshot-bound crosswalk claims.

This is a finite source-quality disposition, never a taxonomy resolver or an instruction to replace one taxid with another. Unknown claims retain their existing emission behavior. A changed source or reviewed witness must receive new review rather than silently bypassing the disposition.

class kg_microbe.transform_utils.microbedecoder.crosswalk_quarantine.CrosswalkDecision(rule_id, source_record, raw_record_sha256, raw_lpsn_id, subject, source_column, source_cell_base64, source_token, object, predicate, relation, primary_knowledge_source, knowledge_level, agent_type, disposition, witness_id, rationale)

Bases: object

One rejected original crosswalk assertion, without a replacement identity.

agent_type: str

disposition: str

property key: tuple[str, str, str]: Identify a literal claim by record, original field and unsplit token.

knowledge_level: str

object: str

predicate: str

primary_knowledge_source: str

rationale: str

raw_lpsn_id: str

raw_record_sha256: str

relation: str

rule_id: str

source_cell_base64: str

source_column: str

source_record: str

source_token: str

subject: str

witness_id: str

class kg_microbe.transform_utils.microbedecoder.crosswalk_quarantine.CrosswalkQuarantine(policy, decisions_input, evidence_input)

Bases: object

Validate reviewed policy, preflight exact raw evidence, then suppress only matched claims.

audit_row(rule, raw_row, claim)

Retain the entire rejected original edge and raw record reversibly, outside the KG.

Return type:: dict[str, str]

match(source_record, row, subject, source_column, token, original_object)

Return only the exact reviewed pre-fold claim, never another field or target.

Return type:: CrosswalkDecision | None

preflight(stream, source_sha256, accepted_lpsn)

Check the full immutable raw snapshot before any graph output is opened.

Return type:: None

require_complete()

Missing emission calls must not publish a partially applied reviewed policy.

Return type:: None

class kg_microbe.transform_utils.microbedecoder.crosswalk_quarantine.QuarantinePolicy(version, source_sha256, source_records, decisions_file, decisions_sha256, decisions_rows, evidence_file, evidence_sha256, evidence_uncompressed_sha256=None, evidence_uncompressed_bytes=None)

Bases: object

Bind one complete source snapshot and its finite reviewed decisions.

decisions_file: str

decisions_rows: int

decisions_sha256: str

evidence_file: str

evidence_sha256: str

evidence_uncompressed_bytes: int | None = None

evidence_uncompressed_sha256: str | None = None

classmethod load(source)

Read a strict, hash-bound policy without any default permissive route.

Return type:: QuarantinePolicy

source_records: int

source_sha256: str

version: int

kg_microbe.transform_utils.microbedecoder.crosswalk_quarantine.canonical_record_json(row)

Preserve every raw field, including surrogate-escaped non-UTF8 bytes.

Return type:: str

kg_microbe.transform_utils.microbedecoder.crosswalk_quarantine.crosswalk_raw_record_sha256(row)

Use the explicitly reviewed crosswalk-record hash, not a chemical-row hash.

Return type:: str

kg_microbe.transform_utils.microbedecoder.crosswalk_witnesses module

Validate finite historical crosswalk witnesses without doing new taxonomy inference.

class kg_microbe.transform_utils.microbedecoder.crosswalk_witnesses.HistoricalTaxonomy(evidence)

Bases: object

Validate exact rank and retired-taxid lines in the reviewed authority snapshot.

alias(original, replacement): Require the precise retired-ID replacement from the saved raw row.

rank(identifier, expected): Require the precise genus/phylum designation from the saved raw row.

class kg_microbe.transform_utils.microbedecoder.crosswalk_witnesses.WitnessObjects(objects)

Bases: object

Resolve only content-addressed composites, rejecting missing references and cycles.

expand(value): Expand and hash-check each referenced object once; retain shared values.

kg_microbe.transform_utils.microbedecoder.crosswalk_witnesses.validate_native(native, taxonomy): Require a complete active lineage with pinned raw rank and retired-ID witnesses.

kg_microbe.transform_utils.microbedecoder.crosswalk_witnesses.validate_witness(witness, rule, taxonomy): Check two independent historical supports and only the reviewed cross-rank conflict.

kg_microbe.transform_utils.microbedecoder.curation module

Resolve reviewed MicrobeDecoder process terms within their source field.

These rules normalize an existing source assertion; they neither establish an independent organism-level observation nor provide global lexical synonyms. Authoritative METPO and GO declarations remain owned by the ontology transform.

class kg_microbe.transform_utils.microbedecoder.curation.ProcessCuration(mapping_input, metpo_nodes_input, go_nodes_input=None)

Bases: object

Load exact rules and fail closed against the supplied ontology export.

Both METPO and GO rules require ontology-owned biolink:BiologicalProcess declarations. Reviewed definitions support process meaning, not a source-created stub or category override. Missing declarations, changed labels/categories and deprecated targets require a reviewed update before these rules can run.

resolve(source_key, literal)

Resolve only the exact role key and literal, without normalization.

Return type:: ProcessMapping | None

class kg_microbe.transform_utils.microbedecoder.curation.ProcessMapping(source_key, source_column, source_literal, target_curie, target_label, target_category, predicate, relation, knowledge_level, agent_type, evidence_type, evidence_uri, curation_rationale)

Bases: object

Retain one exact source-term rule and its separate curation evidence.

agent_type: str

curation_rationale: str

evidence_type: str

evidence_uri: str

knowledge_level: str

predicate: str

relation: str

source_column: str

source_key: str

source_literal: str

target_category: str

target_curie: str

target_label: str

kg_microbe.transform_utils.microbedecoder.gold_crosswalk_dispositions module

Rejoin weakened GOLD crosswalks to complete literal native CSV observations (#1316).

kg_microbe.transform_utils.microbedecoder.gold_crosswalk_dispositions.verify_recorded_inputs(report, report_path, *, admission=None): Require every emitted weakened observation to retain its exact source record and old claim.

kg_microbe.transform_utils.microbedecoder.material_dispositions module

Describe a finite reviewed material cohort without resolving chemical identity.

This catalogue is inventory metadata, never a producer dependency or an alias table. Source records, existing graph endpoints, citations and evidence tiers are matched exactly. Closed-cohort checks are explicit diagnostic operations.

class kg_microbe.transform_utils.microbedecoder.material_dispositions.MaterialDisposition(source_record, raw_record_sha256, subject, source_column, source_literal, source_citation, object_curie, predicate, relation, primary_knowledge_source, knowledge_level, agent_type, value_encoding, disposition, identity_status, identity_approved, evidence_uri, curation_rationale)

Bases: object

One original source use, its unchanged graph witness and unresolved identity.

agent_type: str

curation_rationale: str

disposition: str

evidence_uri: str

identity_approved: str

identity_status: str

property key: tuple[str, str, str]: Identify an exact record/field/token, not a global chemical name.

knowledge_level: str

object_curie: str

predicate: str

primary_knowledge_source: str

raw_record_sha256: str

relation: str

source_citation: str

source_column: str

source_literal: str

source_record: str

subject: str

value_encoding: str

class kg_microbe.transform_utils.microbedecoder.material_dispositions.MaterialDispositionCuration(mapping_input=PosixPath('/home/runner/work/kg-microbe/kg-microbe/mappings/canonical/microbedecoder_material_dispositions.tsv'))

Bases: object

Load exact finite review metadata; never mint or redirect graph identities.

resolve_edge(edge)

Describe a known use, rejecting changed evidence; leave unseen uses unreviewed.

Return type:: MaterialDisposition | None

property rules: tuple[MaterialDisposition, ...]: Expose immutable declarations for complete-cohort diagnostics.

validate_edges(edges_path, *, require_all=True)

Stream an explicit cohort diagnostic without making it a transform dependency.

Return type:: dict

validate_raw(raw_path)

Explicitly verify the full reviewed records against an unchanged raw snapshot.

Return type:: dict

kg_microbe.transform_utils.microbedecoder.microbedecoder module

MicrobeDecoder transform (Hackmann et al., Nucleic Acids Research 2026).

Ingests the wide LPSN-name-record CSV MicrobeDecoder publishes on GitHub — Shiny/MicrobeDecoder/data/database/database.zip — into KGX-format nodes and edges. The database is the actively-maintained successor to FermentationExplorer and pre-joins four curated fermentation-metabolism sources KG-Microbe does not otherwise cover:

Bergey’s Manual of Systematics of Archaea and Bacteria — expert curation of Type_of_metabolism, Major_end_products, Minor_end_products, Substrates_for_end_products
VPI Anaerobe Laboratory Manual — independent second-opinion fermentation profiles for anaerobes
Primary literature — hand-curated end-products with DOI/PMID citations
FAPROTAX — taxon-based functional predictions joined to source records

Also emits a biolink:close_match crosswalk from every lpsn:<LPSN_ID> row to NCBITaxon:, ncbi.assembly: (or GTDB: taxonomy strings), and IMG:. Verified unresolved GOLD referents instead retain the original field observation as audited biolink:related_to, never identity (#1316). BacDive crosswalks use kgmicrobe.strain:bacdive_* subsumption. An LPSN name record is not a strain identity, and its reported traits must not be propagated as universal observations about every member of the taxon.

Design decisions locked from the plan-mode Q&A:

Node identity: normally attach edges to the existing lpsn:<LPSN_ID> node emitted by the LPSN transform. This transform emits edges only (plus terminal nodes for end-products / cross-refs / provisional placeholders); it does not re-emit LPSN taxon nodes. Six contradicted BacDive joins instead preserve nonempty snapshots on record-scoped InformationContentEntity nodes; independent metabolism and FAPROTAX observations retain their LPSN subject (#1363).
BacDive_* replay: ingested with primary_knowledge_source = infores:microbedecoder so the paper’s 2024-11-07 BacDive snapshot stays reproducible. The fresher bacdive transform is authoritative in the merged KG; merge-time dedup keeps both edges around, distinguishable by provenance.

License: CC BY 4.0 (matches the upstream repo).

class kg_microbe.transform_utils.microbedecoder.microbedecoder.MicrobeDecoderTransform(input_dir=None, output_dir=None, chemical_loader=None, process_mappings=None, phenotype_mappings=None, process_scopes=None, crosswalk_quarantine_policy=None, bacdive_join_policy=None)

Bases: Transform

Transform the MicrobeDecoder wide CSV into KGX nodes and edges.

DATA_INPUTS: tuple = ('mappings/kgmicrobe_unified_entity_mappings.sssom.tsv.gz', 'mappings/canonical/microbedecoder_process_mappings.tsv', 'mappings/canonical/microbedecoder_phenotype_mappings.tsv', 'mappings/canonical/microbedecoder_process_scope_definitions.tsv', 'mappings/canonical/microbedecoder_crosswalk_quarantine.json', 'mappings/canonical/microbedecoder_crosswalk_quarantine.tsv', 'mappings/canonical/microbedecoder_crosswalk_quarantine_evidence.json.gz', 'mappings/canonical/microbedecoder_bacdive_join_dispositions.json', 'mappings/canonical/microbedecoder_bacdive_join_evidence.json')

Repo-relative curation files this transform reads, beyond its own data/raw/ download.

Declared so freshness tooling can tell that an output is stale against its data rather than only its code. Without it a mapping correction lands, every consumer keeps reporting FRESH, and a re-merge silently ships the old groundings: #778 corrected 16 isolation-source ids and #786 rewrote the unified chemical SSSOM, and the merged KG built afterwards still asserted 75 organisms isolated from a “Cell Line”, because nothing re-ran the transforms that read those files (#812).

Paths are relative to the repo root. Keep them tracked in git — the freshness check uses commit time, not mtime, because git checkout rewrites mtimes without changing content (#797).

List every curation file read, not a representative one. A partial declaration fails silently and looks identical to a complete one: ontologies_stubs declared 1 of the 11 files it read and was reported fresh after changes to the other ten (#839). Where the set comes from a constant, derive this from it rather than restating it.

REQUIRED_AUDIT_FILES: tuple = ('crosswalk_quarantine.tsv', 'phenotype_normalizations.tsv', 'gold_crosswalk_dispositions.tsv', 'bacdive_join_dispositions.tsv'): Producer-owned disposition reports whose emitted bytes must survive finalization/merge.

REQUIRED_CONSUMED_INPUTS: tuple = ('process_mappings', 'process_authority', 'process_go_authority', 'phenotype_mappings', 'process_scope_definitions', 'chemical_authority', 'crosswalk_policy', 'crosswalk_decisions', 'crosswalk_evidence', 'bacdive_join_policy', 'bacdive_join_evidence', 'bacdive_join_lpsn_gss', 'crosswalk_raw'): Named generated inputs that must actually be read before fresh finalization. Unlike DATA_INPUTS, these are checked after upstream producers have run.

TRANSFORM_INPUTS: tuple = ('lpsn', 'gold', 'gtdb', 'ontologies'): Reads this transform’s output; see Transform.TRANSFORM_INPUTS (#845).

TSV_QUOTING = 3

run(data_file=None, show_status=True)

Emit nodes.tsv and edges.tsv from database.csv.

Return type:: None

Parameters

data_file:: Optional override for the input path. Accepts either a zip (extracted here) or an already-unzipped CSV. Default resolves to <input_base_dir>/microbedecoder_database.zip when the zip is present, else <input_base_dir>/database.csv.
show_status:: Accepted for CLI compatibility; unused because the parse pass is fast enough (~8 K rows) that a progress bar isn’t worth pulling tqdm in for.

verify_native_inputs(*, byte_verified=False, output_dir=None): Retain source-local six-link evidence through finalization and repeat.

classmethod verify_recorded_native_inputs(report, report_path, *, admission=None): Read-only rejoin of each retained GOLD field observation at source/merge admission.

kg_microbe.transform_utils.microbedecoder.phenotype_curation module

Resolve reviewed source values to an Attribute’s native ontology type.

Rules are exact source-column/literal normalizations, not global synonyms or independent organism observations. The producer retains source attributes and their original provenance tier. The native class is an Attribute node’s has_attribute_type slot, never an edge predicate or a taxon phenotype edge. The record-level normalization report retains the evidence for each use.

class kg_microbe.transform_utils.microbedecoder.phenotype_curation.PhenotypeCuration(mapping_input, metpo_nodes_input)

Bases: object

Validate reviewed rules against the supplied native ontology declarations.

The rules use the ontology owner’s exported category without creating or retyping nodes. Their phenotype meaning and field-specific synonym provenance were reviewed against the pinned native ontology, not inferred from a label.

resolve(source_column, literal)

Resolve an exact field and literal without case, sign or value coercion.

Return type:: PhenotypeMapping | None

class kg_microbe.transform_utils.microbedecoder.phenotype_curation.PhenotypeMapping(source_column, source_literal, target_curie, target_label, target_category, evidence_uri, curation_rationale)

Bases: object

Keep exact attribute-type curation separate from the original observation.

curation_rationale: str

evidence_uri: str

source_column: str

source_literal: str

target_category: str

target_curie: str

target_label: str

kg_microbe.transform_utils.microbedecoder.process_scopes module

Load reviewed local process meanings without asserting ontology identity.

The source version identifies the reviewed vocabulary, not a demonstrated build of the saved organism assignment CSV. Original prediction provenance is separate and must be retained by consumers. This module never fetches or rewrites inputs.

class kg_microbe.transform_utils.microbedecoder.process_scopes.ProcessScope(source_column, source_literal, scope_definition, evidence_uri, source_version, archive_sha256, member_sha256, source_first_line)

Bases: object

One field-scoped local process definition and its reviewed source witness.

archive_sha256: str

property curie: str: Identify one exact source-column/literal meaning without global slug collisions.

property description: str: Keep semantic-review evidence distinct from organism assignment evidence.

evidence_uri: str

member_sha256: str

scope_definition: str

source_column: str

source_first_line: str

source_literal: str

source_version: str

class kg_microbe.transform_utils.microbedecoder.process_scopes.ProcessScopeCuration(definitions_input=PosixPath('/home/runner/work/kg-microbe/kg-microbe/mappings/canonical/microbedecoder_process_scope_definitions.tsv'))

Bases: object

Validate the complete finite table; fail closed instead of dropping bad rows.

resolve(source_column, literal)

Match the exact source field and token without additional normalization.

Return type:: ProcessScope | None

property rules: tuple[ProcessScope, ...]: Expose immutable declarations for cross-policy conflict validation.

kg_microbe.transform_utils.microbedecoder.source_annotations module

Preserve finite reviewed non-process group labels without inventing process identity.

These are representation dispositions, NOT ontology mappings. FAPROTAX documents taxon groups, host associations and pathogen groups as well as metabolic functions; the Type_of_metabolism column does not make every value a biological process. Source membership remains a prediction and never becomes a direct infection, habitat, or universal phenotype claim. See docs/microbedecoder_process_curation.md.

kg_microbe.transform_utils.microbedecoder.source_annotations.is_reported_metabolism_annotation(source_column, literal)

Identify reviewed metadata/group membership while retaining its exact source scope.

Return type:: bool

kg_microbe.transform_utils.microbedecoder.utils module

Helpers for the MicrobeDecoder transform.

Column definitions and light per-cell parsers so microbedecoder.py stays focused on row → node/edge emission and cross-ref wiring. All heavy semantic work (label → CURIE resolution, NER fallback) reuses existing utilities in kg_microbe.utils — see the transform module docstring for the full list.

kg_microbe.transform_utils.microbedecoder.utils.LPSN_GENUS_COLUMN = 'LPSN_Genus': Naming columns, used to label stubs for ids the lpsn transform cannot supply (botanical-code cyanobacteria absent from lpsn_gss.csv, #811).

kg_microbe.transform_utils.microbedecoder.utils.crosswalk_curie(local_id, prefix, source_prefix=None)

Normalize a crosswalk token to the identifier its owner transform emits.

Return type:: str

kg_microbe.transform_utils.microbedecoder.utils.format_citation(raw)

Return all recognized PMID: / doi: CURIEs, pipe-separated, or None.

Accepts free text; extracts each explicit DOI or PMID it finds. Returns None when the cell is empty (per is_empty_cell()) or contains no recognisable citation token — the caller can then decide whether to drop the citation attribute or store the raw prose.

Return type:: Optional[str]

kg_microbe.transform_utils.microbedecoder.utils.is_empty_cell(cell)

Report whether a cell value should be treated as absent.

Return type:: bool

kg_microbe.transform_utils.microbedecoder.utils.iter_metabolism_columns(row)

Yield (group_label, fields) for every metabolism source with data.

fields carries the raw cell values for each column-role the group declares (see METABOLISM_GROUPS). Groups whose columns are all empty (via is_empty_cell()) are skipped so the caller only iterates over sources that actually contributed to this row. Missing columns (source doesn’t declare that role) simply are absent from fields.

Return type:: Iterable[Tuple[str, Dict[str, object]]]

kg_microbe.transform_utils.microbedecoder.utils.slugify_label(label)

Produce a stable CURIE-safe slug from a raw source label.

Return type:: str

kg_microbe.transform_utils.microbedecoder.utils.split_multivalue(cell)

Split outside brackets, preserving numeric chemical locants such as 2,3-butanediol.

Parameters:: cell (object) – Raw cell value.
Return type:: List[str]
Returns:: Trimmed, non-empty tokens.

kg_microbe.transform_utils.microbedecoder.utils.split_multivalue_comma_only(cell)

Split a multi-value cell on comma ONLY (preserves in-value semicolons).

Use for identity-crosswalk cells (IMG_Genome_ID and friends) where a raw semicolon is part of the identifier (GTDB_ID rank separator) rather than a value separator.

Return type:: List[str]

kg_microbe.transform_utils.microbedecoder.utils.split_snapshot_values(column, cell)

Preserve a lone assay sign without decoding its biological interpretation.

The generic parser treats ‘-’ as missing, but these two source columns report ‘+’ and ‘-’ assay tokens. A mixed ‘+,-’ already preserved both; the lone ‘-’ must also survive. Other columns retain their existing missing-value and multivalue conventions.

Return type:: List[str]

Module contents

MicrobeDecoder transform (Hackmann & Zhang 2023, Sci Adv).