Evaluation for ncatsgardkg
Evaluator: Cortes et al.
Evaluated on: 2025-08-26
This is a manual evaluation intended to identify potential barriers to reuse.
Evaluation Criteria: This evaluation uses the KG-Registry evaluation rubric as described in Cortes et al. (2025) . The rubric assesses knowledge graphs across multiple dimensions including access, provenance, documentation, maintenance, and fitness for purpose.
Access Level and Types
| Question | Answer | Comment |
|---|---|---|
| Access to data outside of the knowledge graph | Y | The KG enables extraction of disease mappings, disease profiles, and pathogenesis paths, as shown in the case studies (e.g., 3-hop paths from Wilson disease) |
| API or online access to the knowledge graph | Y | Public Neo4j instance available: https://disease.ncats.io |
| Multiple access options available | N | |
| Source code availability | Y | Partially – The in-house framework stitcher is mentioned and linked (https://github.com/ncats/stitcher), but full KG generation scripts for GARD KG are not public |
| Downloadable knowledge graph | Y | Networkx pickle |
Section Score: 4/5
Provenance of Nodes and Edges
| Question | Answer | Comment |
|---|---|---|
| Source list provided | Y | Integrates 34 distinct biomedical resources |
| Source versions information | Y | ORPHADATA v4.0 Gene Ontology Human Phenotype Ontology GO annotations Pathway Commons (PTC) Pharos v3.8.0 |
| Import dependencies | Y | Partially – Tools like stitcher and OWL file parsing are mentioned, but not declared formally like a software repo would |
| Node and edge sources | Y | Each node/edge carries provenance via stitch keys (e.g., N_Name, I_CODE) |
| Edges deduplication | Y | Partially – Mappings and harmonization discussed, but no explicit deduplication process for edges described |
| Triples source details | Y | Object properties defined per source and integration method (e.g., has_phenotype, I_CODE, N_Name) |
| Edge type schema | Y | Meta-ontology schema is described with object properties and their meanings (Table 3) |
Section Score: 7/7
Documented standards, schema, construction
| Question | Answer | Comment |
|---|---|---|
| Biological usable data | Y | Disease profiles, drug-disease associations, and harmonization rules demonstrate biomedical applicability |
| Resolvable IDs | Y | The paper explicitly mentions using "Unique Ingredient Identifier (UNII)," "MONDO ID," and "OMIM ID" for mappings, indicating that external identifiers are a core part of their entity resolution strategy |
| Construction documentation | Y | Detailed methods section describes data collection, mapping strategies, and harmonization |
| Transformation documentation | Y | The paper notes that "data cleanup was performed," mentioning specific examples like restricting prefixes for OMIM IDs and Orphanet IDs. It also discusses the challenges of programmatic annotation, such as preventing the annotation of generic terms like "Disease" or "Syndrome" |
| Schema used | Y | The paper outlines a "meta-ontology" that serves as the schema for the knowledge graph. This includes a clear definition of primary classes and object properties used to structure the data; documented in tables 2–4 |
Section Score: 5/5
Update frequency and versioning
| Question | Answer | Comment |
|---|---|---|
| Stable versions | N | |
| Public tracker information | N | |
| Knowledge graph contact information | Y | Corresponding author: qian.zhu@nih.gov and NCATS/ NIH team affiliation listed |
| Updated annually | N | Suggests updates over time; indicates continually updated |
| Prior versions access | N |
Section Score: 1/5
Evaluation - Metrics and Fitness for Purpose
| Question | Answer | Comment |
|---|---|---|
| Use case provided | Y | Four detailed case studies: disease mapping, disease profiling, data harmonization, pathogenesis exploration |
| Evaluation against other models | Y | The paper compares its approach with other similar efforts, such as the semantic Diseasecard and the Monarch Initiative, placing its own work in the context of the broader field; but no explicit benchmarking |
| Defined scope | Y | Focused on rare diseases, especially those curated in GARD, and integrating relevant biomedical data |
| Multiple evaluation methods | Y | Partially – Demonstrates utility via case studies but lacks formal quantitative evaluation (mapping stats in tables 6,7,10) |
| Accuracy metrics | Y | The paper's semi-automatic mapping process for FDA orphan designations includes a manual curation step where curators labeled mappings as "Done," "Approximate," or "Failed," providing a clear way to measure the confidence or accuracy of the mappings |
Section Score: 5/5
License Information
| Question | Answer | Comment |
|---|---|---|
| License | CC BY 4.0 |