Evaluation for robokop
Evaluator: Not specified
Evaluated on: 2025-08-14
This is a manual evaluation intended to identify potential barriers to reuse.
Access Level and Types
| Question | Answer | Comment |
|---|---|---|
| Access to data outside of the knowledge graph | Y | Can access subgraphs through query searches |
| API or online access to the knowledge graph | Y | Neo4j, ExEmPLAR (experimental tool for queries of Neo4j graphs), website with "question builder" feature, automat queries (cypher and TRAPI) |
| Multiple access options available | Y | Can download or query searches in multiple ways |
| Source code availability | Y | Https://github.com/RobokopU24/ORION (this is not the original GH from the paper but is now the one they are using) |
| Downloadable knowledge graph | Y | Https://stars.renci.org/var/plater/graphs/RobokopKG/6fe13d850fdbf89c/ |
Section Score: 5/5
Provenance of Nodes and Edges
| Question | Answer | Comment |
|---|---|---|
| Source list provided | Y | Metadata contains all source info |
| Source versions information | Y | Metadata includes links and versions for all sources |
| Import dependencies | Y | Github lists docker requirements and graph spec file with example provided |
| Node and edge sources | Y | Node and edges contain primary knowledge source attribute and, if relationship is inferred, any algorithms/methods used |
| Edges deduplication | Y | Not explicitly mentioned in paper or docs, but github for creating KG/metadata records have an "edge_normalization_version" so it appears they do |
| Triples source details | N | One-hop connection schema has summary of all edge types (node 1 type, node 2 type, relationship), and individual edges include the original data source, but sources per triple type were not explicitly stated/summarized |
| Edge type schema | N | One-hop connection schema has summary of all edge types, and individual edges include the original data source, but sources per edge type were not explicitly stated |
Section Score: 5/7
Documented standards, schema, construction
| Question | Answer | Comment |
|---|---|---|
| Biological usable data | Y | Json files |
| Resolvable IDs | Y | Choose one external representative entity ID and map synonyms using existing equivalencies and specailized rules |
| Construction documentation | Y | Github repo for construction with reaonsable documentation on how to use it in general, and metadata files include exact versions of functions that were used when creating their KG |
| Transformation documentation | Y | Not explicitly described but metadata contains all preprocessing steps per source with versions of each normalization strategies used |
| Schema used | Y | BioLink |
Section Score: 5/5
Update frequency and versioning
| Question | Answer | Comment |
|---|---|---|
| Stable versions | N | |
| Public tracker information | Y | Github |
| Knowledge graph contact information | Y | Contact form provided on website and weekly office hours over zoom |
| Updated annually | N | |
| Prior versions access | Y | Accessible for bulk download. No chnage log, but each comes with metadata that documents all source ingests (including versions) and preprocessing steps used for each source |
Section Score: 3/5
Evaluation - Metrics and Fitness for Purpose
| Question | Answer | Comment |
|---|---|---|
| Use case provided | Y | 4 ranging from simple linear queries to discovering mechanistic links |
| Evaluation against other models | N | |
| Defined scope | N | |
| Multiple evaluation methods | N | |
| Accuracy metrics | Y | Informative subgrahs are ranked according to number of supporting publications and literarure co-occurence of entity pairs |
Section Score: 2/5
License Information
| Question | Answer | Comment |
|---|---|---|
| License | Open-Source MIT License |