Evaluation for cl-kg
Evaluator: Automated Evaluation
Evaluated on: 2026-01-06
⚠️ Automated Evaluation: This evaluation was generated automatically using an AI-based system. It is distinct from manual evaluations curated by human experts. Please review findings carefully and report any inaccuracies.
Evaluation Criteria: This evaluation uses the KG-Registry evaluation rubric as described in Cortes et al. (2025) . The rubric assesses knowledge graphs across multiple dimensions including access, provenance, documentation, maintenance, and fitness for purpose.
Access Level and Types
| Question | Answer | Comment |
|---|---|---|
| Access to data outside of the knowledge graph | Y | Cell Ontology via OBO repository and CellXGene integration for single cell transcriptomics data from multiple studies |
| API or online access to the knowledge graph | N | Neo4j instance accessible via browser interface at cellular-semantics.sanger.ac.uk but no documented REST or programmatic API |
| Multiple access options available | N | Single access mechanism through Neo4j browser; no SPARQL endpoint, downloads, or alternative query interfaces documented |
| Source code availability | Y | Complete source code available at github.com/Cellular-Semantics/CL_KG with Apache 2.0 license enabling reproducibility |
| Downloadable knowledge graph | N | No bulk export format documented; Neo4j queries only method to extract data from KG instance |
Section Score: 2/5
Provenance of Nodes and Edges
| Question | Answer | Comment |
|---|---|---|
| Source list provided | Y | Cell Ontology and CellXGene datasets clearly identified as primary sources for cell type and transcriptomics data |
| Source versions information | N | No documentation of specific Cell Ontology version or CellXGene data snapshot dates used in KG construction |
| Import dependencies | Y | GitHub repository shows integration pipeline and dependencies for ontology alignment and annotation |
| Node and edge sources | Y | Cell types traceable to Cell Ontology identifiers; transcriptomics associations sourced from CellXGene studies |
| Edges deduplication | N | No documentation of deduplication strategy for redundant cell type assignments across studies |
| Triples source details | N | Neo4j property graph structure but no RDF schema or formal ontology mappings documented |
| Edge type schema | N | Relationship types between cells and ontology terms not formally mapped to standard ontologies |
Section Score: 3/7
Documented standards, schema, construction
| Question | Answer | Comment |
|---|---|---|
| Biological usable data | Y | Cell type hierarchies and transcriptomics annotations directly applicable to single cell analysis and cell biology research |
| Resolvable IDs | Y | Uses stable Cell Ontology identifiers and CellXGene cell type standardized vocabulary resolvable externally |
| Construction documentation | N | ETL methodology not formally documented beyond source code; integration approach not transparent in publications |
| Transformation documentation | N | How CellXGene transcriptomics hierarchies map to Neo4j structure not documented |
| Schema used | N | Neo4j graph schema not formally published; no RDF or OWL specification available for semantic integration |
Section Score: 2/5
Update frequency and versioning
| Question | Answer | Comment |
|---|---|---|
| Stable versions | Y | GitHub releases available with tagged stable versions enabling reproducibility |
| Public tracker information | Y | GitHub issues tracker active; Cellular Semantics project maintains the resource |
| Knowledge graph contact information | Y | Ugur Bayindir (ORCID 0000-0002-6012-3729) available for contact; active development team |
| Updated annually | Y | Repository shows regular commits and updates reflecting CellXGene data additions |
| Prior versions access | Y | Full Git history and release archives available for reproducibility |
Section Score: 5/5
Evaluation - Metrics and Fitness for Purpose
| Question | Answer | Comment |
|---|---|---|
| Use case provided | Y | Clear use case: cell type annotation and standardization for single cell transcriptomics studies |
| Evaluation against other models | N | No comparison with other cell type ontologies or cell annotation standards (e.g., LSAC, CellTypist) |
| Defined scope | Y | Focused scope on Cell Ontology integration with CellXGene transcriptomics data |
| Multiple evaluation methods | N | No systematic validation of cell type annotation accuracy against independent datasets |
| Accuracy metrics | N | No reported precision/recall for cell type predictions or annotation completeness metrics |
Section Score: 2/5
License Information
| Question | Answer | Comment |
|---|---|---|
| License |