The Reference Model for Disease Progression

Investigators
Jacob Barhak
Contact info (email)
jacob.barhak@gmail.com
1. Define context(s)
aid in FDA decision making
aid in clinical decision making
aid in clinical trial design
Current Conformance Level / Target Conformance Level
Adequate
Primary goal of the model/tool/database

Merge information from multiple models and validate it against observed clinical trial results. This allows accumulating knowledge from models, outcomes, and human interpretation. 

Biological domain of the model
COVID-19 or Diabetes cardiovascular disease
Structure(s) of interest in the model
Ensemble model
Spatial scales included in the model
Individual, population
Time scales included in the model
Days, Weeks, Months, Years,
Other uses for the model (optional)

The model can be extended for other disease processes or time scales.

The model can be used to asses the quality of models according to how those fit trial results

The best fitting model between data sources is created and this model can be used to calculate our cumulative computational gap of knowledge 

The model can merge human interpretations of trial outcomes definitions to reflect standardization level of reported outcomes.

 

Additional comments about the model’s context (optional)

See: https://simtk.org/projects/therefmodel

2. Data for building and validating the model
Data for building the model Published? Private? How is credibility checked? Current Conformance Level / Target Conformance Level
in vitro (primary cells cell, lines, etc.) N/A
ex vivo (excised tissues) N/A
in vivo pre-clinical (lower-level organism or small animal) N/A
in vivo pre-clinical (large animal) N/A
Human subjects/clinical No Yes Tests/Fitness Extensive
Other: ________________________
Data for validating the model Published? Private? How is credibility checked? Current Conformance Level / Target Conformance Level
in vitro (primary cells cell, lines, etc.) N/A
ex vivo (excised tissues) N/A
in vivo pre-clinical (lower-level organism or small animal) N/A
in vivo pre-clinical (large animal) N/A
Human subjects/clinical Yes No Validation Extensive
Other: ________________________
3. Validate within context(s)
Who does it? When does it happen? How is it done? Current Conformance Level / Target Conformance Level
Verification The Model By simulation Optimization Extensive
Validation The Model By simulation Optimization Extensive
Uncertainty quantification Modeler / Model Repeated simulation Reports provide error statistics Adequate
Sensitivity analysis Modeler Repeated simulation Modeler defines different scenario Adequate
Other:__________
Additional Comments
4. Limitations
Disclaimer statement (explain key limitations) Who needs to know about this disclaimer? How is this disclaimer shared with that audience? Current Conformance Level / Target Conformance Level
Default population modelers publication of techniques Partial
Multiple fitting solutions are possible modelers publication of techniques Partial
Different queries can return different best fitting models modelers publication of techniques Partial
Computing power is a bottleneck modelers publication Extensive
Cumulative Computational Knowledge Gap is currently high Everyone publication + graphics Extensive
5. Version control
Current Conformance Level / Target Conformance Level
Partial
Naming Conventions? Repository? Code Review?
individual modeler Yes - in fact model handels human interpretation now Yes - for public simulation tool previous versions- Model is archived Separately with result archived and publication always references those No yet Simulation tool partially public
within the lab
collaborators No Partial - for public simulation tool previous versions No yet Simulation tool partially public
6. Documentation
Current Conformance Level / Target Conformance Level
Code commented? Extensive
Scope and intended use described? Extensive
User’s guide? Partial - Simulation tool is well documented
Developer’s guide? Adequate- Simulation tool is well documented and development story kept as well as published extensively.
7. Dissemination
Current Conformance Level / Target Conformance Level
Extensive
Target Audience(s): “Inner circle” Scientific community Public
Simulations partial - simulations are described in details, yet code or results are proprietary and patented partial - simulations are described in details, yet code or results are proprietary and patented partial - simulations are described in details, yet code or results are proprietary and patented
Models Partial - model is mostly proprietary, yet techniques disclosed through patents and publications Partial - model is mostly proprietary, yet techniques disclosed through patents and publications Partial - model is mostly proprietary, yet techniques disclosed through patents and publications
Software Partial - early versions simulation tool released, yet techniques disclosed through patents and publications Partial - early versions simulation tool released, yet techniques disclosed through patents and publications Partial - early versions simulation tool released, yet techniques disclosed through patents and publications
Results Adequate - Model results regularly published - recently with interactive graphics. However, some elements are not disclosed on purpose since th model is proprietary. However each publication is traceable back to results. Adequate - Model results regularly published - recently with interactive graphics. However, some elements are not disclosed on purpose since th model is proprietary. However each publication is traceable back to results. Adequate - Model results regularly published - recently with interactive graphics. However, some elements are not disclosed on purpose since th model is proprietary. However each publication is traceable back to results.
Implications of results Extensive Extensive Extensive
8. Independent reviews
Current Conformance Level / Target Conformance Level
Extensive
Reviewer(s) name & affiliation: Cureus editor John R. Adler - Stanford, Jeff Shrager - Stanford ,Jerzy Rozenblit - University of Arizona, Robert Smith? University of Ottawa, SciPy 2012 conference committee, Mount Hood diabetes challenge organizers, PyTexas Organizers / MSM/IMAG poster reviewers , AnacondaCon committee, and many others
When was review performed? July 2020 for COVID-19 model andMultiple Conference Publications for MODSIM / SummerSim / SpringSIM / Mount Hood Diabetes Challenge / SciPy 2014/2015
How was review performed and outcomes of the review? Multiple papers accepted - there are over two dozen publications of the model in multiple venues since 2012. Some reviews are blind - some are public.
9. Test competing implementations
Current Conformance Level / Target Conformance Level
Extensive
Yes or No (briefly summarize)
Were competing implementations tested? Yes! the model automates this exact process - also the model competed multiple times in the Mount Hood Diabetes challenge
Did this lead to model refinement or improvement? Yes! The model does this automatically by merging models
10. Conform to standards
Current Conformance Level / Target Conformance Level
Extensive
Yes or No (briefly summarize)
Are there operating procedures, guidelines, or standards for this type of multiscale modeling? Yes - although current level of practice int he diabetes modeling community is very low and it cannot reproduce its own models.
How do your modeling efforts conform? The practices exceed the current state of the art - ModQuality control measure were published and records kept for tractability in every publication. The Reference Model outperforms those by having formula tests before each change that are documented - moreover, it practices continuous improvement and accumulation of data which reduces errors on the long run.
11. (optional) Additional information to support items 1-10

The Reference Model SimTK portal includes links to may publications in t he news sections about the model:

https://simtk.org/projects/therefmodel

An important recent publication on COVID-19 is available at:

The Interactive link visually show results interactively. 

Every publication has a reproducibility section with traceable information. 

The Reference Model is actually a tool that can asses credibility of models. Recently the the ability to asses credibility of human interpretation of data was added. The idea is that our information in not clean. However, accumulating it and locating the most fitting elements and assembling those together improves our understanding.