Investigators
Denise Kirschner, Jennifer Linderman, JoAnne Flynn, Veronique Dartois
Contact info (email)
kirschne@umich.edu
1. Define context(s)
aid in clinical trial design
identify/explore new therapies
reveal new biological insights
Current Conformance Level / Target Conformance Level
extensive
Primary goal of the model/tool/database
TB causes ~8 millions deaths per year. The project aims to integrate state-of-
the-art computational modeling and experimental data from humans, primates and
rabbits to identify optimal antibiotics, and antibiotic regimens, to improve TB treatment.
More details information can be found at http://malthus.micro.med.umich.edu/MSM/
Biological domain of the model
immunology and pharmacology of tuberculosis
Structure(s) of interest in the model
blood, lung, lymph node
Spatial scales included in the model
molecular, intracellular, cellular, tissue, organ, multi-organ
Time scales included in the model
seconds through years
Other uses for the model (optional)
model framework can be adapted to other biological systems, e.g. solid tumors
2. Data for building and validating the model
| Data for building the model | Published? | Private? | How is credibility checked? | Current Conformance Level / Target Conformance Level |
|---|---|---|---|---|
| in vitro (primary cells cell, lines, etc.) | published in the literature | N/A | multiple controls and historical data | extensive |
| ex vivo (excised tissues) | published | private | credibility checked via controls, multiple measurements, power analysis | extensive |
| in vivo pre-clinical (lower-level organism or small animal) | published | private | credibility checked via controls, multiple measurements, power analysis | extensive |
| in vivo pre-clinical (large animal) | published | private | credibility checked via controls, multiple measurements, power analysis | extensive |
| Human subjects/clinical | published in the literature (not our data) | N/A | N/A | N/A |
| Other: ________________________ |
| Data for validating the model | Published? | Private? | How is credibility checked? | Current Conformance Level / Target Conformance Level |
|---|---|---|---|---|
| in vitro (primary cells cell, lines, etc.) | N/A | N/A | N/A | N/A |
| ex vivo (excised tissues) | N/A | N/A | N/A | N/A |
| in vivo pre-clinical (lower-level organism or small animal) | published | private | credibility checked via controls, multiple measurements, power analysis | extensive |
| in vivo pre-clinical (large animal) | N/A | private | credibility checked via controls, multiple measurements, power analysis | extensive |
| Human subjects/clinical | published in the literature (not our data) | N/A | N/A | N/A |
| Other: ________________________ |
3. Validate within context(s)
| Who does it? | When does it happen? | How is it done? | Current Conformance Level / Target Conformance Level | |
|---|---|---|---|---|
| Verification | model verification occurs within Kirschner/Linderman groups as multiple users and full-time computer programmer debug | test each other’s code | test competing implementations | extensive |
| Validation | Kirschner/Linderman/Flynn/Dartois groups | Frequent communication between groups allows iterative comparison of data and model results | Frequent communication between groups allows iterative comparison of data and model results | extensive |
| Uncertainty quantification | Kirschner/Linderman groups | after model built and calibrated | global uncertainty analysis using Latin Hypercube Sampling techniques as published and shared on our website | extensive |
| Sensitivity analysis | Kirschner/Linderman groups | after model built and calibrated | global uncertainty and sensitivity analysis using PRCC as published and shared on our website | extensive |
| Other:__________ | ||||
| Additional Comments |
4. Limitations
| Disclaimer statement (explain key limitations) | Who needs to know about this disclaimer? | How is this disclaimer shared with that audience? | Current Conformance Level / Target Conformance Level |
|---|---|---|---|
| Clinical results measure outcomes at the host level, and our computational model GranSim fundamentally simulates treatment and sterilization at the granuloma level. | The relevance of our results relies on the assumption that treatment at the granuloma scale is indicative of treatment at the host scale | Explained and shared with all audiences | extensive |
| Our model simulates primary granulomas and does not fully capture the full complexity of pulmonary lesions observed in TB disease | It is appreciated that non-replicating and persisting Mtb are critical targets to achieve full sterilization of lesions, and while we observe this in our model, their importance could be amplified in cavitary disease or fibrotic lesions | Explained and shared with all audiences | extensive |
| Directly relating in vitro antimicrobial activity to in vivo efficacy does not necessarily capture the full range of antimicrobial activity that occurs in a granuloma and may partially account for any discrepancies between our simulation results and clinical observations | Explained and shared with all audiences | Explained and shared with all audiences | extensive |
| Our model currently assumes no interaction between antibiotics, and synergistic or antagonistic combinations may be relevant in determining regimen efficacy | We are working to incorporate these interactions in our model now | Explained and shared with all audiences | extensive |
| Using our model to predict treatment efficacy is dependent on the acquisition of relevant data sets for new and repurposed drugs | Explained and shared with all audiences | Explained and shared with all audiences | extensive |
5. Version control
| Current Conformance Level / Target Conformance Level |
|---|
| Extensive. We have been perfecting this process for over a decade using Subversion as our standard |
| We use Subversion for version control for over a decade and once an initial name is assigned to a code, all future versions are indices of that original name. This is true for all individual modelers, within the lab, and with our collaborators. For code review, our experienced full-time programmer performs regular code review of all code written in the groups. Further, in practice that ends up catching not only bugs but also enforcing sustainable coding practices. We have also had two groups examine our code as discussed below in Section 8. |
| Naming Conventions? | Repository? | Code Review? | |
|---|---|---|---|
| individual modeler | |||
| within the lab | |||
| collaborators |
6. Documentation
| Current Conformance Level / Target Conformance Level | |
|---|---|
| Code commented? | We have multiple codes in the lab for each distinct model. Each code for each model is heavily commented. This also is part of our code review process (above) ; namely, that there is complete commenting of each new piece of code added to the program at that stage. |
| Scope and intended use described? | We have multiple codes in the lab for distinct models. Each major code/model has a corresponding website URL -one that is external and one internal. On the external website is housed a clear description of the scope and intended use for each model and an executable and also data for the model. |
| User’s guide? | We have one internal URL for all of our lab codes. Each model is extensively documented there and also the uses and usability is described in detail within. This documentation comprises more than 500 pages of documentation. |
| Developer’s guide? | We have one internal URL for all of our lab codes. Each model is extensively documented there and also the developer’s guide is described in detail within This documentation comprises more than 500 pages of documentation. |
7. Dissemination
| Current Conformance Level / Target Conformance Level |
|---|
| Adequate |
| Target Audience(s): | “Inner circle” | Scientific community | Public |
|---|---|---|---|
| Simulations | All of our work is published with model assumptions, formulations,equations (when using equation-based models), and psuedo-code for agent-based modeling code | Results and implications are shared and disseminated with corresponding websites for each paper that give extra information regarding executables of code, time lapse movies of simulations, parameter values, and model output. | Results and implications are shared and disseminated with corresponding websites for each paper that give extra information regarding executables of code, time lapse movies of simulations, parameter values, and model output. |
| Models | with our ODE models: we share with “Inner circle” , scientific community, public (e.g. SMBL code) | with our ODE models: we share with “Inner circle” , scientific community, public (e.g. SMBL code) | with our ODE models: we share with “Inner circle” , scientific community, public (e.g. SMBL code) |
| Software | We have shared all of our analysis codes online for 15 years for uncertainty and sensitivity analysis in both Matlab and now just released R codes | We have shared all of our analysis codes online for 15 years for uncertainty and sensitivity analysis in both Matlab and now just released R codes | We have shared all of our analysis codes online for 15 years for uncertainty and sensitivity analysis in both Matlab and now just released R codes |
| Results | share | share | share |
| Implications of results | via publications | via publications | via publications |
8. Independent reviews
| Current Conformance Level / Target Conformance Level |
|---|
| extensive |
| Reviewer(s) name & affiliation: | Code reviewing: Dr. Chang Gong , Johns Hopkins University from the Alexander Popel group, has provided and independent evaluation of our code: Evaluation on: ABM, ODE, PDE, C++, object-oriented programming |
|---|---|
| When was review performed? | Meetings: August 11, 2017 and January 30th, 2018 in person and spent 2 days going over our codes in detail. |
| How was review performed and outcomes of the review? | N/A |
| Code and website reviewing | Drs. Blemker and Peirce-Cottler, University of Virginia, are providing a co-review process with us on our projects for all areas of model credibility. 2018 meetings: February 2018 (2 meetings), March 2018 and April 2018. During that time the Peirce-Cottler lab also spent a day on our group website, using our executables, documentation and time-lapse simulations to test the clarity and useability of our publically-shared materials. 2019 meetings: January 2019, March 2019 and April 2019. These discussions all revolve around model evaluation. We focus on model formulations, modeling choices, model credibility planning, code versioning, etc. In addition, we have created a google doc that has cataloged all of our meetings and findings for future reference and reflections on this process. |
| Manuscript reviewing | In October 2019 with the UVA team we reviewed early drafts of each other’s manuscripts (prior to submission). By reviewing manuscripts at this early stage, we have the opportunity to reflect deeply on model choices and assumptions, implementation, model credibility, shared practices, dissemination, etc. Our paper, entitled A computational model tracks whole-lung Mycobacterium tuberculosis infection and predicts factors that inhibit dissemination, was just accepted in PLoS Computational Biology. |
9. Test competing implementations
| Current Conformance Level / Target Conformance Level |
|---|
| Adequate |
| Yes or No (briefly summarize) | |
|---|---|
| Were competing implementations tested? | First, we have had at least 4 independent groups implement our granuloma simulator from our rules and documentation (groups led by: G. Magombedze, Elebeoba E. May, Radda Savic, Ruth Bowness). They were able to successfully implement our model and recapitulate key results (as well as take the work in their own independent directions). |
| Did this lead to model refinement or improvement? | This is a strong indication that our implementations were done well and that key information was shared publicly. |
| Were competing implementations tested? (2) | Second, in our models we implement molecular diffusion (e.g. of cytokines, antibiotics) by solving the relevant partial differential equations (PDEs). Simulating chemical diffusion is our main computational bottleneck. We have been working to optimize our code using different PDE solver algorithms. To date these have provided an aggregate speedup for a single model run of 1.8 when not using antibiotics in a simulation, and 10.1 when using antibiotics. We also implemented an implicit method diffusion algorithm, Crank-Nicolson ADI {Peaceman, 1955}. |
| Did this lead to model refinement or improvement? (2) | We are currently comparing the accuracy of different methods; we now have 5 implemented. We expect providing a choice of solvers in different situations will improve accuracy and performance. |
10. Conform to standards
| Current Conformance Level / Target Conformance Level |
|---|
| N/A |
| Yes or No (briefly summarize) | |
|---|---|
| Are there operating procedures, guidelines, or standards for this type of multiscale modeling? | There are no additional operating procedures, guidelines or standards for this type of MSM beyond what is described above. |
| How do your modeling efforts conform? | There are no additional operating procedures, guidelines or standards for this type of MSM beyond what is described above. |