Webinar: Ten Simple Rules of Credible Practice of Modeling and Simulation in Healthcare: Application to Bone Remodeling and Heart Valve Modeling
The Committee on Credible Practice of Modeling & Simulation in Healthcare (CPMS) was honored to give a webinar in May at the ROSA Worldwide Webinar Series: Impact of Modeling & Simulation in Drug Development on May 17, 2018. THe topic of the webinar discussed the application of our ten simple rules for promoting credibility of modeling and simulation in healthcare.
[WEBINAR] Validating computational models
How do we verify that our models simulate the important features of a biological system?
Neural Encoding Model System (NEMS)
10.1371/journal.pcbi.1007430
Microconnectomics of motor cortex: a multiscale computer model
We developed a model of primary motor cortex (M1) microcircuits [1] with over 10,000 biophysically detailed neurons and 30 million synaptic connections. It simulates a cylindric cortical volume with a depth of 1350 μm and a diameter of 300 μm. Neuron densities, classes, morphology and biophysics, and connectivity at the long-range, local and dendritic scale were derived from experimental data published in over 30 studies. The model was developed using the NetPyNE tool, which facilitated the integration of this complex experimental data at multiple scales. Our model exhibited spontaneous neural activity patterns and oscillations consistent with M1 data. Neural activity depended on cell class, cortical layer and sublaminar location. Different output dynamics were seen when the network was driven by brief activation of particular long-range inputs, or in the setting of different neuromodulatory conditions. Results yielded insights into circuit information pathways, oscillatory coding mechanisms and the role of HCN in modulating corticospinal output.
LFP revealed physiological oscillations in delta (0.5-4 Hz) and high beta to low gamma (25-40 Hz) ranges across layers and populations. Oscillations occurred in the absence of rhythmic external inputs, emergent from neuronal biophysical properties and circuit connectivity. Filtering the LFP signal from the electrode located in upper L5B revealed phase-amplitude coupling of fast oscillations on delta wave phase. LFP spectrogram demonstrated that the fast oscillations occurred robustly during the time course of simulations. Strong LFP beta and gamma oscillations are characteristic of motor cortex activity, and have been found to enhance signal transmission in mouse neocortex. Phase-amplitude coupling may help integrate information across temporal scales and across networks.
Analysis of firing dynamics and information flow in our model confirmed and extended our understanding of information flow in cortical microcircuits. Consistent with existing models, sensory-related long-range inputs targeted superficial layers which in turn projected to deeper layers. Our simulations, however, provided further details: information flow was cell-class specific, going unidirectionally from IT to PT cells; sublaminar-specific, with superficial ITs targeting primarily the upper portion of L5B PT cells; and oscillation frequency-specific, with Granger causality peaks occurring at shifted beta/gamma range frequencies for different internal connections.
The model parameters were constrained by literature using data from over 40 publications. The model results have been compared against in vitro and in vivo experimental data. See publication: https://www.biorxiv.org/content/10.1101/201707v4
The tool (NetPyNE) to develop the model has been validated with a methods publication: https://elifesciences.org/articles/44494
NetPyNE tool for multiscale modeling
Dura-Bernal S, Neymotin SA, Suter BA, Shepherd GMG, Lytton WW. Multiscale dynamics and information flow in a data-driven model of the primary motor cortex microcircuit. bioRxiv. 2019. p. 201707. doi:10.1101/201707
Dura-Bernal S, Suter BA, Gleeson P, Cantarelli M, Quintana A, Rodriguez F, et al. NetPyNE, a tool for data-driven multiscale modeling of brain circuits. eLife. 2019;8. doi:10.7554/eLife.44494
Gleeson P, Cantarelli M, Marin B, Quintana A, Earnshaw M, Sadeh S, et al. Open Source Brain: A Collaborative Resource for Visualizing, Analyzing, Simulating, and Developing Standardized Models of Neurons and Circuits. Neuron. 2019. doi:10.1016/j.neuron.2019.05.019
Peng GCY, Alber M, Buganza Tepole A, Cannon WR, De S, Dura-Bernal S, et al. Multiscale Modeling Meets Machine Learning: What Can We Learn? Arch Comput Methods Eng. 2020. doi:10.1007/s11831-020-09405-5
Sivagnanam S, Gorman W, Doherty D, Neymotin SA, Fang S, Hovhannisyan H, et al. Simulating large-scale models of brain neuronal circuits using Google Cloud Platform. 2020. doi:10.31219/osf.io/m8vza. PEARC'20
DOI: 10.1101/201707
DOI: 10.7554/eLife.44494
Virtual Population Obesity Prevention (VPOP) Labs: Computational, Multi-Scale Models for Obesity Solutions
The VPOP (Virtual Populations for Obesity Prevention) project has involved developing various multi-scale computational simulation models to help decision makers address questions and issues related to obesity prevention and control.
As described in Nutrition Reviews and in Forbes, obesity is a multi-scale systems problem. The relevant scales are interconnected and include factors such as physiology, behavior, culture, social networks, environment, policy, and economics. Changing only parts of the system may have little effect—or, even worse, unintended consequences.
While the human brain can see single cause and single effect outcomes, unaided, it is difficult to understand the multi-scale factors and relationships with obesity. Multi-scale systems approaches, models, and tools that the VPOP project has developed designs and tests various obesity prevention measures, policies and interventions within the safety of a computer before trying them in real life, which can save considerable time, effort, and resources.
Here are some examples of questions that the VPOP project has used multi-scale modeling to answer and address:
- Can sugar-sweetened beverage (SSB) warning labels impact obesity prevalence? The VPOP project built virtual representations of Baltimore, Philadelphia, and San Francisco to help city officials determine the potential impact of introducing SSB warning labels. The results from the VPOP’s multi-scale simulation model show when SSB warning labels appeared in all store locations in the three cities and reduced the chances of someone purchasing a sugary drink by 8%, obesity prevalence went down overtime by 1.69% in Baltimore, 4.08% in San Francisco, and 2.17% in Philadelphia while overweight prevalence dropped by 1.39% in Baltimore, 3.1% in San Francisco, and 0.36% in Philadelphia. Full paper published in AJPM.
- Can following the current formula-feeding and solid food recommendations result in infants who are overweight or have obesity? Since most of the available formula-feeding and solid food guidelines for infants vary significantly, the VPOP project was utilized to evaluate how this can affect infant weight gain. The National Institutes of Health (NIH) and media outlets covered the results, which call for more consistent guidelines across the board. Specifically, the findings indicate that following each of the formula-feeding guidelines in the study led to infants becoming overweight/with obesity by 6 months old. Findings also suggested that following each of the solid food guidelines in the study for just a few months would drive the average body mass index (BMI) for the infants into the overweight category. Full papers published in Pediatric Research and AJPM.
- What is the impact of youth physical inactivity in the U.S. and what is the value of increasing physical activity among children to different degrees? VPOP has developed a multi-scale, national model to estimate the cost of youth physical inactivity. Results from the model showed even modest increases in youth’s physical activity levels could avert billions in healthcare and other costs. For example, if half of the youth in the U.S. aged 8-11 years old were physically active, VPOP’s multi-scale model results showed this would avert $8.1 billion in direct medical costs and $13.8 billion in productivity losses. Increasing to 75% would avert $16.6 billion and $23.6 billion, respectively. The findings sparked heavy media coverage in outlets including New York Times, USA Today, and Yahoo News. Leaders in the field continue to bring the VPOP project’s generated results to the table to show decision makers the substantial savings attributed to getting and keeping youth physically active. Full paper published in Health Affairs.
- What is the value of programs to increase youth sports participation in different locations? The Aspen Institute’s Sports & Society Program worked with the VPOP project for its ProjectPlay State of Play Report. The VPOP’s multi-scale computational simulation models were utilized at local levels to generate numbers for East Harlem, New York and Southeast Michigan in order to better understand the impact of increasing youth sport participation. The projections are used to show stakeholders the benefits of more youth meeting the CDC’s physical activity guidelines. Results from the VPOP project showed if 50% of youth in East Harlem get and stay active into adulthood, $36.1 million in direct medical costs and $38.5 million in productivity losses would be averted. If 50% of youth in Southeast Michigan get and stay active into adulthood, the VPOP project’s results show this will avert $1.8 billion in direct medical costs and $1.9 billion in productivity losses.
- What is the value of increasing physical activity in a pre-existing program? Girl Scouts of Central Maryland (GSCM) worked with the VPOP projectto better understand the impact of its original Fierce & Fit program and how changes to the sessions could result in even greater benefits. Based on the results from the VPOP project, GSCM decided to change its Fierce & Fit program sessions six-fold. Results showed that this updated version of the program could save $84,828 in lifetime direct medical costs and $81,365 in lifetime productivity losses. By doubling the length, frequency and increasing the amount of physical activity during its sessions six-fold, the program is expected to deliver cost savings nearly 2.5 times greater than the cost of implementing the program. Full paper published in Obesity.
- Can crime have an impact on physical activity levels and obesity? The National Heart, Lung, and Blood Institute (NHLBI) wanted to better understand the impact crime has on African American women’s physical activity levels and obesity prevalence in Washington D.C. Results from the VPOP project showed that reducing crime so that more physical activity locations were safely accessible could help decrease the annual rise in obesity prevalence. For example, results from VPOP’s multi-scale computational model suggest that reducing crime in Washington D.C. so more physical activity locations were accessible (increasing from 10% to 50%) decreased the annual rise in obesity prevalence in African American women by 2.69%. Full paper published in Obesity.
- What is the value of reducing an adult’s weight at different ages? The VPOP project built a multi-scale computational simulation model to represent the U.S. adult population to show the lifetime costs and health effects for an individual with obesity, overweight, and healthy weight statuses at ages 20 through 80. The results from the VPOP project show that a 20-year-old adult who goes from having obesity to being overweight would save an average of $17,655 in direct medical costs and productivity losses over their lifetime. If the same person were to go from having obesity to a healthy weight, the average savings would be $28,020. The expected average savings are even higher—$18,262 and $31,447, respectively—for a 40-year-old. Full paper published in Obesity.
For more information on VPOP multi-scale computer simulation models, email Bruce Y. Lee, MD, MBA at BruceLeeMDMBA@gmail.com.
We have organized the model credibility as face validity, criterion validity and convergence and divergence validity.
Face Validity: Involves developing the model in close collaboration with stakeholders who have intimate knowledge of the system
Our team currently works with the Baltimore City Health Department, New York City health department, Laureus Sport for Good, Aspen Institute, National Heart Lung and Blood Institute, Girl Scouts of Central Maryland and the Philadelphia Health Department. We regularly discuss model development and results with each partner, to ensure the model is reflecting the system based on their on-the ground understanding of it.
Criterion Validity: Reproducing observed trends at multiple levels
|
Sample Factors |
Sample Sources |
|
Overweight and obesity prevalence
|
National Health And Nutrition Examination Survey (NHANES)
Youth Risk Behavior Surveillance System (YRBSS)
Parish State of Youth Sports and Physical Activity New Orleans Sport for Development Coalition 2015.
Rivera JÁ. Review Childhood and adolescent overweight and obesity in Latin America: a systematic review. THE LANCET Diabetes & Endocrinology. 2014;2:321-332.
Baltimore City: Demographics and Social Determinants of Health (2005-2009). In: Maryland Department of Health and Mental Hygiene.
|
|
Proportion of students purchasing unhealthy snacks
|
NHANES
Lent MR, Vander Veur S, Mallya G, et al. Corner store purchases made by adults, adolescents and children: items, nutritional characteristics and amount spent. Public Health Nutr
|
|
Average distance students regularly walk to store
|
How far will people walk to facilities in their local neighbourhoods. AUSTRALIA: WALKING THE 21ST CENTURY, INTERNATIONAL CONFERENCE, 2001, PERTH, WESTERN AUSTRALIA, VOL 3; 2001.
Burke M, Brown A. Distances people walk for transport. Road & Transport Research: A Journal of Australian and New Zealand Research and Practice 2007;16(3):16
|
|
Average daily SSB consumption levels
|
NHANES |
|
Likelihood of individuals using health apps to meet physical activity recommendations |
National Cancer Institute’s 2015 Health Information National Trends Survey – HINTS |
|
Proportion of population exercising |
Behavioral Risk Factor Surveillance System National Survey – BRFSS)
Parish State of Youth Sports and Physical Activity New Orleans Sport for Development Coalition 2015.
|
|
Time spent in PE class |
Gharib H, Galavíz KI, Lee RE, et al. The Influence of Physical Education Lesson Context and Teacher Behaviour on Student Physical Activity in Mexico. 2015;2041:160-164
|
Convergence and Divergence Validity: Comparing with other models and calculations
We have adapted the metabolic model from Hall et al(1) and Rhamandad et al(2) and validated using the methods described below:
The Rahmandad model was validated extensively for infants and children using empirical energy requirements and expenditure, weight, fat mass and fat free mass, and basal metabolic rate data from sources including Butte(3), Torun(4), and Ellis, et al(5). Our team followed the same validation process upon implementing the metabolic model, validating our baseline scenario against the energy inputs and growth trajectories reported in Butte et al, before, and also after we made changes to the model.We also compared our results against data from the 2004-2014 National Health and Nutrition Examination Survey (NHANES) and the model-estimated weights were not statistically different from those reported in NHANES.
- Hall, K.D., et al., Quantifying the Dynamics of Childhood Growth and Obesity.Lancet Diabetes and Endocrinology 2013. 1(2): p. 97-105.
- Rahmandad H. Human growth and body weight dynamics: an integrative systems model. PLoS ONE. 2014;9(12):e114609.
- Butte, N.F., et al., Body composition during the first 2 years of life: an updated reference.Pediatr Res, 2000. 47(5): p. 578-85.
- Torun, B., Energy requirements of children and adolescents.Public Health Nutrition, 2005. 8(7A): p. 968-993.
Ferguson, M.C., Morgan, M.J., O’Shea, K.J., Winch, L., Siegmund, S.S., Solano Gonzales, M., Randall, S., Hertenstein, D.L., Montague, V., Woodberry, A., Cassatt, T. and Lee, B.Y. (2020), Using Simulation Modeling to Guide the Design of the Girl Scouts Fierce & Fit Program. Obesity, 28: 1317-1324.
Lee, B. Y., A. Adam, E. Zenkov, D. Hertenstein, M. C. Ferguson, P. I. Wang, M. S. Wong, P. Wedlock, S. Nyathi, J. Gittelsohn, S. Falah-Fini, S. M. Bartsch, L. J. Cheskin and S. T. Brown (2017). "Modeling the economic and health impact of increasing children's physical activity in the United States." Health Aff (Millwood) 36(5): 902-908.
Lee, B. Y., M. C. Ferguson, D. L. Hertenstein, A. Adam, E. Zenkov, P. I. Wang, M. S. Wong, J. Gittelsohn, Y. Mui and S. T. Brown (2017). "Simulating the Impact of Sugar-Sweetened Beverage Warning Labels in Three Cities." Am J Prev Med, 54(2) 197-204.
Powell‐Wiley, T.M., Wong, M.S., Adu‐Brimpong, J., Brown, S.T., Hertenstein, D.L., Zenkov, E., Ferguson, M.C., Thomas, S., Sampson, D., Ahuja, C., Rivers, J. and Lee, B.Y. (2017), Simulating the Impact of Crime on African American Women's Physical Activity and Obesity. Obesity, 25: 2149-2155.
Ferguson, M. C., K. J. O'Shea, L. D. Hammer, D. L. Hertenstein, N. J. Schwartz, L. E. Winch, S. S. Siegmund and B. Y. Lee (2019). "The Impact of Following Solid Food Feeding Guides on BMI Among Infants: A Simulation Study." American Journal of Preventive Medicine 57(3): 355-364.
Ferguson, M. C., K. J. O’Shea, L. D. Hammer, D. L. Hertenstein, R. M. Syed, S. Nyathi, M. S. Gonzales, M. Domino, S. S. Siegmund, S. Randall, P. Wedlock, A. Adam and B. Y. Lee (2020). "Can following formula-feeding recommendations still result in infants who are overweight or have obesity?" Pediatric Research.