I'd love to have a discussion about how model credibility and model/parameter uncertainty can be handled together? I would think uncertainty quantification should be included in the model analysis (although maybe it doesn't necessarily fit into "model credibility" per se).

The credibility practices provided are fantastic for the development for the use of models for the forward problem (e.g., simulation). How do we develop or incorporate credibility practices into the inverse problem (e.g., model calibration, accounting for error correlation or model discrepancy, etc)?

My first blush at this is to suggest that its similar to the forward problem, it is the outcomes of the model use (i.e. the context of use) that guides how the credible practices approaches are tailored.  For instance  - Model calibration is often considered a part of testing and validation (at least in the engineering application domain).  In an inverse problem).  An inverse problem will still require verification, version control and review by the community, as well as pedigree of the information that is applied.   In that sense, it may not be the same process for representing credible practice as in a forward problem, however, the categories of credible practice information is still very applicable, even though the approach may need tailored to the environment.

How do we approach rules 2 and 3 in a significantly data-limited environment? We may have physiologically-based mechanisms within a model, but verification & validation of the model using real life data/evidence, especially for intended use cases, can be challenging. Are there recommendations or guidelines for how to approach verification and validation in these cases?

A bit tangential to this topic, but I am curious about current standards, infrastructure, and research around data repositories. Are there guidelines or templates that are being developed and used to guide organization, metadata management, and documentation?

Neat study on the compliance with the 10 rules across a lot of MSM credibility plans.

Q: were the two reviewers' scores/evaluations well correlated with each other?

https://www.imagwiki.nibib.nih.gov/content/10-simple-rules-conformance-…

Many of my potential grant applicants find this form useful to help them shape their grant application so it is more understandable by the grant reviewer, and helps them better communicate their model development process in general.  Good for publication writing too!

One of the biggest challenges for peer review is to actually understanding the model from the context of the PI (otherwise they infer and fill in the blanks based on their own experiences!)

How do we know when we've done enough to establish model credibility? Are there target conformance levels that modelers should aim to achieve (e.g., is "adequate" enough?), or is this primarily dependent on the end user's wants/needs?

How do we engage stakeholders in creating and maintaining standards? How do we encourage the use of these standards? 

Agree credibility is very important not just for reproducibility but also utility to repurpose for uses that were not anticipated.

I think that we are NOT supported to make out modelers reusable- that takes time and effort to put it in a sharing format.

SO, we prefer collaboration ---- i think that is a better way in the case of complex models.

Also--wetlab experiments ar NOT re-performed when those papers are reviewed...why are we holding models to a different more difficult review standard?

Thank you for your great talks. In addition to the provide example on bone fracture I would be interested in learning more about other examples of where the Ten Simple Rules have been used to achieve model credibility. Are there other examples that could be shared broadly to the group?

Reproducibility can be a difficult bar when involving long simulation times, large unwieldy datasets, complex software dependencies, etc.
Can replicating the findings and reaching the same conclusions be more suitable in these edge cases?

Ah a problem near and dear to my work.  FWIW Astronaut data is VERY sparse, so lots of challenges with validation.  

There is not simple answer.  Credibility is not really a "grade for the model" its more a representation of model testing, evaluation rigor with respect to what is achievable and what is needed for it intended use.    So let's say the M&S is used to gain insight into physical mechanisms of the physiological system, then the validation is likely at a gross level i.e. measurable large scale response.  In that case gaps may be analyzed by evaluating alternative modeling hypothesis.  It may be that the validation will remain low until more information for a referent can be obtained. At that point its with respect to the gap in the real world, the referent and the model results. 

Would love to discuss this more and see what ideas bubble up! 

I would say that one should think about this as model and simulation credibility.  In that sense, credibility is a part of the entire life cycle of development and during tis use history.  In a latter the credibility is a quality of the model simulation within the context of that current use.  We recommend it not be treated as a check box , that ones model is credible from some point to infinity.  

Are there any model types that may be difficult to code in SBML?  What alternatives would you recommend for those cases?

The semantic layer IS THE MODEL. Everything else is just implementation details.

Model credibility really should be done on the SEMANTIC LAYER. (Sauro talked ONLY about validation on the code).

Can the approaches used in MEMOTE be applied in areas outside of Systems Biology (and for files other than SBML)? If so, what would be required to do so? And is there documentation available on how MEMOTE was built and/or how it works?

Just a side note: 50% reproducibility appears to compare well with the percent of reproducible experimental papers, based on a quick search.