GSR: Certification Program

The Genetic Simulator Resources (GSR) catalogs more than 100 genetic simulators for a wide variety of applications. These simulators differ not only in simulation-related features such as simulation methods, input and output formats, assumptions, and applications, but also in usability measures such as documentation, user-friendliness, support, and level of adoption by the research community. To encourage the development and dissemination of high quality genetic simulation programs and help end users identify the most appropriate simulator for their research topics and encourage the development and dissemination of high quality simulation resources, a GSR-certification program was proposed in a National Cancer Institute (NCI) sponsored workshop "Genetic Simulation Tools for Post-Genome Wide Association Studies of Complex Diseases", Bethesda, Maryland on March 11–12, 2014" and was subsequently implemented by the GSR team.

What are GSR certifications?

The GSR certification program sets standards for genetic data simulation programs that are developed and used by researchers in related disciplines such as evolutionary and population genetics, bioinformatics, and genetic epidemiology. A GSR certified simulator ensures users that the simulator is reasonably easy to install and use, has decent documentation and support, and has been used by other researchers in the community. A genetic data simulator can receive four certificates if it meets criteria in the areas of accessibility, documentation, application, and support, respectively. A GSR simulator will be evaluated for each certificate (e.g. documentation) individually, but only simulators which are considered certified for all four criteria are classified as "GSR certified".

Why should my simulator be certified?

GSR Certified simulators will be listed prominently on the GSR website. As the author of a genetic data simulator, having your simulator certified by GSR can

  1. Improve the usability and visibility of your simulator by exposing details of your simulator in GSR and prepare your documentation with specified guidelines
  2. Help dissemination of your simulator because a user would much prefer a certified simulator to other simulators with unknown quality
  3. Help advance the science of genetic simulation by fostering standards for simulation programs for the field

Authors of GSR certified simulators can also place a special "GSR Certified" tile on their websites:

GSR-certified

What are the requirements for GSR Certification?

Accessibility:

This certificate ensures that the simulator is openly available to all interested users, and is easy to install and use. Simulators with this certificate should meet the following criteria:

  1. Accessible homepage and download link. We encourage but do not require simulator to be hosted on public servers such as sourceforge.net and github.com.
  2. Clear license information and accessible link to source resource or source code repository. Source code that is not publicly accessible, even if it is available upon request, is not considered eligible.
  3. Accessible and clear instructions on how to download and install the program. Installation instructions should be available directly from the website.
  4. Easy installation on supported platforms, following commonly used installation process. A typical installation process should take less than 15 minutes to complete, including time to download source or binary packages, and required third party libraries and programs. Authors of complex simulation packages are recommended to provide binary installers for supported platforms, to bundle third-party libraries and tools, and/or use well accepted package distribution channels such as CRAN, CPAN, pip, and conda.

Criteria 3 and 4 are not applicable to simulators that provide only a web interface, and java (or similar) applications that start directly from a web browser.

Documentation:

This certificate ensures that the simulator is well documented so that users can quickly determine if the simulator provides needed features and learn how to use it. Simulators with this certificate should meet the following criteria:

  1. Complete GSR entry with accurate description of simulation targets, simulation methods, input/output, programming language, and license. The entry should reflect the most recent release and provide a complete list of features.
  2. Easily accessible documentation that contains information on
    1. Supported platform, dependency, and installation instructions.
    2. Type of license
    3. Target of simulation (DNA, RNA, type of genetic variants etc)
    4. Typical application areas with clear description of background and research questions that could be tackled by the simulator.
    5. Input and output formats. If the simulator uses application-specific format, it should provide sample input and output files with details description of the file formats.
  3. The documentation can be in any format but should be accessible directly from the simulator's homepage.

Application:

This certificate ensures that the software simulator is peer reviewed, is reasonably user-friendly to be useful to peer researchers, and has been used by researchers in the scientific community. The application certificate will be evaluated based on the application references you provide in GSR. Simulators with this certificate should meet the following criteria:

  1. Published in peer-reviewed journals. The publication does not have to focus on the simulator, but should describe the simulation method and applications clearly.
  2. List typical applications in the "Applications" tab of the GSR entry of the simulator. The author needs to identify typical application(s) of the simulator and list publications that use the simulator on GSR. The publications listed should use the simulator to simulate data. Publications that only mention the simulator (e.g. in a discussion section or review article) should not be included because these publications do not reflect use of the simulator, or an application.
  3. Used and cited by at least 5 external users in the past 5 years, or twice in the past year, as demonstrated by citations listed in the Applications tab. External users refer to people or researchers who are not one of the simulator authors or directly affiliated with the authors.

Support:

This certificate ensures that the authors of the simulator are actively maintaining the simulator and are actively addressing users’ questions, bug reports and feature requests. Simulators with this certificate should meet the following criteria:

  1. Available contact information of authors, maintainers, or mailing list
  2. Evidence of response to user inquiries and support for users, as shown from publicly available sources such as mailing list archives and forum posts.
  3. Response to GSR communications such as comments raised by visitors of GSR.

How to get certified?

The GSR certificates are awarded by the GSR certification committee after an initial review by the GSR team. The certification process is author-initiated. If you are the author of a genetic simulation program, you can:

  1. Register your simulator in GSR or update information for your simulation program if it has already been registered. An up-to-date GSR entry is required for many components of certification.
  2. Collect usage information of your simulator and enter citations to the Applications tab of the GSR entry of your simulator.
  3. Review your simulator with respect to the certification criteria and provide any additional information that might assist in the evaluation of your simulator (e.g., location of license information and URL to mailing list archive).
  4. Send an email to gsr@mail.nih.gov and express your interest in having your simulation program certified. Please state clearly the name of the simulation program to be considered, the certificates you want to be (re)evaluated, and any additional information that might assist in the evaluation of your simulator (e.g., location of license information and URL to mailing list archive). Please refer to the requirements section as a guide.

A GSR moderator will verify the information and forward the simulator to the GSR certification committee. The GSR moderator might contact you for further details or suggest changes to the simulator or website for certification. Decisions regarding GSR certificates must be unanimous by the committee.

After a program has been certified, the GSR certification may be removed from a program if the simulator no longer meets criteria of the certification program. Prior to removal of the GSR certificates, authors of such simulators will be notified of potential changes in status of the simulator and will be provided with a two-month grace period to re-certify the program. Review of certified simulators is anticipated annually.