The GSR Certification Program

The Genetic Simulator Resources (GSR) catalogs more than 100 genetic simulators for a wide variety of applications. These simulators differ not only in simulation-related features such as simulation methods, input and output formats, assumptions, and applications, but also in usability measures such as documentation, user-friendliness, support, and level of adoption by the research community. To encourage the development and dissemination of high quality genetic simulation programs and help end users identify the most appropriate simulator for their research topics and encourage the development and dissemination of high quality simulation resources, a GSR-certification program was proposed in a National Cancer Institute (NCI) sponsored workshop "Genetic Simulation Tools for Post-Genome Wide Association Studies of Complex Diseases", Bethesda, Maryland on March 11–12, 2014" and was subsequently implemented by the GSR team.

What are GSR certifications?

The GSR certification program sets standards for genetic data simulation programs that are developed and used by researchers in related disciplines such as evolutionary and population genetics, bioinformatics, and genetic epidemiology. A GSR certified simulator ensures users that the simulator is reasonably easy to install and use, has decent documentation and support, and has been used by other researchers in the community. A resource can receive four certificates if it meets criteria in the areas of accessibility, documentation, application, and support, respectively. A GSR resource will be evaluated for each certificate (e.g. documentation) individually, but only resources which are considered certified for all four criteria are classified as "GSR certified".

Return to top

Why should my resource be certified?

GSR Certified simulators will be listed prominently on the GSR website. As the author of a simulation resource, having your simulator certified by GSR can

  1. Improve the usability and visibility of your simulator by exposing details of your resource in GSR and prepare your documentation with specified guidelines
  2. Help dissemination of your simulator because a user would much prefer a certified simulator to other simulators with unknown quality
  3. Help advance the science of genetic simulation by fostering standards for simulation programs for the field

Authors of GSR certified resources can also place a special "GSR Certified" tile on their websites:


Return to top

What are the requirements for GSR Certification?

Accessibility:

This certificate ensures that the software resource is openly available to all interested users, and is easy to install and use. Simulators with this certificate should meet the following criteria:

  1. Accessible homepage and download link. We encourage but do not require resources to be hosted on public servers such as sourceforge.net and github.com.
  2. Clear license information and accessible link to source resource or source code repository. Source code that is not publicly accessible, even if it is available upon request, is not considered eligible.
  3. Accessible and clear instructions on how to download and install the program. Installation instructions should be available directly from the website.
  4. Easy installation on supported platforms, following commonly used installation process. A typical installation process should take less than 15 minutes to complete, including time to download source or binary packages, and required third party libraries and programs. Authors of complex simulation packages are recommended to provide binary installers for supported platforms, to bundle third-party libraries and tools, and/or use well accepted package distribution channels such as CRAN, CPAN, pip, and conda.

Criteria 3 and 4 are not applicable to resources that provide only a web interface, and java (or similar) applications that start directly from a web browser.

Documentation:

This certificate ensures that the software resource is well documented so that users can quickly determine if the resource provides needed features and learn how to use it. Simulators with this certificate should meet the following criteria:

  1. Complete GSR entry with accurate description of simulation targets, simulation methods, input/output, programming language, and license. The entry should reflect the most recent release and provide a complete list of features.
  2. Easily accessible documentation that contains information on
    1. Supported platform, dependency, and installation instructions.
    2. Type of license
    3. Target of simulation (DNA, RNA, type of genetic variants etc)
    4. Typical application areas with clear description of background and research questions that could be tackled by the simulator.
    5. Input and output formats. If the simulator uses application-specific format, it should provide sample input and output files with details description of the file formats.
  3. The documentation can be in any format but should be accessible directly from the resource homepage.

Application:

This certificate ensures that the software resource is peer reviewed, is reasonably user-friendly to be useful to peer researchers, and has been used by researchers in the scientific community. The application certificate will be evaluated based on the application references you provide in GSR. Simulators with this certificate should meet the following criteria:

  1. Published in peer-reviewed journals. The publication does not have to focus on the simulator, but should describe the simulation method and applications clearly.
  2. List typical applications in the "Applications" tab of the GSR entry of the simulator. The author needs to identify typical application(s) of the simulator and list publications that use the simulator on GSR. The publications listed should use the simulator to simulate data. Publications that only mention the software resources (e.g. in a discussion section or review article) should not be included because these publications do not reflect use of the resource, or an application.
  3. Used and cited by at least 5 external users in the past 5 years, or twice in the past year, as demonstrated by citations listed in the Applications tab. External users refer to people or researchers who are not one of the resource authors or directly affiliated with the authors.

Support:

This certificate ensures that the authors of the software resource is actively maintaining the resource and are actively addressing users’ questions, bug reports and feature requests. Simulators with this certificate should meet the following criteria:

  1. Available contact information of authors, maintainers, or mailing list
  2. Evidence of response to user inquiries and support for users, as shown from publicly available sources such as mailing list archives and forum posts.
  3. Response to GSR communications such as comments raised by visitors of GSR.

Return to top

How to get certified?

The GSR certificates are awarded by the GSR certification committee after an initial review by the GSR team. The certification process is author-initiated. If you are the author of a genetic simulation program, you can:

  1. Register your simulator in GSR or update information for your simulation program if it has already been registered. An up-to-date GSR entry is required for many components of certification.
  2. Collect usage information of your simulator and enter citations to the Applications tab of the GSR entry of your simulator.
  3. Review your resource with respect to the certification criteria and provide any additional information that might assist in the evaluation of your resource (e.g., location of license information and URL to mailing list archive).
  4. Send an email to gsr@mail.nih.gov and express your interest in having your simulation program certified. Please state clearly the name of the simulation program to be considered, the certificates you want to be (re)evaluated, and any additional information that might assist in the evaluation of your resource (e.g., location of license information and URL to mailing list archive). Please refer to the requirements section as a guide.

A GSR moderator will verify the information and forward the resource to the GSR certification committee. The GSR moderator might contact you for further details or suggest changes to the resource or website for certification. Decisions regarding GSR certificates must be unanimous by the committee.

After a program has been certified, the GSR certification may be removed from a program if the simulator no longer meets criteria of the certification program. Prior to removal of the GSR certificates, authors of such simulators will be notified of potential changes in status of the software resource and will be provided with a two-month grace period to re-certify the program. Review of certified resources is anticipated annually.

Return to top