PrecisionFDA Receives FDA Commissioner’s Award for Outstanding Achievement

Today, the precisionFDA Next Generation Sequencing (NGS) Team received the FDA Commissioner’s Special Citation Award for Outstanding Achievement and Collaboration in the development of the precisionFDA platform promoting innovative regulatory science research to modernize regulation of NGS-based genomic tests. This award recognizes superior achievement of the Agency’s mission through teamwork, partnership, shared responsibility, and fostering collaboration to achieve the FDA goals.

 

PrecisionFDA is an online, cloud-based, virtual research space where members of the genomics community can experiment, share data and tools, collaborate, and define standards for evaluating and validating analytical pipelines. This open-source community platform, which has become a global reference standard for variant comparison, includes members from academia, industry, healthcare, and government, all working together to further innovation and develop regulatory standards for NGS-based drugs and devices. Launched in December 2015, the precisionFDA community includes nearly 5,000 users across 1,200 organizations, with more than 38 terabytes of genomic data stored.

To date, the precisionFDA NGS Team has engaged the genomics community through a series of community challenges:

  • The Consistency Challenge (Feb-Apr 2016): Invited participants to manipulate datasets with their software pipelines and conduct performance comparisons.
  • The Truth Challenge (Apr-May 2016): Gave participants the unique opportunity to test their NGS pipelines on an uncharacterized sample (HG002) and publish results for subsequent evaluation against a newly-revealed ‘truth’ dataset.
  • App-a-thon in a Box (Aug-Dec 2016): Invited the community to contribute NGS software to the precisionFDA app library, enabling the community to explore new tools.
  • Hidden Treasures Competition (Jul-Sep 2017): Participants beta-tested the in-silico analyses of NGS datasets for the purpose of determining the reliability and accuracy of different NGS tests.
  • CFSAN Pathogen Detection Challenge (Feb-Apr 2018): Participants helped to improve bioinformatics pipelines for detecting pathogens in samples sequenced using metagenomics.

We are thrilled that precisionFDA has been recognized for its efforts in fostering shared responsibility for the evaluation and validation of analytical pipelines. PrecisionFDA’s proven success has driven other scientific communities such as St. Jude Cloud to promote pediatric cancer research, and the Mosaic microbiome platform for advancing microbial strains analysis, to establish their own collaborative ecosystem for members to contribute and innovate. DNAnexus is proud to be the platform that powers precisionFDA and other community portals to advance scientific research through a secure and collaborative online environment.

To learn more about DNAnexus community portals please visit: http://go.dnanexus.com/community-portals.

Once in a Blue Moon Competition: precisionFDA Truth Challenge

The FDA, the Global Alliance for Genomics and Health (GA4GH)  and National Institute for Standards and Technology (NIST) recently teamed up to create a once-in-a-blue-moon challenge for genomic scientists! Dubbed the precisionFDA Truth Challenge, genomic innovators were invited to test their informatics pipelines on two datasets, the well-characterized Genome in a Bottle’s (GiaB) NA12878 (HG001) reference sample and a new reference sample HG002, of which the results were unknown.

PrecisionFDA_TruthChallenge_Image

PrecisionFDA is an online, cloud-based, virtual research space where members of the genomics community can experiment, share data and tools, collaborate, and define standards for evaluating analytical pipelines. Community members span academia, industry, healthcare organizations and government.  All of these organizations are working together to further innovation and develop regulatory science around NGS tests. So far, the community currently includes more than 1,500 users across 600 organizations, with more than 10 terabytes of genetic data stored.

This is the second challenge issued through precisionFDA, following the precisionFDA Consistency Challenge.   The Truth Challenge is about discovering the consistency and accuracy of informatics pipelines when analyzing a human sample whose truth data is unknown. NIST and GiaB released the truth data May 26, 2016, after the close of the challenge.

What makes this challenge so exciting?

NIST released NA12878 in 2014, the first gold standard whole human reference genome, in collaboration with GiaB and the FDA. Since then,  it has arguably become one of the most studied biospecimens. Researchers from around the world use NA12878 as training data for assessing pipeline performance.

Since many pipelines use some sort of machine learning algorithm when trying to determine whether a reported mutation is real or not,  the difficulty that arises is ensuring a pipeline doesn’t overfit the training data. Pipelines can ultimately be tuned, in order to maximize performance on the training dataset, and if the test data happens to be similar to the training data the pipeline’s performance would be abnormally consistent and accurate. A great resource in understanding why scientists split data into train and test roles in order to assess the accuracy, reliability, and credibility of their predictive models (the algorithm that goes into a pipeline) can be found here.

In order to test performance of pipelines in real-life, scientists needed a second reference sample and associated truth callset of which NGS pipelines have not been trained on. This is exactly what NIST and GiaB have provided in reference sample HG002.

Scientists can now evaluate algorithms using test data that is separate from the training data, an attribute  that is broadly accepted as fundamental to the evaluation methodology. Moreover, unlike NA12878, the new reference sample HG002 is male, which poses new challenges to algorithms since there is only one copy of the X chromosome, and brings new opportunity for evaluating NGS methods along this dimension.

The winners

As the clock struck midnight EST on May 25, 2016, the precisionFDA Truth Challenge closed with 36 entries across 21 teams, spanning 5 countries;  truly an international competition of epic proportions!

The winners of the Truth Challenge will be announced at the upcoming Festival of Genomics in Boston on June 29th at 8:45am EST by Elizabeth Mansfield, PhD, Deputy Director for Personalized Medicine in FDA’s Center for Devices and Radiological Health’s Office of In Vitro Diagnostics and Radiological Health. Registration is free. Want to learn more about precisionFDA?  Stop by the DNAnexus booth (# 240)  during the Festival to receive a demo of the precisionFDA platform from a member of the precisionFDA team.

Want to recreate the Truth Challenge for yourself? Join the precisionFDA community today and evaluate a pipeline of your choice against HG002. Happy testing!

Cloud-based Genomics at the White House

The Launch of precisionFDA Consistency Challenge

Last week the White House held a Precision Medicine Initiative (PMI) Summit, where government agencies discussed their progress on genomics-based personalized care, real people shared their success stories with precision medicine, and President Obama himself reiterated his vision for precision medicine. It was thrilling for DNAnexus to hear precisionFDA, the “online, cloud-based portal” that DNAnexus built for the FDA, based upon FDA requirements, specifically acknowledged (29:45-30:33). You can watch full coverage of the PMI Summit here.

PMI_Obama

It is no surprise that we have finally reached the era where cloud-based genomics is making it into White House announcements. The announcement also included the launch of the first precisionFDA consistency challenge, calling on members of the genomics community to assess their bioinformatics software on supplied reference human datasets. Called the precisionFDA Consistency Challenge, the goal is to engage genomics innovators to improve reproducibility and accuracy of next-generation sequencing (NGS) pipelines in order to achieve more consistent results from genetic tests and advance precision medicine.

PURPOSE OF THE CHALLENGE
Currently a single NGS test can identify genetic variants ranging from thousands to in the millions. The results, in some instances, are already being used to diagnose and treat disease. While NGS tests are currently used in clinical applications in oncology, non-invasive prenatal testing (NIPT), and rare disease, there is still room to improve consistency as they become more broadly adopted in clinical practice. A better understanding of accuracy and reliability of the results for specific NGS tests will help us get closer to personalized treatments and improve patient care.

Multiple sequencing technologies (Illumina XTen, Illumina HiSeq, PacBio RSII) can be used for human whole genome sequencing, and each has their own accuracy or reproducibility error profiles for different parts of the genome. The means to establish methodologies to assess the accuracy of a technology for specific variant types or regions across the genome would help advance evaluation of novel NGS-based tests for clinical applications. Unfortunately, there are still inconsistencies where an NGS-based test can report differing results. However, by establishing appropriate standards, inter-test variation can be minimized, allowing patients and physicians to place greater confidence in test results and the resulting treatments.

The FDA has delivered a new approach, precisionFDA, to help establish standards around secondary analysis – the process of mapping, aligning, and calling the variants of DNA sequence data. To jumpstart the engagement and improve techniques on the precisionFDA platform, the FDA’s first challenge is focused on consistency.

CHALLENGE DETAILS
The process of human WGS pipeline development and validation typically relies on mapping the sequenced reads to a well-known reference genome, then identifying the differences between the results and the reference dataset. Participants who join the challenge are asked to download two FDA-provided datasets (one contributed by the Garvan Institute of Medicine, and one by Human Longevity Inc. — both corresponding to sequencing of the well-characterized NA12878 sample), process it through their pipelines, upload results back to precisionFDA, and compare them to other files. The challenge provides a common frame of reference for measuring some of the aspects of reproducibility and accuracy of the participant’s pipeline.

precisionFDA Challenge

WHO CAN PARTICIPATE?
The challenge is open to all innovators in the field of human genomics. If you’re not already a member of precisionFDA, you will need to request access to get started. You have until April 25th to submit your software assessments.

DETERMINING WINNERS*
Results will be ranked on the precisionFDA website for achievements in eight categories. See the Determining Winners section on the challenge webpage for full details. In addition to exclusive bragging rights, your results, comparisons, and methods will be featured on the precisionFDA platform highlighting your technical contributions.

DNAnexus is incredibly excited about this challenge – the idea of genomics innovators working together to advance quality standards is something that gets us fired up. As of this blog post, precisionFDA hosts more than 1000 community members on the platform representing nearly 500 organizations. We are proud to support this novel community-contribution model for evaluating bioinformatics pipelines to help address the challenges of precision medicine.

* Winning a precisionFDA category is an acknowledgement by the precisionFDA community and does not imply FDA endorsement of any organization, tool, software, etc.