Security Update: Meltdown and Spectre Vulnerabilities

On January 3rd, a new class of security flaw was reported that impacts most processors including those that are used by Cloud Service Providers (CSPs), such as Amazon AWS and Microsoft Azure.  The issue exploits the speculative execution optimizations in processors as a side-channel attack that leak kernel memory (Meltdown, CVE-2017-5754) or user memory (Spectre, CVE-2017-5715, CVE-2017-5753).  

At this point, we have no evidence that this flaw has been exploited at DNAnexus.  

Patching Process and Status

We are actively working to address this flaw while minimizing any interruption in the DNAnexus service.  We are working with our CSPs and vendors to receive, test, and deploy patches efficiently and reliably.  Once available, patches are rapidly deployed in our staging environment where automated functional and scalability tests are performed.  When the patch is verified, it is deployed into our production environment without any expected downtime for the DNAnexus service.

On January 3rd, the CSPs have patched their hypervisors to prevent this class of flaw from leaking information between their cloud virtual instances.  This required a reboot of all DNAnexus servers which was completed that same day.

We have been working with Canonical, the organization that supports the Ubuntu operating system used at DNAnexus.  Canonical has released a Meltdown patch that we are in the process of testing.  We will be deploying the patch in two phases.  To ensure Meltdown cannot be exploited by a malicious DNAnexus user app, we will patch the worker fleet across all regions and clouds followed by deployment of the patch across all supporting systems.  Once the patch has been verified, it will be deployed into the worker fleet within 1 hour.  All new worker instances will take the patch.  All currently executing jobs will be allowed to complete to minimize disruption.  Then, we will initiate the patching process of our supporting systems, which is expected to take 1 week.

To address Spectre, given the nature of the flaw, we expect to receive multiple patches in the future.  We will work closely with our vendors to ensure the patches are deployed quickly while maintaining our high quality of service.

Profiling the Impact on Compute Performance for Standard Genomics Tools

The patches developed to mitigate this security flaw may cause certain applications to run slower. This will impact all patched work, whether conducted in DNAnexus, on local machines, or in other cloud environments.

Typical guidance from non-genomics areas is a slowdown from 5-30%, depending on domain.  The degree of impact depends on the type of computational operations and the only way to reliably determine this is empirically.  We have performed the exact same analyses on Meltdown patched machines with several popular genomic tools to assess the impact.

Our initial analysis indicates that most genomic analyses require around 5% more compute with the Meltdown patch, with a range of 5%-10%.  We expect this to generalize to the most common types of genomic analysis.   Fortunately, this suggests genomic workflows are less impacted than some other reported areas.

If you have any concerns, please contact DNAnexus at support@dnanexus.com.

CIO Webinar Series: Genomic Data Privacy in the Cloud

Join our two-part webinar series focusing on infrastructure requirements to scale geno-pheno analysis and realize genomic-based clinical trials. Can’t make it? Register anyway and we’ll send you the recording.

Advances in DNA sequencing have created tremendous volumes of whole-genome sequence and multi-omics data, creating new opportunities to explore how the genome plays a role in human disease. As the use of human genomic information becomes more prevalent in research and clinical care, it is important to understand the responsibilities for handling of data in these contexts. The inclusion of genomic information has also shown reduction in costs and time, and improved results of clinical trials. The reduction in sequencing costs and increasing value of NGS in clinical trials is leading some organizations to incorporate NGS into the majority of their trials.

Webinar 1: Understanding Security, Privacy, and the Regulatory Landscape for Genomics in Research and Clinical Settings
January 23rd, 2018
10:00am PST/1:00pm EST

Loren Buhle, PhD, VP Security, Quality & Compliance
Loren is a seasoned leader with over three decades of experience working in the regulated space of life sciences, clinical, and basic research. He brings an unusual combination of scientific, commercial, regulatory, quality, and IT disciplines to identify and manage security, quality, and compliance issues. 

 

Webinar 2: Major IT Considerations for Genomics in Healthcare
February 28th, 2018
10:00am PST/1:00pm EST

Omar Serang, Chief Cloud Officer
Omar has decades of experience building global operations teams and infrastructures, including cloud computing at Amazon Web Services, social web real-time analysis services at Topsy Labs, and messaging and messaging security services at Cloudmark and Critical Path. 

 

 

Hosted in partnership with Microsoft Azure

Updated DNAnexus Impact Assessment for Cloudbleed: No evidence of exploitation.

As described in our February 27, 2017 blog post regarding the Cloudflare information leak (“Cloudbleed”), a  bug within the code running on Cloudflare edge servers was discovered by a Google security researcher.

Upon further investigation into the use of Cloudflare on DNAnexus we found, on February 27th at 2:39 PM PST, that contrary to what we had indicated in our blog post, HTTP requests to platform.dnanexus.com served by Cloudflare edge servers in some cases included session tokens with authentication information. We revoked all customer session tokens at 5:06 PM PST that same day, at which point all requests to DNAnexus required re-authentication. All existing tokens were unusable after this time.

On February 23rd Cloudflare provided their most recent update and stated that there was no evidence of exploitation; there have been no updates since that deviate from this information. Additionally, Cloudflare has completed analysis of edge server log data, and on March 3rd confirmed that platform.dnanexus.com was not found to have been impacted.

Our CDN usage design has been reviewed and we continue to believe no customer has been impacted by the incident. Any potential new exposure has been eliminated and there continues to be no evidence of exploitation.

We know how critical information security is to our customers so if you have any questions about your account, please do not hesitate to contact our customer support team at support@dnanexus.com.