On January 3rd, a new class of security flaw was reported that impacts most processors including those that are used by Cloud Service Providers (CSPs), such as Amazon AWS and Microsoft Azure. The issue exploits the speculative execution optimizations in processors as a side-channel attack that leak kernel memory (Meltdown, CVE-2017-5754) or user memory (Spectre, CVE-2017-5715, CVE-2017-5753).
At this point, we have no evidence that this flaw has been exploited at DNAnexus.
Patching Process and Status
We are actively working to address this flaw while minimizing any interruption in the DNAnexus service. We are working with our CSPs and vendors to receive, test, and deploy patches efficiently and reliably. Once available, patches are rapidly deployed in our staging environment where automated functional and scalability tests are performed. When the patch is verified, it is deployed into our production environment without any expected downtime for the DNAnexus service.
On January 3rd, the CSPs have patched their hypervisors to prevent this class of flaw from leaking information between their cloud virtual instances. This required a reboot of all DNAnexus servers which was completed that same day.
We have been working with Canonical, the organization that supports the Ubuntu operating system used at DNAnexus. Canonical has released a Meltdown patch that we are in the process of testing. We will be deploying the patch in two phases. To ensure Meltdown cannot be exploited by a malicious DNAnexus user app, we will patch the worker fleet across all regions and clouds followed by deployment of the patch across all supporting systems. Once the patch has been verified, it will be deployed into the worker fleet within 1 hour. All new worker instances will take the patch. All currently executing jobs will be allowed to complete to minimize disruption. Then, we will initiate the patching process of our supporting systems, which is expected to take 1 week.
To address Spectre, given the nature of the flaw, we expect to receive multiple patches in the future. We will work closely with our vendors to ensure the patches are deployed quickly while maintaining our high quality of service.
Profiling the Impact on Compute Performance for Standard Genomics Tools
The patches developed to mitigate this security flaw may cause certain applications to run slower. This will impact all patched work, whether conducted in DNAnexus, on local machines, or in other cloud environments.
Typical guidance from non-genomics areas is a slowdown from 5-30%, depending on domain. The degree of impact depends on the type of computational operations and the only way to reliably determine this is empirically. We have performed the exact same analyses on Meltdown patched machines with several popular genomic tools to assess the impact.
Our initial analysis indicates that most genomic analyses require around 5% more compute with the Meltdown patch, with a range of 5%-10%. We expect this to generalize to the most common types of genomic analysis. Fortunately, this suggests genomic workflows are less impacted than some other reported areas.
If you have any concerns, please contact DNAnexus at support@dnanexus.com.
