Dot: An Interactive Dot Plot Viewer for Comparative Genomics

Author: Maria Nattestad, Scientific Visualization Lead

 

 

 

Introduction

Next week, DNAnexus will be at the Plant and Animal Genome conference (PAG) in San Diego (booth 431). As part of an ongoing effort to expand our visualization capabilities, we will present an open-source tool called Dot that helps scientists visualize genome-genome alignments through a rich, interactive dot plot.

In addition to its scientific contribution, Dot encourages community development of new visualization tools by providing a template that can be used for new visualization tools in other areas of bioinformatics. This would allow bioinformaticians to focus on the bioinformatics and visualization without needing to master web programming intricacies such as reading data from local and remote servers, which is all handled by Dot’s modular and reusable inner workings.

Importance of Dot Plots

Constructing a genome assembly is fundamental to studying the biology of a species. In recent years, advances in long-read sequencing and scaffolding technologies have led to unprecedented quality and quantity of genome assemblies. Better reference genomes contribute to better gene annotations, evolutionary understanding, and biotech opportunities.

Comparing new assemblies to existing genomes of related species is crucial to understanding differences between organisms across the tree of life. Genome assemblies are never perfect and always have to be evaluated critically by comparing against other assemblies or reference genomes, whether of the same or a closely related species. Comparative genomics is also how assemblies of two species’ genomes can be compared and contrasted to look for features that represent functional differences or inform the study of their evolution.

The classic method for visualizing genome-genome alignments is the dot plot, which provides an excellent overview of alignments from the perspective of both genomes. Dot plots place the reference genome on one axis and the query genome (that is aligned against the reference) on the other axis. Alignments between the two genomes are placed according to their coordinates on both genomes. Whereas genome browsers (such as IGV and the UCSC Genome Browser) plot data in one dimension on one genome, dot plots use two dimensions to show alignments in two genomes’ coordinates spaces simultaneously. This is necessary when representing large genome alignment data where the query coordinates matter just as much as the reference coordinates for a particular alignment.

However, dot plots have barely changed in the past decade and are still generated from the command-line as static images, limiting detailed investigation. We decided to tackle this problem as an open-source science project at DNAnexus.

Introducing Dot

Here we present Dot, an interactive dot plot viewer that allows genome scientists to visualize genome-genome alignments in order to evaluate new assemblies and perform exploratory comparative genomics.

Dot supports the output of MUMmer’s nucmer aligner the most commonly used software method for aligning genome assemblies. A quick script called DotPrep.py converts the delta file to a more streamlined coordinates file with an index that enables Dot to read in more alignments in certain regions on demand.

Interactivity and features

Dot adds a number of useful features on top of the classic dot plot concept. The index enables a quick plot of an overview that includes the longest 1000 alignments. From here, users can zoom in to look at particular regions and load all the alignments for regions of interest.

In addition to showing alignments, Dot allows scientists to load annotations for either or both genomes to show additional context  (e.g. understanding how sequence differences map to gene differences). Annotation tracks are a common feature of one-dimensional genome browsers, but to translate this concept to the two-dimensional dot plot, we enable annotation tracks on both axes. This is a major benefit of Dot that makes it possible to compare gene annotations visually alongside the alignments of the DNA sequences.

Moreover, users can jump to the same region of the reference genome in the UCSC Genome Browser to quickly see additional context for a region of interest. This allows scientists to explore how known repetitive elements in the reference genome could potentially affect assembly quality in specific regions.

Details for developers

By leveraging D3 and canvas in JavaScript, Dot combines the benefits of interactivity with scalability, enabling scientists to explore large genomes. The UI on the right side panel is built using an open-source SuperUI.js [https://github.com/marianattestad/superui] plugin, and the input handling and basic page navigation is set up through a special VisToolTemplate [https://github.com/MariaNattestad/VisToolTemplate] plugin we developed to enable others to create new visualization tools more easily. We encourage developers to utilize and build on Dot and these open-source projects to create their own visualization tools. Dot is very modular and can be used as a template to build new visualization tools. The template handles complex and necessary components like reading input data files from various sources, thereby letting developers focus only on the visualization itself.

Dot is open source

Dot is free to use online at [https://dnanexus.github.io/dot/] and open source at [https://github.com/dnanexus/dot]. For DNAnexus users, there is a package available among the featured projects with (1) an applet for running MUMmer’s nucmer aligner that includes DotPrep.py, (2) a shortcut  to Dot to send files from DNAnexus quickly, and (3) example data and results.

 

Security Update: Meltdown and Spectre Vulnerabilities

On January 3rd, a new class of security flaw was reported that impacts most processors including those that are used by Cloud Service Providers (CSPs), such as Amazon AWS and Microsoft Azure.  The issue exploits the speculative execution optimizations in processors as a side-channel attack that leak kernel memory (Meltdown, CVE-2017-5754) or user memory (Spectre, CVE-2017-5715, CVE-2017-5753).  

At this point, we have no evidence that this flaw has been exploited at DNAnexus.  

Patching Process and Status

We are actively working to address this flaw while minimizing any interruption in the DNAnexus service.  We are working with our CSPs and vendors to receive, test, and deploy patches efficiently and reliably.  Once available, patches are rapidly deployed in our staging environment where automated functional and scalability tests are performed.  When the patch is verified, it is deployed into our production environment without any expected downtime for the DNAnexus service.

On January 3rd, the CSPs have patched their hypervisors to prevent this class of flaw from leaking information between their cloud virtual instances.  This required a reboot of all DNAnexus servers which was completed that same day.

We have been working with Canonical, the organization that supports the Ubuntu operating system used at DNAnexus.  Canonical has released a Meltdown patch that we are in the process of testing.  We will be deploying the patch in two phases.  To ensure Meltdown cannot be exploited by a malicious DNAnexus user app, we will patch the worker fleet across all regions and clouds followed by deployment of the patch across all supporting systems.  Once the patch has been verified, it will be deployed into the worker fleet within 1 hour.  All new worker instances will take the patch.  All currently executing jobs will be allowed to complete to minimize disruption.  Then, we will initiate the patching process of our supporting systems, which is expected to take 1 week.

To address Spectre, given the nature of the flaw, we expect to receive multiple patches in the future.  We will work closely with our vendors to ensure the patches are deployed quickly while maintaining our high quality of service.

Profiling the Impact on Compute Performance for Standard Genomics Tools

The patches developed to mitigate this security flaw may cause certain applications to run slower. This will impact all patched work, whether conducted in DNAnexus, on local machines, or in other cloud environments.

Typical guidance from non-genomics areas is a slowdown from 5-30%, depending on domain.  The degree of impact depends on the type of computational operations and the only way to reliably determine this is empirically.  We have performed the exact same analyses on Meltdown patched machines with several popular genomic tools to assess the impact.

Our initial analysis indicates that most genomic analyses require around 5% more compute with the Meltdown patch, with a range of 5%-10%.  We expect this to generalize to the most common types of genomic analysis.   Fortunately, this suggests genomic workflows are less impacted than some other reported areas.

If you have any concerns, please contact DNAnexus at support@dnanexus.com.

Gatk4 on DNAnexus

Authors:

The Broad Institute’s Genome Analysis Toolkit (GATK) is one of the most popular and well regarded repositories of best practices variant calling workflows, and DNAnexus has consistently provided optimized support of these pipelines on our platform. Announced January 9th, GATK4 is the latest release of the toolkit, and this release is particularly significant. GATK has been completely re-architected and is available fully open source. The best practices workflows descriptions are now also explicitly specified and distributed in an open format called the Workflow Description Language (WDL).

At DNAnexus, we are excited about supporting open, portable, and reproducible ways to share not only these new best practices workflows, but also general bioinformatics workflows written in WDL.  As such, to execute explicit GATK4 workflow definitions written in WDL and maintained by the Broad, we use a new utility that we developed called dxWDL. With this tool, a GATK WDL workflow can be used just like any other workflow on the platform with all of the additional benefits our platform provides (e.g. provenance tracking, reproducibility, organization management, project collaboration, and security). As we did with GATK3, we are in the process of optimizing the performance of GATK4 on our platform and future posts will go into more detail about how it performs in terms of efficiency and accuracy. In the interim, we are pleased to announce the launch of the DNAnexus GATK4 Pilot Program, to be offered to a limited number of interested users, with broader access to the tool in the coming months. To request early access to GATK4 on DNAnexus please sign up here.

As an example of using GATK4 with dxWDL, we successfully ran a single sample haplotype workflow and Broad’s production germline variant calling workflow written in WDL on DNAnexus. For the production workflow, the version run on DNAnexus was modified slightly since Broad’s original pipeline has some Google cloud specific references.  The following figure shows what the haplotype workflow looks like on DNAnexus: 

After execution, the timeline of tasks can be easily visualized showing Broad’s more complex production germline variant calling workflow:

Using dxWDL for GATK4 marks a change in how we will be executing these and other workflows written to be portable across platforms.  In contrast to our previous approach of maintaining our own GATK applications, we will be directly supporting open and portable languages, such as WDL and CWL.  Portability through languages such as WDL not only enables research in our field to be better critiqued and improved upon, but it also significantly reduces friction when communicating method details to collaborators and regulatory agencies. Often times while the details of a specific method in a workflow are not changed, some subtleties of in the workflow definition do change leading to reproducibility challenges, such as the changes we needed to make in the production pipeline described above.  With our adoption of open workflow languages like WDL, we will more easily share these workflow-level differences with the community and work with one another towards a single representation that runs portably across a variety of execution platforms.

DNAnexus is proud to be one of the first genome informatics platforms to support WDL. As a member of the core team to govern future developments in WDL, we look forward to continuing work with the Broad and the broader community so that the the best practices WDL workflows can be run as efficiently and portably as possible.