Tag: Basic Local Alignment Search Tool (BLAST)

BLAST+ 2.13.0 now available with SRA BLAST, ARM Linux executables, and database metadata

BLAST+ 2.13.0 now available with SRA BLAST, ARM Linux executables, and database metadata

BLAST+ 2.13.0  includes several important new features including SRA BLAST programs, ARM Linux executables, and the ability to produce database metadata as well as some important improvements, and a few bug fixes.  You can download the new BLAST release from the FTP site.

New features

SRA / WGS BLAST (blastn_vdb, tblastn_vdb)

Beginning with this release, the BLAST distribution now includes the SRA BLAST programs  blastn_vdb and tblastn_vdb that can directly search SRA and WGS projects without the need to build a BLAST database. See the BLAST documentation on how to use these programs with WGS projects.

ARM Linux executables

This release also includes executables compiled under ARM Linux for the first time. Please let us know if you find any issues with ARM Linux programs.

Database metadata in JSON format

Starting with BLAST+ 2.13.0, the makeblastdb program generates an additional file with the file extension .njs for nucleotide databases or .pjs  for protein databases. These files contain BLAST database metadata in JSON format. See the BLAST database metadata section in the BLAST User Manual for an example. This file can be easily read by many tools and makes the BLAST database more compliant with FAIR principles.

See the release notes for more details on improvements and bug fixes for the release.

Important reminder about usage reporting

As we announced previously, BLAST can report limited usage information back to NCBI. This information shows us whether BLAST+ is being used by the community, and therefore is worth being maintained and developed.  It also allows us to focus our development efforts on the most used aspects of BLAST+.  Please help us improve BLAST by allowing BLAST to share information about your search. The BLAST privacy statement  provides details on the information collected, how it is used, and how to opt-out of reporting if you don’t want to participate.

Introducing ElasticBLAST – BLAST® is now easier, bigger, and faster on the Cloud!

Introducing ElasticBLAST – BLAST® is now easier, bigger, and faster on the Cloud!

ElasticBLAST is a new tool that helps you run BLAST searches on the cloud. ElasticBLAST is perfect for you if you have thousands to millions of queries to our Basic Local Alignment Search Tool (BLAST ®), or if you want to use cloud infrastructure for your searches. ElasticBLAST can handle large searches that are not appropriate for NCBI web BLAST, and it runs them more quickly than stand-alone BLAST+.

ElasticBLAST works on two of the current NIH Science and Technology Research Infrastructure for Discovery, Experimentation, and Sustainability (STRIDES) partners- Amazon Web Services (AWS) and Google Cloud Platform (GCP).  ElasticBLAST works by distributing your searches across multiple cloud instances to process them in tandem. The ability to scale resources in this way allows you to process large numbers of queries in a shorter time than you could with BLAST+. ElasticBLAST can handle millions of queries, and it also supports most BLAST+ options and programs.

Making it easier to run BLAST on the cloud

ElasticBLAST reduces the barrier to using the cloud by creating and managing cloud resources for you. It manages the software and database installation, handles partitioning of the BLAST workload among the various instances and deallocates cloud resources when the searches are done. For example, ElasticBLAST will select the best cloud instance type for your search based on the database metadata that provides database size and memory needs (Figure 1). You can also manually select the instance type if you prefer.

Fig. 1: JSON metadata for the 16S_ribosomal_RNA database. The “bytes-to-cache” information helps ElasticBLAST pick out an instance with the appropriate capacity.

Selecting Databases

ElasticBLAST can access the 28 NCBI databases available on AWS and GCP. These are the same databases that are also available from the NCBI FTP site. For instance, databases available on the two cloud providers include the RefSeq Eukaryotic Representative Genomes database, 16S database based on Targeted Loci, and Human and mouse genomes databases.

You can also provide your own databases, and you can produce the metadata needed to select an instance through a Python script that comes with ElasticBLAST.

Example Runs

ElasticBLAST can perform a variety of searches with query sets that range from hundreds to millions of sequences and BLAST databases of all sizes.  Table 1 shows ElasticBLAST searches with query sets that range up to billions of letters using a variety of BLAST databases.

Table 1: Sample ElasticBLAST searches.  This table demonstrates the breadth of searches supported by ElasticBLAST.  Additionally, the first row demonstrates the ability of ElasticBLAST to use many CPUs (3200) on a cloud provider at once to complete a task in hours that would have taken days on a single machine.

Costs

Because ElasticBLAST runs on cloud providers, using it will incur some cost. Based on current cost structures on AWS and GCP, in most cases these costs are quite small. For example, a protein search with a query of about 20 million residues against a database of about 20 billion residues can cost less than $5. Even a larger search with a query of 3-4 billion DNA bases can cost only around $50. Both cloud services include the option to bid on instances for less than full price, which can result in significant savings. ElasticBLAST can be configured to request such instances. Your costs will obviously vary based on many factors, and we encourage you to explore these options with the individual cloud providers. Also, both AWS and GCP offer a free tier or time-limited trial of their cloud services, and you can find information about using ElasticBLAST with the free tiers here.

Welcome to ElasticBLAST!

Go ahead and run your first ElasticBLAST search! We are sure you’ll love how ElasticBLAST accelerates your research.

Your feedback is crucial to the development and support of ElasticBLAST. If you have any questions or suggestions, please reach out to us at blast-help@ncbi.nlm.nih.gov. We’d love to hear from you.

ElasticBLAST is a cloud-native package developed by the National Center for Biotechnology Information (NCBI) at the National Library of Medicine (NLM) with support from the NIH Science and Technology Research Infrastructure for Discovery, Experimentation, and Sustainability (STRIDES) Initiative.

Feb 16 Webinar: accelerating your alignments in the Cloud with ElasticBLAST

Feb 16 Webinar: accelerating your alignments in the Cloud with ElasticBLAST

Join us on February 16, 2022 at 12 PM US eastern time to learn about ElasticBLAST, a new tool that runs your BLAST searches on cloud hardware, using the standard BLAST command-line package. You will hear about the benefits of ElasticBLAST, which include speed and ease of use. You will also see some practical applications of this tool and how you can try it out yourself.

    • Date: Wed, February 16, 2022
    • Time: 12:00 PM – 12:45 PM EDT
    • Register

After registering, you will receive a confirmation email with information about attending the webinar. A few days after the live presentation, you can view the recording on the NCBI webinars playlist on the NLM YouTube channel. You can learn about future webinars on the Outreach Events page.

Petabyte-Scale Sequence Search: Metagenomics Benchmarking Codeathon Highlights

Petabyte-Scale Sequence Search: Metagenomics Benchmarking Codeathon Highlights

The National Institutes of Health (NIH) Office of Data Science Strategy (ODSS), the National Library of Medicine’s (NLM’s) National Center for Biotechnology and Information (NCBI), and the Department of Energy’s (DOE’s) Office of Biological and Environmental Research (BER) hosted scientists from around the world for a virtual Petabyte-Scale Sequence Search: Metagenomics Benchmarking Codeathon. The codeathon, held September 27-October 1, 2021, attracted experts from national laboratories including the Los Alamos National laboratory, research institutions including the Joint Genome Institute, and students from universities across the world to develop benchmarking approaches to address challenges in conducting large-scale analyses of metagenomic data.

Continue reading “Petabyte-Scale Sequence Search: Metagenomics Benchmarking Codeathon Highlights”

Web IgBLAST can now determine immunoglobulin isotypes

We have added a new function to IgBLAST on the Web. You can now search immunoglobulin (Ig) nucleotide sequences against the Constant region (C) gene database (Figure 1) to determine the Ig isotypes including subtypes (IgM, IgG, IgA1, etc.). The isotype information is reported in the rearrangement summary table, and the C gene region is displayed in the alignment section. This feature is now available on the IgBLAST web service for human and mouse sequences with possible expansion to other organisms in the future.  The feature is not yet implemented for the standalone IgBLAST package.

Figure 1.  IgBLAST constant region database selection and rearrangement summary table showing the top C gene match, IgHM in this case.  The NCBI C genes database is based on the the current NCBI human Reference Genome annotation.

NCBI events at the Bioinformatics Open Science Conference 2021 (BOSC 2021)

NCBI events at the Bioinformatics Open Science Conference 2021 (BOSC 2021)

Come visit us virtually to learn about new NCBI data access, tools and best practices at the Bioinformatics Open Science Conference  part of the ISMB/ECCB online conference from July 29 – 30, 2021. We will be presenting virtual posters on NCBI resources, offering a Birds of a Feather discussion, and participating in the BOSC  (CoFest) following the conference where you can take part in a hands-on evaluation of ElasticBLAST.

NCBI Posters, July 29, 2021, 11:20 – 12:20 PM EDT

All posters will be presented on Thursday afternoon. You can see complete abstracts on the ISMB/ECCB BOSC schedule.

Nuala O’Leary will talk about NCBI Datasets, a new resource for fast, easy access to NCBI sequence data.  You will learn about the new interface and new tools to access reference genomes, genes, and orthologs using web-based and programmatic tools.

Adelaide Rhodes will present Open access NCBI cloud resources to accelerate scientific insights where you can learn about recent developments in transferring > 20 petabytes of NCBI Sequence Read Archive (SRA) data to the cloud.

Deacon Sweeney will describe the web RAPT service for assembling and annotating bacterial genomes at the click of a button in RAPT, The Read assembly and Annotation Pipeline Tool: building a prokaryotic genome annotation package for users of all backgrounds.

Roberto Vera Alvarez will talk about best practices for using cloud tools for transcriptomics in his poster Transcriptome annotation in the cloud: complexity, best practices, and cost.

Greg Boratyn will discuss improvements to the BLAST-based short read aligner, Magic-Blast, in Recent improvements in Magic-BLAST 1.6.

Visit Christiam Camacho’s poster ElasticBLAST: Using the power of the cloud to speed up science to get an introduction to  ElasticBLAST, a Kubernetes-based approach for high throughput BLAST tasks. Join us following the conference in the CoFest to try out ElasticBLAST yourself and provide input. See the section on the CoFest below and our companion post.

Birds of a Feather, July 29, 2021, 11:20 – 12:20 PM EDT

We will host a Birds of Feather public feedback session on Thursday, where you can provide feedback and participate in discussions on all aspects of NCBI’s new data access options: NCBI Datasets, SRA, BLAST, and the Genome Data Viewer (GDV) — our genome browser for sequence visualization. We welcome your input!  Come and see us!

CollaborationFest (CoFest), July 31 – August 1, 2021

The ElasticBlast team will attend the BOSC CoFest following the conference. Sign up to participate on July 31 and August 1 to get an in-depth orientation and opportunity to test the capabilities of ElasticBlast on the Amazon Web Services (AWS) cloud. You do not have to register for the conference to attend the CoFest. See our post on the CoFest for more information.

 

Try out ElasticBLAST at the BOSC2021 CoFest!

Try out ElasticBLAST at the BOSC2021 CoFest!

Join the BLAST team at the virtual CollaborationFest (July 31 -August 1, 2021) after the BOSC 2021 conference to help test and improve ElasticBLAST, a new cloud-based tool designed to speed up high throughput BLAST searches. We would love to have your help with real world testing of our alpha release of ElasticBLAST with you own workflows and data. You may sign up for the CoFest even if you aren’t registered for BOSC 2021.

Here are suggestions for how you can participate. See the FAQs below for additional information.

  1. Try it out and let us know how well it works. You can be blunt.
  2. Help us improve the documentation.
  3. Write a script to make ElasticBLAST part of your workflow.
  4. Try to process ElasticBLAST results with cloud-native tools. Here is an example.
  5. Bring your own high throughput BLAST search problem to use with ElasticBLAST!  Please discuss it with us first to make sure you don’t blow our budget and get the ElasticBLAST team in trouble!

Continue reading “Try out ElasticBLAST at the BOSC2021 CoFest!”

Introducing the new NCBI Datasets Genomes page

The updated NCBI Datasets Genomes page now has genome data for all domains of life, including bacterial and viral genomes.

The genomes table (Figure 1) now offers filters for:

  • Reference genomes — switch it on to only show reference or representative genomes
  • Annotated — switch it on to only show annotated genomes
  • Assembly level — use the assembly level slider to select higher-quality genomes
  • Year released — use the slider to limit your results to recent genomes

In addition, the new Actions column connects you to NCBI’s Genome Data Viewer, BLAST, and Assembly. The Text filter box lets you search by the name of the assembly, species/infraspecies, or submitter.Figure 1. The new Datasets Genomes page with primate assemblies showing the STATUS switches (reference genomes, annotated); expanded filters section with ASSEMBLY LEVEL and YEAR RELEASED sliding selectors; and the Actions column menu with access to Assembly details, BLAST, the Genome Data Viewer, and Download options. Continue reading “Introducing the new NCBI Datasets Genomes page”

BLAST+ 2.12.0 now available with more efficient multithreaded searches

BLAST+ 2.12.0  programs feature better multithreaded searches and support a different threading model, threading by query, that can be more efficient in some situations.  The new release is also fully compatible the increase in the numeric range for the GI identifier, which will take effect in the nucleotide database later this year.  The list below shows details of the new features and bug fixes.  You can download the new BLAST release from the FTP site.

Continue reading “BLAST+ 2.12.0 now available with more efficient multithreaded searches”

June 2 Webinar: Quickly upload and view your own data in genomic context at NCBI

June 2 Webinar: Quickly upload and view your own data in genomic context at NCBI

Join us on June 2, 2021 at 12PM eastern time to learn how to how to upload and display your own genomic data in the context of annotated genome assemblies. You will use the Genome Data Viewer and the Sequence viewer to visualize your own uploaded data (indexed BAM, VCF, BED, wig, GFF formats), data from public track hubs, and your BLAST and Primer-BLAST results. You will also learn to take advantage of features of the viewers including optimizing display settings, sharing a view with collaborators, exporting images, and downloading genes or other features in the view.

  • Date and time: Wed, June 2, 2021 12:00 PM – 12:45 PM EDT
  • Register

After registering, you will receive a confirmation email with information about attending the webinar. A few days after the live presentation, you can view the recording on the NCBI webinars playlist on the NLM YouTube channel. You can learn about future webinars on the Webinars and Courses page.