Tag: DNA-Seq

We want to hear from you about changes to NIH’s Sequence Read Archive data format and storage

RFI_SRA_largeNIH’s Sequence Read Archive (SRA) is the largest, most diverse collection of next generation sequencing data from human, non-human and microbial sources. Hosted by the National Center for Biotechnology Information (NCBI) at the National Library of Medicine (NLM), SRA data is also available on the Google Cloud Platform (GCP) and Amazon Web Services (AWS) as part of the NIH Science and Technology Research Infrastructure for Discovery, Experimentation, and Sustainability (STRIDES) Initiative.

SRA currently contains more than 36 petabytes (PB) of data and is projected to grow to 43 PB by 2023. Though the value of this resource grows with each new sample, the exponential growth experienced over the last decade (Figure 1) threatens SRA sustainability. The storage footprint is growing more costly to maintain and the data more difficult to use at scale. The situation has reached a tipping point. SRA must be refactored to support FAIR data principles into the future.

Sra_growthFigure 1. SRA data has grown exponentially over the last decade.

NIH remains committed to the SRA and hopes to establish a long-range plan for sustained resource growth. Considerations include a model wherein normalized working files without Base Quality Scores (BQS) are readily available through cloud platforms and NCBI FTP sites, and larger source files and normalized files with base quality scores will be distributed on cloud platforms based on prevalent use cases and usage demands. Further details regarding data formats are available here.

It is critical that as an SRA user, you  participate in the review and testing of proposed data formats and infrastructure by commenting on how these developments impact your data usage. NIH has prepared a Request for Information (RFI) that details planned developments and would greatly appreciate feedback from the scientific community.

Continue reading “We want to hear from you about changes to NIH’s Sequence Read Archive data format and storage”

May 20 webinar: Exploring SRA metadata in the cloud with BigQuery

May 20 webinar: Exploring SRA metadata in the cloud with BigQuery

Join us on May 20th to learn how to use Google’s BigQuery to quickly search the data from the Sequence Read Archive (SRA) in the cloud to speed up your bioinformatic research and discovery projects. BigQuery is a tool for exploring cloud-based data tables with SQL-like queries. In this webinar, we’ll introduce you to using BigQuery to mine SRA submitter-supplied metadata and the results of taxonomic analysis for SRA runs. You’ll see real-world case studies that demonstrate how to find key information about SRA runs and identify data sets for your own analysis pipelines.

  • Date and time: Wed, May 20, 2020 12:00 PM – 12:45 PM EDT
  • Register

After registering, you will receive a confirmation email with information about attending the webinar. A few days after the live presentation, you can view the recording on the NCBI YouTube channel. You can learn about future webinars on the Webinars and Courses page.

April 8 Webinar: Accelerate genomics discovery with SRA in the cloud

April 8 Webinar: Accelerate genomics discovery with SRA in the cloud

On Wednesday, April 8, 2019 at 12 PM, NCBI staff will show you how to leverage the cloud to speed up your research and discovery. You’ll be introduced to new and existing tools and data including BigQuery, SRA Toolkit, and more. You’ll hear about real workflows in the cloud featuring an example of the work NCBI was able to accomplish in the cloud using SRA data and a case study from an SRA cloud customer

By the end of this webinar, you will know where to look for new cloud products from NCBI, access help information to get you started, and will see how to run your analyses efficiently in the cloud.

  • Date and time: Wed, Apr 8, 2020 12:00 PM – 12:45 PM EDT
  • Register

After registering, you will receive a confirmation email with information about attending the webinar. A few days after the live presentation, you can view the recording on the NCBI YouTube channel. You can learn about future webinars on the Webinars and Courses page.

The entire corpus of the Sequence Read Archive (SRA) now live on two cloud platforms!

The National Library of Medicine (NLM) is pleased to announce that all controlled-access and publicly available data in SRA is now available through Google Cloud Platform (GCP) and Amazon Web Services (AWS). To access the data please visit our SRA in the Cloud webpage where you will find links to our new SRA Toolkit and other access methods.

The SRA data available in the two clouds currently totals more than 14 petabytes and consists of all data in the SRA format as well as some data in its original submission format.  Since May 2019, NCBI has been putting all submitted SRA data on the GCP and AWS clouds in both the submitted format and our converted SRA format. We have also been moving previously submitted original format data to the clouds and expect to complete that process in 2021. Continue reading “The entire corpus of the Sequence Read Archive (SRA) now live on two cloud platforms!”

NCBI on YouTube: Get the most out of NCBI resources with these videos

Check out the latest videos on YouTube to learn how to best use NCBI graphical viewers, SRA, PGAP, and other resources.

Genome Data Viewer: Analyzing Remote BAM Alignment Files and Other Tips

This video shows you how to upload remote BAM files, and succinctly demonstrates handy viewer settings, such as Pileup display options, and highlights the very helpful tooltips in the Genome Data Viewer (GDV). There’s also a brief blog post on the same topic.

Continue reading “NCBI on YouTube: Get the most out of NCBI resources with these videos”

Request for proposals: Single Cell in the Cloud codeathon at NYGC in January

Request for proposals: Single Cell in the Cloud codeathon at NYGC in January

The New York Genome Center is hosting an NCBI  Single Cell in the cloud codeathon from January 15-17, 2020. Submissions for project proposals are due December 2nd.

Please submit your proposal and apply here.

What topics are in scope?

This codeathon will focus on single cell data, including RNA, DNA, and chromatin accessibility.  We are particularly interested in proposals for pipelines and analysis of SRA data, data interoperability, and using machine learning techniques in clustering.  We also welcome proposals for tutorial pipelines and educational tools. You will have access to computational resources in the Cloud to turn your idea into a working prototype.   Visit our website for examples of previous codeathon projects.

Continue reading “Request for proposals: Single Cell in the Cloud codeathon at NYGC in January”

November 13 NCBI Minute: Resources for next-gen sequence analysis

November 13 NCBI Minute: Resources for next-gen sequence analysis

On Wednesday, November 13, 2019 at 12 PM, NCBI staff will present a webinar on NCBI resources for next-gen sequence analysis.  You will learn about key  resources that support multiple aspects of next-gen sequence analyses, including quality control, alignment, data visualization and interpreting results. You will also see how to access and apply these resources for both SRA and your own RNASeq/DNASeq datasets. Whether you’re embarking on your first analysis or already have a background in bioinformatics, you’ll find tools that meet your needs!

  • Date and time: Wed, Nov 13, 2019 12:00 PM – 12:45 PM EDT
  • Register

After registering, you will receive a confirmation email with information about attending the webinar. A few days after the live presentation, you can view the recording on the NCBI YouTube channel. You can learn about future webinars on the Webinars and Courses page.

Magic-BLAST version 1.5.0 is here!

Magic-BLAST version 1.5.0 is here!

We’ve just released a new version of Magic-BLAST with several new, user-driven enhancements like:

  • Nanopore sequence alignment
  • Improved multithreading performance
  • Support for the new BLAST database version, BLASTDBv5, that allows you to limit your search by taxonomy
  • More reliable placements of reads

The new executables are available on the NCBI FTP site.

graphic.png

A new paper (PMID: 31345161), published in July 2019 by BMC Bioinformatics, presents the usage accuracy of Magic-BLAST.

Magic-BLAST aligns next generation DNA- and RNA-Seq sequencing reads. Read more about the latest version of Magic-BLAST in the release notes.

NCBI’s First Hackathon: Advanced Bioinformatic Analysis of Next-Gen Sequencing Data

This blog post is geared toward genomics professionals.

From January 5th-7th, 2015, NCBI, in conjunction with the NIH Office of Data Science, held a genomics hackathon, where genomics professionals gathered to write useful, efficient pipelines for people new to genomics.

After we announced the hackathon, over 130 qualified applicants expressed interest in attending. Four team leads chose 23 attendees from this pool, then assigned initial predefined roles and provided biological guidance for a product in one of four subject areas: DNA-Seq, RNA-Seq, Epigenomics and Metagenomics. Continue reading “NCBI’s First Hackathon: Advanced Bioinformatic Analysis of Next-Gen Sequencing Data”