Tag: STRIDES

Tackling Petabyte Scale Sequence Search Challenges

The volume of biological data being generated by the scientific community is growing exponentially, reflecting technological advances and research activities. This increase in available data has great promise for pushing scientific discovery but also introduces new challenges that scientific communities need to address. The National Institutes of Health’s (NIH) Sequence Read Archive (SRA), which is maintained by the National Library of Medicine’s National Center for Biotechnology Information (NCBI), is a rapidly growing public database that researchers use to improve scientific discovery across all domains of life. As part of the Science and Technology Research Infrastructure for Discovery, Experimentation, and Sustainability (STRIDES) Initiative, over 36 petabytes of “next generation” (raw and SRA-formatted) sequencing data is accessible to anybody via two cloud service providers.

To help address the challenges of conducting large-scale analysis of -omic data in the SRA and similar databases, the Department of Energy (DOE) Office of Biological and Environmental Research (BER), the NIH Office of Data Science Strategy (ODSS), and NCBI, held a virtual workshop on June 8, 2021, on Emerging Solutions in Petabyte Scale Sequence Search. The workshop brought together experts from DOE national labs, research institutions, and universities across the world.

SRA data growth over time. Databases like the NIH Sequence Read Archive are growing rapidly and are used extensively by scientific communities. As these databases grow, so do their potential scientific value, but work must be done to ensure ease of access.

Continue reading “Tackling Petabyte Scale Sequence Search Challenges” →

NIH’s COVID-focused Sequence Read Archive (SRA) datasets are now open access on AWS!

While searching for SARS-CoV-2 sequences, have you longed for a COVID-focused SRA dataset? Great news — now there is one! We are happy to announce the addition of COVID-focused datasets (including source and normalized SRA file formats) to the AWS Public Dataset Program. These data can now be explored at the Registry of Open Data on AWS.

Researchers can now access more than 13K SRA runs that include Coronaviridae (CoV) content identified by a kmer-based approach to organismal content identification using the SRA Taxonomy Analysis Tool.

Continue reading “NIH’s COVID-focused Sequence Read Archive (SRA) datasets are now open access on AWS!” →

We want to hear from you about changes to NIH’s Sequence Read Archive data format and storage

NIH’s Sequence Read Archive (SRA) is the largest, most diverse collection of next generation sequencing data from human, non-human and microbial sources. Hosted by the National Center for Biotechnology Information (NCBI) at the National Library of Medicine (NLM), SRA data is also available on the Google Cloud Platform (GCP) and Amazon Web Services (AWS) as part of the NIH Science and Technology Research Infrastructure for Discovery, Experimentation, and Sustainability (STRIDES) Initiative.

SRA currently contains more than 36 petabytes (PB) of data and is projected to grow to 43 PB by 2023. Though the value of this resource grows with each new sample, the exponential growth experienced over the last decade (Figure 1) threatens SRA sustainability. The storage footprint is growing more costly to maintain and the data more difficult to use at scale. The situation has reached a tipping point. SRA must be refactored to support FAIR data principles into the future.

Figure 1. SRA data has grown exponentially over the last decade.

NIH remains committed to the SRA and hopes to establish a long-range plan for sustained resource growth. Considerations include a model wherein normalized working files without Base Quality Scores (BQS) are readily available through cloud platforms and NCBI FTP sites, and larger source files and normalized files with base quality scores will be distributed on cloud platforms based on prevalent use cases and usage demands. Further details regarding data formats are available here.

It is critical that as an SRA user, you participate in the review and testing of proposed data formats and infrastructure by commenting on how these developments impact your data usage. NIH has prepared a Request for Information (RFI) that details planned developments and would greatly appreciate feedback from the scientific community.

Continue reading “We want to hear from you about changes to NIH’s Sequence Read Archive data format and storage” →

May 20 webinar: Exploring SRA metadata in the cloud with BigQuery

Join us on May 20th to learn how to use Google’s BigQuery to quickly search the data from the Sequence Read Archive (SRA) in the cloud to speed up your bioinformatic research and discovery projects. BigQuery is a tool for exploring cloud-based data tables with SQL-like queries. In this webinar, we’ll introduce you to using BigQuery to mine SRA submitter-supplied metadata and the results of taxonomic analysis for SRA runs. You’ll see real-world case studies that demonstrate how to find key information about SRA runs and identify data sets for your own analysis pipelines.

Date and time: Wed, May 20, 2020 12:00 PM – 12:45 PM EDT
Register

After registering, you will receive a confirmation email with information about attending the webinar. A few days after the live presentation, you can view the recording on the NCBI YouTube channel. You can learn about future webinars on the Webinars and Courses page.

April 8 Webinar: Accelerate genomics discovery with SRA in the cloud

On Wednesday, April 8, 2019 at 12 PM, NCBI staff will show you how to leverage the cloud to speed up your research and discovery. You’ll be introduced to new and existing tools and data including BigQuery, SRA Toolkit, and more. You’ll hear about real workflows in the cloud featuring an example of the work NCBI was able to accomplish in the cloud using SRA data and a case study from an SRA cloud customer

By the end of this webinar, you will know where to look for new cloud products from NCBI, access help information to get you started, and will see how to run your analyses efficiently in the cloud.

Date and time: Wed, Apr 8, 2020 12:00 PM – 12:45 PM EDT
Register

Request for proposals: Single Cell in the Cloud codeathon at NYGC in January

The New York Genome Center is hosting an NCBI Single Cell in the cloud codeathon from January 15-17, 2020. Submissions for project proposals are due December 2nd.

Please submit your proposal and apply here.

What topics are in scope?

This codeathon will focus on single cell data, including RNA, DNA, and chromatin accessibility. We are particularly interested in proposals for pipelines and analysis of SRA data, data interoperability, and using machine learning techniques in clustering. We also welcome proposals for tutorial pipelines and educational tools. You will have access to computational resources in the Cloud to turn your idea into a working prototype. Visit our website for examples of previous codeathon projects.

Continue reading “Request for proposals: Single Cell in the Cloud codeathon at NYGC in January” →

Virus hunting in the cloud: A hackathon story at ASV 2019

Are you going to ASV 2019?

If you are, join us in a few days for a workshop on the virus hunting hackathon we helped run earlier this year.

Session: Workshop #19: Virus Discovery

Program Number: W-19-8

Time: Sunday, July 21, 7:00 PM CDT

Location: Mayo Auditorium

In this workshop, Dr. Rodney Brister will talk about how 41 scientists from 21 organizations worked to improve the usability of SRA data, identifying datasets that included known viruses and viral signals. Not only is that information now being integrated into a public search interface, but the approach used is also being refined in future hackathons so it can be applied to all SRA datasets.

We hope to see you there!

Bring your own data to the computational virology workshop at LSU New Orleans

Attention all aspirational computational virologists and cloud-curious bioinformaticians! NCBI is hosting a free workshop in New Orleans, Louisiana April 23 and 24.

Choose your own adventure: participants may bring their own data and/or work with public data housed at NCBI.

Day 1 will consist of a short cloud-onboarding session, introduction to Jupyter notebooks, SRA and BLAST intros, and more! On day 2, we’ll roll our sleeves up in a working session around phylogenetic clustering of sequences where we’ll look for unknown viruses.

BYOD (bring your own data) and apply today! Please fill out the application form in its entirety by Tuesday, April 9th.

Attendee insights will be made publicly available on GitHub.