Handling trillions of supercomputer files just got simpler

March 15, 2019 by Nancy Ambrosiano, Los Alamos National Laboratory
Gary Grider, left, and Brad Settlemyer discuss the new Los Alamos and Carnegie Mellon software product, DeltaFS, released to the software distribution site GitHub this week. Credit: Los Alamos National Laboratory

A new distributed file system for high-performance computing available today via the software collaboration site GitHub provides unprecedented performance for creating, updating and managing extreme numbers of files.

"We designed DeltaFS to enable the creation of trillions of ," said Brad Settlemyer, a Los Alamos computer scientist and project leader. Los Alamos National Laboratory and Carnegie Mellon University jointly developed DeltaFS. "Such a tool aids researchers in solving classical problems in high-performance computing, such as particle trajectory tracking or vortex detection."

DeltaFS builds a file system that appears to the user just like any other file system, doesn't require specialized hardware, and is exactly tailored to assisting the scientist in new discoveries when using a high-performance computing platform.

"One of the foremost challenges, and primary goals of DeltaFS, was scaling across thousands of servers without requiring a portion of them be dedicated to the file system," said George Amvrosiadis, assistant research professor at Carnegie Mellon University and a coauthor on the project. "This frees administrators from having to decide how to allocate resources for the file system, which will become a necessity when exascale machines become a reality."

Credit: Los Alamos National Laboratory

The brings about two important changes in computing. First, DeltaFS enables new strategies for designing the supercomputers themselves, dramatically changing the cost of creating and managing files. In addition, DeltaFS radically improves the performance of highly selective queries, dramatically reducing time to .

DeltaFS is a transient, software-defined service that allows data to be accessed from a handful up to hundreds of thousands of computers based on the user's performance requirements.

"The storage techniques used in DeltaFS are applicable in many scientific domains, but we believe that by alleviating the metadata bottleneck we have really shown a way for designing and procuring much more efficient HPC storage systems," Settlemyer said.

Explore further: Large Hadron Collider pushing computing to the limits

More information: GitHub link: github.com/pdlfs/deltafs/

Related Stories

Large Hadron Collider pushing computing to the limits

March 1, 2019

At the end of 2018, the Large Hadron Collider (LHC) completed its second multi-year run ("Run 2") that saw the machine reach a proton–proton collision energy of 13 TeV, the highest ever reached by a particle accelerator. ...

Fujitsu releases high-performance file system

October 17, 2011

Fujitsu today announced the launch of FEFS (Fujitsu Exabyte File System), a scalable file system software package for building file systems for x86 HPC clusters in Japan.

A user-controlled file security scheme for cloud services

May 22, 2017

By securing data files with a 'need-to-know' decryption key, researchers at Singapore's Agency for Science, Technology and Research (A*STAR) have developed a way to control access to cloud-hosted data in real time, adding ...

Recommended for you

Coffee-based colloids for direct solar absorption

March 22, 2019

Solar energy is one of the most promising resources to help reduce fossil fuel consumption and mitigate greenhouse gas emissions to power a sustainable future. Devices presently in use to convert solar energy into thermal ...

Paleontologists report world's biggest Tyrannosaurus rex

March 22, 2019

University of Alberta paleontologists have just reported the world's biggest Tyrannosaurus rex and the largest dinosaur skeleton ever found in Canada. The 13-metre-long T. rex, nicknamed "Scotty," lived in prehistoric Saskatchewan ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.