Jefferson Lab's newest cluster makes Top500 list of fastest supercomputers

November 17, 2016
The SciPhi-XVI supercomputer was just listed as a TOP500 Supercomputer Site on Nov. 14, placing 397th on the 48th edition of the list of the world's top supercomputers. Credit: DOE's Jefferson Lab

For the third time in its history, Thomas Jefferson National Accelerator Facility is home to one of the world's 500 fastest supercomputers. The SciPhi-XVI supercomputer was just listed as a TOP500 Supercomputer Site on November 14, placing 397th on the 48th edition of the list of the world's top supercomputers.

The Top500 list was announced Monday, Nov. 14, at The International Conference for High Performance Computing, Networking, Storage and Analysis, SC16, in Salt Lake City. It's announced semi-annually and is comprised of systems that are ranked according to their computation speeds as determined by the LINPACK Benchmark, which is a calculation of a linear system represented by a large double precision dense matrix.

SciPhi-XVI was delivered to Jefferson Lab and installed in mid-August and was deployed on Sept. 30 with 200 nodes. An additional 64 nodes were added to the machine in late October, and the High Performance Computing group then benchmarked the machine. The SciPhi-XVI machine benchmarked at 425.8 Teraflops using 256 of its 264 nodes.

The cluster's 264 nodes are comprised of the second-generation Intel Xeon Phi chips, which were released in 2016 and are codenamed Knights Landing. Each node has 64 computing cores, and individual nodes within the machine are connected via the Intel Omni-Path Architecture, which supports a 100 Gigabit/second data transfer rate between the nodes.

"A special Intel distributed LINPACK benchmark software, HPL (High Performance Linpack), was used to rank the Jefferson Lab cluster, and it is based on an open source distribution with performance enhancement for Intel Xeon Phi processors," said Jie Chen, a member of Jefferson Lab's High Performance Computing group. "To achieve the performance number for the SciPhi-XVI cluster, a set of HPL running parameters has to be determined through many trials. In particular, the size of the matrix representing the linear equations solved by the HPL was chosen to be 2,457,600."

Jefferson Lab conducts discovery-caliber nuclear physics research in exploring the atomic nucleus and its fundamental constituents, such as protons and neutrons and these particles' building blocks: quarks and gluons. The theory of how these particles interact—Quantum Chromodynamics, or QCD, is calculated in the form of lattice QCD, or LQCD.

Scientists will use the SciPhi-XVI machine for LQCD calculations, as well as a test system for the use of the Xeon Phi processors for analyzing data that is generated in Jefferson Lab's experimental physics program. The Chroma code, which is used to solve LQCD on the Xeon Phi processor platform, will be used with an open source library of equation solvers for LQCD. The High Performance Computing group has been modernizing the freely available Chroma code, so that it will run faster and provide better results, more quickly on the new generations of hardware.

"There are always challenges to bringing online state-of-the-art hardware systems. We are very proud of our staff for their hard work and dedication that made this achievement possible," said Jefferson Lab Chief Information Office, Amber Boehnlein. "This hardware achievement, together with our efforts to continually optimize the performance of our software for the newest systems, positions the laboratory to continue performing great science in high-performance computation."

Jefferson Lab work on Chroma is funded through the U.S. Department of Energy Office of Science. The research is carried out in collaboration with partner institutions, such as DOE's National Energy Research Scientific Computing Center (NERSC) Exascale Science Application Program (NESAP) and the Scientific Discovery through Advanced Computing (SciDAC) program. NERSC is a DOE Office of Science User Facility at Lawrence Berkeley National Laboratory.

Supercomputer computation time on SciPhi-XVI is allocated based on the recommendations of the United States Lattice Gauge Theory Computational Program (USQCD), a consortium of top LQCD theorists in the United States that spans both high-energy physics and nuclear physics. The USQCD consortium developed the QUDA code library that is used with the Chroma code.

Explore further: K computer takes first place on HPCG benchmark

Related Stories

K computer takes first place on HPCG benchmark

November 16, 2016

On November 16, the K computer took first place in the HPCG benchmark, a new index developed to create a more realistic view of supercomputer performance compared to the commonly used LINPACK benchmark. This success, which ...

Supercomputer benchmark gains adherents

December 23, 2015

A software program that ranks supercomputers on their ability to solve complex problems rather than on raw speed alone continues to gain traction in the high-performance computing community.

Tianhe-2 supercomputer at 31 petaflops is title contender

June 10, 2013

( —How is this for bragging rights in the always-on title grab for the world's fastest supercomputer: China's Tianhe-2 supercomputer, aka Milkyway-2, recently measured at speeds of 31 petaflops (30.65) out of a ...

OSC's Oakley Cluster delivers on performance efficiency

May 30, 2012

The Ohio Supercomputer Center's newest system would fall within the top half of the list of the world's most powerful supercomputers based purely on speed, but the cluster would rank even higher – ninth in the United ...

Recommended for you

Wireless power could enable ingestible electronics

April 27, 2017

Researchers at MIT, Brigham and Women's Hospital, and the Charles Stark Draper Laboratory have devised a way to wirelessly power small electronic devices that can linger in the digestive tract indefinitely after being swallowed. ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.