What to do with 15 million gigabytes of data

November 3, 2008

When it is fully up and running, the four massive detectors on the new Large Hadron Collider (LHC) at the CERN particle-physics lab near Geneva are expected to produce up to 15 million gigabytes, aka 15 petabytes, of data every year. Andreas Hirstius, manager of CERN Openlab and the CERN School of Computing, explains in November's Physics World how computer scientists have risen to the challenge of dealing with this unprecedented volume of data.

When CERN staff first considered how they might deal with the large volume of data that the huge collider would produce when its two beams of protons collide, in the mid-1990s, a single gigabyte of disk space still cost a few hundred dollars and CERN's total external connectivity was equivalent to just one of today's broadband connections.

It quickly became clear that computing power at CERN, even taking Moore's Law into account, would be significantly less than that required to analyse LHC data. The solution, it transpired during the 1990s, was to turn to "high-throughput computing" where the focus is not on shifting data as quickly as possible from A to B but rather from shifting as much information as possible between those two points.

High-performance computing is ideal for particle physics because the data produced in the millions of proton-proton collisions are all independent of one another - and can therefore be handled independently. So, rather than using a massive all-in-one mainframe supercomputer to analyse the results, the data can be sent to separate computers, all connected via a network.

From here sprung the LHC Grid. The Grid, which was officially inaugurated last month, is a tiered structure centred on CERN (Tier-0), which is connected by superfast fibre links to 11 Tier-1 centres at places like the Rutherford Appleton Laboratory (RAL) in the UK and Fermilab in the US. More than one CD's worth of data (about 700 MB) can be sent down these fibres to each of the Tier-1 centres every second.

Tier 1 centres then feed down to another 250 regional Tier-2 centres that are in turn accessed by individual researchers through university computer clusters and desktops and laptops (Tier-3).

As Andreas Hirstius writes, "The LHC challenge presented to CERN's computer scientists was as big as the challenges to its engineers and physicists. The computer scientists managed to develop a computing infrastructure that can handle huge amounts of data, thereby fulfilling all of the physicists' requirements and in some cases even going beyond them."

Source: Institute of Physics

Explore further: How the insights of the Large Hadron Collider are being made open to everyone

Related Stories

Three questions climate science must answer

January 17, 2017

The signing of the Paris agreement in 2015 may give the impression that the major questions concerning climate change have at last been answered and that climate science has met its challenges.

LHC Computing Centres Join Forces for Global Grid Challenge

April 25, 2005

Today, in a significant milestone for scientific grid computing, eight major computing centres successfully completed a challenge to sustain a continuous data flow of 600 megabytes per second (MB/s) on average for 10 days ...

CERN openlab adds a new dimension to Grid computing

July 6, 2004

Geneva, Switzerland 5 July 2004. The CERN openlab for DataGrid applications, a partnership between CERN , the European Organization for Nuclear Research, and five leading IT companies – Enterasys Networks, HP, IBM, Intel ...

CERN openlab boosts the performance of LHC computing

October 6, 2008

The LHC Grid Fest, held last Friday at CERN in Geneva, Switzerland, and at several sites around the world, commemorated the readiness of the Worldwide LHC Computing Grid (WLCG). At full capacity, the Large Hadron Collider ...

Recommended for you

Researchers control soft robots using magnetic fields

March 29, 2017

A team of engineering researchers has made a fundamental advance in controlling so-called soft robots, using magnetic fields to remotely manipulate microparticle chains embedded in soft robotic devices. The researchers have ...

How to outwit noise in quantum communication

March 29, 2017

How to reliably transfer quantum information when the connecting channels are impacted by detrimental noise? Scientists at the University of Innsbruck and TU Wien (Vienna) have presented new solutions to this problem.

Testing the performance of semiconductors—with light

March 29, 2017

Semiconductors are the cornerstone of modern electronics. They're used in solar cells, light emitting diodes (LEDs), microprocessors in laptops and cell phones, and more. Most of them are made of silicon, but silicon has ...


Adjust slider to filter visible comments by rank

Display comments: newest first

4.8 / 5 (4) Nov 03, 2008
Awesome! It's great to see scientists and engineers from all different fields coming together to work towards a single goal. This makes me happy =)
5 / 5 (1) Nov 03, 2008
Science has become very inter-disciplinary. While it is possible to study your own little niche and stay there, to be successful you need to branch out (or collaborate with other fields). Some of the most successful scientists are those who are excellent in their field but look at other fields for ideas.
5 / 5 (3) Nov 03, 2008
Sounds like a very smart plan and a very cool network they built.

Too bad the whole place will be destroyed soon by strangelets or stable quantum black holes that don't evaporate via Hawking radiation.

I am kidding... I hope!
2 / 5 (1) Nov 04, 2008
This story is a year old.
It is , no doubt,even older for
the people working on the issue.

not rated yet Nov 04, 2008
As a tree reaches up and out to intertwine with the atmosphere and it's root system grows a strong anchor into the earth so will it be with the sciences, growing, helping, supporting one another.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.