What to do with 15 million gigabytes of data

November 3, 2008

When it is fully up and running, the four massive detectors on the new Large Hadron Collider (LHC) at the CERN particle-physics lab near Geneva are expected to produce up to 15 million gigabytes, aka 15 petabytes, of data every year. Andreas Hirstius, manager of CERN Openlab and the CERN School of Computing, explains in November's Physics World how computer scientists have risen to the challenge of dealing with this unprecedented volume of data.

When CERN staff first considered how they might deal with the large volume of data that the huge collider would produce when its two beams of protons collide, in the mid-1990s, a single gigabyte of disk space still cost a few hundred dollars and CERN's total external connectivity was equivalent to just one of today's broadband connections.

It quickly became clear that computing power at CERN, even taking Moore's Law into account, would be significantly less than that required to analyse LHC data. The solution, it transpired during the 1990s, was to turn to "high-throughput computing" where the focus is not on shifting data as quickly as possible from A to B but rather from shifting as much information as possible between those two points.

High-performance computing is ideal for particle physics because the data produced in the millions of proton-proton collisions are all independent of one another - and can therefore be handled independently. So, rather than using a massive all-in-one mainframe supercomputer to analyse the results, the data can be sent to separate computers, all connected via a network.

From here sprung the LHC Grid. The Grid, which was officially inaugurated last month, is a tiered structure centred on CERN (Tier-0), which is connected by superfast fibre links to 11 Tier-1 centres at places like the Rutherford Appleton Laboratory (RAL) in the UK and Fermilab in the US. More than one CD's worth of data (about 700 MB) can be sent down these fibres to each of the Tier-1 centres every second.

Tier 1 centres then feed down to another 250 regional Tier-2 centres that are in turn accessed by individual researchers through university computer clusters and desktops and laptops (Tier-3).

As Andreas Hirstius writes, "The LHC challenge presented to CERN's computer scientists was as big as the challenges to its engineers and physicists. The computer scientists managed to develop a computing infrastructure that can handle huge amounts of data, thereby fulfilling all of the physicists' requirements and in some cases even going beyond them."

Source: Institute of Physics

Explore further: 'Maiden Flight' for LHC Computing Grid Breaks Gigabyte-per-Second Barrier

Related Stories

CERN openlab adds a new dimension to Grid computing

July 6, 2004

Geneva, Switzerland 5 July 2004. The CERN openlab for DataGrid applications, a partnership between CERN , the European Organization for Nuclear Research, and five leading IT companies – Enterasys Networks, HP, IBM, Intel ...

Large Hadron Collider pauses protons; enters new phase

November 4, 2010

(PhysOrg.com) -- Proton running for 2010 in the Large Hadron Collider at CERN came to a successful conclusion today. Since the end of March, when the first collisions occurred at a total energy of 7 TeV, the machine and experiment ...

CERN prepares its long-term future

February 6, 2014

Particle physics takes the long-term view. Originally conceived in the 1980s, the LHC took another 25 years to come into being. This accelerator, which is unlike any other, is just at the start of a programme which is expected ...

Recommended for you

Test racetrack dipole magnet produces record 16 tesla field

November 30, 2015

A new world record has been broken by the CERN magnet group when their racetrack test magnet produced a 16.2 tesla (16.2T) peak field – nearly twice that produced by the current LHC dipoles and the highest ever for a dipole ...

Turbulence in bacterial cultures

November 30, 2015

Turbulent flows surround us, from complex cloud formations to rapidly flowing rivers. Populations of motile bacteria in liquid media can also exhibit patterns of collective motion that resemble turbulent flows, provided the ...

'Material universe' yields surprising new particle

November 25, 2015

An international team of researchers has predicted the existence of a new type of particle called the type-II Weyl fermion in metallic materials. When subjected to a magnetic field, the materials containing the particle act ...

CERN collides heavy nuclei at new record high energy

November 25, 2015

The world's most powerful accelerator, the 27 km long Large Hadron Collider (LHC) operating at CERN in Geneva established collisions between lead nuclei, this morning, at the highest energies ever. The LHC has been colliding ...


Adjust slider to filter visible comments by rank

Display comments: newest first

4.8 / 5 (4) Nov 03, 2008
Awesome! It's great to see scientists and engineers from all different fields coming together to work towards a single goal. This makes me happy =)
5 / 5 (1) Nov 03, 2008
Science has become very inter-disciplinary. While it is possible to study your own little niche and stay there, to be successful you need to branch out (or collaborate with other fields). Some of the most successful scientists are those who are excellent in their field but look at other fields for ideas.
5 / 5 (3) Nov 03, 2008
Sounds like a very smart plan and a very cool network they built.

Too bad the whole place will be destroyed soon by strangelets or stable quantum black holes that don't evaporate via Hawking radiation.

I am kidding... I hope!
2 / 5 (1) Nov 04, 2008
This story is a year old.
It is , no doubt,even older for
the people working on the issue.

not rated yet Nov 04, 2008
As a tree reaches up and out to intertwine with the atmosphere and it's root system grows a strong anchor into the earth so will it be with the sciences, growing, helping, supporting one another.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.