Big Data exploration in the era of Gaia

June 20, 2018, Netherlands Research School for Astronomy
This screenshot of an image is based on 1 billion rides of yellow cab users in New York. Credit: Breddels & Veljanoski (RUG)

Two astronomers from the University of Groningen (The Netherlands) have developed a software library that can effortlessly generate visualisations based on hundreds of millions of data points. Maarten Breddels and Jovan Veljanoski initially developed their software to handle the enormous quantity of data from the Gaia mission. However, the software can also show patterns in other large data files. The software is open source and free to use. The researchers explain the ins and outs in an article that has been accepted for publication in the journal Astronomy & Astrophysics.

Breddels and Veljanoski call their Vaex, which stands for "visualisation and exploration of big tabular datasets." The interactive software can generate visualisations of billions of data points in only one second. It behaves similarly to Google Maps. When panning or zooming, an updated or more detailed map appears almost immediately. However, Google Maps runs on fast, powerful servers, while Vaex works on a laptop.

The power of Vaex lies in the combination of several smart techniques. First, it uses a smart algorithm that maximises all available computing power. Then, it reads only the required data from the hard disk and sends it directly to the main memory of the computer. Finally, it is extremely memory efficient, and the working memory does not store unnecessary copies of the data.

Breddels has showcased Vaex live at several conferences. As an example, he used a dataset consisting of 1 billion entries related to the Yellow Cab taxis in New York City. He shows which taxi rides are the most lucrative, and where the taxis should wait in any part of the day to maximise their profit. This example shows how Vaex can be interesting and beneficial for general applications outside of astronomy.

Explore further: SHINE software shows data using virtual reality

More information: Vaex: Big Data exploration in the era of Gaia.

Related Stories

Software capable of quickly producing 3-D building models

December 10, 2014

Researchers at the UT have developed software which enables users to quickly produce 3D models of buildings for relatively little money. 3D models are used for navigation purposes, training purposes, urban planning and safety, ...

Recommended for you

Semimetals are high conductors

March 18, 2019

Researchers in China and at UC Davis have measured high conductivity in very thin layers of niobium arsenide, a type of material called a Weyl semimetal. The material has about three times the conductivity of copper at room ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.