New platform to help business tap full potential of big data

The EU JUNIPER project was launched in December 2012 to develop, test and evaluate prototype technologies that could aid big data analytical software applications.

Final industrial evaluations of the platform were assessed just before completion of the project at the end of November 2015, and the project team believe that the finalised platform has the potential to contribute positively to supporting projected growth of data streams and stored data. Financial and web streaming case studies were used to provide industrial data and data volumes and to effectively evaluate the newly developed technologies.

The results of the project could be of significant benefit to a number of sectors. The term 'big data' describes streams of information that are so large and complex that traditional data processing applications often cannot cope. By finding patterns through using advanced analytics however, new business opportunities and smarter applications could be opened up in numerous fields.

For example, patterns in big data can be analysed to better understand customer behaviour and preferences by including social media data, browser logs and text analytics. Retailers can use big data analytics to optimise their stock based on social media-generated predictive models, while big data analytics can also help machines and devices become smarter and more autonomous (in operating self-driving cars, for example).

There is therefore huge business potential in effectively handling big data. A key issue up to now however has been how to effectively manage such large and complex information streams, with online information outstripping network capacity. The internet now performs millions of tasks, from online banking to tsunami monitoring, and data traffic volumes are expected to grow twelve-fold by 2018.

Typically big data has been processed by two main components. Firstly, a data generator to produce large streams of information that need to be filtered prior to storage in order to reduce volume and secondly an application that can reply to an end users request; a financial transaction seeking authorisation from a banking database for example. However, real time constraints are often placed on big data streams and on processing, i.e. the user needs an answer quickly.

The JUNIPER project sought to address this challenge by developing a real time platform capable of supporting a wide range of high-performance big data applications. The ultimate goal has been to ensure that demands for information from end users can be met through the real time exploitation of streaming data sources and stored data.

The project brought together a number of leading industrial organisations involved in the development of products and services that utilise big data systems, leading software developers and technology companies operating in the field of advanced computing systems.