Massive data management for the Digital Single Market

If the Digital Single Market launched by the European Commission in May this year is to become a reality, the fragmented field of data management needs to be addressed quickly. Current cloud computing practices often sacrifice data consistency for size, in order to cope with the huge volumes of Big Data being handled. It makes the building of apps – where strong safeguards are necessary – a very difficult and time-consuming task.

This has so far held back Big Data and Internet of Things developers, since it takes them months to produce apps that can fully update across all the different data stores they need to access in real-time. These might be mission-critical machine-to-machine applications dealing with millions of events per second. For example, telecoms operators updating detailed call and billing records, banks handling payment transactions, or IoT modules in consumer items like cars, collecting route, traffic, and weather information.

Making life easier for developers

COHERENTPAAS – a three-year project involving 10 partners in six European countries, which has received a EUR 4.95 million investment from FP7 – is dedicated to changing this complex scenario by providing the data consistency so far missing from cloud computing. It resolves the dilemma by harmonising the SQL (Structured Query Language) and other query languages used by cloud data stores, simplifying them into one language for developers.

'By integrating our ultra-scalable transactional management with different cloud data stores, SQL and NoSQL (Not-only Structured Query Language), we can guarantee full data consistency, preventing any data loss during operational failures,' explained Dr. Ricardo Jimenez-Peris, technical coordinator of COHERENTPAAS. 'This in effect could enable the data management necessary for the Digital Single Market.'

Enterprises today have moved to this world of so-called 'polyglot persistence' (a term minted by Martin Fowler in his book 'NoSQL Distilled'), where they continually add to the soup by introducing new kinds of SQL and NoSQL document-oriented, key-value and graph-oriented data store technologies. But by developing integrated data management, COHERENTPAAS has come up with a simple solution that shields developers from the complexity of manually combining all these technologies, so allowing them to produce apps in a matter of hours instead of the current months.

The transactional processing that forms the basis of the COHERENTPAAS platform is ultra-scalable, guaranteeing 'all or nothing' semantics on a big scale, which means it never loses data when the system fails. If there is a failure during a transaction across data stores it will, as in banking, either successfully complete or have no effect at all, meaning the application will never be left in the inconsistent partial state that is currently the bane of 'polyglot persistence'.

European startup ready to compete in global marketplace

Some partners in the project have just spun off a startup company, LeanXcale, to offer the platform in the fast-moving cloud and big data marketplace, initially creating five jobs, while COHERENTPAAS will continue until 30 September 2016. LeanXcale is already talking to big banks such as Santander and BBVA, and working with COHERENTPAAS partner Portugal Telecom, about how these corporations can use its Cloud Big Data platform. The start-up is also targeting companies working in the fields of insurance, transport, logistics, Internet of Things and smart networks.