IU 'Twister' software improves Google's MapReduce for large-scale scientific data analysis

Mar 16, 2010

(PhysOrg.com) -- "Twister," a new software tool released by Indiana University, supports faster execution of many data mining applications implemented as MapReduce programs. Developed by researchers from the Pervasive Technology Institute at IU, the tool extends the functionality of MapReduce, a distributed programming technique patented by Google for large-scale data processing in datacenter environments.

Twister allows MapReduce to achieve higher performance, perform faster data transfers, and reduce the time it takes to process vast sets of data for data mining and machine learning applications.

"MapReduce is an exceptionally valuable tool for finding meaning in very large scientific data sets," said Xiaohong "Judy" Qiu, Associate Director of the Community Grids Lab within the PTI Digital Science Center and lead on the project (Service Aggregated Linked Sequential Activities, or SALSA) that produced the Twister . "Twister makes MapReduce even more powerful for data-intensive disciplines such as physics, chemistry and the medical and life sciences."

Applications that currently use Twister include: K-means clustering, Google's page rank, Breadth first graph search, Matrix multiplication, and Multidimensional scaling. Twister also builds on the SALSA team's work related to commercial MapReduce runtimes, including Microsoft Dryad software and open source Hadoop software. SALSA project work is funded in part by an award from Microsoft, Inc.

"Twister is especially effective for applications with iterative MapReduce Computations," said Jaliya Ekanayake, lead developer on the Twister project. "The architecture is based on pub/sub messaging that enables it to perform faster data transfers, minimizing the overhead of the runtime. Also, the support for long running processes improves the efficiency of the runtime for many iterative MapReduce computations."

To access these papers or to learn more about Twister, please visit www.iterativemapreduce.org .

To watch a video about Twister, please visit pti.iu.edu/video/twister.

Explore further: OrangeSec pair said Cortana visited Android

Related Stories

See what I see -- machines with mental muscle

Oct 01, 2008

(PhysOrg.com) -- The way we use and interact with machines is undergoing a profound change as computers are programmed to learn from experience and see more how we see. European research into machine learning ...

Data Travels Six Times Faster in the Clouds

Feb 26, 2009

(PhysOrg.com) -- The National Center for Data Mining (NCDM) at the University of Chicago at Illinois established a cloud computing system that can quickly compile data from widely geographically distributed ...

Recommended for you

OrangeSec pair said Cortana visited Android

13 hours ago

Can, did, Cortana work on Android? A talked-about act at droidcon 2015: a presentation titled "Cracking Cortana." The OrangeSec team arrived at the Turin, Italy, event to show their work in a CortanaProxy ...

Report: Trusted Voice option rolling out for some

Apr 15, 2015

Smart Lock is arguably the best new feature in Android Lollipop, wrote Droid Life founder "Kellex" in March. With a secure lock screen set, he said, the user gets a number of options in Smart Lock to keep the phone or tablet quickly unlockable w ...

Redirect to SMB vulnerability in Windows discovered

Apr 14, 2015

News stories on tech spots on Monday reported that the Irvine, California, security company Cylance's SPEAR research team discovered a vulnerability relating to all versions of Windows including the Windows ...

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.