Cambridge (ip-192.com): Researchers from the Broad Institute and Harvard University have developed a tool that can tackle large data sets in a way that no other software program can. Part of a suite of statistical tools called MINE, which stands for Maximal Information-based Nonparametric Exploration, is able to analyze a broad spectrum of patterns. It can tease out multiple patterns hidden in health information from around the globe, statistics amassed from a season of major league baseball, data on the changing bacterial landscape of the gut, and much more.
From Facebook to physics to the global economy, the world is filled with data sets that could take a person hundreds of years to analyze by eye. Sophisticated computer programs can search these data sets with great speed, but fall 
short when researchers attempt to even-handedly detect different kinds of patterns in large data collections.
"There are massive data sets that we want to explore, and within them, there may be many relationships that we want to understand," said senior author Pardis Sabeti, Assistant Professor in the Department of Organismic and Evolutionary Biology and the Center for Systems Biology at Harvard and an associate member of the Broad Institute. "The human eye is the best way to find these relationships, but these data sets are so vast that we can't do that. This toolkit gives us a way of mining the data to look for relationships."
The researchers tested their analytical toolkit on several large data sets, including one provided by Peter Turnbaugh, a Bauer Fellow at the Harvard Center for Systems Biology, who is interested in the trillions of microorganisms that live in the gut. Working with Turnbaugh, the research team harnessed MINE to make more than 22 million comparisons and narrowed in on a few hundred patterns of interest that had not been observed before.
"The goal of this statistic is to take data with a lot of different dimensions and many possible correlations and pick out the top ones," said senior author Michael Mitzenmacher, Gordon McKay Professor of Computer Science at the Harvard School of Engineering and Applied Sciences. "We view this as an exploration tool," he said. "It can find patterns and rank them in an equitable way."
One of the tool's greatest strengths is that it can detect a wide range of patterns and characterize them according to a number of different parameters a researcher might be interested in. Other statistical tools work well for searching for a specific pattern in a large data set, but they cannot score and compare different kinds of possible relationships.
"Standard methods will see one pattern as signal and others as noise," said David Reshef, a co-first author of the paper who is currently a graduate student in the Harvard-MIT Health Sciences and Technology (HST) program and also worked on this project as a graduate student in the department of statistics at the University of Oxford. "There can potentially be a variety of different types of relationships in a given data set. What's exciting about our method is that it looks for any type of clear structure within the data, attempting to find all of them."
Not only does MINE attempt to identify any pattern within the data, but it also attempts to do so with an eye toward capturing different types of patterns equally well. "This ability to search for patterns in an equitable way offers tremendous exploratory potential in terms of searching for patterns without having to know ahead of time what to search for," said Reshef, whose brother Yakir is also a co-lead author of the paper.
MINE is especially powerful in exploring data sets with relationships that may harbor more than one important pattern. As a proof of concept, the researchers applied MINE to social, economic, health, and political data from the World Health Organization and its partners. When they compared the relationship between household income and female obesity, they found two contrasting trends in the data. Many countries follow a parabolic rate, with obesity rates rising with income but peaking and tapering off after income reaches a certain level. But in the Pacific Islands, where female obesity is a sign of status, countries follow a steep trend, with the rate of obesity climbing as income increases.
"Many data sets will contain these types of complicated relationships that are guided by multiple drivers," said Sabeti. MINE is able to identify these. "This greatly extends our capability to find interesting relationships in data."
Researchers can use MINE to generate new ideas and connections that no one has thought to look for before. "Our tool is a hypothesis generator," said Yakir Reshef, a Fulbright scholar at the Weizmann Institute of Science. "The standard paradigm is hypothesis-driven science, where you come up with a hypothesis based on your personal observations. But by exploring the data, you get ideas for hypotheses that would never have occurred to you otherwise."
The paper was first published in the journal Science.
A new tool, Maximal Information-based Nonparametric Exploration or MINE, developed by SEAS computer scientist Michael Mitzenmacher, provides a new toolkit for recognizing relationships in large data sets. Image: Harvard University
Recent Comments