Researchers examine inaccuracies in mobile app maturity ratings

(Phys.org) —As smartphones, tablets and mobile applications (apps) continue to become more widespread, there is a rising concern among parents who have experienced unreliable content maturity ratings for mobile apps that result in their children's exposure to inappropriate content.

Researchers at Penn State's College of Information Sciences and Technology (IST) and College of Engineering have developed mechanisms to verify the maturity ratings of mobile apps and investigate possible reasons behind the incorrect ratings, which could have implications for platform providers (e.g. Google or Apple) as well as for regulatory bodies and application developers.

"Is This App Safe for Children? A Comparison Study of Maturity Ratings on Android and IOS Applications," was accepted to WWW2013—the 22nd International World Wide Web Conference—which will be held May 13 to 17 in Rio de Janeiro, Brazil. The paper was written by Ying Chen, a doctoral student in the Department of Computer Science and Engineering (CSE) in the College of Engineering, Heng Xu, an associate professor at the College of IST, Sencun Zhu, an associate professor of both IST and CSE, and Yilu Zhou, an assistant professor in the Department of Information Systems and Technology Management at George Washington University.

"This article can be a starting point in developing standardization of mobile app maturity ratings," said Xu, whose research projects deal with the impact of novel technologies on individuals' privacy concerns, strategic management of firms' information privacy practices, and design and empirical evaluations of privacy-enhancing technologies.

Among smartphone and tablet operating systems, the researchers reported, Android and Apple's iOS dominate the U.S. smartphone market by 52.5 percent and 34.3 percent, respectively. In order to help parents determine age-appropriate mobile apps for their children, both Android and iOS apps come with maturity ratings, which "examine the existence and intensity of mature themes such as mature content, violence, offensive language, sexual content and drug usage within each app."

However, they added, while the movie and video game industries have official rating organizations such as the Motion Picture Association of America (MPAA) and the Entertainment Software Rating Board (ESRB), mobile apps do not. Instead of having standard rating rules across platforms, each platform establishes its own rating policy and strategy. Android's maturity rating policy contains four maturity-rating levels: "Everyone," "Low Maturity," "Medium Maturity," and "High Maturity," while iOS's policy provides four maturity-rating levels based on the suitable age of the audience: "4+," "9+," "12+" and "17+." The classification rules for both rating systems for each level are similar except for some minor differences.

In terms of implementing maturity rating policy, the researchers said, the main difference between iOS and Android platforms is who determines the actual ratings. iOS rates each app submitted according to its own policies. Apple first requires developers to select from a list of objectionable content and indicate the intensity of the content to generate the maturity rating. According to Apple's "App Store Review Guidelines," Apple examines the content of apps and adjusts any inappropriate ratings during a review process before the app becomes available to users.

In contrast, Chen said, the maturity ratings for Android's apps are generated solely by the developers, with no centralized maturity rating system in place. In addition, Google does not verify each app's maturity rating unless there a number of user complaints. The laxity of Android's maturity rating policy has been examined in the media, and there has also been a "rising concern among parents who have experienced that the maturity ratings of the apps are unreliable."

"Our paper tries to highlight that the platform of Android is problematic," Xu said.

To analyze the problems with Android's maturity rating policy and its implementation, the researchers developed a text mining algorithm to automatically predict apps' actual maturity ratings from app descriptions and user reviews. Since the iOS apps are subject to a review process, Chen said, the researchers operated under the assumption that they are more accurate than the Android app ratings. Using iOS ratings as a baseline, Chen said, the researchers designed a method to automatically detect discrepancies between ratings of Android and iOS apps and examine the types of apps which tend to be misclassified. They propose a text-mining based Automatic Label of Maturity Ratings (ALM) algorithm, a learning algorithm that processes apps' descriptions and user reviews to determine maturity ratings.

"From our algorithm, parents can know (detailed information) more about the app, rather than just the rating," Chen said.

Through the use of ALM, the researchers discovered that over 30 percent of Android apps have unreliable maturity ratings, among which 20 percent of the apps are overrated (i.e. their maturity ratings on Android were higher than on iOS) and 10 percent of the apps are underrated (i.e. their ratings were lower than on iOS.)

Many Android apps are overrated, they said, because "developers are under the illusion that the maturity rating is also the criterion to judge users' capabilities or intelligence levels." For example, a chess game is rated as "Medium Maturity" but not "Everyone," because the developer may think that children younger than 12 are not equipped to play chess. In addition, the researchers said, Android's maturity policy has vague guidelines regarding the meanings of "simulated gambling," "violence" and "mature and suggestive themes."

While overrated apps can be misleading, the researchers said, "the underrated apps may directly harm children's mental health, because those apps conceal their actual maturity levels to parents and minors." Zhu said, developers of popular free apps are more likely to underrate their products to reach a broader audience. In addition, the researchers found that developers with lower levels of privacy awareness are more likely to underrate the cross-platform apps, for the purpose of reaching a wider user population and harvesting users' personal information.

The results of their study, Chen, Xu and Zhu said, have the potential to impact the mobile app industry. By adopting a stricter mobile app rating policy, Google would be playing a more important role than simply responding to consumer demand. While many parents are concerned about their children being exposed to inappropriate content via mobile apps, a sizable portion of parents may be unfamiliar with the technology required to properly monitor their children's activities. A more centralized review system, the researchers said, would provide parents with the tools to address those concerns.

"Google should probably take the initiative to re-examine its maturity rating policy for mobile apps," Xu said.