Study: Words can deceive, but tone of voice cannot

A new computer algorithm can predict whether you and your spouse will have an improved or worsened relationship based on the tone of voice that you use when speaking to each other with nearly 79 percent accuracy.

In fact, the algorithm did a better job of predicting marital success of couples with serious marital issues than descriptions of the therapy sessions provided by relationship experts, reported in a study in Proceedings of Interspeech on September 6, 2015.

Researchers recorded hundreds of conversations from over one hundred couples taken during marriage therapy sessions over two years, and then tracked their marital status for five years.

An interdisciplinary team—led by Shrikanth Narayanan and Panayiotis Georgiou of the USC Viterbi School of Engineering with their doctoral student Md Nasir and collaborator Brian Baucom of University of Utah—then developed an algorithm that broke the recordings into acoustic features using speech-processing techniques. These included pitch, intensity, "jitter" and "shimmer" among many - things like tracking warbles in the voice that can indicate moments of high emotion.

"What you say is not the only thing that matters, it's very important how you say it. Our study confirms that it holds for a couple's relationship as well," Nasir said.

Taken together, the vocal acoustic features offered the team's program a proxy for the subject's communicative state, and the changes to that state over the course of a single therapy and across therapy sessions.

Importantly, these features weren't analyzed in isolation - rather, the impact of one partner upon the other and longitudinally over multiple therapy sessions was studied.

"It's not just about studying your emotions," Narayanan said. "It's about studying the impact of what your partner says on your emotions."

"Looking at one instance of a couple's behavior limits our observational power," Georgiou said. "However, looking at multiple points in time and looking at both the individuals and the dynamics of the dyad can help identify trajectories of the their relationship. Sometimes those are for the best or sometimes they are towards relationship deterioration."

The power of such methods is to help identify how domain experts can better advise couples towards improved relationships, Georgiou said.

"Psychological practitioners and researchers have long known that the way that partners talk about and discuss problems has important implications for the health of their relationships. However, the lack of efficient and reliable tools for measuring the important elements in those conversations has been a major impediment in their widespread clinical use. These findings represent a major step forward in making objective measurement of behavior practical and feasible for couple therapists," Baucom said.

Once it was fine-tuned, the program was then tested against behavioral analyses made by human experts who studied the couples. Those behavioral codes (positive qualities like "acceptance" or negative qualities like "blame"). The team found that studying voice directly - rather than the expert-created behavioral codes - offered a more accurate glimpse at a couple's future.

Next, using behavioral signal processing - a framework for computationally understanding human behavior - the team plans to use language (e.g., spoken words) and nonverbal information (e.g., body language) to improve the prediction of how effective treatments will be.