Why don't my document photos rotate correctly?

June 27, 2017, The Korea Advanced Institute of Science and Technology (KAIST)
Phone's rotation tracking with a rotation sensor. Credit: KAIST

John, an insurance planner, took several photos of a competitors' new brochures. At a meeting, he opened a photo gallery to discuss the documents with his colleagues. He found, however, that the photos of the document had the wrong orientation; they had been rotated in 90 degrees clockwise. He then rotated his phone 90 degrees counterclockwise, but the document photos also rotated at the same time. After trying this several times, he realized that it was impossible to display the document photos correctly on his phone. Instead, he had to set his phone down on a table and move his chair to show the photos in the correct orientation. It was very frustrating for John and his colleagues, because the document photos had different patterns of orientation errors.

Professor Uichin Lee and his team at KAIST have identified the key reasons for such errors and proposed novel techniques to solve this problem efficiently. Interestingly, it was due to a software glitch in screen rotation-tracking algorithms, and all smartphones on the market suffer from this error.

When taking a photo of a , your smartphone generally becomes parallel to the , as shown in the figure above (right). Professor Lee said, "Your fails to track the orientation if you make any rotation changes at that moment." This is because software engineers designed the rotation tracking software in conventional smartphones with the following assumption: people hold their phones vertically either in portrait or landscape orientations. Orientation tracking can be done by simply measuring the gravity direction using an acceleration sensor in the phone (for example, whether gravity falls into the portrait or landscape direction).

Professor Lee's team conducted a controlled experiment to discover how often orientation errors happen in document-capturing tasks. Surprisingly, their results showed that landscape document photos had error rates of 93%. Smartphones' camera apps display the current orientation using a camera-shaped icon, but users are unaware of this feature, nor do they notice its state when they take document photos. This is why we often encounter rotation errors in our daily lives, with no idea of why the errors are occurring.

Making a micro-tilt toward a photographer. Credit: KAIST

The team developed a technique that can correct a phone's orientation by tracking the rotation sensor in a phone. When people take document photos their smartphones become parallel to the documents on a flat surface. This intention of photographing documents can be easily recognizable because gravity falls onto the phone's surface. The current orientation can be tracked by monitoring the occurrence of significant rotation.

In addition, the research team discovered that when taking a document photo, the user tends to tilt the phone, just slightly, towards the user (called a "micro-tilt phenomenon"). While the tilting degree is very small—almost indistinguishable to the naked eye—these distinct behavioral cues are enough to train machine-learning models that can easily learn the patterns of gravity distributions across the phone.

The team's experimental results showed that their algorithms can accurately track phone orientation in document-capturing tasks at 93% accuracy. Their approaches can be readily integrated into both Google Android and Apple iPhones. The key benefits of their proposals are that the correction software works only when the intent of photographing documents is detected, and that it can seamlessly work with existing orientation tracking methods without conflict. The research team even suggested a novel user interface for photographing documents. Just like with photocopiers, the capture interface overlays a document shape onto a viewfinder so that the user can easily double-check possible orientation errors.

Professor Lee said, "Photographing documents is part of our daily activities, but orientation errors are so prevalent that many users have difficulties in viewing their documents on their phones without even knowing why such errors happen." He added, "We can easily detect users' intentions to photograph a document and automatically correct orientation changes. Our techniques not only eliminate any inconvenience with orientation errors, but also enable a range of novel applications specifically designed for document capturing." This work, supported by the Korean Government (MSIP), was published online in the International Journal of Human-Computer Studies in March 2017. In addition, their US patent application was granted in March 2017.

Explore further: Researchers find accelerometers may pose security risk for smartphones

More information: Jeungmin Oh et al, Understanding mobile document capture and correcting orientation errors, International Journal of Human-Computer Studies (2017). DOI: 10.1016/j.ijhcs.2017.03.004

Related Stories

Google Drive sports new view and scan enhancements

May 23, 2013

(Phys.org) —Google Drive has a new look and functions. The makeover in Google Drive features scanning and interface enhancements that put the user into "card" mode. The enhancements make it easy for the user to create and ...

Recommended for you

Two new planets discovered using artificial intelligence

March 26, 2019

Astronomers at The University of Texas at Austin, in partnership with Google, have used artificial intelligence (AI) to uncover two more hidden planets in the Kepler space telescope archive. The technique shows promise for ...

Infertility's roots in DNA packaging

March 26, 2019

Pathological infertility is a condition affecting roughly 7 percent of human males, and among those afflicted, 10 to 15 percent are thought to have a genetic cause. However, pinpointing the precise genes responsible for the ...

Facebook is free, but should it count toward GDP anyway?

March 26, 2019

For several decades, gross domestic product (GDP), a sum of the value of purchased goods, has been a ubiquitous yardstick of economic activity. More recently, some observers have suggested that GDP falls short because it ...

Droughts could hit aging power plants hard

March 26, 2019

Older power plants with once-through cooling systems generate about a third of all U.S. electricity, but their future generating capacity will be undercut by droughts and rising water temperatures linked to climate change. ...

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

5 / 5 (1) Jun 27, 2017
After trying this several times, he realized that it was impossible to display the document photos correctly on his phone.

On an Android phone, there's a button at the top menu to disable autorotation, and then then in the gallery app just rotate the picture 90 degrees whichever way you want.

Problem exists between hat and collar.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.