Transforming ordinary photos into clean, high-resolution 3D worlds

Transforming ordinary photos into clean, high-resolution 3D worlds
Picture of a bust, created from 100 single pictures. Credit: Vienna University of Technology

It is no longer unusual for computers to display our world in three dimensions. 3D scanners can scan faces, buildings or entire landscapes, and the data can be used to generate 3D models. In most cases though, this process is very complex. In the 'Harvest4D' research project, led by TU Wien (under the direction of Prof. Michael Wimmer from TU Wien's Institute of Computer Graphics and Algorithms), algorithms are therefore being developed that can be used to generate these three-dimensional worlds much more easily, using existing image data not necessarily collected for this purpose. Even the development of these worlds over time can be studied on the computer in this way. This type of software solution can be applied to a range of different fields, from archaeology to flood research.

Creating 3D worlds from 2D photos

"Up until now 3D of reality have mainly been created using extremely complicated methods such as laser scanners," Michael Wimmer explains. "Experts have to plan a scan campaign carefully, take and painstakingly edit the data at the end." That is the only way to generate beautiful, smooth surfaces and clear shapes from a three-dimensional point cloud.

"We are envisioning a radical paradigm shift in this area," Michael Wimmer says. "A range of data is available to us that is more comprehensive than ever before, and the right algorithms enable us to use this ." Devices that can take high-resolution images are becoming more and more commonplace. Almost everyone has a camera phone with them these days. A few years ago, quadcopters with webcams were still expensive, high-tech equipment; today they are an affordable gadget.

The appropriate calculation methods can be used to generate three-dimensional worlds from large collections of data. "The images are not just pieced together on the computer; we use them to compute a complete 3D model," project team member Reinhold Preiner explains. You can freely move around an object on the screen and view it from any angle, including those that were not originally present in the images. Excavations can be studied and analysed virtually; they can even be made accessible in high quality to users at home.

Error-tolerant image analysis

Transforming ordinary photos into clean, high-resolution 3D worlds
Change detection: Comparison of two 3D scans. Credit: Vienna University of Technology

In order to achieve this, it is necessary to develop programs that tolerate errors, as the data is never perfect. Images are sometimes out of focus, they show objects in varying light conditions, and scans can include artefacts that interfere with the reconstruction. Despite this, the calculation methods developed in the 'Harvest 4D' make it possible to process this type of image data automatically. Archaeological excavations have, for example, been visualised in 3D in this way, where the raw data consisted of completely ordinary photos. In this case, the rough structure was also measured with a laser scan. The level of detail of the visualisation does not always have to be the same: for ordinary stonemasonry a more basic representation suffices, whereas you may want to look at valuable frescos on the wall in more detail. High-resolution images of frescos can therefore be additionally incorporated into 3D models.

If you do not need the data for scientific purposes, you do not necessarily need to use the maximum level of detail. If you are satisfied with a bit less precision, even end devices such as laptops and smartphones can present smooth 3D models. TU Wien's Institute of Computer Graphics designed the potree viewer especially for this purpose. It uses specialist techniques to display large point clouds that do not fit into the memories of low-power end devices.

Changes in 4D

Transforming ordinary photos into clean, high-resolution 3D worlds
Model created from 3D-scanner data and photographs. Credit: Vienna University of Technology

If you know the time the images were taken, you can add a temporal dimension to the three spatial dimensions, and investigate how the observed 3D world changes. This is also a complex task for computers: have the depicted objects actually changed, or do the images just look different because they were taken in different , from different angles or with different devices? Human beings usually find it pretty easy to answer such questions, but it is a big challenge to teach this skill to a computer. Algorithms such as this could even prove interesting for flood research. "Our algorithms have the potential to be applied almost everywhere, and the opportunities to apply them keep on increasing," project team member Stefan Ohrhallinger says.

Transforming ordinary photos into clean, high-resolution 3D worlds
3D scanner data with camera positions (colored). Credit: Vienna University of Technology

Explore further

Helping computers learn to tackle big-data problems outside their comfort zones

Citation: Transforming ordinary photos into clean, high-resolution 3D worlds (2016, May 30) retrieved 19 January 2020 from
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Feedback to editors

User comments