Transforming ordinary photos into clean, high-resolution 3D worlds

It is no longer unusual for computers to display our world in three dimensions. 3D scanners can scan faces, buildings or entire landscapes, and the data can be used to generate 3D models. In most cases though, this process is very complex. In the 'Harvest4D' research project, led by TU Wien (under the direction of Prof. Michael Wimmer from TU Wien's Institute of Computer Graphics and Algorithms), algorithms are therefore being developed that can be used to generate these three-dimensional worlds much more easily, using existing image data not necessarily collected for this purpose. Even the development of these worlds over time can be studied on the computer in this way. This type of software solution can be applied to a range of different fields, from archaeology to flood research.

Creating 3D worlds from 2D photos

"Up until now 3D images of reality have mainly been created using extremely complicated methods such as laser scanners," Michael Wimmer explains. "Experts have to plan a scan campaign carefully, take high-resolution images and painstakingly edit the data at the end." That is the only way to generate beautiful, smooth surfaces and clear shapes from a three-dimensional point cloud.

"We are envisioning a radical paradigm shift in this area," Michael Wimmer says. "A range of data is available to us that is more comprehensive than ever before, and the right algorithms enable us to use this image data." Devices that can take high-resolution images are becoming more and more commonplace. Almost everyone has a camera phone with them these days. A few years ago, quadcopters with webcams were still expensive, high-tech equipment; today they are an affordable gadget.

The appropriate calculation methods can be used to generate three-dimensional worlds from large collections of data. "The images are not just pieced together on the computer; we use them to compute a complete 3D model," project team member Reinhold Preiner explains. You can freely move around an object on the screen and view it from any angle, including those that were not originally present in the images. Excavations can be studied and analysed virtually; they can even be made accessible in high quality to users at home.

Error-tolerant image analysis

In order to achieve this, it is necessary to develop programs that tolerate errors, as the data is never perfect. Images are sometimes out of focus, they show objects in varying light conditions, and scans can include artefacts that interfere with the reconstruction. Despite this, the calculation methods developed in the 'Harvest 4D' research project make it possible to process this type of image data automatically. Archaeological excavations have, for example, been visualised in 3D in this way, where the raw data consisted of completely ordinary photos. In this case, the rough structure was also measured with a laser scan. The level of detail of the visualisation does not always have to be the same: for ordinary stonemasonry a more basic representation suffices, whereas you may want to look at valuable frescos on the wall in more detail. High-resolution images of frescos can therefore be additionally incorporated into 3D models.

If you do not need the data for scientific purposes, you do not necessarily need to use the maximum level of detail. If you are satisfied with a bit less precision, even end devices such as laptops and smartphones can present smooth 3D models. TU Wien's Institute of Computer Graphics designed the potree viewer especially for this purpose. It uses specialist techniques to display large point clouds that do not fit into the memories of low-power end devices.

Changes in 4D

If you know the time the images were taken, you can add a temporal dimension to the three spatial dimensions, and investigate how the observed 3D world changes. This is also a complex task for computers: have the depicted objects actually changed, or do the images just look different because they were taken in different light conditions, from different angles or with different devices? Human beings usually find it pretty easy to answer such questions, but it is a big challenge to teach this skill to a computer. Algorithms such as this could even prove interesting for flood research. "Our algorithms have the potential to be applied almost everywhere, and the opportunities to apply them keep on increasing," project team member Stefan Ohrhallinger says.