Thermodynamics of visual images may help us see the world

Thermodynamics of visual images may help us see the world
(a) A grayscale image of a forest. Photo by Dan Ruderman. (b) The same image after it is quantized into two equally populated levels of black and white. The researchers found that small patches within this quantized image retain substantial local structure. This finding led them to discover that the photo is scale-invariant—its structure stays the same as its scale changes. Credit: Greg J. Stephens, et al. ©2013 American Physical Society

(—Although researchers know that a large portion of the brain is devoted to visual processing, exactly how we interpret the complex patterns within natural scenes is far from understood. One question scientists ask is, is there something about the structure of the visual world itself that enables our brains to process and understand our visual surroundings, and is this structure something that can be described quantitatively?

A team of researchers at Princeton University has taken a closer look at images of nature and proposed that the scale invariance of images closely resembles the thermodynamics of physical systems at a critical point, with the distribution of pixels in the images analogous to the distribution of particle states in a physical system such as a . The parts of an image that correspond to the low-, or local minima, have surprisingly interpretable structure, and these thermodynamic characteristics may help the brain see.

The researchers, Greg J. Stephens, Thierry Mora, Gašper Tkačik, and William Bialek, at Princeton University, have published their study on the thermodynamics of images in a recent issue of .

In their study, the scientists analyzed an ensemble of photographs taken in a forest at Hacklebarney State Park in New Jersey. The researchers converted the grayscale camera images to binary (black and white) images. Although intensity information was lost in the quantization, many details such as the structure of the trees and a body of water could still be identified.

Thermodynamics of visual images may help us see the world
(a) 4 x 4 patches from the quantized forest image with the lowest energy states, starting with the lowest energy states of all: solid black and white blocks. The other patches are local minima, and many of them can be interpreted as lines and edges. The scientists speculate that the visual system might build neurons that identify these local minima in order to build a representation of the world. In part (b), the researchers computed the average light-intensity images that correspond to those in part (a). These average images resemble those that trigger neuron responses in the primary visual cortex. Credit: Greg J. Stephens, et al. ©2013 American Physical Society

The researchers then divided each binary image into much smaller patches composed of 3 x 3 and 4 x 4 pixels and examined the distribution of black and white pixels in these patches.

To quantify how much structure is present in these tiny segments of natural images, the researchers measured the entropy of the distribution of pixels. Randomly distributed pixels would give an entropy level of 9 and 16 bits, respectively, for the 3 x 3 and 4 x 4 pixel regions. But the researchers found that the entropy levels of the same-sized regions from the photo were only 6.5 and 11.2 bits, suggesting that substantial local structure remains in the tiny patches.

To explore how local image structure changes with scale, the researchers averaged neighboring pixels within each image and repeated their patch analysis. After such "coarse-graining," the image had lower resolution, but remarkably both the and pixel distribution were unchanged from the original image. Even after repeating this coarse-graining process four times, the pixel distributions in the small square regions remained the same, indicating that the photo is scale-invariant—its structure stays the same as its scale changes.

The scientists saw this scale invariance as a hint that natural images may have something in common with a physical system at a critical point. In physical systems, scale invariance emerges only when the temperature reaches a critical value, at which point a phase transition occurs between two phases characterized by different forms of order.

To examine whether the ensemble of natural images has its own critical point, the researchers treated the distribution of pixels as the Boltzmann distribution for a physical system, where the patterns of pixels in the small patches are associated with different energy levels according to their probability. Remarkably, as the patch size increased so too did a peak in the specific heat, a thermodynamic variable that characterizes fluctuations in the energy of the ensemble. These results suggest a sharp transition in the thermodynamic limit of large patch sizes, similar to how a physical system reaches this limit at a critical temperature.

The researchers found that this approach to the thermodynamics of images also shares similarities with Zipf-like distributions. According to Zipf's law, elements in a group (for example, words in a book) that are sorted from most common to least common will follow a pattern where the second most common element is 1/2 as common as the first, the third most common element is 1/3 as common as the first, etc. Zipf-like distributions have been found to hold for many different situations, and here the scientists found that they also closely describe the distribution of the size of pixel patches ranked by the structure as determined by their black and white pixels.

Perhaps the most interesting implication of viewing natural images from a thermodynamics perspective is what it reveals about the nature of image patches that correspond to the low energy states. The patches with the absolute lowest energy states are those that are either all black or all white. However, a small number of patches have pixels in both states yet are considered local minima, since flipping any single pixel would increase the energy. Looking closer at these patches, the researchers found that many of them have distinct patterns, such as edges between dark and light regions.

The researchers speculate that the importance of these local minima in natural images may help us and other creatures "see" our surroundings, even when our eyes don't absorb every pixel. The visual system may build neurons that are tuned to these "basins of attraction." In other words, these low-energy patches may assist the brain in filling in the details using some kind of error-correcting code based on the of the visual world.

Explore further

The worlds smallest 3D HD display

More information: Greg J. Stephens, et al. "Statistical Thermodynamics of Natural Images." PRL 110, 018701 (2013). DOI: 10.1103/PhysRevLett.110.018701
Journal information: Physical Review Letters

Copyright 2013
All rights reserved. This material may not be published, broadcast, rewritten or redistributed in whole or part without the express written permission of

Citation: Thermodynamics of visual images may help us see the world (2013, February 13) retrieved 9 April 2020 from
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Feedback to editors

User comments