Scene statistics

Scene statistics is a discipline within the field of perception. It is based on the premise that a perceptual system is designed to interpret scenes.

Biological perceptual systems have evolved in response to physical properties of natural environments. Therefore natural scenes receive a great deal of attention.

Natural scene statistics can play an important role in Ideal Observer Analysis. They are useful for defining the behavior of an Ideal Observer in a naturalistic task, typically by incorporating signal detection theory, information theory, or estimation theory.

Within-domain vs. across-domain

Geisler (2008) distinguishes between four kinds of domains: (1) Physical environments, (2) Images/Scenes, (3) Neural responses, and (4) Behavior.

Within the domain of images/scenes, one can study the characteristics of information related to redundancy and efficient coding.

Across-domain statistics determine how an autonomous system should make inferences about its environment, process information, and control its behavior. To study these statistics, it is necessary to sample or register information in multiple domains simultaneously.

The image above was generated from a database of segmented leaves that simultaneously registers natural images (scene information) with the exact locations of leaf boundaries (information about the physical environment). Such a database can be used to study across-domain statistics.

Still monocular images

Achromatic

In 1987, Field conducted a classic investigation of luminance contrast in natural images by showing that the amplitude of image intensity is inversely proportional to frequency in the Fourier domain. This suggested a type of redundancy in the way images tend to be coded. In simple terms, the result means that as the distance between any two pixels increases, the expected difference in pixel intensities also increases.

In 1994, Ruderman and Bialek later expanded on Field’s work by showing that the amplitude spectrum for a log-luminance image captured in a forested environment tends to be proportional to (1/f)^0.91 where f is the spatial frequency. For any single image, only small deviations from this rule are expected.

Brady & Field (2000) measured the distribution of contrasts in natural scenes and derived an optimal contrast response function based on a histogram equalization technique. The derived function matched the observed contrast response function of cortical cells studied by Albrecht & Hamilton (1982) demonstrating that neural processing reflects the statistics of natural images.

Frazor & Geisler (2006) also explored the range of local contrast information present in natural images. They showed that the dynamic range of luminance in natural images, including 95% of luminance samples, is 1.0 log unit (base 10), implying that luminance tends to vary by a factor of 10 in a given natural image. Similarly, the dynamic range of local RMS contrast is 1.5 log units, implying that local RMS contrast tends to vary by a factor of 30 in a given natural image. There was no correlation in the luminance of two image patches separated by the distances consistent with the typical lengths of human saccades. There was also no correlation in local contrast. In addition, local luminance and contrast are generally not correlated with each other within a given image (except that patches of sky stand out with high luminance and low contrast), implying that luminance and contrast are two independent sources of information. Mante et al. (2005) demonstrated that neural processing reflected these findings by demonstrating that gain control for the two sources of information are independent in the lateral geniculate nucleus (LGN).

Local patterns of contrast in natural images tend to be of a specific variety, and this has important implications for efficient coding of natural images. Using a technique similar to Blind Source Separation, Bell & Sejnowski (1997) revealed that the most informative components of natural scenes, which they called independent components, are similar to two-dimensional log-Gabor filters. Olshausen & Field (1997) also achieved the same result using a similar technique. These independent components look like edges, so both groups argue that if the goal of the visual system is to encode contrast, then an efficient method for doing so would be to encode these edge-like patterns. The visual system may follow this principle because these independent components resemble the receptive fields in V1, but their spatial frequency is too high on average (but this may be a result of the whitening procedures used, the whitening method is unclear).

Real images, however, are not really composed of disjoint edge elements subtending small spatial scales. Instead they are composed of long and smooth contours. Sigman et al. (2001) demonstrated that edges tend to be mutually arranged as if they come from contours of constant curvature, also called cocircular contours. Similarly, Hoyer & Hyvarinen (2002) discovered that the independent components of edge activation ensembles correspond to extended contours and end-stop formations (since some contours tend to reach an end point). Both studies are important because they demonstrate that contours, like those found in natural environments, are a higher order source of variation in natural images than disjoint edge elements.

Geisler et al. (2001) took this a step further by exploring the inferential power of edge ensembles with respect to predicting the path of contours in natural images. By analyzing the co-occurrence statistics of pairs of edges, not only did Geisler et al. confirm the observations of Sigman et al. about cocircular contours, but they precisely quantified the inferential power of the related across-domain statistics. Elder & Goldberg (2002) later confirmed the robustness of that result by using a slightly different method to arrive at a similar result. Geisler et al. also demonstrated that the statistics were important by showing how they accounted for approximately 80% of the variance of human performance in a contour completion task exploring a broad range of contour shapes. The work demonstrated that the human visual system is capable of behaving in accordance with the statistics of contour grouping in natural images, so it has led to a viable hypothesis describing the nature of algorithms in early areas of visual processing.

Chromatic

Krinov (1947) gathered natural materials and demonstrated that the reflectance spectra of these materials tend to vary smoothly and regularly as a function of wavelength to the extent that only three principal components are required to capture most of this variation. Since irradiance spectra and reflectance spectra are so regular, radiance spectra are also very regular. The human visual system may be limited to three kinds of photoreceptors because an additional type of receptor would not add any more knowledge about radiances typically observed in the environment. This leads to the idea of trichromacy, that the color of something can be described by three numbers.

Ruderman, Cronin, & Chaio (1998) studied trichromatic images captured in the natural forest environment. The basis for the trichromatic signal was to filter the image through the wavelength sensitivity functions of human L (long wavelength), M (medium wavelength), and S (short wavelength) cones (as measured by Stockman, MacLeod, and Johnson, 1993) to mimic the trichromatic signal received by the human visual system. They found that for a random image pixel, the expected log of cone responses was Gaussian and contained three principal components. The first principal component was denoted $l$ because of its resemblance to the classic idea of luminance and was roughly proportional to $\log(L)+\log(M)+\log(S)$; the second principal component, $\alpha$ , was roughly proportional to $\log(L)+\log(M)-2\log(S)$ and resembled a yellow-blue opponent channel; and the third principal component, $\beta$ , was roughly proportional to $\log(L)-\log(M)$ and resembled a red-green opponent channel. Therefore, color variation in a forested environment is well described by three numbers: $l$ , $\alpha$, and $\beta$; because the numbers are distributed as Gaussian and not correlated with each other.

Lee et al. (2002) were interested in explaining why the cone responses are highly correlated. One reason for this must be that the spectral sensitivity functions of the cones overlap, but another explanation might be that natural radiance functions tend to be smooth (as opposed to choppy). To address this question, Lee et al. invented cone sensitivity functions that did not overlap and then simulated the perception of natural radiances with those sensitivity functions. The correlational structure was still apparent in the simulated cone responses. This was important because it demonstrated that the smoothness of natural radiance functions was a major reason why this correlational structure exists.

The three-dimensional distribution of $l$, $\alpha$, and $\beta$ values may be Gaussian in forested environments, but Fine, MacLeod, and Boynton (2003) showed that for a random pair of image pixels, the expected difference in $l$, $\alpha$, and $\beta$ values is leptokurtic. In other words, when moving around an image, the values of $l$, $\alpha$, and $\beta$ either change by a small or large magnitude, with relatively few changes of medium sized magnitude (relative to Gaussian normality). In addition, there were slight correlations in the magnitude of change between $l$, $\alpha$, and $\beta$; suggesting that when one value changes, the other values are likely to change. This may be attributable to the fact that when both pixels lie on the same surface, the radiance is relatively constant, causing no change in the values; but when the pixels lie on different surfaces, the radiance changes drastically, causing a change in all the values.

Lewis and Zhaoping (2006) later demonstrated that the shape of the cone sensitivity functions is suboptimal for capturing the variance in natural radiance spectra. One possible explanation is that it was difficult for evolution to produce the required sensitivity functions, but a different explanation is that the sensitivity functions are optimized for a specific task that is important for primate survival. One possibility is that the shape helps in discriminating red surfaces (e.g. fruit) from green surfaces (e.g. leaves) given all the variation in the environment due to shading. Lovell et al. (2005) studied this problem with respect to hyperspectral images containing red fruit and green backgrounds under various lighting conditions and concluded that the L (long wavelength) and M (medium wavelength) cones are nearly optimal for making this discrimination because their sensitivity functions are broadly tuned with adjacent peaks. The starling (a type of bird), on the other hand, has narrower sensitivity functions with a large gap between the peaks, and this proves to be a less optimal design for solving the task.

A third possible explanation for the shape of cone sensitivity functions may be that it helps to isolate the sources of image contrast due to shading. Endler (1993) studied the phenomenon of shading by measuring the irradiant in several different forest environments, ranging from dense shade (no direct sunlight) to no shade (direct sunlight), from cloudy to sunny days, and through all times of day. The result was that the irradiant varied principally along one major axis, ranging from the color of blue sky at one extreme to green foliage at the other. Another axis of variation was introduced by orange, red, or purple sunsets; but that was considered relatively minor and therefore insignificant. Since the primary variation could be accounted for in the range of colors from blue to green, the relative activity of the S and M cones is more strongly affected by this variation while the relative activity between the M and L cones is more immune. Therefore, information about total changes in luminance ($l$) relative to changes in red-green levels ($\beta$) should be valuable for isolating the artifacts of shading in images.

Wachtler et al. (2001) studied the possibility that the spatial distribution of color in images might show regularity. They analyzed image patches, square arrangements of 49 pixels (7 x 7 arrangements), encoded with respect to the log of L, M, and S cone responses. A principal component analysis revealed eigenvectors that were segregated along the dimensions of $l$, $\alpha$, and $\beta$. Within these three groups, a spatial arrangement resembling checkerboard patterns was observed. The size of the checkers (from large to small) predicted the rank order of the proportion of variance accounted for (from large to small) by each eigenvector, exactly as one would expect from a principal component analysis of grayscale images (encoded with only one color channel). In other words, the principal components of color were completely separable from the principal components of spatial variation.

However, a different result emerges when an independent component analysis is run. Wachtler et al. (2001) borrowed the technique of Bell & Sejnowski (1997), but used color image patches instead of grayscale. The independent components did not show segregation along three color channels as the principal components had. Therefore, unlike the principal components, the independent components of natural images reflect a mixed relationship between spatial variation and color variation. Wachtler suggests that the representation of color in LGN, with segregated $l$, $\alpha$, and $\beta$ color channels, is more like the representation uncovered by a principal component analysis; but the representation in V1, with mixed color channels, is more like the representation uncovered by an independent component analysis, especially in light of the high similarity between independent components and V1 (referring to Van Hateren & Ruderman 1998). Regardless of the possible similarity to neural receptive fields, independent components suggest that edges may have a specific color signature, mixing information across several color channels at once.

Work unrelated to natural scene statistics has shown that human color vision is notoriously difficult to characterize. Researchers are unable to uncover a single color space amenable to the psychophysical phenomena related to color perception. Instead, researchers have uncovered several mysterious phenomena that defy simple explanation. Long, Yang, & Purves (2006) tried to explain these mysteries by demonstrating that the 3-dimensional frequency distribution of trichromatic values in natural images, when mapped onto any color space, is highly irregular (a hyper-surface with irregularly distributed hills and valleys). Histogram equalization is a way of warping this surface to a non-Euclidian space so that it becomes a hyper-plane. This warping seems to predict most of the mysterious phenomena (to extents that were not measured). It seems that the visual system somehow represents color in an efficient manner as the warped space would suggest. This demonstrates that the human visual system is built around the statistics of color in natural images.

Still binocular images

[Content needed.]

Moving monocular images (movies)

Van Hateren & Ruderman (1998) ran an Independent Component Analysis of moving image patches and showed that when the goal is to encode movies instead of still images, the independent components are moving spatial gratings with properties that strongly and precisely mimic the properties of V1 receptive fields.

Moving binocular images

[Content needed.]

Properties of 2-dimensional images and depth

Potetz & Lee (2003) demonstrated a relationship between depth and optical image properties by assembling a database of depth images simultaneously registered with grayscale optical images. Some of the scenes were natural while others contained manmade objects. The images were always captured with the horizontal axis parallel to the horizon. There was a -0.32 correlation between optical intensity and relative depth, but only for rural images. They performed ridge regression on 25 x 25 square image patches and predicted 21% of the variance in relative depth (i.e. log depth). This work demonstrated that there exists a relationship between properties of two-dimensional optical images and the depth at locations within those images.

A different approach was taken by a group who collected depth images that were not simultaneously registered with any optical images. These depth data could be used to justify several classic optical illusions. Three dimensional geometry could provide reasons why vertical lines are judged to be longer than horizontal lines of the same length (Howe & Purves 2002), why the perception of angles is biased towards the right angle (Howe & Purves 2005a), several illusions involving arranged circles (Howe & Purves 2004), the Muller-Lyer illusion (Howe & Purves 2005b), and the Pogendorff illusion (Howe et al. 2005). These findings serve to demonstrate that several biases in the human visual system may be adaptive for making judgments about three-dimensional geometry from the information contained in two-dimensional images.

Monaural auditory scenes

[Content needed.]

Binaural auditory scenes

[Content needed.]

Notes

This article was originally created on Feb. 4, 2009 using excerpts from Almon Ing's PhD Dissertation (as a service to the community).