Why I Want A Gradient Camera
Jack Tumblin, Amit Agrawal & Ramesh Raskar
CVPR 2005


Abstract: We propose a camera that measures static gradients instead of static intensities. Quantizing sensed intensity differences between adjacent pixel values permits an ordinary A/D converter to measure detailed high contrast (HDR) scenes. We measure alternating ‘cliques’ of sensors (small groups) that locally determine their own best exposure, and reconstruct the image using a Poisson solver. This intrinsically differential design suppresses common-mode noise,hides and smoothes quantization, and can correct for its own saturated sensors. Simulations demonstrate these capabilities in side-by-side comparisons.

Here is an intuitive explanation: Consider our high dynamic range image as a height field over a rectangular grid. Consider the A/D converter as a ruler by which we

will measure this signal. The height of the ruler can be changed in two ways (a) By changing the number of bits/quantization levels (b) By changing the quantization step. Suppose the number of bits is fixed, to lets say 8, giving 2^8=256 quantization levels. The quantization step should be big enough so that the top of the ruler reaches the maximum intensity that we want to measure. In natural scenes, the dynamic range is around 1:1 million or 6 log units. So
If we naively want to measure the high dyanmic range signal, each quantization step will be 10^6/256. This will completely wipe out the fine details in both high and low intensity regions.

Log based cameras try to overcome this problem by measuring the log of the input intensity signal. Thus in log units, one needs to have a quantization step of 6/256=0.0234 to be able to measure the entire dynamic range. Although this preserves details in low intensity regions, details in high intensity regions are washed out
due to quantization.

The Gradient Camera that we propose solves this problem by measuring the differences of log intensities rather than the absolute value of log intensities. In our scheme, one end of A/D ruler is not tied to 0 (or ground). Instead, it is in some sense tied to the intensity of the neighboring pixel. Thus we don't have to care about increasing the quantization step to measure a high dynamic range signal. In fact, the gradient camera can do as well as human eye. It has been found emperically that human beings can sense a contrast change of 2%. Thus in a Gradient Camera we can have a quantization level of log(1.02) = 0.0086. Considering 8 bits, the dynamic range of Gradient camera is only 1.02^256 = 159.09 in intensity units or 0.0086*256=2.2 in log units !!! (as opposed to 6 for log camera). But remember that this dynamic range is used to measure differences of neighboring pixel intensities, not the absolute pixel intensity. In images of natural scenes, the distribution of the gradients has a strong peak at zero. Thus most of the gradients in natural images are low valued. By measuring these differences and summing them up (in 1D) we can measure the entire dynamic range. In 2D, it amounts to doing a 2D integration on the estimated gradients by solving a Poisson Equation.

The following image shows a high dynamic range signal in 1D. The signal has details in both high and low intensity regions. A normal intensity camera will start saturating at some point as shown in green. Log based cameras perform well in low intensity region but due to quantization, details in high intensity regions will be lost. A gradient camera can capture details well at both high and low intensity regions as only the gradients are measured.

How to measure gradients:

The trick lies in the following equation:  log(K*I(x+1)) - log(K*I(x)) =  log(I(x+1)) - log(I(x)).   Thus one can measure the difference of log intensities with ANY K value without need for storing that particular K value. Here the gain K plays the role of local adaptation to the signal. Simply, it can be thought of as the common mode signal which is discarded and only the relative differences are measured.

Note that this is a very important difference from previously proposed Artifical Retina kind of designs. In those designs, the local adaption is removed from the signal, essentially removing the low frequency components from the signal. These designs need to store the local adaption value also if they want to fully reconstruct the original signal. Instead by taking gradients, we do not have to store the K values and still we can recover both the low and high frequency components of the signal.

Consider a simple 1D Gradient Camera as shown here. To ensure local adaptation, one needs to measure two times instead of one. In the first set of measurement (A cliques), pixels in A cliques will be measured with a different gain K for each clique. Again, K need not be stored and only the differences needs to be stored. Thus one can get all the even set of differences. In next set of measurements, pixels in C cliques will be measured with a different gain K for each clique. Thus odd set of differences are obtained. Thus with this two pass strategy, one can measure all local differences, each one measured according to the local adaption. In order to obtain the absolute pixel values, one can just sum up these differences from left to right.

The strategy is exactly the same in 2D. The A and C cliques now contains a box set of 4 pixels. The final image is obtained by a 2D integration process.


1D Gradient Camera

2D Gradient Camera

Saturation in gradient camera:

The gradient camera will also saturate as a normal intensity camera whenever the gradients are extremely high. However, there are two important differences. Firstly,  in a high dynamic range signal, places where gradients are saturated will be much less as compared to places where image intensities get saturated. Secondly, and fortunately most of these saturated gradients can be recovered. This is due to the fact that the gradient field of an image should be integrable, i.e. the integral along any closed loop should be equal to zero. This property can be used to locally recover most of the saturated gradients.

And thats why I want a gradient camera. Hope you also want one now.

Also see http://www.merl.com/projects/gradient/