Dappled Photography: Mask Enhanced Cameras for
Heterodyned Light Fields and Coded Aperture Refocusing

Ashok Veeraraghavan, Ramesh Raskar, Amit Agrawal, Ankit Mohan and Jack Tumblin
ACM SIGGRAPH 2007

Low res pdf
High res pdf
Slides


Frequently Asked Questions


Matlab Code for computing 4D Light Field from 2D Image

Copyright 2010 Mitsubishi Electric Research Laboratories
All Rights Reserved.

Permission to use, copy and modify this software and its documentation without fee for educational, research and non-profit purposes, is hereby granted, provided that the above copyright notice and the following three paragraphs appear in all copies.

To request permission to incorporate this software into commercial products contact: Vice President of Marketing and Business Development; Mitsubishi Electric Research Laboratories (MERL), 201 Broadway, Cambridge, MA 02139 or .

IN NO EVENT SHALL MERL BE LIABLE TO ANY PARTY FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOST PROFITS, ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS DOCUMENTATION, EVEN IF MERL HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

MERL SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE SOFTWARE PROVIDED HEREUNDER IS ON AN "AS IS" BASIS, AND MERL HAS NO OBLIGATIONS TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS OR MODIFICATIONS.

The following code takes as input the 2D image captured by our heterodyne light field camera. It then computes the 4D light field from the captured 2D image.

=======================================================================

% Source Code for Computing 4D Light-Field from Captured 2D Photo
% Mask contains Cosines with 4 Harmonics leading to 9X9 Angular Samples
m = 2133; n=1719 % Size of Captured Image

nAngles = 9; cAngles = (nAngles+1)/2; 
% Number of Angular Samples

F1Y = 237;  F1X = 191; 
%Cosine Frequency in Pixels from Calibration Image
phi1 = 300;  phi2 = 150; % PhaseShift due to Mask In-Plane Transltn wrt Sensor

F12X = floor(F1X/2); F12Y = floor(F1Y/2); 

%Compute Spectral Tile Centers, Peak Strengths and Phase
for i=1:nAngles ; 
    for j=1:nAngles
        CentY(i,j) = (m+1)/2 + (i-cAngles)*F1Y;CentX(i,j)  = (n+1)/2  + (j-cAngles)*F1X;
        Mat(i,j) = exp(sqrt(-1)*((phi1*pi/180)*(i-cAngles) + (phi2*pi/180)*(j-cAngles)));
    end
end

Mat(cAngles,cAngles) = Mat(cAngles,cAngles) * 20;

f = fftshift(fft2(imread(‘InputCones.png’)));
%Read Photo and Perform 2D FFT

%Rearrange Tiles of 2D FFT into 4D Planes to obtain FFT of 4D Light-Field
for i = 1: nAngles;
    for j = 1: nAngles
           FFT_LF(:,:,i,j) =  f(CentY(i,j)-F12Y:CentY(i,j)+F12Y,
                                           CentX(i,j)-F12X:CentX(i,j)+F12X)/Mat(i,j);
    end
end

LF     =    ifftn(ifftnshift(FFT_LF)); %Compute Light-Field by 4D Inverse FFT

%%%%End of Code
=======================================================================







Graphical Illustration of the matlab code for computing 4D Light Field from 2D Image






Consider a scene consisting of several cones at different depths. If we take an image with a traditional camera, we get the photo shown above. The magnitude
of the 2D FFT of the image shows high energy at low frequencies. This is a well known fact that most energy in natural images is concentrated
in low frequencies.


Now consider taking a 2D photo of the same scene with our Heterodyne Light Field Camera. In this new camera, we place a cosine mask near the
sensor. We get an image shown as below.







Although this image looks similar to the previous photo captured using a traditional camera, notice the effect of mask on the input photo. The mask
casts a soft shadow on the sensor. It dapples the light reaching the sensor. In theory, mask modulates the incoming 4D light field to make spectral
replicas.


Now consider the 2D FFT of this new image. Notice that the spectral replicas are clearly seen!!!. Compare this FFT image with the previous FFT image to
visualize the differences.
These replicas are due to the cosine mask placed near the sensor. The cosine mask modulates the incoming 4D light field.
In this example, the mask has 4 harmonics, or 4*2+1 = 9 impulses in its frequency response. Thus, we get 9*9=81 replicas in both x and y direction.


Now lets see how we can obtain the 4D light field from the modulated 2D photo captured from our Heterodyne Light Field Camera. The entire algorithm
is shown graphically below.










Steps:

1. Compute the 2D FFT of the captured 2D photo.
2. We know that we will get 9*9 replicas due to the physical mask placed near the sensor. Rearrange these 81 tiles into 4D.
3. Compute the inverse 4D FFT to get the light field.


Click on the above image to see a video (ViewCones1.avi) of different 'views' obtained from the light field. These views are essentially images that we
would obtain if we look through a narrow aperture on the lens at different positions.


Email agrawal at merl dot com for questions/bugs etc

Back to Dappled Photography.....
Home