Reinterpretable Imager: Towards Variable Post-Capture

Space, Angle and Time Resolution in Photography


Amit Agrawal, Ashok Veeraraghavan and Ramesh Raskar
Eurographics 2010


Take a photo: Decide later if it stays a photo, becomes a video or turns into a lightfield. Ultimate post-capture control in software.


We describe a novelmultiplexing approach to achieve tradeoffs in space, angle and time resolution in photography. We explore the problem of mapping useful subsets of time-varying 4D lightfields in a single snapshot. Our design is based on using a dynamic mask in the aperture and a static mask close to the sensor. The key idea is to exploit scene-specific redundancy along spatial, angular and temporal dimensions and to provide a programmable or variable resolution tradeoff among these dimensions. This allows a user to reinterpret the single captured photo as either a high spatial resolution image, a refocusable image stack or a video for different parts of the scene in post-processing.

A lightfield camera or a video camera forces a-priori choice in space-angle-time resolution. We demonstrate a single prototype which provides flexible post-capture abilities not possible using either a single-shot lightfield camera or a multi-frame video camera. We show several novel results including digital refocusing on objects moving in depth and capturing multiple facial expressions in a single photo.

Paper (Preprint)
High res pdf (62 MB), Low res pdf (5MB)

Video, avi, 50 MB 

Talk Slides


1. Capturing a video in a single shot. In this example, we trade-off spatial resolution to capture temporal resolution in a single photo.

(Left) A single captured photo using our camera, where a person is making facial expressions within the exposure time. (Right) Low spatial resolution video frames recovered from the photo.

2. This example shows how we can have different resolution for different parts of the image, not possible with a conventional camera. In this example, there are several static dolls and static flowerpot in the back. During the exposure of the photo, a doll is rotated on the right. From the single captured image, we can obtain a video for the rotating doll and lightfield information for the static parts of the scene.

Single Captured Photo

GIF animation shwoing obtained video (9
frames for rotating doll)

GIF animation shwoing digital refocusing
on static dolls and flowerpot. The focus is
changing from back to front in 4 steps.

Optical Design

The optical design of Reinterpretable Imager is simple to understand. In a Coded Aperture camera, a static mask is placed in the aperture for applications such as extended depth of field. This design was showed by our group in SIGGRAPH 2007. To capture lightfields, one can insert a high frequency mask close to the sensor to achieve optical heterodyning. This design was also showed by our group in SIGGRAPH 2007. Reinterpretable imager uses a dynamic mask in the aperture and a static near-sensor mask. For dynamic aperture mask, we either move a pinhole or a slit across the aperture during the exposure time of the camera to capture the photo.

By moving the pinhole in the aperture during the expsoure time of the camera, the temporal ray variations in the scene are mapped to angular variations in the aperture. These are captured by the mask close to the sensor as a light field. Thus, if the scene only consist of temporal variations (a dynamic scene in focus), temporal frames can be captured as sub-aperture "views" of the light field. For static scenes, the sub-aperture "views" of the light field correspond to the angular variations, using which digital refocusing can be done. Thus, depending on the scene, the captured photo can be "reinterpreted" as a short video or a light field.

Similarly, by moving a vertical slit in the apertur, the temporal variations in the scene are mapped to horizontal angular variations in the aperture. And angular variations in the scene are mapped to vertical angular variations in the aperture. Thus, the captured light field views correpsond to temporal variations in one dimension and angular variations in other dimension. This can be used to digitally refocus (in 1D) on an object moving in depth.


Our implementation is as shown above. We use a medium format Mamiya camera as the sensor. We insert a high frequency mask on top of the CCD for capturing lightfields via optical heterodyning. This arrangement is similar to as shown previously in our SIGGRAPH 2008 paper on glare reduction. The more interesting part is the implementation of the dynamic mask in the aperture. We use a motor drive plastic wheel, on which printed plastic pattern as shown above are attached. Note that we capture only a single photo. During the exposure time of the camera, the motor rotates the wheel so that light is gathered through each pinhole (or slit) for equal amount of time. This video shows the motion of the wheel during the exposure time.

Back to my homepage