Light-Field Camera (Plenoptic)

A camera that records not just where light landed, but which direction it was travelling — letting you choose the focus after the shutter has closed.

A plenoptic camera inserts a microlens array — a sheet of hundreds of thousands of tiny lenses — between the main camera lens and the image sensor. Each microlens captures a small sub-image that encodes both the position of the incoming light on the main lens aperture (angle) and the position of the scene point (spatial location). The resulting raw image is a mosaic of thousands of tiny circular sub-images, one per microlens. Post-processing software untangles this mosaic to: (1) digitally refocus the image at any depth after capture; (2) generate multiple perspective views of the same scene; and (3) compute a depth map from parallax between views. The core cost is a fundamental trade-off: every pixel devoted to recording angular information is a pixel no longer available for spatial resolution, so effective image resolution is a small fraction of raw sensor resolution.

In the plenoptic 1.0 design (Adelson & Wang 1992; Ng et al. 2005), the microlens array is placed at the sensor's native image plane. Each microlens re-images the main lens aperture onto the N×N pixels beneath it. The pixel location within each micro-image encodes ray angle, and the identity of the microlens encodes ray position. Extracting a single viewpoint ('sub-aperture image') means assembling the pixel at a fixed position within every micro-image across the array. If each microlens covers N×N pixels, the camera captures N² views but only as many spatial pixels as there are microlenses — a steep resolution cost. Plenoptic 2.0 (focused plenoptic) moves the microlens array away from the native image plane, allowing each microlens to image a patch of the intermediate image rather than the aperture, recovering higher spatial resolution at the cost of a more complex depth-decoding pipeline. Raytrix cameras extend this further with a multi-focal microlens array using three interleaved focal lengths, extending depth of field.

In plain terms

Imagine you are trying to take a photo of a garden, but you have thousands of tiny magnifying glasses lined up in a grid right in front of the film. Each little magnifying glass records what the garden looks like from slightly different angles. Later, on a computer, you can pick any angle you like and assemble a sharp picture of just the flowers, or just the bees — all from the one photo you already took.

Also called: plenoptic camera, plenoptic 1.0, plenoptic 2.0, focused light-field camera, refocusable camera.

Why it matters

Captures a 4D light field — two spatial dimensions and two angular dimensions — in a single camera exposure with no moving parts.
Enables post-capture refocusing, extended depth of field, and stereo depth maps from a standard camera body with only sensor modification.
The same microlens array principle underpins light-field microscopy, where post-capture refocusing reduces phototoxic illumination in biological specimens.
Industrial plenoptic cameras (Raytrix) provide single-shot depth maps for inline metrology, useful where illuminating a part with structured light is impractical.
Honest limitation: the spatial-resolution penalty is severe. A 41.5 MP raw sensor yields only ~10 MP effective resolution in the Raytrix R42, and each level of angular oversampling directly subtracts from spatial pixel budget.

Where it lives

Ng et al. 2005 hand-held prototype (Stanford): 296×296 microlens array, 125 µm pitch, f/4, ~4000×4000 raw pixels
Lytro first-generation consumer camera (2012): ~100,000 microlenses, 13.9 µm pitch (now discontinued; Lytro wound down in 2018)
Lytro Illum (2014): 40-megaray sensor, ~5 MP effective spatial resolution, 1" sensor, f/2 fixed lens (discontinued)
Raytrix R42 industrial camera: 41.5 MP raw (7728×5368, 1.1 µm pixels), up to 10 MP effective, multi-focal MLA, 7 fps
Raytrix R12/R25/R26 series: 12–26 megarays, up to 6.5 MP effective, 60–80 fps
Light-field microscope (Levoy et al., SIGGRAPH 2006): adapted research microscope with MLA at intermediate image plane

Key numbers

Microlens pitch (Ng 2005 prototype): 125 µm (Adaptive Optics Associates MLA, 296×296 lenslets, f/4)
Lytro 1st-gen microlens pitch: ~13.9 µm (~100,000 microlenses; original consumer device)
Lytro Illum sensor: 40 megarays → ~5 MP spatial (1" CMOS sensor; spatial resolution ≈ 12.5% of raw pixel count)
Raytrix R42 raw / effective: 41.5 MP raw / ~10 MP effective (7728×5368 sensor, 1.1 µm pixels; multi-focal MLA)
Raytrix R42 frame rate: 7 fps at full resolution (Higher rates at reduced spatial resolution)
Directional samples (Ng prototype): ~14×14 per microlens (N≈12–14 pixels across each microlens sub-image; determines refocus range)

Surprising facts

Adelson and Wang's 1992 paper at MIT coined the word 'plenoptic' — from Latin plen (full/complete) and optic — because the camera captures the complete plenoptic function: intensity as a function of position, angle, wavelength, and time.
Ng's 2005 Stanford prototype used a microlens array originally made for adaptive optics (Adaptive Optics Associates, part 0125-0.5-S) with 296×296 lenslets at 125 µm pitch — roughly the diameter of a thin human hair each.
The Lytro Illum's 40-megaray sensor recorded 40 million individual light-ray samples per shot yet delivered a final spatial resolution of only ~5 megapixels — 87% of the raw pixel budget was spent recording angular information, not scene content.
Lytro raised approximately $215 million in venture capital over 12 years, shipped two consumer light-field camera generations, then shut down in March 2018 — a cautionary tale about a technically correct product searching for a mainstream use case.

Open questions

The resolution collapse that no algorithm fully recovers: In the plenoptic 1.0 design, spatial resolution is fundamentally bounded by the number of microlenses — independent of how large the underlying pixel count grows. Researchers have applied super-resolution algorithms that exploit the micro-parallax between adjacent sub-aperture views, and in principle these should partially recover the lost resolution. In practice, the recoverable gain depends sensitively on scene texture, depth gradients, and microlens calibration errors in ways that no published model fully predicts. Cameras built a decade apart with sensors ten times larger show only modest improvement in reconstructed spatial resolution, a persistent gap between theory and experiment that is widely acknowledged but incompletely explained.

A short history

The intellectual roots trace to Gabriel Lippmann's 1908 integral photography proposal (Comptes Rendus 146:446-451) and Frederic Ives's 1903 lenticular parallax patent (US 725,567). The modern plenoptic camera was conceived by Edward Adelson and John Wang at MIT's Media Lab in their 1992 IEEE Transactions on Pattern Analysis and Machine Intelligence paper 'Single Lens Stereo with a Plenoptic Camera' (vol. 14, no. 2, pp. 99-106), which placed a lenticular array at the sensor plane to capture multiple simultaneous viewpoints. Marc Levoy and Ren Ng at Stanford extended the design into a hand-held prototype described in Ng's 2005 PhD thesis and the associated SIGGRAPH 2005 paper, demonstrating digital post-capture refocusing. The plenoptic 2.0 (focused plenoptic) design was introduced by Georgiev and Lumsdaine around 2009, recovering higher spatial resolution. Lytro commercialised the concept in 2012 for consumer photography; the company raised approximately $215 million (Crunchbase: $215.8 million) but wound down operations in March 2018, with some intellectual property acquired by Google.

Sources

Adelson, E.H. & Wang, J.Y.A. — Single Lens Stereo with a Plenoptic Camera, IEEE T-PAMI 14(2):99-106 (1992) (https://persci.mit.edu/pub_pdfs/plenoptic.pdf)
Ng, R. et al. — Light Field Photography with a Hand-held Plenoptic Camera, Stanford Tech Report CSTR 2005-02 (https://graphics.stanford.edu/papers/lfcamera/lfcamera-150dpi.pdf)
ORNL / OSTI — Light Field Cameras Offer Surveillance a New Dimension (Raytrix R42 evaluation) (https://www.osti.gov/servlets/purl/1524858)
CNET — Google acquires some Lytro employees as company shuts down (2018) (https://www.cnet.com/tech/tech-industry/google-may-acquire-light-field-tech-company-lytro-for-vr-ar/)
Xiao, X. et al. — Advances in 3D integral imaging: sensing, display, and applications, Appl. Opt. 52(4):546-560 (2013) (https://pubmed.ncbi.nlm.nih.gov/23385893/)