Simulating raw data

Check out this Medium post for a detailed explanation of how to simulate raw data of lensless cameras, given a digital image of a scene and point spread function (PSF).

In short, there are several scripts inside the scripts/sim folder that can be used to simulate raw data. Behind the scenes, code from the waveprop library is used with the following simulation steps:

  1. Prepare object plane: resize and pad the original image according to the physical dimensions of the setup and camera sensor.

  2. Convolve with PSF.

  3. (Optionally) downsample: perhaps you use a higher resolution PSF than your actual camera sensor.

  4. Add noise: e.g. shot noise to replicate noise at the sensor or Gaussian noise.

  5. Quantize according to the bit depth of the sensor.

PyTorch support is available to speed up simulation on GPU, and to create Dataset and DataLoader objects for training and testing!

FarFieldSimulator

A wrapper around waveprop.simulation.FarFieldSimulator is implemented as lensless.utils.simulation.FarFieldSimulator. It handles the conversion between the HWC and CHW dimension orderings so that the convention of LenslessPiCam can be maintained (namely HWC).

class lensless.utils.simulation.FarFieldSimulator(object_height, scene2mask, mask2sensor, sensor, psf=None, output_dim=None, snr_db=None, max_val=255, device_conv='cpu', random_shift=False, is_torch=False, quantize=True, **kwargs)[source]

LenslessPiCam-compatible wrapper for FarFieldSimulator (source code on GitHub).

__init__(object_height, scene2mask, mask2sensor, sensor, psf=None, output_dim=None, snr_db=None, max_val=255, device_conv='cpu', random_shift=False, is_torch=False, quantize=True, **kwargs)[source]
Parameters

  • psf (np.ndarray or torch.Tensor, optional.) – Point spread function. If not provided, return image at object plane.

  • object_height (float or (float, float)) – Height of object in meters. Or range of values to randomly sample from.

  • scene2mask (float) – Distance from scene to mask in meters.

  • mask2sensor (float) – Distance from mask to sensor in meters.

  • sensor (str) – Sensor name.

  • snr_db (float, optional) – Signal-to-noise ratio in dB, by default None.

  • max_val (int, optional) – Maximum value of image, by default 255.

  • device_conv (str, optional) – Device to use for convolution (when using pytorch), by default “cpu”.

  • random_shift (bool, optional) – Whether to randomly shift the image, by default False.

  • is_torch (bool, optional) – Whether to use pytorch, by default False.

  • quantize (bool, optional) – Whether to quantize image, by default True.

propagate_image(obj, return_object_plane=False)[source]
Parameters

  • obj (np.ndarray or torch.Tensor) – Single image to propagate of format HWC.

  • return_object_plane (bool, optional) – Whether to return object plane, by default False.

set_point_spread_function(psf)[source]

Set point spread function.

Parameters

psf (np.ndarray or torch.Tensor) – Point spread function.

Simulating 3D data

Check out this other Medium post.

In short, the process is divided in the following steps:

  • Generate the different PSFs corresponding to the normal map of the surface of the diffuser to simulate, with light sources at different depths, or take existing ones.

  • Split the scene to render in different depth layers corresponding to the light sources we used to simulate the PSFs. Convolve the radiance coming from each layer with the corresponding PSF and sum the result on a single image.

The corresponding code will likely be added soon in waveprop.