Evaluation

After performing reconstruction, we are typically interested in evaluating the quality of the reconstruction. To this end, four metrics are made available:

  • Mean squared error (MSE): lower is better with a minimum of 0.

  • Peak signal-to-noise ratio (PSNR): higher is better with values given in decibels (dB).

  • Structural similarity index measure (SSIM): higher is better with a maximum of 1.

  • Learned Perceptual Image Patch Similarity (LPIPS): perceptual metrics that used a pre-trained neural network on patches. Lower is better with a minimum of 0. NB: only for RGB!

Note that in the examples below, YAML configuration files are read from the configs directory. --help can be used to see the available options.

On a single file

The script scripts/compute_metrics_from_original.py shows how to compute the above metrics for a single file by (1) extraction a region of interest and (2) comparing it to a reference file.

After downloading the example files:

wget https://drive.switch.ch/index.php/s/NdgHlcDeHVDH5ww/download -O data.zip
unzip data.zip -d data
cp -r data/*/* data/
rm -rf data/LenslessPiCam_GitHub_data
rm data.zip

The script can be run with:

python scripts/compute_metrics_from_original.py

Default parameters will be used from the configs/compute_metrics_from_original.yaml file.

More information can be found in this Medium article.

DiffuserCam Lensless Mirflickr Dataset (DLMD)

The DiffuserCam Lensless Mirflickr Dataset (DLMD) comes with (lensed, lensless) image pairs, namely an image that is captured with a conventional lensed camera and a corresponding image that is captured with the diffuser-based camera.

The original dataset is quite large (25000 files, 100 GB). So we’ve prepared this subset (200 files, 725 MB).

After downloading the data, you can run ADMM on the subset with the following script.

python scripts/evaluate_mirflickr_admm.py

The default parameters can be found in the configs/evaluate_mirflickr_admm.yaml file.

It is also possible to set the number of files.

python scripts/evaluate_mirflickr_admm.py n_files=10 save=True

The save option will save a viewable image for each reconstruction.

You can also apply ADMM on a single image and visualize the iterative reconstruction.

python scripts/apply_admm_single_mirflickr.py

The default parameters can be found in the configs/apply_admm_single_mirflickr.yaml file.

Benchmarking with PyTorch

It may be useful to benchmark reconstruction algorithms with PyTorch, e.g. with a parallel dataset of lensless and corresponding lensed images.

ParallelDataset is a PyTorch Dataset object that can be used to load a parallel dataset of lensless and corresponding lensed images. The function benchmark() can be used to evaluate a reconstruction algorithm on a parallel dataset in batches.

Running the following file will evaluate ADMM on a subset of DLMD:

python lensless/benchmark.py

Evaluation API

Metrics

lensless.eval.metric.mse(true, est, normalize=True)[source]

Compute the mean-squared error between two images. The closer to 0, the closer the match.

Parameters

  • true (ndarray) – Ground-truth image, same shape as est.

  • est (ndarray) – Test image.

  • normalize (bool) – Whether to normalize such that maximum value is 1.

Returns

mse – The mean-squared error (MSE) metric.

Return type

float

lensless.eval.metric.psnr(true, est, normalize=True)[source]

Compute the peak signal to noise ratio (PSNR) for an image. The higher the value, the better the match.

Parameters

  • true (ndarray) – Ground-truth image, same shape as est.

  • est (ndarray) – Test image.

  • normalize (bool) – Whether to normalize such that maximum value is 1.

Returns

psnr – The PSNR metric.

Return type

float

lensless.eval.metric.ssim(true, est, normalize=True, channel_axis=2, data_range=None, **kwargs)[source]

Compute the mean structural similarity index between two images. Values lie within [0, 1]. The closer to 1, the closer the match.

Parameters

  • true (ndarray) – Ground-truth image, same shape as est.

  • est (ndarray) – Test image.

  • normalize (bool, optional) – Whether to normalize such that maximum value is 1.

  • channel_axis (int or None, optional) – If None, the image is assumed to be a grayscale (single channel) image. Otherwise, this parameter indicates which axis of the array corresponds to channels.

  • data_range (float or None, optional) – The data range of the input image (distance between minimum and maximum possible values). By default, this is estimated from the image data-type.

Returns

ssim – The mean structural similarity index over the image.

Return type

float

lensless.eval.metric.lpips(true, est, normalize=True)[source]

Compute a perceptual metric (LPIPS) between two images. Values lie within [0, 1]. The closer to 0, the closer the match.

GitHub: https://github.com/richzhang/PerceptualSimilarity

Parameters

  • true (ndarray) – Ground-truth image, same shape as est.

  • est (ndarray) – Test image.

  • normalize (bool) – Whether to normalize such that maximum value is 1.

Returns

lpips – The LPIPS metric.

Return type

float

lensless.eval.metric.extract(estimate, original, vertical_crop=None, horizontal_crop=None, rotation=0, verbose=False)[source]

Utility function to extract matching region in reconstruction and in original image. Later will also be resized to the same dimensions as the estimate.

Parameters

  • estimate (ndarray) – Reconstructed image from lensless data.

  • original (ndarray) – Original image.

  • vertical_crop ((int, int)) – Vertical region (in pixels) to keep.

  • horizontal_crop ((int, int)) – Horizontal region (in pixels) to keep.

  • rotation (float) – Degrees to rotate reconstruction.

  • verbose (bool) – Whether to print extracted and resized shapes.

Returns

  • estimate (ndarray) – Cropped and rotated image estimate.

  • img_resize (ndarray) – Original image resized that the dimensions of estimate.

Benchmarking

lensless.eval.benchmark.benchmark(model, dataset, batchsize=1, metrics=None, crop=None, save_idx=None, save_intermediate=False, output_dir=None, unrolled_output_factor=False, pre_process_aux=False, return_average=True, snr=None, use_wandb=False, label=None, epoch=None, use_background=True, pnp=None, swap_channels=False, **kwargs)[source]

Compute multiple metrics for a reconstruction algorithm.

Parameters

  • model (ReconstructionAlgorithm) – Reconstruction algorithm to benchmark.

  • dataset (ParallelDataset) – Parallel dataset of lensless and lensed images.

  • batchsize (int, optional) – Batch size for processing. For maximum compatibility use 1 (batchsize above 1 are not supported on all algorithm), by default 1

  • metrics (dict, optional) – Dictionary of metrics to compute. If None, MSE, MAE, SSIM, LPIPS and PSNR are computed.

  • save_idx (list of int, optional) – List of indices to save the predictions, by default None (not to save any).

  • output_dir (str, optional) – Directory to save the predictions, by default save in working directory if save_idx is provided.

  • crop (dict, optional) – Dictionary of crop parameters (vertical: [start, end], horizontal: [start, end]), by default None (no crop).

  • unrolled_output_factor (bool, optional) – If True, compute metrics for unrolled output, by default False.

  • return_average (bool, optional) – If True, return the average value of the metrics, by default True.

  • snr (float, optional) – Signal to noise ratio for adding shot noise. If None, no noise is added, by default None.

  • use_background (bool, optional) – If dataset has background, use it for reconstruction, by default True.

  • pnp (dict, optional) – Dictionary of parameters for (Parameterize and perturb) algorithm, by default None. Required keys: “mu” (distance from original parameters), “lr” (SGD learning rate), “n_iter” (number of iterations), “model_path” (original model path).

Returns

A dictionary containing the metrics name and average value

Return type

Dict[str, float]