Leveraging Multispectral Sensors for Color Correction in Mobile Cameras

Abstract

Recent advances in snapshot multispectral (MS) imaging have enabled compact, low-cost spectral sensors for consumer and mobile devices. By capturing richer spectral information than conventional RGB sensors, these systems can enhance key imaging tasks, including color correction. However, most existing methods treat the color correction pipeline in separate stages, often discarding MS data early in the process. We propose a unified, learning-based framework that (i) performs end-to-end color correction and (ii) jointly leverages data from a high-resolution RGB sensor and an auxiliary low-resolution MS sensor. Our approach integrates the full pipeline within a single model, producing coherent and color-accurate outputs. We demonstrate the flexibility and generality of our framework by refactoring two different state-of-the-art image-to-image architectures. To support training and evaluation, we construct a dedicated dataset by aggregating and repurposing publicly available spectral datasets, rendering under multiple RGB camera sensitivities. Extensive experiments show that our approach improves color accuracy and stability, reducing error by up to 50% compared to RGB-only and MS-driven baselines.

Dataset

We build a physically grounded synthetic dataset based on two publicly available hyperspectral datasets containing densely sampled spectral reflectance images (see KAUST and BJTU-UVA). From these data, we simulate corresponding RGB and multispectral (MS) measurements across a broad range of illuminants and camera spectral sensitivities, and we render ground-truth color references under the standard D65 illuminant.

To mimic geometric inconsistencies typical of dual-sensor systems, we further create a misaligned version of the dataset by introducing spatial offsets between RGB–MS image pairs. Realistic warping transformations are estimated from the Zurich dataset and applied to our synthesized data.

Dataset is available for download here.

Method

To leverage the complementary strengths of the high-resolution RGB sensor and the low-resolution MS sensor, we adopt two state-of-the-art image-to-image architectures as our base models: cmKAN and LPIENet. We refactor these architectures to integrate the MS data effectively, enabling end-to-end color correction. The resulting models are trained on our curated dataset, which includes diverse scenes rendered under various RGB camera sensitivities. By jointly processing the RGB and MS inputs, our framework learns to produce color-corrected outputs that are both accurate and visually coherent, outperforming traditional pipelines that treat these stages separately. All the models are extremely lightweight, with a parameter count spanning from 18K (cmKAN) to 220K (LPIENet), making them suitable for deployment on mobile devices.

Visual Results

Select a scene and two methods to compare. Move the sliders to compare each method with the Ground Truth.

Baseline

Ours

Synchronize sliders

Method 1 GT

FC⁴ — ΔE00: --

Method 2 GT

cmKAN — ΔE00: --

Citation

@inproceedings{leveraging2026cogo,
  title={Leveraging Multispectral Sensors for Color Correction in Mobile Cameras}, 
  author={Luca Cogo, Marco Buzzelli, Simone Bianco, Javier Vazquez-Corral, Raimondo Schettini},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2026}
}

Acknowledgments

JVC acknowledges grants PID2021-128178OB-I00 and PID2024-162555OB-I00 funded by MCIN/AEI/10.13039/501100011033 and by ERDF "A way of making Europe", and by the Generalitat de Catalunya CERCA Program. JVC also acknowledges the 2025 Leonardo Grant for Scientific Research and Cultural Creation from the BBVA Foundation. The BBVA Foundation accepts no responsibility for the opinions, statements and contents included in the project and/or the results thereof, which are entirely the responsibility of the authors. This work was partially supported by the MUR under the grant "Dipartimenti di Eccellenza 2023-2027" of the Department of Informatics, Systems and Communication of the University of Milano-Bicocca, Italy.