Tutorial on Transforming Hyperspectral Images to RGB

Tutorial on Transforming Hyperspectral Images to RGB Colour Images

This page provides an introduction to hyperspectral images and how hyperspectral reflectance or radiance image data can be transformed to RGB colour images.

A longer tutorial article on hyperspectral imaging in color vision research is available on the JOSA website here or locally here. See also Spotlight Summary.

1. Background

Hyperspectral images provide both spatial and spectral representations of scenes, materials, and sources of illumination. They differ from images obtained with a conventional RGB colour camera, which divides the light spectrum into broad overlapping red, green, and blue image slices that when combined seem realistic to the eye. By contrast, a hyperspectral camera effectively divides the spectrum into very many thin image slices, the actual number depending on the camera and application [1] [2]. This fine-grained slicing reveals spectral structure that may not be evident to the eye or to an RGB camera but which does become apparent in a range of visual and optical phenomena, including metamerism [3] and colour constancy [4].

To help understand its organisation, a hyperspectral image may be represented as a cube containing two spatial dimensions (pixels) and one spectral dimension (wavelength), as illustrated in Fig. 1. In this example (the original hyperspectral image is available here), the spectrum has been sampled at 10-nm intervals over 400-720 nm. At each sample wavelength, there is a complete intensity (grey-level) representation of the reflectance or radiance in the scene.

Figure 1. Hyperspectral image cube

The same data can be rendered as a conventional RGB colour image, as in Fig. 2. The effective spectral reflectances at selected pixels 1, 2, and 3 are indicated in Fig. 3.


Figure 2. RGB scene image with three selected pixels	Figure 3. Normalized reflectances at selected pixels of Fig. 2

The aim of this tutorial is to show how hyperspectral images of reflectance data such as those downloadable here and here can be transformed into reflected radiance data and how, in turn, these radiance data can be transformed into RGB images.

Computations are performed in MATLAB (The MathWorks Inc.), which draws on the Matlab Image Processing Toolbox. Some supporting routines, data, and a small test hyperspectral image are also needed, all of which can be downloaded as a 15 MB zipped package here.

This package should be unzipped into a directory on the MATLAB path. Some familiarity with elementary MATLAB operations and basic colour science is assumed. For an introduction to both, see [5].

If you want to experiment with other hyperspectral images, then sets of reflectance files are available here and here and separate, larger sets of radiance files are available here and here. With some hyperspectral images, the edges may need trimming to remove pixels with zero or padded values that were introduced during postprocessing.

2. Contents of the package(download here)

ref4_scene4.mat is a test hyperspectral image consisting of an array of spectral reflectances of size 255 x 355 x 33. The first two coordinates represent spatial dimensions (pixels), in row-column format, and the third coordinate represents wavelength (400, 410, ..., 720 nm), as in Fig. 1. For ease of handling in this tutorial, the image has been reduced spatially to about a quarter of the size of those downloadable here.

illum_25000.mat, illum_6500.mat, and illum_4000.mat are three illuminant spectra, each vectors of length 33 (i.e. 400, 410, ..., 720 nm), representing the spectra of blue skylight with correlated colour temperature (CCT) 25000 K, daylight with CCT 6500 K, and evening sunlight with CCT 4000 K [6]. Formulae for generating daylight spectra are provided in [6].

xyzbar.mat contains the CIE 1931 colour-matching functions [6]. Tablulated values are also available here.

XYZ2sRGB_exgamma.m is a routine for converting tristimulus values XYZ to the default RGB colour space sRGB, but without gamma correction . A copy of the document IEC_61966-2-1.pdf explaining the full conversion of XYZ to sRGB is also included.

3. Reflectance data

Load ref4_scene4.mat and test the size of the resulting array reflectances.

load ref4_scene4.mat;

size(reflectances)

It should have size 255 x 335 x 33. To inspect reflectances, try taking a slice at a middle wavelength, say 560 nm (17th in the sequence 400, 410, , 720 nm), and displaying it. The image should look like Fig. 4 (except for the added red square, left).

slice = reflectances(:,:,17);

figure; imagesc(slice); colormap('gray'); brighten(0.5);


Figure 4. Slice from hyperspectral reflectance image at 560 nm	Figure 5. As Fig. 4 with intensity levels clipped to level of red square in Fig. 4

The command brighten is used to compensate roughly for the gamma of the monitor or display (i.e. the nonlinear input-output function), and an equivalent effect can be produced with slice.^0.5, or slice.^0.4, in place of slice as the argument to imagesc. The image still appears dark, however, because the exposure used during image acquisition by the hyperspectral camera was determined by the specular highlight at the top right of the scene. For presentational purposes, therefore, the intensity levels of the image in Fig. 4 may be truncated by clipping to a level, z say, of a less-specular region (all intensity levels above z are then set equal to z), and then rescaling. The clip level z was taken from the pixel with row-column coordinates (100, 39) in the red square. The image should now look like Fig. 5. Detail in the darker areas is now much clearer than in Fig. 4.

z = max(slice(100, 39));

slice_clip = min(slice, z)/z;

figure; imagesc(slice_clip.^0.4);
colormap('gray');

Now choose a different pixel in the scene, say with row-column coordinates (141, 75), and plot its spectral reflectance. The plot should look like Fig. 6.

reflectance =
squeeze(reflectances(141, 75,:));

figure; plot(400:10:720, reflectance);

xlabel('wavelength, nm');

ylabel('unnormalized reflectance');

Figure 6. Unnormalized spectral reflectance at pixel (141, 75)

The value of reflectance exceeds unity at some wavelengths, as in Fig. 6. This is because the reflectances were normalized against an area on the grey sphere at the bottom of the image, and the intensity of light reflected from some other surfaces, at different angles, is greater, e.g. at the pixel with row-column coordinates (100, 39). The array reflectances may be normalized by dividing by the maximum reflectance taken over all pixels and wavelengths.

reflectances =
reflectances/max(reflectances(:));

For more a detailed explanation of normalization, see Section 6.2 later; also the technical notes section here and Appendix A of [3].

4. Radiance data

Load one of the illuminant spectra, say illum_25000. Multiply each reflectance in the scene by this illuminant spectrum to get the reflected radiances radiances_25000 (a for-loop is less efficient than MATLAB's bsxfun function, but it makes little difference here). Using the same pixel as for Fig. 6, plot the reflected radiant spectrum radiance_25000 (notice the different spelling). The spectrum should look like curve 1 in Fig. 7.

load illum_25000.mat;

radiances_25000 = zeros(size(reflectances)); % initialize array

for i = 1:33,

  radiances_25000(:,:,i) = reflectances(:,:,i)*illum_25000(i);

end

radiance_25000 = squeeze(radiances_25000(141, 75, :));

figure; plot(400:10:720, radiance_25000, 'b'); % blue curve

xlabel('wavelength, nm');

ylabel('radiance, arbitrary units');

hold
on;

Repeat with another illuminant spectrum, e.g. illum_4000. The reflected radiance spectrum should look like curve 2 in Fig. 7. Notice how the reflected spectra differ from each other and from the spectral reflectance in Fig. 6.

load illum_4000.mat;

radiances_4000 = zeros(size(reflectances)); % initialize array

for i = 1:33,

  radiances_4000(:,:,i) = reflectances(:,:,i)*illum_4000(i);

end

radiance_4000 = squeeze(radiances_4000(141, 75, :));

      plot(400:10:720, radiance_4000, 'r'); % red
curve

Figure 7. Reflected radiance spectra at pixel (141, 75) for illuminants with CCTs (1) 25000 K and (2) 4000 K

5. Converting hyperspectral data to CIE XYZ and sRGB representations

Suppose you want to make an RGB representation of the scene in ref4_scene4.mat under a global illuminant with CCT 6500 K. Load the illuminant spectrum illum_6500 and obtain reflected radiances radiances_6500.

load illum_6500.mat;

radiances_6500 = zeros(size(reflectances)); % initialize array

for i = 1:33,

  radiances_6500(:,:,i) = reflectances(:,:,i)*illum_6500(i);

end

For convenience, put radiances = radiances_6500. The next step is to convert the radiance data radiances into tristimulus values XYZ.

5.1. CIE XYZ image

Record the size of the array radiances, and then reshape it as a matrix of w columns for matrix multiplication with the colour-matching functions. If you are working with an independent set of radiances, e.g. from here or here, then just rename the hyperspectral image as radiances.

[r
c w] =
size(radiances);

radiances = reshape(radiances,
r*c, w);

Load the CIE 1931 colour matching functions xyzbar and take the matrix product of xyzbar with radiances to get the tristimulus values XYZ at each pixel. For simplicity XYZ is not normalized.

load xyzbar.mat;

XYZ = (xyzbar'*radiances')';

Correct the shape of XYZ so that it represents the original 2-dimensional image with three planes (rather than a matrix of 3 columns), and ensure that the values of XYZ range within 0-1 (conventional normalization so that Y = 100 is unnecessary here).

XYZ
= reshape(XYZ, r, c, 3);

XYZ = max(XYZ, 0);

XYZ = XYZ/max(XYZ(:));

5.2. RGB colour image

To convert this XYZ representation of the image to the default RGB colour representation sRGB (IEC_61966-2-1), excluding any gamma correction, apply the transformation XYZ2sRGB_exgamma. Ensure that the RGB values range within 0-1 by clipping to 0 and 1 (or, if they greatly exceed 1, perhaps scaling by the maximum).

RGB
= XYZ2sRGB_exgamma(XYZ);

RGB = max(RGB, 0);

RGB = min(RGB, 1);

Finally, display the RGB image, including a rough gamma correction. The image should look like Fig. 8 (except for the added red square, bottom left).

figure;
imshow(RGB.^0.4, 'Border','tight');


Figure 8. RGB image of reflected hyperspectral radiance image	Figure 9. As Fig. 8 but intensity levels clipped to level of red square in Fig. 8

The image appears dark for the same reason mentioned in Section 3, namely the hyperspectral camera exposure having to accommodate the specular highlight on the top right. Again for presentational purposes, the RGB levels of the image may be uniformly truncated by clipping to the level of a less-specular region, and then rescaling. The clip level was taken from the pixel with row-column coordinates (244, 17) in the red square at the bottom left. The image should now look like Fig. 9. As with Fig. 5, detail in the darker areas is now much clearer. The same procedure may be used to obtain sRGB images with the other two illuminant spectra with CCTs of 25000 K and 4000 K.

z
= max(RGB(244,17,:));

RGB_clip = min(RGB, z)/z;

figure; imshow(RGB_clip.^0.4, 'Border','tight');

Nevertheless, clipping to the same value for all three RGB levels can lead to artifacts, and a more principled gamut-mapping procedure would preserve, for example, the hue and lightness of the clipped values [7].

Other examples of conversions from hyperspectral data to RGB images are available with reflectance files here and here and radiance files here and here.

6. Technical notes

Two issues need to be considered more carefully: one has to do with the non-smoothness of natural reflectance spectra and the other to do with effective spectral reflectance values. Further discussion can be found here or here.

6.1. Non-smoothness of natural reflectance spectra

Sample spectra from natural scenes recorded with a hyperspectral camera may appear less smooth than spectra from single pigments recorded with a spectroradiometer in the laboratory. The reason is that the physics of the two situations is very different. Natural scenes usually are complex, have non-uniform illumination, and are recorded remotely by the camera, whereas laboratory pigment spectra usually are simple, have uniform illumination, and are recorded with a non-imaging device that averages signals over large homogeneous areas of the specimen. Where imaging conditions coincide, hyperspectral imaging and spectroradiometric data can match closely (see e.g. spectra of acrylic paints imaged under laboratory illumination in Fig. 1 of [3]).

The complexity of natural spectra is due to several factors.

1. Many natural scenes contain foliage, and foliage contains pigments such as chlorophylls and carotenoids, and anthocyanins in autumn, which give multiple large and small absorbance peaks distributed over the visible spectrum (see e.g. [8]. Depending on the particular combination of these pigments, their presence can produce a variety of complex reflectance spectra from region to region.

2. There are important physical surface effects associated with reflection [9] [10], which cause the reflected spectrum to vary markedly across different microscopic areas. More generally, natural organic and inorganic materials are likely to be spatially nonuniform, and may be contaminated by dirt and moisture, which along with weathering and biological degredation contribute to the complexity of the recorded spectrum.

3. Natural illumination is often variegated. Direct illumination from the sun may be combined with indirect illumination from the sky, together with local nonlinear mutual illumination, transillumination, and occlusion. Time-lapse sequences of hyperspectral images of natural scenes showing changing patterns of illumination over the course of the day are available here, and described in [11]. Thirty hyperspectral radiance images of natural scenes with multiple embedded probe spheres showing variation in local illumination estimates are available here and described in [12].

4. A further complication with hyperspectral imaging of natural scenes comes from having to limit the camera exposure duration to avoid the effects of fluctuating illumination and movement within the scene. For a combination of reasons, the signal-to-noise ratio may be low at the extremes of the visible range. In particular, the transmittance of the tunable liquid crystal filter used in some hyperspectral imaging devices is poorest at short wavelengths, where the incident radiant power from the sun is also least. Image slices at 400 nm may be particularly noisy (see examples here and here). Images with higher signal-to-noise ratios, obtained by averaging spatially registered replicates, are available here, and described in [11].

Further discussion of natural spectra can be found in Section 3.C of [13].

6.2. Effective reflectance values

The spectral reflectance of a surface element in a hyperspectral image is defined essentially as the quotient of the recorded radiant spectrum by the recorded radiant spectrum from a matt reference surface embedded in the scene, multiplied by the known spectral reflectance of the reference surface [3], Appendix A). From these effective spectral reflectance values, spectral radiance values can be recovered as outlined in Section 4.

Some of the spectral reflectance values available here and here exceed unity (also as in Fig. 6 here), despite the image being normalized with respect to a reference surface in the scene, usually a well-illuminated area (although see e.g. Fig. 2, bottom right). These excessive spectral reflectance values are a result of the optics of the scene and are intrinsic to these radiance calculations.

1. Excessive spectral reflectance values are due mainly to the nature of the bidirectional reflectance distribution function (BRDF) sampled by the scene and camera geometry. Thus when the reference surface is flat, it is almost always oriented with its surface normal towards the camera and away from the sun. But other surfaces in the scene may be oriented more favourably and be capable of reflecting more light to the camera. Even when the reference surface is spherical so that there is an optimal surface normal at some point (e.g. Fig. 8, bottom left), specular reflections from other surfaces in the scene may reflect more light to the camera (e.g. from water, as in Fig. 8, top right).

2. As explained in the technical notes section here, excessive spectral reflectance values are not important for a given scene and camera geometry, since it is the estimation of the reflected spectral radiance that matters. But for the analysis of the spectral reflectances themselves, it may be best to uniformly scale all the spectral reflectances in a hyperspectral image so that the greatest value does not exceed unity(Section 3, end). Unlike the manipulations for image presentation (Figs 5 and 9), spectral reflectances should not be scaled by ranked pixel values, e.g. the 90th or 99th centile, as this kind of clipping can substantially inflate both the spectral reflectance and accompanying noise estimates.

3. A fuller discussion of effective reflectances and effective illuminants is available in the technical notes section here, in Appendix A of [3], and in Sections 3.A and 3.B of [13].

4. A comparison of simulated global changes in illumination with natural changes in illumination in which uneven changes in illumination geometry were excluded at sample points is available in [11, Sect. 3.6]. Depending on the sampling constraints, deviations in ratios of cone excitations across pairs of sample points were 3.1%–4.8% with simulated global changes in illumination spectrum and 3.1%–5.8% with natural changes in illumination.

7. Hyperspectral imaging in color vision research

Reference [13] provides an introduction to terrestrial and close-range hyperspectral imaging and some of its uses in human color vision research. The main types of hyperspectral cameras are described together with procedures for image acquisition, postprocessing, and calibration for either radiance or reflectance data. Image transformations are defined for colorimetric representations, color rendering, and cone receptor and postreceptor coding. Several example applications are also presented. These include calculating the color properties of scenes, such as gamut volume and metamerism, and analyzing the utility of color in observer tasks, such as identifying surfaces under illuminant changes. The effects of noise and uncertainty are considered in both image acquisition and color vision applications.

If you use this material in published research, please cite the publication in full: Foster, D.H., & Amano, K. (2019). Hyperspectral imaging in color vision research: tutorial. Journal of the Optical Society of America A, 36, 606-627.

8. References

J. Y. Hardeberg, F. Schmitt, and H. Brettel, Multispectral color image capture using a liquid crystal tunable filter, Optical Engineering, vol. 41, pp. 2532-2548, 2002.
H. F. Grahn and P. Geladi, Techniques and Applications of Hyperspectral Image Analysis, Chichester, England: John Wiley & Sons, Ltd, 2007.
D. H. Foster, K. Amano, S. M. C. Nascimento, and M. J. Foster, Frequency of metamerism in natural scenes, Journal of the Optical Society of America A - Optics Image Science and Vision, vol. 23, pp. 2359-2372, 2006. [download PDF here]
D. H. Foster, K. Amano, and S. M. C. Nascimento, Color constancy in natural scenes explained by global image statistics, Visual Neuroscience, vol. 23, pp. 341-349, 2006. [download PDF here]
S. Westland and C. Ripamonti, Computational Colour Science using Matlab, Chichester, England: John Wiley & Sons, 2004.
CIE, Colorimetry, 4th Edition, CIE Publication 015:2018 (CIE Central Bureau, Vienna, 2018).
J. Morovič, Color Gamut Mapping, Chichester, England: John Wiley & Sons, 2008.
D. A. Sims and J. A. Gamon, Relationships between leaf pigment content and spectral reflectance across a wide range of species, leaf structures and developmental stages, Remote Sensing of Environment, vol. 81, pp. 337-354, 2002.
J. B. Clark and G. R. Lister, Photosynthetic action spectra of trees. 2. Relationship of cuticle structure to visible and ultraviolet spectral properties of needles from 4 coniferous species, Plant Physiology, vol. 55, pp. 407-413, 1975.
J. H. McClendon, The micro-optics of leaves. 1. Patterns of reflection from the epidermis, American Journal of Botany, vol. 71, pp. 1391-1397, 1984.
Foster, D.H., Amano, K., and Nascimento, S.M.C. (2016). Time-lapse ratios of cone excitations in natural scenes. Vision Research, 120, 45-60, http://dx.doi.org/10.1016/j.visres.2015.03.012. [download PDF here]
Nascimento, S.M.C., Amano, K., and Foster, D.H. (2016). Spatial distributions of local illumination color in natural scenes. Vision Research, 120, 39-44, http://dx.doi.org/10.1016/j.visres.2015.07.005. [download PDF here]
Foster, D.H. and Amano, K. (2019). Hyperspectral imaging in color vision research: tutorial. Journal of the Optical Society of America A, 36, 606-627. http://dx.doi.org/10.1364/JOSAA.36.000606. [download PDF here]

A Concise Guide to Communication in Science & Engineering

		How to write research articles, literature reviews, technical reports, dissertations and theses How to deal with mathematical material, statistical inference, data fitting, plotting graphs How to submit papers for publication, measure impact, present posters, give lectures and talks How to handle ethical issues, manage data, avoid plagiarism The advice in this guide is based on several hundred sources. It includes practical checklists at the ends of chapters for quick reference and verification, and it has over 400 examples of good and bad scientific writing and graphing from published research articles.
Published by Oxford University Press, November 2017, 408 pages. ISBN: 9780198704249. Available as hardback, paperback, or ebook.

d.h.foster@manchester.ac.uk | +44 (0)161 306 3888 | www.eee.manchester.ac.uk/d.h.foster
Department of Electrical and Electronic Engineering, University of Manchester, Manchester M13 9PL, UK