Sensor fusion for imaging

Integration and combined analysis of sensors adds value to information. .

ingemar.petermann (at) acreo.se

Today’s technical equipment and systems are ­filled with a large number of sensors to gather data from entities necessary for the field of operation. ­Ana­lysing the output of each of these sensors sep­arately yields information about the respective measure­ment target. However, by integrating the sensors and analysing their output in relation to each other, new possibilities open and even more information may be obtained. This is often re­ferred to as sensor fusion or, when discussing analysis ­techniques and algorithms, data fusion.

A good example of a system utilizing sensor fusion is the human body. Our impression of the outside world is formed by a vast amount of sensory data that is mixed and processed in the brain. For instance, the data from each eye, when processed separately, will only give a two-dimensional image of the surroundings, whereas combining the sensory input from both eyes enables stereo vision, yielding an additional perceived dimension. The list of other interacting sensory systems is virtually endless. It comes as no surprise that a large part of the general research on sensor fusion has its roots in disciplines associated with robotics, where perception of the surroundings is one of the key issues.

Leaving biology and turning the focus towards technical systems, we find sensor fusion-based techniques in a wide spectrum of areas ranging from simple dedicated sensors, for gas detection or security installations for example, to large systems found in automotive, space and aerial survey applications. In terms of imaging solutions, which is the focus of this text, multi- and hyperspectral imaging constitute a particularly interesting class of sensor fusion techniques. The common point of these techniques is that spectral data is obtained for each pixel of an acquired image. Differences between the responses in different spectral ranges may e.g. give information on the material composition and distribution in the image.

Hyperspectral imaging
Ordinary imaging in its simplest form results in an image where each pixel has a value corresponding to the radiation intensity in that position. Today, ­colour video cameras are common and these are actual examples of multispectral imaging systems with simul­taneous detection of intensity values from three ­closely spaced wavelength bands in the visible range.

However, what is usually referred to as “multispectral” imaging are systems that are capable of detecting radiation in each pixel from two or a few spectral bands somewhat further spaced, such as wavelengths from the ultra-violet (UV), visible and mid-wavelength infrared (MWIR) regions. Extend­ing the capabilities even further to encompass up to several hundred detected wavelength regions per pixel is referred to as hyperspectral imaging. There are also ambitions to acquire detailed spectrometer data in each pixel, which is then called full spectral imaging. In the two latter cases, one also speaks about imaging spectrometers.

The data produced by a hyperspectral imager consists of a data cube where two of the dimensions are used for the spatial and the third for the spectral information. One of the challenges with sensor ­fusion is to be able to handle the large amount of data that is produced, and data fusion is a research field of its own. Without diving into any details, the general idea is to arrange data from an acquisition system with N parameters into an N-dimensional parameter space and then classify each acquired observation based on its position in this space. The theory necessary for analysis and classification is the same as the theory used within artificial intelligence and pattern recognition.

Realizing a hyperspectral imaging system
There are basically three different (mutually not excluding) ways to increase the sensory capabilities of an imaging system, each of which has its own advantages:

Multiple discrete sensors: the hardware fusion is limited to packaging and assembly of otherwise separate sensors.
The simplest way to accumulate more sensory data from an imaging system is to physically place different sensor systems together (e.g. a CCD cam­era in the visible and an IR camera) and then acquire the data separately followed by data fusion in the analysis software. Even though the result can be very useful this kind of system tends to be rather bulky and it is furthermore hard to align the different parts to yield a well-­determined pixel-to-pixel correspondence between the sensors.

Another possibility is to use separate sensors mounted in a common housing, possibly including further optics to divide and guide the input to each sensor. This will result in a more compact system that is easier to calibrate and handle but also re­quires a more advanced packaging and assembly tech­nology. However, for systems that can be limited to two or just a few wavelength ranges, this may currently be the most cost-effective solution.

Time-resolved parameter acquisition: each para­meter (e.g. wavelength range) is acquired one at a time by the same sensor, which is re-tuned for acquisition of the next parameter.

One of the most common solutions for creating multispectral images is to mount either a filter wheel or an electronically tunable filter in front of the sensor and acquire a sequence of monochrome ­images with different filter settings, thus obtaining the hyperspectral data cube by scanning in the spectral domain.

Spatial scanning solutions are suitable when either the object to be imaged or the imager itself is moving, which often is the case for industrial applications with objects on conveyor belts or for air- or spaceborne surveying. In this case one of the spatial dimensions of a sensor matrix is used for spatial and the other for spectral image information. The light is divided into spectral bands by aid of a grating, prism or other dispersive element mounted in front of the sensor. As the object/imager moves, new spatial lines and spectra may be ­acquired. This kind of spatial scanning is also referred to as ”push-broom” scanning.

Both the above schemes have the drawback that the final image cube is composed of parts acquired at different points in time, which may pose a problem for high-speed applications.

Sensor integration: one single sensor is de­signed to simultaneously acquire several parameters. The sensor can either be fully integrated or consist of a central sensing unit with external components such as dispersive filters.

There are techniques available for simultaneous dual wavelength acquisition with one single sensor component, which is interesting for applications where the imaging is tailored for spectroscopic recognition of a predefined substance such as a specific gas. However, a substantial extension of the number of simultaneously detectable wavelengths is not possible with existing technologies.

Some solutions for adding more spectral sensing capabilities to single sensors have been presented during the past decades, all of which trade spatial resolution for additional spectral information. Either different configurations of dispersive micro elements project spectral lines for each image pixel on the sensor chip, or the image as a whole is multiplexed both spatially and spectrally on the chip by dis­persive optics and then analysed with well-known computed tomography methods. Large parts of the sensor array are in both cases used to capture spectral and/or multiplexed image data, which means that the spatial resolution is considerably reduced. However, if the spatial resolution is of less importance for the application, this kind of solution offers fast and instantaneous acquisition of full hyper­spectral data cubes.

Applications
Looking at current literature, products and ongoing research projects, the largest application area for hyper­spectral imaging is found in air- and spaceborne sensing. This is used for earth surface characterization, geology, environmental and mineral ­mapping, and environmental health investigations. As an example of the latter, it is possible to reveal vermin infestation in forest areas at an early stage, which enables early counter-actions when the problems are still at a manageable size.

Another large application area is found in the security sector, where multispectral-based surveillance systems automatically locate possible threats. Within the field of robotics and autonomous devices there are furthermore many applications concerning positioning and perception of the surroundings (e.g. stereo vision and three-dimensional imaging).

There are also a number of industrial applications, including material inspection and sorting, detection of surface contaminants and monitoring of coating application and mixtures in tanks. An example of a fairly compact commercially available product for monitoring and maintenance is a system for identification and location of corona discharges and hot spots in electrical equipment that uses triple-band sensing in the NIR, visible and UV regions.

Apart from the more general areas listed above, there are also a number of more niched applications that benefit from different kinds of multi- and hyperspectral imaging. These include gas flow visualization and detection devices, systems for identification of counterfeit money, as well as recovery and restoration of art and historical documents.

Flexible hyperspectral systems are very useful in research and development of more dedicated sensors, but the trend is that flexibility is traded for cost and ease-of -use in the final application-specific products.

Conclusion
It is hard to define a general class of sensor fusion-based solutions. Given the multitude of possibilities, the technique has to be addressed with a given prob­lem in mind. However, it is evident that the field of sensor fusion is vast with many unexplored paths and corners and that there is much to be gained by learning to use this kind of technology and way of thinking.  