Imaging from temporal data via spiking convolutional neural networks

Kirkland, Paul and Kapitany, Valentin and Lyons, Ashley and Soraghan, John and Turpin, Alex and Faccio, Daniele and Di Caterina, Gaetano; Buller, Gerald S. and Hollins, Richard C. and Lamb, Robert A. and Laurenzis, Martin and Camposeo, Andrea and Farsari, Maria and Persano, Luana and Busse, Lynda E., eds. (2020) Imaging from temporal data via spiking convolutional neural networks. In: Emerging Imaging and Sensing Technologies for Security and Defence V; and Advanced Manufacturing Technologies for Micro- and Nanosystems in Security and Defence III. SPIE, Bellingham, Washington. ISBN 9781510638938 (https://doi.org/10.1117/12.2573484)

[thumbnail of Kirkland-etal-SPIE-2020-Imaging-from-temporal-data-via-spiking-convolutional]
Preview
 Text. Filename: Kirkland_etal_SPIE_2020_Imaging_from_temporal_data_via_spiking_convolutional.pdf
Accepted Author Manuscript
Download (742kB)| Preview

Abstract

A new approach for imaging that is solely based on the time of flight of photons coming from the entire imaged scene, combined with a novel machine learning algorithm for image reconstruction: a spiking convolutional neural network (SCNN) named Spike-SPI (Spiking - Single Pixel Imager). The approach uses a single point detector and the corresponding time-counting electronics, which provide the arrival time of photons in the form of spikes distributed over time. This data is transformed into a temporal histogram containing the number of photons per arrival time. A SCNN that converts the 1D temporal histograms into a 3D image (2D image with depth map) by exploiting the feature extraction capabilities of convolutional neural networks (CNNs), the high dimensional compressed latent space representations of a variational encoder-decoder network structure, and the asynchronous processing capabilities of a spiking neural network (SNN). The performance of the proposed SCNN is analysed to demonstrate the state-of-the-art feature extraction capabilities of CNNs and the low latency asynchronous processing of SNNs that offer both higher throughput and higher accuracy in image reconstruction from the ToF data, when compared to standard ANNs. The results of Spike-SPI show an increase in spatial accuracy of 15% over then ANN, using the Intersection of Union (IoU) for the objects in the scene. While also delivering a 100% increase over then ANN in object reconstruction signal to noise ratio (RSNR) from ~3dB to ~6dB. These results are also consistent across a range of IRF (Instrument Response Functions) values and photo counts, highlighting the robust nature of the new network structure. Moreover, the asynchronous processing nature of the spiking neurons allow for a faster throughput and less computational overhead, benefiting from the operational sparsity in the single point sensor.

ORCID iDs

Kirkland, Paul ORCID logoORCID: https://orcid.org/0000-0001-5905-6816, Kapitany, Valentin, Lyons, Ashley, Soraghan, John ORCID logoORCID: https://orcid.org/0000-0003-4418-7391, Turpin, Alex, Faccio, Daniele and Di Caterina, Gaetano ORCID logoORCID: https://orcid.org/0000-0002-7256-0897; Buller, Gerald S., Hollins, Richard C., Lamb, Robert A., Laurenzis, Martin, Camposeo, Andrea, Farsari, Maria, Persano, Luana and Busse, Lynda E.
CORE (COnnecting REpositories)