Image sensors that behave like biological retinas

Ever since the invention of the first camera obscura and the advent of photography in the 19th century, scientists have been fascinated by the use of light sensors to capture the world around us from the perspective of a man-made machine. Most recently, all eyes have been on image sensors relying on CCD or CMOS technology. These state-of-the-art camera devices can convert optical images into an electronic signal, and are used in applications for sectors including healthcare, automotive, media or security.

According to a recent report by MarketsandMarkets, image sensors will be worth some EUR 13.24 billion (USD 15.77 billion) by 2020. But while the fight for market share has led to considerable advances in terms of pixel size, pixel density, resolution and performance, there is still a long way to go before these technologies can go toe-to-toe with biological retinas.

The SEEBETTER (Seeing Better with Hybrid BSI Spatio-Temporal Silicon Retina) project is organised around the conviction that these technologies are dragged down by the way they produce redundant sequences of images at a limited frame rate. The Imec-led consortium has spent the last three years trying to overcome this problem by 'realising an advanced silicon retina with the superior quantum efficiency and spatiotemporal processing of biological retinas'. In other words, they have been studying the functional roles of various retinal ganglion cells to better understand retinal vision, after which they attempted to recreate its capacity to generate data according to timely changes in the amount of light picked up.

David San Segundo Bello, coordinator of the project, accepted to discuss the state of the project, the strengths and weaknesses of the SEEBETTER technology and its potential applications, including the hypothetical impact it could have on retinal prosthetics.

What is the main objective of the project?

The objective of SEEBETTER is fourfold: to better understand the functional role of the major classes of retinal ganglion cells; to model mathematically and computationally retinal vision processing from the perspective of biology, machine vision and future retinal prosthetics; to design and build a high-performance silicon retina with a heterogeneous array of pixels specialised for both spatial and temporal visual processing; and to use silicon back-side processing technology to increase the sensitivity of the sensor.

Each of the project partners is an expert in achieving one of these four objectives.

Artificial vision has been gaining momentum lately. What would you say are the main advantages of your technology compared to other existing solutions?

The first thing to bear in mind is that we need to be careful about the nomenclature. Our silicon retina is an image sensor manufactured in silicon, which works in a similar way to biological retinas. In this sense, it is very different from a silicon retina to be implanted in a patient as a retinal prosthesis.

Whilst our 'silicon retina sensor' could indeed be used in such an implantable 'artificial' retina, our project does not directly target this field of applications. I could say, however, that the main advantage in such a usage scenario would be that the sensor works in a similar way to the biological retina, so it could be 'easier' to hook it up to the visual nerve, but this is far beyond my area of expertise so I'm merely speculating.

If by 'artificial vision' you refer to applications of image sensors in so-called 'machine vision', then the main advantage of our sensor is its larger dynamic range compared to standard sensors. Concretely, the dynamic range of an image sensor can be defined as the difference between the lowest amount of light that can be detected before reaching the noise floor of the system, and the highest amount of light before the pixel saturates. In standard sensors, increasing the dynamic range requires a lot of effort and trade-offs related to the photodetection element, the pixel readout electronics, and the control of the pixels. In our sensor, the main limitation is in the amount of pulses that can be processed, i.e. the speed of the electronics. But since no data is generated when there are no changes in the scene, this results in lower power consumption and data rates many applications could benefit from.

How do your silicon retinas work exactly?

A silicon retina sensor works in a completely different way from most existing CCD or CMOS image sensors. Standard image sensors generate data proportionally to the amount of light landing on the sensor's pixels. The information is in the 'amplitude' of the pixel signal and the pixels are active and read at regular time intervals defined by the frame rate and/or the exposure time. The information relevant to the application will be extracted from these pixel values for each image or frame.

Our sensor, on the other hand, is based on the 'Dynamic vision sensor' (DVS) principle, which is itself inspired by the way biological retinas work. Instead of generating data proportional to the amount of light, each pixel is sensitive to timely changes impacting this amount of light. The data generated consists of asynchronous digital pulses, and the information lies in the amount of time between pulses. For example, a slow change in the amount of light on a particular pixel will generate pulses at a relatively slow rate, while sudden changes will generate pulses at a high rate. At the same time, if there are no changes in the scene being captured by the sensor, no data is generated.

How close would you say these sensors are to biological retinas?

Well naturally real biological retinas are more complex, with many different types of pixels (cells) which are also communicating with their neighbours. Such properties would be very complicated or impossible to develop with standard CMOS technology. With our project, we add some additional functionality to the pixels with respect to existing 'silicon retina sensors', but it is a small increment. Nevertheless, we are convinced that this limited functionality in comparison to real retinas can be very useful in a lot of vision applications.

Of course nothing comes for free, and these extra functionalities require larger pixels: between 10 and 20 micrometre pitches depending on the technology node and the functionality included on the pixel. In contrast, state-of-the-art standard image sensors currently feature pixels from two to five micrometres, with some manufacturers already offering pixel pitches very close to one micrometre.

What would be the most groundbreaking discovery you have made in your research so far?

In terms of groundbreaking discoveries, our biology partner, the Friedrich Miescher Institute, is the one with the most visible results, having improved our understanding of how cone cells in the retina work. This has led to several publications in high-impact journals such as Science and Cell. With regards to the sensor itself, the University of Zurich has demonstrated the first silicon retina sensor with embedded 'standard' pixels enabling more complex visual processing and extending the utility of these sensors. Imperial College developed a hardware emulator of a retina sensor using standard off-the-shelf cameras. Finally, Imec successfully developed and implemented silicon back-side processing for image sensors which can be used for high-volume applications.

Where do you stand with the production of high-performance silicon retinas?

We manufacture our sensor with a major silicon foundry. If the sensor were to be used in large quantities, almost all of the pieces would be in place for its production in large volumes.

What are the next steps for the project, and do you have any follow-up plans after its end?

We are in the last months of the project. The final device has been manufactured and is currently starting to be tested. There are no plans for the current consortium to follow up on this project, but all members will continue working on the technologies developed and the discoveries made over the course of this project.