Like the Human Eye

"Cameras we use today have an array of pixels: 1024 by 768," explained Isidoros Doxas, an AI Systems Architect at Northrop Grumman. "And each pixel essentially measures the amount of light or number of photons falling on it. That number is called the flux. Now, if you display the same numbers on a screen, you will see the same image that fell on your camera."

By contrast, neuromorphic cameras only report changes in flux. If the rate of photons falling on a pixel doesn't change, they report nothing.

"If a constant 1,000 photons per second is falling on a pixel, it basically says, 'I'm good, nothing happened.' But, if at some point there are now 1,100 photons per second falling on the pixel, it will report that change in flux," Doxas said.

"Surprisingly, this is exactly how the human eye works," he added. "You may think that your eye reports the image that you see. But it doesn't. All that stuff is in your head. All the eye reports are little blips saying 'up' or 'down.' The image we perceive is built by our brains."

Advantages of Neuromorphic Cameras

Doxas identified several advantages in neuromorphic imaging, beginning with reduced power requirements.

"You need a lot of energy to send a number from pixel to computer or from eye to brain," he said. "And you don't want to spend all that energy. In fact, that's why people started thinking about building neuromorphic cameras. They require much less energy because the pixels just report changes and not actual values."

He continued, "Another important advantage is speed. If you have a million pixels, and you have to send the computer a thousand frames per second, that's one billion numbers per second. However, usually nothing changes in a scene from one millisecond to the next, so you don't need to report that entire image."

Doxas likened the process to compression methods for video entertainment, noting that a 4K-resolution movie represents 8 million pixels, times three colors, times 30 frames per second.

"That's over a gigabyte per second," he said. "Yet, you can watch that over an internet connection that's only a few megabits per second. That's because little changes from one frame to the next. They leverage that fact and compress frames in the same way. The difference here is that neuromorphic cameras do the compression."

Decompression is accomplished by computers, where reporting speeds accelerating from one to tens of thousands of frames per second allow for millisecond reaction times. This paves the way for a huge range of different applications — from active combat systems detecting and defeating a bullet, to self-driving cars interpreting dangerous situations almost instantly.

Doxas said that Northrop Grumman has been involved with the technology for more than a decade, highlighting a collaboration with Johns Hopkins University that resulted in the recent design of a readout integrated circuit as well as "the brains that go behind that circuitry."

Future efforts will include increasing the number of pixels, further lowering power and increasing resolution.