Event camera

From Wikipedia, the free encyclopedia
Jump to navigation Jump to search

An event camera, also known as a neuromorphic camera, silicon retina or dynamic vision sensor, is an imaging sensor that responds to local changes in brightness. Event cameras do not capture images using a shutter as conventional cameras do. Instead, each pixel inside an event camera operates independently and asynchronously, reporting changes in brightness as they occur, and staying silent otherwise. Modern event cameras have microsecond temporal resolution, 120 dB dynamic range, and do not suffer under/overexposure or motion blur[1][2].

Functional description[edit]

Event cameras contain pixels that independently respond to changes in brightness as they occur[1]. Each pixel stores a reference brightness level, and continuously compares it to the current level of brightness. If the difference in brightness exceeds a preset threshold, that pixel resets its reference level and generates an event; a discrete packet of information containing the pixel address and timestamp. Events may also contain the polarity (increase or decrease) of a brightness change, or an instantaneous measurement of the current level of illumination[3]. Thus, event cameras output an asynchronous stream of events triggered by changes in scene illumination.

Typical characteristics of image sensors
Sensor Dynamic

range (dB)

Equivalent

framerate* (fps)

Spatial

resolution (MP)

Power

consumption (mW)

Human eye 30–40 50 - 10[4]
High-end DSLR camera (Nikon D850) 44.6[5] 120 2–8 -
Ultrahigh-speed camera (Phantom v2640)[6] 64 12,500 0.3–4 -
Event camera[7] 120 1,000,000 0.1–0.2 30

*Indicates temporal resolution since human eyes and event cameras do not output frames.

Types[edit]

While all event cameras respond to local changes in brightness, there are a few variants. Temporal contrast sensors[1] produce events that indicate polarity (increase or decrease in brightness), while temporal image sensors[3] indicate the instantaneous intensity with each event. The DAVIS[8] (Dynamic and Active-pixel Vision Sensor) contains a global shutter active pixel sensor (APS) in addition to the dynamic vision sensor (DVS) that shares the same photosensor array. Thus, it has the ability to produce image frames alongside events. Many event cameras additionally carry an inertial measurement unit (IMU).

Event cameras
Name Event output Image frames Color IMU Manufacturer Commercially available
DVS128[1] Polarity No No No Inivation No
DAVIS240[8] Polarity Yes No Yes Inivation Yes
DAVIS346[9] Polarity Yes Yes Yes Inivation Yes
SEES[10] Polarity Yes No Yes Insightness Yes
Samsung DVS[11] Polarity No No Yes Samsung No
Onboard[3] Polarity No No Yes Prophesee Yes
Celex[12] Intensity Yes No Yes CelePixel Yes

Algorithms[edit]

Image Reconstruction[edit]

Night run reconstruction
A pedestrian runs in front of car headlights at night. Left: image taken with a conventional camera exhibits severe motion blur and underexposure. Right: image reconstructed by combining the left image with events from an event camera[13].

Image reconstruction from events has the potential to create images and video with high dynamic range, high temporal resolution and minimal motion blur. Image reconstruction can be achieved using temporal smoothing, e.g. high-pass or complementary filter[13]. Alternative methods include optimization[14] and gradient estimation[15] followed by Poisson integration.

References[edit]

  1. ^ a b c d Lichtsteiner, P.; Posch, C.; Delbruck, T. (February 2008). "A 128$\times$128 120 dB 15μs Latency Asynchronous Temporal Contrast Vision Sensor" (PDF). IEEE Journal of Solid-State Circuits. 43 (2): 566–576. Bibcode:2008IJSSC..43..566L. doi:10.1109/JSSC.2007.914337. ISSN 0018-9200.
  2. ^ Longinotti, Luca. "Product Specifications". iniVation. Retrieved 2019-04-21.
  3. ^ a b c Posch, C.; Matolin, D.; Wohlgenannt, R. (January 2011). "A QVGA 143 dB Dynamic Range Frame-Free PWM Image Sensor With Lossless Pixel-Level Video Compression and Time-Domain CDS". IEEE Journal of Solid-State Circuits. 46 (1): 259–275. Bibcode:2011IJSSC..46..259P. doi:10.1109/JSSC.2010.2085952. ISSN 0018-9200.
  4. ^ Skorka, Orit (2011-07-01). "Toward a digital camera to rival the human eye". Journal of Electronic Imaging. 20 (3): 033009–033009–18. Bibcode:2011JEI....20c3009S. doi:10.1117/1.3611015. ISSN 1017-9909.
  5. ^ DxO. "Nikon D850 : Tests and Reviews | DxOMark". www.dxomark.com. Retrieved 2019-04-22.
  6. ^ "Phantom v2640". www.phantomhighspeed.com. Retrieved 2019-04-22.
  7. ^ Longinotti, Luca. "Product Specifications". iniVation. Retrieved 2019-04-22.
  8. ^ a b Brandli, C.; Berner, R.; Yang, M.; Liu, S.; Delbruck, T. (October 2014). "A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision Sensor". IEEE Journal of Solid-State Circuits. 49 (10): 2333–2341. Bibcode:2014IJSSC..49.2333B. doi:10.1109/JSSC.2014.2342715. ISSN 0018-9200.
  9. ^ Taverni, Gemma; Paul Moeys, Diederik; Li, Chenghan; Cavaco, Celso; Motsnyi, Vasyl; San Segundo Bello, David; Delbruck, Tobi (May 2018). "Front and Back Illuminated Dynamic and Active Pixel Vision Sensors Comparison" (PDF). IEEE Transactions on Circuits and Systems II: Express Briefs. 65 (5): 677–681. doi:10.1109/TCSII.2018.2824899. ISSN 1549-7747.
  10. ^ "Insightness – Sight for your device". Retrieved 2019-04-22.
  11. ^ Son, Bongki; Suh, Yunjae; Kim, Sungho; Jung, Heejae; Kim, Jun-Seok; Shin, Changwoo; Park, Keunju; Lee, Kyoobin; Park, Jinman (February 2017). "4.1 A 640×480 dynamic vision sensor with a 9µm pixel and 300Meps address-event representation". 2017 IEEE International Solid-State Circuits Conference (ISSCC). San Francisco, CA, USA: IEEE: 66–67. doi:10.1109/ISSCC.2017.7870263. ISBN 9781509037582.
  12. ^ Chen, Shoushun; Tang, Wei; Zhang, Xiangyu; Culurciello, Eugenio (December 2012). "A 64 $\times$ 64 Pixels UWB Wireless Temporal-Difference Digital Image Sensor". IEEE Transactions on Very Large Scale Integration (VLSI) Systems. 20 (12): 2232–2240. doi:10.1109/TVLSI.2011.2172470. ISSN 1063-8210.
  13. ^ a b Scheerlinck, Cedric; Barnes, Nick; Mahony, Robert (2019). "Continuous-Time Intensity Estimation Using Event Cameras". Computer Vision – ACCV 2018. Lecture Notes in Computer Science. Springer International Publishing. 11365: 308–324. arXiv:1811.00386. doi:10.1007/978-3-030-20873-8_20. ISBN 9783030208738.
  14. ^ Dai, Yuchao; Liu, Miaomiao; Hartley, Richard; Yu, Xin; Scheerlinck, Cedric; Pan, Liyuan (2018-11-26). "Bringing a Blurry Frame Alive at High Frame-Rate with an Event Camera". arXiv:1811.10180v2 [cs.CV].
  15. ^ Scheerlinck, Cedric; Barnes, Nick; Mahony, Robert (April 2019). "Asynchronous Spatial Image Convolutions for Event Cameras". IEEE Robotics and Automation Letters. 4 (2): 816–822. arXiv:1812.00438. doi:10.1109/LRA.2019.2893427. ISSN 2377-3766.