AI accelerator

From Wikipedia, the free encyclopedia
Jump to navigation Jump to search

An AI accelerator is a class of microprocessor[1] or computer system[2] designed to accelerate artificial neural networks, machine vision and other machine learning algorithms for robotics, internet of things and other data-intensive or sensor-driven tasks.[3] They are often manycore designs and generally focus on low-precision arithmetic, novel dataflow architectures or in-memory computing capability.[4] A number of vendor-specific terms exist for devices in this space.

History of AI acceleration[edit]

Computer systems have frequently complemented the CPU with special purpose accelerators for specialized tasks, most notably video cards for graphics, but also sound cards for sound, etc. As Deep learning and AI workloads rose in prominence, specialized hardware units were developed or adapted from previous products to accelerate these tasks.

Early attempts[edit]

As early as 1993, DSPs were used as neural network accelerators e.g. to accelerate OCR software,.[5] In the 1990s, there were also attempts to create parallel high throughput systems for workstations aimed at various applications, including neural network simulations.[6][7][8] FPGA-based accelerators were also first explored in the 1990s for both inference[9] and training.[10] ANNA was a neural net CMOS accelerator developed by Yann LeCun.[11]

Heterogeneous computing[edit]

Heterogeneous computing began the incorporation of a number of specialized processors in a single system, or even a single chip, each optimized for a specific type of task. Architectures such as the Cell microprocessor[12] have features significantly overlapping with AI accelerators including: support for packed low precision arithmetic, dataflow architecture, and prioritizing 'throughput' over latency. The Cell microprocessor was subsequently applied to a number of tasks[13][14][15] including AI.[16][17][18]

CPUs themselves also gained increasingly wide SIMD units (driven by video and gaming workloads) and support for packed low precision data types.[19]

Use of GPU[edit]

Graphics processing units or GPUs are specialized hardware for the manipulation of images. As the mathematical basis of neural networks and image manipulation are similar, embarrassingly parallel tasks involving matrices, GPUs became increasingly used for machine learning tasks.[20][21][22] As such, as of 2016 GPUs are popular for AI work, and they continue to evolve in a direction to facilitate deep learning, both for training[23] and inference in devices such as self-driving cars.[24] - and gaining additional connective capability for the kind of dataflow workloads AI benefits from (e.g. Nvidia NVLink).[25] As GPUs have been increasingly applied to AI acceleration, GPU manufacturers have incorporated neural network specific hardware to further accelerate these tasks.[26] Tensor cores are intended to speed up the training of neural networks.[26]

Use of FPGA[edit]

Deep learning frameworks are still evolving, making it hard to design custom hardware. Reconfigurable devices like field-programmable gate arrays (FPGA) make it easier to evolve hardware, frameworks and software alongside each other.[9][10][27]

Microsoft has used FPGA chips to accelerate inference.[28][29] The application of FPGAs to AI acceleration has also motivated Intel to purchase Altera with the aim of integrating FPGAs in server CPUs, which would be capable of accelerating AI as well as general purpose tasks.[citation needed]

Emergence of dedicated AI accelerator ASICs[edit]

Whilst GPUs and FPGAs perform far better than CPUs for these AI related tasks, a factor of 10 in efficiency[30][31] can still be gained with a more specific design, via an application-specific integrated circuit (ASIC).[citation needed] These include differences in memory use[citation needed] and the use of lower precision numbers.[32][33]


As of 2016, the field is still in flux and vendors are pushing their own marketing term for what amounts to an "AI accelerator", in the hope that their designs and APIs will dominate. There is no consensus on the boundary between these devices, nor the exact form they will take, however several examples clearly aim to fill this new space, with a fair amount of overlap in capabilities.

In the past when consumer graphics accelerators emerged, the industry eventually adopted Nvidia's self-assigned term, "the GPU",[34] as the collective noun for "graphics accelerators", which had taken many forms before settling on an overall pipeline implementing a model presented by Direct3D.


Stand alone products[edit]

GPU based products[edit]

  • Nvidia Tesla is Nvidia's line of GPU derived products marketed for GPGPU and AI tasks.
    • Nvidia Volta is a microarchitecture which augments the Graphics processing unit with additional 'tensor units' targeted specifically at accelerating calculations for neural networks[38]
    • Nvidia DGX-1 is a Nvidia workstation/server product which incorporates Nvidia brand GPUs for GPGPU tasks including machine learning.[39]
  • Radeon Instinct is AMD's line of GPU derived products for AI acceleration.[40]

AI accelerating co-processors[edit]

  • The processor in Qualcomm's mobile platform Snapdragon 845 contains a Hexagon 685 DSP core for AI processing in camera, voice, XR and gaming applications
  • PowerVR 2NX NNA (Neural Net Accelerator) is an IP core from Imagination Technologies licensed for integration into chips.[41]
  • Neural Engine is an AI accelerator core within the Apple A11 Bionic SoC.[42]
  • Cadence Tensilica Vision C5 is a neural networks optimized DSP IP core[43]
  • The Neural Processing Unit is a neural network accelerator within the HiSilicon Kirin 970[44]
  • January 2018 CEVA, Inc. launched a family of four AI processors called NeuPro, each containing one programmable vector DSP and one hardwired implementation of 8-bit or 16-bit neural network layers supporting neural nets with performances ranging from 2 TOPS thru 12.5 TOPS. [45]

Universal Multifunction Accelerator[edit]

Universal Mutifunction Accelerator ( UMA) developed and silioconised by Manjeera Digital Systems under the aeges of Centre for Innovation Engineering and Entrepreneurship Incubator in Indian Institute of Information Technolgy ( IIIT), Hyderabad is a breakthrough innovation in Acceleration in the computation of any application with a proprietary architecture based on Middle Stratum Operations.[46][47][48]

Research and unreleased products[edit]

  • In December 2017 Tesla Motors confirmed a rumour that it is developing an AI chip for autonomous driving. Jim Keller has been working in this project since at least early 2016.[49]
  • Eyeriss is an accelerator design aimed explicitly at convolutional neural networks, using a scratchpad and on chip network architecture.[50]
  • Nullhop is an accelerator designed at the Institute of Neuroinformatics of ETH Zürich and University of Zürich based on sparse representation of feature maps. The second generation of the architecture is commercialized by the university spin-off Synthara Technologies.[51][52]
  • Kalray is an accelerator for convolutional neural nets.[53]
  • SpiNNaker is a many-core design specialized for simulating a large neural network.
  • Graphcore IPU is a graph-based AI accelerator.[54]
  • DPU, by wave computing, a dataflow architecture[55]
  • STMicroelectronics at the start of 2017 presented a demonstrator SoC manufactured in a 28 nm process containing a deep CNN accelerator.[56]
  • NM500 is the latest as of 2016 in a series of accelerator chips for Radial Basis Function neural nets from General Vision.[57]
  • TrueNorth is a manycore design based on spiking neurons rather than traditional arithmetic.[58][59]
  • Intel Loihi is an experimental neuromorphic chip.[60]
  • BrainChip in September 2017 introduced a commercial PCI Express card with a Xilinx Kintex Ultrascale FPGA running neuromorphic neural cores applying pattern recognition on 600 video images per second using 16 watts of power.[61]
  • IIT Madras is designing a spiking neuron accelerator for big-data analytics.[62]
  • Several memristor-based AI accelerators have been proposed which leverage in-memory computing capability of memristor.[4]
  • AlphaICs is designing an agent-based Co-Processor called Real AI Processor (RAP) to enable perception and decision making in a chip. [63]

Potential applications[edit]

See also[edit]


  1. ^ "Intel unveils Movidius Compute Stick USB AI Accelerator". 
  2. ^ "Inspurs unveils GX4 AI Accelerator". 
  3. ^ "google developing AI processors". google using its own AI accelerators.
  4. ^ a b "A Survey of ReRAM-based Architectures for Processing-in-memory and Neural Networks", S. Mittal, Machine Learning and Knowledge Extraction, 2018
  5. ^ "convolutional neural network demo from 1993 featuring DSP32 accelerator". 
  6. ^ "design of a connectionist network supercomputer". 
  7. ^ "The end of general purpose computers (not)". This presentation covers a past attempt at neural net accelerators, notes the similarity to the modern SLI GPGPU processor setup, and argues that general purpose vector accelerators are the way forward (in relation to RISC-V hwacha project. Argues that NN's are just dense and sparse matrices, one of several recurring algorithms)
  8. ^ "SYNAPSE-1: a high-speed general purpose parallel neurocomputer system". 
  9. ^ a b "Space Efficient Neural Net Implementation" (PDF). 
  10. ^ a b "A Generic Building Block for Hopfield Neural Networks with On-Chip Learning" (PDF). 
  11. ^ Application of the ANNA Neural Network Chip to High-Speed Character Recognition
  12. ^ "Synergistic Processing in Cell's Multicore Architecture". 
  13. ^ "Performance of Cell processor for biomolecular simulations" (PDF). 
  14. ^ "Video Processing and Retrieval on Cell architecture". 
  15. ^ "Ray Tracing on the Cell Processor". 
  16. ^ "Development of an artificial neural network on a heterogeneous multicore architecture to predict a successful weight loss in obese individuals" (PDF). 
  17. ^ "Parallelization of the Scale-Invariant Keypoint Detection Algorithm for Cell Broadband Engine Architecture". 
  18. ^ "Data Mining Algorithms on the Cell Broadband Engine". 
  19. ^ "Improving the performance of video with AVX". 
  20. ^ "microsoft research/pixel shaders/MNIST". 
  21. ^ "how the gpu came to be used for general computation". 
  22. ^ "imagenet classification with deep convolutional neural networks" (PDF). 
  23. ^ "nvidia driving the development of deep learning". 
  24. ^ "nvidia introduces supercomputer for self driving cars". 
  25. ^ "how nvlink will enable faster easier multi GPU computing". 
  26. ^ a b Harris, Mark (May 11, 2017). "CUDA 9 Features Revealed: Volta, Cooperative Groups and More". Retrieved August 12, 2017. 
  27. ^ "FPGA Based Deep Learning Accelerators Take on ASICs". The Next Platform. 2016-08-23. Retrieved 2016-09-07. 
  28. ^ "microsoft extends fpga reach from bing to deep learning". 
  29. ^ "Accelerating Deep Convolutional Neural Networks Using Specialized Hardware" (PDF). 
  30. ^ "Google boosts machine learning with its Tensor Processing Unit". 2016-05-19. Retrieved 2016-09-13. 
  31. ^ "Chip could bring deep learning to mobile devices". 2016-02-03. Retrieved 2016-09-13. 
  32. ^ "Deep Learning with Limited Numerical Precision" (PDF). 
  33. ^ Rastegari, Mohammad; Ordonez, Vicente; Redmon, Joseph; Farhadi, Ali (2016). "XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks". arXiv:1603.05279Freely accessible [cs.CV]. 
  34. ^ "NVIDIA launches the World's First Graphics Processing Unit, the GeForce 256,". 
  35. ^ Kampman, Jeff (17 October 2017). "Intel unveils purpose-built Neural Network Processor for deep learning". Tech Report. Retrieved 18 October 2017. 
  36. ^ "Intel Nervana Neural Network Processors (NNP) Redefine AI Silicon". Retrieved 20 October 2017. 
  37. ^ "The Evolution of EyeQ". 
  38. ^ "Nvidia goes beyond the GPU for AI with Volta". 
  39. ^ "nvidia dgx-1" (PDF). 
  40. ^ Smith, Ryan (12 December 2016). "AMD Announces Radeon Instinct: GPU Accelerators for Deep Learning, Coming in 2017". Anandtech. Retrieved 12 December 2016. 
  41. ^ "The highest performance neural network inference accelerator". 
  42. ^ "The iPhone X's new neural engine exemplifies Apple's approach to AI". The Verge. Retrieved 2017-09-23. 
  43. ^ "Cadence Unveils Industry's First Neural Network DSP IP for Automotive, Surveillance, Drone and Mobile Markets". 
  44. ^ "HUAWEI Reveals the Future of Mobile AI at IFA 2017". 
  45. ^ "A Family of AI Processors for Deep Learning at the Edge". 
  46. ^ Manjeera Digital System, UMA. "Universal Multifunction Accelerator". Manjeera Digital Systems. Retrieved 28 June 2018. 
  47. ^ Manjeera Digital Systems, Universal Multifunction Accelerator. "Revolutionise Processing". Indian Express. Retrieved 28 June 2018. 
  48. ^ AI Chip, UMA (10 May 2018). "AI Chip from Hyderabad" (News Paper). Telangana Today. Retrieved 28 June 2018. 
  49. ^ Lambert, Fred (December 8, 2017). "Elon Musk confirms that Tesla is working on its own new AI chip led by Jim Keller". 
  50. ^ Chen, Yu-Hsin; Krishna, Tushar; Emer, Joel; Sze, Vivienne (2016). "Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks". IEEE International Solid-State Circuits Conference, ISSCC 2016, Digest of Technical Papers. pp. 262–263. 
  51. ^ Aimar, Alessandro; et al. "NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps" (PDF). 
  52. ^ "Synthara Technologies". 
  53. ^ "kalray MPPA" (PDF). 
  54. ^ "Graphcore Technology". 
  55. ^ "Wave Computing's DPU architecture". 
  56. ^ "A 2.9 TOPS/W Deep Convolutional Neural Network SoC in FD-SOI 28nm for Intelligent Embedded Systems" (PDF). 
  57. ^ "NM500, Neuromorphic chip with 576 neurons". 
  58. ^ "yann lecun on IBM truenorth". argues that spiking neurons have never produced leading quality results, and that 8-16 bit precision is optimal, pushes the competing 'neuflow' design
  59. ^ "IBM cracks open new era of neuromorphic computing". TrueNorth is incredibly efficient: The chip consumes just 72 milliwatts at max load, which equates to around 400 billion synaptic operations per second per watt — or about 176,000 times more efficient than a modern CPU running the same brain-like workload, or 769 times more efficient than other state-of-the-art neuromorphic approaches 
  60. ^ "Intel's New Self-Learning Chip Promises to Accelerate Artificial Intelligence". 
  61. ^ "BrainChip Accelerator". 
  62. ^ "India preps RISC-V Processors - Shakti targets servers, IoT, analytics". The Shakti project now includes plans for at least six microprocessor designs as well as associated fabrics and an accelerator chip 
  63. ^ "AlphaICs". 
  64. ^ "drive px". 
  65. ^ "design of a machine vision system for weed control" (PDF). 
  66. ^ "qualcomm research brings server class machine learning to every data devices". 
  67. ^ "movidius powers worlds most intelligent drone". 

External links[edit]