New architectures emerge in AI chipset race

As AI chipset market is getting crowded, many AI companies have started creating  solutions that caters to a niche market. The needs for chipset power, performance, software etc. vary greatly depending on the nature of application. For instance IoT edge market needs ultra low power (in milliwatts), mobile phones can work well with power consumption of up to 1W, drones can consume a bit more, automotive can go from 10-30W and so on.

Today two most prominent architectures are CPU and GPU. Both have been around for decades and have been extremely successful. While CPU is a general purpose compute architecture, GPU is developed for graphics in mind. When it comes to AI, both have their own limitations and that’s the area where startups are trying to innovate.

Many architectural solutions have been proposed as solution in the academia for some time to solve AI acceleration problem. Each architecture has its own advantages and disadvantages and some of them eventually boil down to physics of the semiconductor process node. The most popular architecture being deployed by ASIC companies today essentially involves a large array of processing elements that tries to minimize memory access thus increasing compute and reducing power.

However neural networks are getting increasingly complex and application specific.  Number of weights as well as operations per pass are increasing and so is the optimization level. The companies who are coming late to the market have chosen to innovate at the architecture level to take the network acceleration to next level. These approaches represent some of the fundamental ways to approach computing These include:

• Optical computing: In optical computing, light is used to perform matrix multiplication rather than the digital MAC. The advantage of this approach is that the multiplication is carried out in almost zero time thus increasing the overall performance. The downside is that memory is still required to store the result which may limit the performance. Two companies out of MIT, Lightmatter and Lightelligence are taking this approach and both have received funding.
• Analog: In analog computing, similar approach is taken and the matrix multiplication is carried out using an analog circuit. In essence two signals are multiplied using a transistor based analog amplifier. The power consumption in the analog multiplication is much less than the digital counterpart. The analog multiplication results are not always accurate however NNs are notoriusly good ar generating good results at lower bit width and thus the argument is that the analog multiplication will perform very well for smaller NNs given that the error will not multiply. Irvine based company, Syntiant is taking this approach.
• Processing in Memory (PIM): The processing in memory removes the cost of data transfer from RAM to ASIC. In essence PIM architecture takes an array of Flash memory and inserts compute elements in between. The weights are permanently stored in the Flash and the incoming signal simply goes from input to output. PIMs work very well for inferencing. Start-up out of Austin, Mythic and Gyrfalcon are taking this approach.
• Neuromorphic: Neuromorphic chips try to simulate behavior of brain by mimicking neurons and synapse. Neuromorphic compute has been around for some time and can be done in digital as well as analog way. Several large companies such as IBM and Intel have announced neuromorphic chipsets while many start-ups are coming online.

Of course, there are many pros and cons for each of this approach and it remains to be seen how the progress from academia to industry pans out. Oher than neuromorphic, none of the chipsets have been released and neuromorphic chipsets have had limited success. Of course all the hardware need a good software support and these companies will also have to innovate on that front when they go to market.

New architectures emerge in AI chipset race

As AI chipset market is getting crowded, many AI companies have started creating  solutions that caters to a niche market. The needs for chipset power, performance, software etc. vary greatly depending on the nature of application. For instance IoT edge market needs ultra low power (in milliwatts), mobile phones can work well with power consumption of up to 1W, drones can consume a bit more, automotive can go from 10-30W and so on.

Today two most prominent architectures are CPU and GPU. Both have been around for decades and have been extremely successful. While CPU is a general purpose compute architecture, GPU is developed for graphics in mind. When it comes to AI, both have their own limitations and that’s the area where startups are trying to innovate.

Many architectural solutions have been proposed as solution in the academia for some time to solve AI acceleration problem. Each architecture has its own advantages and disadvantages and some of them eventually boil down to physics of the semiconductor process node. The most popular architecture being deployed by ASIC companies today essentially involves a large array of processing elements that tries to minimize memory access thus increasing compute and reducing power.

However neural networks are getting increasingly complex and application specific.  Number of weights as well as operations per pass are increasing and so is the optimization level. The companies who are coming late to the market have chosen to innovate at the architecture level to take the network acceleration to next level. These approaches represent some of the fundamental ways to approach computing These include:

• Optical computing: In optical computing, light is used to perform matrix multiplication rather than the digital MAC. The advantage of this approach is that the multiplication is carried out in almost zero time thus increasing the overall performance. The downside is that memory is still required to store the result which may limit the performance. Two companies out of MIT, Lightmatter and Lightelligence are taking this approach and both have received funding.
• Analog: In analog computing, similar approach is taken and the matrix multiplication is carried out using an analog circuit. In essence two signals are multiplied using a transistor based analog amplifier. The power consumption in the analog multiplication is much less than the digital counterpart. The analog multiplication results are not always accurate however NNs are notoriusly good ar generating good results at lower bit width and thus the argument is that the analog multiplication will perform very well for smaller NNs given that the error will not multiply. Irvine based company, Syntiant is taking this approach.
• Processing in Memory (PIM): The processing in memory removes the cost of data transfer from RAM to ASIC. In essence PIM architecture takes an array of Flash memory and inserts compute elements in between. The weights are permanently stored in the Flash and the incoming signal simply goes from input to output. PIMs work very well for inferencing. Start-up out of Austin, Mythic and Gyrfalcon are taking this approach.
• Neuromorphic: Neuromorphic chips try to simulate behavior of brain by mimicking neurons and synapse. Neuromorphic compute has been around for some time and can be done in digital as well as analog way. Several large companies such as IBM and Intel have announced neuromorphic chipsets while many start-ups are coming online.

Of course, there are many pros and cons for each of this approach and it remains to be seen how the progress from academia to industry pans out. Oher than neuromorphic, none of the chipsets have been released and neuromorphic chipsets have had limited success. Of course all the hardware need a good software support and these companies will also have to innovate on that front when they go to market.

However, one thing is clear. The need for specialization beyond what is available today in traditional digital platform seems to have been recognized by the market. We are still a few (or even several years) to see these products go into production as the transition from academia to industry takes place. In the short term, current architectures will continue to sell well but given the compute intensive nature of AI applications, we can potentially see some fundamental changes to the computer architecture in the long term.