Binary-Weight Convolution CMOS Image Sensor for Neural Network

Abstract

Recently, a CMOS image sensor (CIS) has widely used in the industrial applications such as self-driving vehicles, surveillance, biomedical diagnosis, and internet-of-things (IOT). Especially, these applications are most effective using neural networks for an image classification. For many applications, a system which consists of a CIS and a convolutional neural network (CNN) is required. However, the system is difficult to fabricate with a single chip because a large pixel array and readout circuits occupy most of area in a high resolution CIS and the CNN mainly consists of memories and many processing units for weights and neural computations. To reduce the computational complexity for the CNNs, the binary neural networks (BNN) has been also researched in recent. It is two types of BNN. One binarizes both the feature maps and the weights, and another binarizes only the weights. The former can massively reduce the memory and the computational amount but it severely undergoes its decreasing accuracy because the feature maps the weights are all binary number. However, the latter is almost same as the accuracy of CNNs because, although its weights are binarized, the feature maps are not binarized and it have the image information. In this dissertation, we suggest a CMOS image sensor (CIS) that can perform on-chip binary weight convolution. The CIS can greatly reduce memory usage and computational complexity by directly generating a feature map for a binary neural network (BNN). The pixel readout of the CIS is performed in the column-parallel fashion using incremental delta-sigma analog-to-digital converters (ADCs). The CIS operates in two different modes: convolution and normal modes. When the column ADC is working in the convolution mode, it works as a first-order delta-sigma ADC and generates convolved images using a binary kernel. In the normal operation mode, the ADC is switched to a second-order delta-sigma ADC with little hardware modification and used to capture high-quality images. Therefore, the CIS cannot only capture a normal scene but also generate feature map image with simple switching.