LAUSR

Deep learning has achieved remarkable success across a wide range of applications, such as language modeling, computer vision, recommendation systems, and robotics. However, the growing size of models and their increasing computational demands pose significant challenges, particularly for resource-constrained devices. A promising approach to address these challenges is extreme quantization, exemplified by binary and ternary neural networks. These techniques significantly reduce model size by quantizing weights and activations to 1 or 1.58 bit, while simplifying computation, making them well-suited for efficient deployment in resource-limited environments. This article presents a comprehensive review of extreme quantization techniques, organized into three key areas: 1) a comparative analysis of quantizing only the weights [e.g., binary weight networks (BWNs) and ternary weight networks (TWNs)] versus quantizing both weights and activations (e.g., binary neural networks and ternary neural networks), along with a discussion of the progress and tradeoffs of their approaches; 2) an examination of how extreme quantization, initially applied to convolutional neural networks (CNNs), has been extended to Transformer architectures; and 3) an overview of compute-in-memory architectures optimized for binarization and ternarization, including designs based on advanced bit-cell technologies.