Abstract
Advances in neural networks and computing power have led to a series of incredible achievements in computer vision and brain modeling previously impossible just a few decades ago. The neural networks facilitating these innovations have generally relied on a hierarchical structure of several (deep) layers and were named deep neural networks. However, in the light of these advancements, a mystery arose as to why these models achieved such great performance since a strong theoretical basis was missing. Furthermore, issues arose when researchers demonstrated that the great image classification performance of these models could be significantly impaired with the addition of small perturbations to images as well as the generation of artificial images. The objective of this dissertation was to provide a first-principles approach to deriving and understanding deep neural networks with a sparse coding theoretical framework. With this theoretical framework, we derived a new neural network called a neural inhibition network and performed simulations to demonstrate the utility of this model that leverages sparse coding in a neural network. We also show how the main computation of many deep neural networks, a ReLU of an affine transformation, may be derived from sparse coding theory and motivates the neural inhibition network. Finally, we describe how inference via sparse coding may be possible, and we show empirically how it may be achieved. These results suggest deep neural networks perform a form of sparse coding, adding neural inhibition analogous to lateral inhibition in the brain may aid classification with neural networks, and tradeoffs exist for solving vision tasks that may not appear intuitive at first glance.