Adding with Alternative Carries: An Investigation of the Inductive Biases of Artificial Neural Networks

Abstract

As artificial neural networks race past human levels in a variety of specific domains, the human mind remains unrivaled in its general intelligence. Arguably the hallmark of that intelligence is our capacity for radical generalization, which we largely owe to our skill at recognizing and exploiting symmetries. In this thesis, we investigate a little-studied yet paradigmatic example of radical generalization via utilization of symmetry: base addition. We present a formal treatment that reveals a hidden world of alternative carries—a fertile test-bed for probing the inductive biases of neural networks in symmetry learning. We introduce some complexity measures to quantitatively describe these carries' intricate, highly-varied structure, and then we train neural networks to add using these carries and compare learning speeds. We find that even a very small neural network can achieve radical generalization with the right input format, and that learning speed is correlated with carry complexity, which is relevant both to machine learning and to psychology.