It has long been debated whether Newton's theory of color and, more generally, his optical investigations are mathematical, but Newton himself always insisted that his theory of color and that of the colors of thin plates are mathematical theories. He based his claim on his ability when utilizing these theories to predict and calculate all the phenomena of interest. Newton favored mathematical theories in order to avoid invoking speculative physical causes. He repeatedly claimed that his aim was to establish properties of light and not hypothetical physical causes. Newton desired causal theories but would not adopt those causes that he judged to be hypothetical, such as forces or vibrating aethers. I will argue that he believed that his theory of color did not provide a physical cause for the appearance of spectral colors after refraction, but rather an explanation, namely, that the rays are separated by their property of unequal refrangibility. I will describe the nature of these mathematical theories and how they were more a program than full-fledged mathematical physical theories. In contrast to these theories, I will briefly present his theory of colored bodies, which was his only optical theory that was explicitly cast in causal form. Here too calculation and prediction played an important role for Newton.