What does it mean for an artificial intelligence (AI) to think? Will AIs ever be sentient? How would we even know, when they have been trained to fake it so well? Large language models and other generative AI systems are usually described as predicting the next token in a sequence --- forecasting the most likely next word. However, they actually do a bit more than this. A combination of clever high-dimensional embedding of the tokens and simultaneous prediction at points throughout the token sequence gives an effective illusion of relevance and coherence. Such systems are still prone to fabrications and, as a consequence, are often claimed to be ``creative.'' Nonetheless, they fail to grasp basic logic. These systems can perform computation, but when they do so, it is extremely expensive and prone to error.
How can we build systems to perform logical deductive reasoning? How would we test their performance? I will argue that the ability to ``do mathematics'' (but not simply reproducing versions of known problems, perhaps with substitutions) is such a test. The efforts to apply machine learning to mathematics are a testament to that progress. I will review the current state-of-the-art of application of AI to mathematics and challenge what it means ``to think'' and to ``do mathematics.'' I do not believe that AI can ever achieve sentience --- without a fundamental change in the hardware.