Proof Ingredients: Is AI going to replace software developers?

How good is AI at software coding?
Carl Brown, founder of the TH-cam channel Internet of Bugs, has made a name for himself by holding AI claims up to scrutiny. So we asked the physicist and software developer to help us assess how good AI is at coding tasks.
Using our AI testing software, which simultaneously queries five leading AI models, Brown asked the models coding questions. He published the results on his TH-cam channel and spoke with Proof founder Julia Angwin for our Ingredients video interview series.
Ingredients
Hypothesis: Generative AI cannot replace software engineers, but it can do parts of the job.
Sample size: A dozen questions were asked to five AI models: OpenAI’s GPT-4, Anthropic’s Claude 3 Opus, Google’s Gemini, Mistral’s Mixtral, and Meta’s LLama 2.
Techniques: Posed three types of questions to models: ones that require recent coding knowledge, ones that have multiple solutions, and tasks that require planning.
Key findings: AI models often produced generic answers instead of producing tailored solutions to or plans to execute the specific task at hand, and overall, fell short of what one would expect of a human software engineer.
Limitations: Questions were limited to those that someone who does not code would likely understand. The sample size was small and models may perform differently after updates.
Why we think news needs an ingredients label
• What's in your news?
Links
Carl Brown's video about this investigation
• AI Coding Crap: More E...
Carl's video debunking Devin
• Debunking Devin: "Firs...
Carl's TH-cam channel, Internet of Bugs
/ @internetofbugs
www.proofnews....
/ proof_news
/ proof__news
Join us in making trustworthy, verifiable information the new baseline:
www.proofnews....