Bill Gates Watched ChatGPT Ace An AP Bio Exam And Went Into ‘A State Of Shock’

Bill Gates, co-chairman of the Bill and Melinda Gates Foundation, during the EEI 2023 event in Austin, Texas, US, on Monday, June 12, 2023

Even tech icon Bill Gates says he was caught off-guard by the rapid development of artificial intelligence programs like ChatGPT.

On Thursday, Gates opened up about “the most stunning demo I’ve ever seen in my life” — a demonstration of ChatGPT’s capabilities on an AP Biology exam in August 2022, he said on an episode of his “Unconfuse Me” podcast featuring Khan Academy CEO Sal Khan.

The story began last June when Gates first tested the AI-powered chatbot — developed by Microsoft partner OpenAI — and came away unimpressed. “I was like, ‘Yeah, it’s kind of an idiot savant. I don’t think it’s practical,’” Gates said he remarked at the time.

He issued what he believed to be a rather difficult challenge to OpenAI CEO Sam Altman: Bring ChatGPT back to him once it could exhibit advanced, human-level competency by achieving the highest possible score on the AP Biology exam.

“I thought, ‘OK, that’ll give me three years to work on HIV and malaria,’” Gates joked on the podcast.

Two months later, OpenAI’s developers came back, and Gates watched ChatGPT achieve the top score of five on the test. The program had gone from not being able to read or write “in the sense humans do,” Gates said, to being able to do both at nearly a human-like level.

“I’m still, personally, in a state of shock at ‘Wow, it is so good,’” Gates said.

ChatGPT’s success comes with “lots of footnotes about hallucination and things like that,” Gates said, noting that even the most advanced AI models can make significant mistakes or fabricate information altogether.

Still, the rapid development left Gates thoroughly impressed, and excited about the technology’s potential applications: “Let’s see where we can put it to good use,” he said.

AI tools could help ‘close the gap’ in education

Throughout the podcast episode, Gates and Khan discussed whether education could be one of those “good uses.”

Gates, an optimist on this front, has predicted that AI chatbots like ChatGPT or Google’s Bard could start helping kids learn to read and write as soon as next year. Khan agreed, saying his company is developing an experimental chatbot tutor called Khanmigo for that very purpose.

Khanmigo can already “act like a fairly good human tutor,” Khan said. “There’s moments with it that I think would pass the Turing test, where you’d think that there’s a good human on the other side of the chat.”

Of course, moments of brilliance don’t necessarily translate to consistent human-level output. AI tutors are unlikely to replace teachers anytime soon, Khan and Gates both said: Rather, they could help make teachers’ lives easier, and provide cheaper tutoring options to underserved students in low-income areas in the U.S. and overseas.

“It makes mistakes, to be clear,” Khan said of OpenAI’s latest large language model, GPT-4, which rolled out this summer. But, he added that the model is “dramatically better” than previous iterations, with OpenAI noting it can surpass 90% of human test-takers on the SAT.

Those mistakes are reasons for caution, Barnard College child psychologist Tovah Klein told CNBC Make It last month. Children shouldn’t come to rely on AI tools as their only source of information, and AI models need to get better at explaining how they arrive at certain answers, she said.

“If we have the science showing that that kind of learning, in addition to a teacher, is useful, then I think there is a role for AI,” said Klein. “I think part of the problem is, we don’t really know [yet].”

Still, Gates has seen enough of ChatGPT’s development — including its growing ability to explain its reasoning — to predict a game-changing impact on education within the next decade, he said.

“If we think about the next 10 years, [in terms of] both the absolute level of learning and the gap with lower-income, minority students ... these new tools can both close the gap and raise up the overall level of achievement,” Gates said.