Zapier’s Mike Knoop launches ARC Prize to Jumpstart New Ideas for AGI

As impressive as LLMs are, the growing consensus is that language, scale and compute won’t get us to AGI. Although many AI benchmarks have quickly achieved human-level performance, there is one eval that has barely budged since it was created in 2019. Google researcher François Chollet wrote a paper that year defining intelligence as skill-acquisition efficiency—the ability to learn new skills as humans do, from a small number of examples. To make it testable he proposed a new benchmark, the Abstraction and Reasoning Corpus (ARC), designed to be easy for humans, but hard for AI. Notably, it doesn’t rely on language. Zapier co-founder Mike Knoop read Chollet’s paper as the LLM wave was rising. He worked quickly to integrate generative AI into Zapier’s product, but kept coming back to the lack of progress on the ARC benchmark. In June, Knoop and Chollet launched the ARC Prize, a public competition offering more than $1M to beat and open-source a solution to the ARC-AGI eval. In this episode Mike talks about the new ideas required to solve ARC, shares updates from the first two weeks of the competition, and shares why he’s excited for AGI systems that can innovate alongside humans. Hosted by: Sonya Huang and Pat Grady, Sequoia Capital Mentioned: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models : The 2019 paper that first caught Mike’s attention about the capabilities of LLMs On the Measure of Intelligence : 2019 paper by Google researcher François Chollet that introduced the ARC benchmark, which remains unbeaten