Sample geometry problem
The Allen Institute for Artificial Intelligence (AI2) today announced a significant breakthrough in AI research in the form of GeoS, a new AI system that is capable of solving SAT geometry problems as well as the average American 11th grade student through a combination of visual processing and textual analysis. The creation of GeoS was led by AI2, in collaboration with UW faculty and students with expertise in artificial intelligence, natural language processing and computer vision at the University of Washington. AI2 – established by investor, philanthropist, and technologist Paul G. Allen in 2004 to advance artificial intelligence research for the public good – is led by former UW CSE professor Oren Etzioni.
Oren Etzioni
From the press release:
“[GeoS] uses a combination of computer vision to interpret diagrams, natural language processing to read and understand text and a geometric solver to achieve a 49 percent accuracy on official SAT test questions. If these results were extrapolated to the entire Math SAT test, the computer roughly achieved an SAT score of 500 (out of 800), the average test score for 2015….
“‘Unlike the Turing Test, standardized tests such as the SAT provide us today with a way to measure a machine’s ability to reason and to compare its abilities with that of a human,’ said Oren Etzioni, CEO of AI2. ‘Much of what we understand from text and graphics is not explicitly stated, and requires far more knowledge than we appreciate. Creating a system to be able to successfully take these tests is challenging, and we are proud to achieve these unprecedented results.'”
Ali Farhadi
“Said Ali Farhadi, senior research manager for Vision at AI2 and UW assistant professor of computer science and engineering, ‘We are excited about GeoS’s performance on real-world tasks. Our biggest challenge was converting the question to a computer-understandable language. One needs to go beyond standard pattern-matching approaches for problems like solving geometry questions that require in-depth understanding of text, diagram and reasoning.'”
In addition to Etzioni and Farhadi, the research team included UW CSE Ph.D. student Minjoon Seo, who contributed to the project as part of an internship at AI2; UW EE professor (and UW CSE adjunct professor) Hanna Hajishirzi; and UW B.S. alum Clint Malcolm. The team presented its technical paper at the Conference on Empirical Methods in Natural Language Processing (EMNLP) in Lisbon, Portugal.
Read the press release here and the research paper here. Check out John Markoff’s New York Times article on how the GeoS breakthrough fits into the broader landscape of AI research, with additional quotes from Oren and Ali, here. The Washington Post also published a nice piece on its “Speaking of Science” blog here.
Congratulations to Paul, Oren, Ali, Hanna, Minjoon, and Clint on this phenomenal achievement!