How to write science questions that are easy for people and hard for computers

Research output: Contribution to journalArticle

Abstract

As a challenge problem for AI systems, I propose the use of hand-constructed multiple-choice tests, with problems that are easy for people but hard for computers. Specifically, I discuss techniques for constructing such problems at the level of a fourth-grade child and at the level of a high school student. For the fourth-grade-level questions, I argue that questions that require the understanding of time, of impossible or pointless scenarios, of causality, of the human body, or of sets of objects, and questions that require combining facts or require simple inductive arguments of indeterminate length can be chosen to be easy for people, and are likely to be hard for AI programs, in the current state of the art. For the high school level, I argue that questions that relate the formal science to the realia of laboratory experiments or of real-world observations are likely to be easy for people and hard for AI programs. I argue that these are more useful benchmarks than existing standardized tests such as the SATs or New York Regents tests. Since the questions in standardized tests are designed to be hard for people, they often leave many aspects of what is hard for computers but easy for people untested.

Original languageEnglish (US)
Pages (from-to)13-22
Number of pages10
JournalAI Magazine
Volume37
Issue number1
StatePublished - 2016

Fingerprint

Students
Experiments

ASJC Scopus subject areas

  • Artificial Intelligence

Cite this

How to write science questions that are easy for people and hard for computers. / Davis, Ernest.

In: AI Magazine, Vol. 37, No. 1, 2016, p. 13-22.

Research output: Contribution to journalArticle

@article{ebd85272795a4e7b9b71e0bb2c298c9a,
title = "How to write science questions that are easy for people and hard for computers",
abstract = "As a challenge problem for AI systems, I propose the use of hand-constructed multiple-choice tests, with problems that are easy for people but hard for computers. Specifically, I discuss techniques for constructing such problems at the level of a fourth-grade child and at the level of a high school student. For the fourth-grade-level questions, I argue that questions that require the understanding of time, of impossible or pointless scenarios, of causality, of the human body, or of sets of objects, and questions that require combining facts or require simple inductive arguments of indeterminate length can be chosen to be easy for people, and are likely to be hard for AI programs, in the current state of the art. For the high school level, I argue that questions that relate the formal science to the realia of laboratory experiments or of real-world observations are likely to be easy for people and hard for AI programs. I argue that these are more useful benchmarks than existing standardized tests such as the SATs or New York Regents tests. Since the questions in standardized tests are designed to be hard for people, they often leave many aspects of what is hard for computers but easy for people untested.",
author = "Ernest Davis",
year = "2016",
language = "English (US)",
volume = "37",
pages = "13--22",
journal = "AI Magazine",
issn = "0738-4602",
publisher = "AI Access Foundation",
number = "1",

}

TY - JOUR

T1 - How to write science questions that are easy for people and hard for computers

AU - Davis, Ernest

PY - 2016

Y1 - 2016

N2 - As a challenge problem for AI systems, I propose the use of hand-constructed multiple-choice tests, with problems that are easy for people but hard for computers. Specifically, I discuss techniques for constructing such problems at the level of a fourth-grade child and at the level of a high school student. For the fourth-grade-level questions, I argue that questions that require the understanding of time, of impossible or pointless scenarios, of causality, of the human body, or of sets of objects, and questions that require combining facts or require simple inductive arguments of indeterminate length can be chosen to be easy for people, and are likely to be hard for AI programs, in the current state of the art. For the high school level, I argue that questions that relate the formal science to the realia of laboratory experiments or of real-world observations are likely to be easy for people and hard for AI programs. I argue that these are more useful benchmarks than existing standardized tests such as the SATs or New York Regents tests. Since the questions in standardized tests are designed to be hard for people, they often leave many aspects of what is hard for computers but easy for people untested.

AB - As a challenge problem for AI systems, I propose the use of hand-constructed multiple-choice tests, with problems that are easy for people but hard for computers. Specifically, I discuss techniques for constructing such problems at the level of a fourth-grade child and at the level of a high school student. For the fourth-grade-level questions, I argue that questions that require the understanding of time, of impossible or pointless scenarios, of causality, of the human body, or of sets of objects, and questions that require combining facts or require simple inductive arguments of indeterminate length can be chosen to be easy for people, and are likely to be hard for AI programs, in the current state of the art. For the high school level, I argue that questions that relate the formal science to the realia of laboratory experiments or of real-world observations are likely to be easy for people and hard for AI programs. I argue that these are more useful benchmarks than existing standardized tests such as the SATs or New York Regents tests. Since the questions in standardized tests are designed to be hard for people, they often leave many aspects of what is hard for computers but easy for people untested.

UR - http://www.scopus.com/inward/record.url?scp=85020005543&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85020005543&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:85020005543

VL - 37

SP - 13

EP - 22

JO - AI Magazine

JF - AI Magazine

SN - 0738-4602

IS - 1

ER -