Study: A cognitive framework for policy argument

Policy analysis is based on a type of reasoning called deliberation or policy argument.  The knowledge and skills required for policy argument have remained ill-defined making it difficult to study and to teach. We’ve developed a cognitive framework for policy argument defining the skills of policy argument as a process of: identifying questions; searching for information; comprehending and evaluating information; constructing representations of information synthesizing information with prior beliefs, weighing political values; and finally interpreting the synthesized model to determine which policy intervention will most effectively achieve the outcomes most valued.

This framework provides a basis for all our subsequent work on policy argumentation and informs educational psychology in other causal reasoning domains such as science, history and law.

People

Matt Easterday, Northwestern
Vincent Aleven, Carnegie Mellon
Richard Scheines, Carnegie Mellon

Papers

Constructing causal diagrams to learn deliberation. IJAIED 2009, 19(4), 425-445.
Easterday, Aleven, Scheines & Carver

 

Study: Bias and reasoning challenges in synthesis

Previous research suggests that confirmation bias causes students to search only for information that supports their beliefs and to interpret that information in a biased way. How does bias affect reasoning within the policy argument framework? Using a novel paradigm in which we poll students beliefs after encountering each new piece of evidence, we found that students often do change their beliefs in a “rational” way–increasing their confidence in the position after reading a confirming piece of evidence, and decreasing their confidence after disconfirming evidence.

However, students begin with extremely high confidence in their original beliefs and change that belief only slightly after each piece of evidence. Furthermore, students incorrectly recall the mass of evidence, believing that the majority of evidence supports their current belief, i.e., rationalizing their original position. In other words, through finer grained measurement, we see that the most difficult reasoning challenge may be in syn- thesis of information, rather than search or evaluation.

People

Matt Easterday, Northwestern
Vincent Aleven, Carnegie Mellon
Richard Scheines, Carnegie Mellon
Sharon Carver, Carnegie Mellon

Papers

Will Google destroy western democracy? Bias in policy problem solving. AIED, 2009
Easterday, Aleven, Scheines & Carver

 

Study: Improving synthesis using diagrams

The previous study showed that synthesizing information is difficult because it requires one to integrate a large amount of conflicting information that may also contradicts one’s initial beliefs. To overcome this challenge, we can alter the synthesis task: just as we improve algebraic reasoning by using equations, we may be able to improve policy argument by using causal diagrams. In an experiment asking students to synthesize and interpret a complex set of claims without diagrams, with diagrams and with diagramming tools, we found that causal diagrams improved students’ ability to make evidence-based policy recommendations and that unlike many other studies, practice constructing diagrams improved students ability to comprehend causal claims.

Furthermore, by studying novices and experts, we found that background knowledge can play a normative role as when experts overrule new information when they have more credible previous knowledge, and in unpredicted, non-normative ways as when novices misinterpret a diagram to make it consistent with their prior beliefs.  This shows again that we must take a more nuanced view of confirmation bias when looking at the interaction of knowledge, representations and diagramming skill in policy argument.

People

Matt Easterday, Northwestern
Vincent Aleven, Carnegie Mellon
Richard Scheines, Carnegie Mellon
Sharon Carver, Carnegie Mellon

Papers

Constructing causal diagrams to learn deliberation. IJAIED, 2009, 19(4), 425-445.
Easterday, Aleven, Scheines, & Carver



The logic of Babel: Causal reasoning from conflicting sources. Workshop on Ill-Defined Domains, AIED 2007
Easterday, Aleven, & Scheines

 

Study: Policy World–a cognitive game for teaching policy argument

pheonix molly tutor
 screen1  screen3  synth

Policy world embeds an intelligent tutors in a game environment to teach policy argument.

How might we help increase learners’ policy argumentation skills? Intelligent computer tutors are now nearly as effective as one-on-one human tutors, but only for well-defined do- mains like Algebra (VanLehn 2006; 2011). Fortunately, the previous work on policy argument provides sufficient definition to create intelligent tutors for policy argumentation. Policy World is a cognitive game combining an intelligent tutor and game-based environment to teach policy argumentation. As described in Easterday (2012b): in Policy World the learner plays a policy analyst who must defend the public against the handsome but unscrupulous corporate lobbyist Mr. Harding.  

Learners must persuade the Senator to adopt policies based on evidence on topics such as carbon emissions, national health care and childhood obesity.  They search for policy information, analyzing that information, and debate policy recommendations against a computer opponent. Experiments show that Policy World increases students’ ability to analyze policy problems (comprehend, evaluate, diagram and synthesize policy information).

People

Matt Easterday, Northwestern

Papers

Will Google destroy western democracy? Bias in policy problem solving. AIED, 2009
Easterday, Aleven, Scheines & Carver

 

Study: Design principles for cognitive games

In designing cognitive game like Policy World, there is a tension between designing intelligent tutors, which provide a great deal of guidance to promote learning, and games, which often withhold guidance to create interesting choices where the player must explore and take risks. How does step-level tutoring and game penalties affect learning and interest?”

In two experiments, we found that cognitive games that provide more step-level feedback and decrease game penalties increase both learning and intrinsic interest. While this contradicts the conventional wisdom of game design, it means we can maximize both fun and learning.

People

Matt Easterday, Northwestern
Yelee Jo, Northwestern
Vincent Aleven, Carnegie Mellon
Richard Scheines, Carnegie Mellon
Sharon Carver, Carnegie Mellon

Papers

Using tutors to improve educational games. AIED, 2011
Easterday, Aleven, Scheines & Carver



Toward a framework for the analysis and design of educational games. DIGITEL, 2010
Aleven, Myers, Easterday & Ogan

 

Study: Extending policy argument with finer-grained causal comprehension tutors

The Policy World studies showed that comprehending causal claims in text is extremely difficult. It is a surprising complex task: the learner must recognize variables and the causal relation between them which may involve inferring variables from the causal relationship, chained sequences of causation, relations with multiple interacting causes and effects, implied but not stated variables, and so on. To teach these comprehension sub-skills, I’ve developed the Comprehension Tutor, an intelligent tutor that provides step-level feedback and scaffolding on the complex process of reading causal claims based on coding schemes developed by political psychologists.

We have conducted a pilot study with the next scheduled for early 2014. This line of work shows that we can provide even finer grained scaffolding of comprehending causal claims and might soon improve learners’ comprehension skills.

People

Matt Easterday, Northwestern

 

Study: Extending policy argument with games for weighing values

libertarian panel

Perspective Detective teaches students to reason about different political values.

Part of the policy argumentation framework is weighing political values, that is, reasoning about the political perspectives of oneself or others. Unfortunately, there are no existing measures that allow us to assess political perspective taking skills, and therefore no data about whether this presents a significant learning challenge. We have recently developed such a measure and have found that, consistent with our anecdotal classroom observations, students have great difficulty reasoning about different political perspectives, even students majoring in political science.

We are now piloting an educational game called Perspective Detective in a social policy course to teach political perspective skills. This work: expands the breadth of the policy argumentation framework; provides practical measurement instruments for political science policy and ethics; and produces educational games for an important domain. Future work on Perspective Detective will also test whether we can apply the same cognitive game approach to weighing political values.

People

Matt Easterday, Northwestern
Selwa Barhumi, Northwestern
Yana Krupnikov Northwestern
Colin Fitzpatrick, Northwestern

Papers

Perspective taking, political ideologies and digital games. Digital Ethics, 2013
Fitzpatrick, Hope, Barhumi, Krupnikov & Easterday