Agile Research Studios: Orchestrating Communities of Practice to Advance Research Training at ScaleCSCW 2017

Haoqi Zhang, Matthew W. Easterday, Elizabeth Gerber, Daniel Rees Lewis, Leesha Maliakal

Agile Research Studios (ARS) is a socio-technical system that greatly expands research training opportunities by supporting research communities of practice without increasing faculty mentoring resources.

Understanding Trust amid Delays in CrowdfundingCSCW 2017

Yongsung Kim, Aaron Shaw, Haoqi Zhang, Elizabeth Gerber


Studying the Effects of Task Notification Policies on Participation and Outcomes in On-the-go CrowdsourcingHCOMP 2016

Yongsung Kim, Emily Harburg, Shana Azria, Aaron Shaw, Elizabeth Gerber, Darren Gergle, Haoqi Zhang

Telescope: Fine-Tuned Discovery of Interactive Web UI Feature ImplementationUIST 2016

Joshua Hibschman, Haoqi Zhang

Telescope traces public website runtimes to create interactive low-barrier learning material for developers. It features composite code views and visual links to help developers overcome barriers in unfamiliar code.

Habitsourcing: Sensing the Environment through Immersive, Habit-Building ExperiencesUIST 2016

Katherine Lin, Henry Spindell, Scott Cambo, Yongsung Kim, Haoqi Zhang

We introduce habitsourcing, which harnesses habit-building practices to collect environmental data. We present interaction techniques for acquiring sensing data, and ExperienceKit, an iOS library that helps build habitsourcing applications.

Designing for Inclusion: Supporting Gender Diversity in Independent Innovation TeamsGROUP 2016

Julie Hui and Shelly Farnham

We study how independent innovation teams develop an environment of inclusion to support gender diversity.

Designing crowdcritique systems for formative feedback.International Journal of Artificial Intelligence in Education.

Easterday, M. W., Rees Lewis, D., & Gerber, E. M.

Describes 3 design-based implementations, creating a hybrid crowdsourcing systems that combine the intelligence of multiple novices in face-to-face settings might provide an alternate approach for providing intelligent formative feedback.


Enabling Physical Crowdsourcing On-the-Go with Context-Sensitive NotificationsHCOMP Poster 2015

Yongsung Kim, Emily Harburg, Shana Azria, Elizabeth Gerber, Darren Gergle, Haoqi Zhang

This paper introduces the idea of presenting physical tasks into people’s existing routine so that people can contribute small tasks on-the-go with minimal effort. We present two on-the-go crowdsourcing systems: Libero for package delivery, and CrowdFound for finding lost items.

Using Anonymity and Communal Efforts to Improve Quality of Crowdsourced FeedbackHCOMP 2015

Julie Hui, Amos Glenn, Rachel Jue, Elizabeth Gerber, Steven Dow

An experiment to test the effect of online anonymity on feedback quality and quantity. We also test a communal feedback method—distributing between teams feedback providers from the class’s collective social networks.

Unravel: Rapid Web Application Reverse Engineering via Interaction Recording, Source Tracing, and Library DetectionUIST 2015

J Hibschman, H Zhang

We present Unravel, an extension of the Chrome Developer Tools for quickly tracking and visualiz- ing HTML changes, JavaScript method calls, and JavaScript libraries. Unravel injects an observation agent into websites to monitor DOM interactions in real-time without functional interference or external dependencies.

CrowdFound: A Mobile Crowdsourcing System to Find Lost Items On-the-GoCHI Extended Abstract 2015

Emily Harburg, Yongsung Kim, Elizabeth Gerber, Haoqi Zhang

We present CrowdFound, a mobile crowdsourcing system to find lost items. CrowdFound allows users to input lost item descriptions on a map and then sends notifications to users passing near tagged areas. To assess the system’s efficacy, we conducted interviews and user testing on CrowdFound.

GAZE: Using Mobile Devices to Promote Discovery and Data CollectionCHI SRC 2015

Zak Allen, Haoqi Zhang

We developed Gaze, an approach for people to gather information about physical environments while on the go. Gaze uses mobile devices to promote data collection through low-effort interactions.

Libero: On-the-go Crowdsourcing for Package DeliveryCHI SRC 2015

Yongsung Kim, Haoqi Zhang

We introduce a system called Libero, which utilizes people’s existing routine for package delivery by incorporating just-in-time notifications to reduce task distance and having a community support itself.

Remote Paper Prototype Testing CHI 2015

Kevin Chen, Haoqi Zhang

We develop a method and tool that supports remote wizard-of-oz prototype testing; allows mobile app designers to test paper prototypes in realistic scenarios.

Crowdfunding Science: Sharing Research with an Extended AudienceCSCW 2015

Julie Hui, Elizabeth Gerber

A qualitative study of scientists using crowdfunding, motivations for use, and struggle they face. We offer guidelines for the future design of interfaces.

Building Support Tools to Connect Novice Designers with Professional CoachesCreativity and Cognition

D Rees Lewis, E Harburg, E Gerber, M Easterday

Creativity support tools help learners undertake creative work, such as facilitating coaching by creative professionals. How might we design creativity support tools that in-crease learners’ access to coaching by creative professionals?

CrowdCheer: Situational Crowdsourcing of Motivation for RunnersGrace Hopper SRC 2015

Leesha Maliakal

We studied the effects of crowdsourced and targeted motivation for runners in an attempt to understand how to use existing crowds to deliver targeted encouragement and thus improve runners’ performances.


Frenzy: Collaborative Data Organization for Creating Conference SessionsCHI 2014

Lydia B. Chilton, Juho Kim, Paul Andre, Felicia Cordeiro, James A. Landay, Daniel S. Weld, Steven P. Dow, Robert C. Miller, Haoqi Zhang

best paper honorable mention
We present a collaborative web application called Frenzy to draw on the efforts and knowledge of an entire program committee to collectively categorize accepted papers and create sessions.

Learning to Fail: Experiencing Public Failure Online Through CrowdfundingCHI 2014

Michael Greenberg, Elizabeth Gerber

A qualitative and quantitative study of failed projects on Kicsktarter. We look at which projects relaunch, what project creators change upon relaunch and what project creatorsl learn through the process of failing.

Understanding the Role of Community in Crowdfunding WorkCSCW 2014

Julie Hui, Michael Greenberg, Elizabeth Gerber

best paper honorable mention
A qualitative study to understand community efforts to support crowdfunding work, such as providing mentorship to novices and giving feedback on campaign presentation.

Pair Research: Matching People for Collaboration, Learning, and ProductivityCSCW 2014

Robert Miller, Haoqi Zhang, Elizabeth Gerber, Eric Gilbert

To increase productivity, informal learning, and collaborations within and across research groups, we have been experimenting with a new kind of interaction that we call pair research.

Computer Supported Collective Actioninteractions 2014

Aaron Shaw, Haoqi Zhang, Andres Monroy-Hern ́andez, Sean Munson, Benjamin Mako Hill, Elizabeth Gerber, Peter Kinnaird, and Patrick Minder

We investigate the design challenges of computer supported collective action (CSCA) systems that connects crowds and communities of participants, lowers the cost of communication and deliberation, and helps to coordinate action, thereby enabling new forms of collaboration unimaginable a decade ago.

Crowdfunding: Motivations and Deterrents for ParticipationTOCHI 2014

Elizabeth Gerber, Julie Hui

A grounded theory of why people are motivated and deterred to participate in crowdfunding. We present guidelines for the design of interfaces for crowdfunding to enhance performance and engagement.

WeDo: End-To-End Computer Supported Collective Action.ICWSM

Haoqi Zhang, Andrés Monroy-Hernández, Aaron D Shaw, Sean A Munson, Elizabeth M Gerber, Benjamin Mako Hill, Peter Kinnaird, Shelly Diane Farnham, Patrick Minder

We introduce WeDo, a system aimed at promoting simple forms of participatory, end-to-end collective action. Pilot deployments of WeDo illustrate that sociotechnical systems can support automated transitions through different phases of end-to-end collective action, but that challenges, such as the elicitation of leadership and the accommodation of existing group norms, remain.

Computer supported novice group critiqueDIS 2014

Matthew W. Easterday, Daniel Rees Lewis, Colin Fitzpatrick, Elizabeth M. Gerber

Groups of novice critiquers can sometimes provide feedback that is as good as a single expert. Unfortunately, we do not know how to create such systems for novice group critique in design education. The study provides improved principles for developing computer-supported novice group critique systems in design.

Design-Based research process: Problems, phases, and applicationsICLS 2014

Matthew W. Easterday, Daniel Rees Lewis, Elizabeth M. Gerber

Since the first descriptions of design-based research (DBR), there have been continued calls to better define DBR and increase its rigor. Here we address four uncertainties about DBR: (a) the phases of the DBR process, (b) what distinguishes DBR from other forms of research, (c) what distinguishes DBR from design, and (d) the characteristics of DBR that make it effective for answering certain types of questions. By better articulating the process of DBR, this definition helps us to better craft, improve, communicate, and teach design-based research.


Community clustering: Leveraging an academic crowd to form coherent conference sessionsHCOMP 2013

Paul Andre, Haoqi Zhang, Juho Kim, Lydia B. Chilton, Steven P. Dow, and Robert C. Miller

notable paper
To help conference organizers create sessions, we tap into the interest and expertise of committee members and authors. At CHI 2013 we evaluate alternative approaches and collect 10,000+ affinity pairs from community members.

Cobi: A Community-Informed Conference Scheduling ToolUIST 2013

Juho Kim, Haoqi Zhang, Paul André, Lydia Chilton, Wendy MacKay, Michel Beaudouin-Lafon, Robert C. Miller, Steven P. Dow

Cobi engages an entire academic community in planning a large-scale conference. The mixed-initiative scheduling tool helped CHI organizers resolve 100+ newly surfaced scheduling conflicts in just hours.

Game penalties decrease learning and interestAIED 2013

Easterday, M. W. & Jo, Y.

Educational game designers can improve learning and interest by embedding cognitive tutors in games and removing penalties such as loss of progress.

Formative feedback in digital lofts: Learning environments for real world innovationAIED workshop, 2013

Easterday, M. W., Rees Lewis, D., & Gerber, E.

Digital Lofts for teaching design process (in engineering, science in law) can support learning through 3 interelated feedback loops: (a) a crowd critique loop in which learners receive formative feedback on their innovation work from a broader community, (b) a case development loop in which examples of student work are semi-automatically created to provide instruction, and (c) a learner-driven instructional loop, in which self-assessments determine which group instruction is provided.

Automated Workflow SynthesisAAAI 2013

Haoqi Zhang, Eric Horvitz, and David Parkes

We introduce a method aimed at ideally harnessing human efforts in solving a problem by learning about a crowd’s performance on tasks and synthesizing an optimal workflow for solving a problem.

Cobi: Communitysourcing Large-Scale Conference SchedulingCHI Interactivity 2013

Haoqi Zhang, Paul André, Lydia Chilton, Juho Kim, Steven P. Dow, Robert C. Miller, Wendy MacKay, and Michel Beaudouin-Lafon

Cobi presents an alternative approach to conference scheduling that engages the entire community to take active roles in the planning process. This paper provides an overview of the Cobi system.

A Pilot Study of Using Crowds in the ClassroomCHI 2013

Steven Dow, Elizabeth Gerber, Audris Wong

We explore if and how online crowds can support student learning in the classroom and how scalable, diverse, immediate (and often ambiguous and conflicting) input from online crowds affects student learning and motivation for project-based innovation work. We share early evidence and discuss implications for creating a socio-technical infrastructure to more effectively use crowdsourcing in education.

The Future of Crowd WorkCSCW 2013

Niki Kitur, Michael Bernstein, Jeff Nickerson, Elizabeth Gerber, Aaron Shaw, Matt Lease, John Zimmerman, John Horton

Drawing on theory from organizational behavior and distributed computing, as well as direct feedback from workers, we outline a framework that will enable crowd work that is complex, collaborative, and sustainable. The framework lays out research challenges in twelve major areas: workflow, task assignment, hierarchy, real-time response, synchronous collaboration, quality control, crowds guiding AIs, AIs guiding crowds, platforms, job design, reputation, and motivation.

Perspective taking, political ideologies and digital gamesInternational symposium on digital ethics, 2013

Fitzpatrick, C., Hope, A., Barhumi, S., Krupnikov, Y., & Easterday, M. W.

Students struggle to predict whether different ideologies support or oppose policy interventions. Some ideologies are more difficult to reason about than others. Experience in domains such as political science do not seem to be correlated with improved reasoning ability.

Affect and Creative Performance on Crowdsourcing PlatformsACII 2013

Robert Morris, Mira Dontcheva, Adam Finkelstein, Elizabeth Gerber

We examine two affective techniques to boost creativity on crowdsourcing platforms – affective priming and affective pre-screening. We present guidelines for the design of human-computer interaction scenarios that involve affect and creative performance.


TurkServer: Enabling Synchronous and Longitudinal Online ExperimentsHCOMP 2012

Andrew Mao, Yiling Chen, Krzysztof Gajos, David Parkes, Ariel Procaccia, and Haoqi Zhang

We introduce TurkServer, a platform supporting synchronous and longitudinal experiments involving real-time interaction betwen participants on Amazon Mechanical Turk.

Hallucination: a Mixed-Initiative Approach for Efficient Document ReconstructionHCOMP 2012

Haoqi Zhang, John Lai, and Moritz Baecher

We introduce a mixed-initiative approach for document reconstruction that can significantly reduce the amount of time and effort required to reassemble a document from shredded pieces or an artifact from broken fragments.

Task Routing for Prediction TasksAAMAS 2012

Haoqi Zhang, Eric Horvitz, Yiling Chen, and David Parkes

We propose methods for routing a prediction task on a network where each participant can contribute information and route the task onwards. We introduce routing scoring rules that induce effective solving and routing behavior in equilibrium.

Human Computation Tasks with Global ConstraintsCHI 2012

Haoqi Zhang, Edith Law, Krzysztof Gajos, Eric Horvitz, Rob Miller, and David Parkes

best paper honorable mention
Mobi is a system for crowdsourcing itinerary planning. We introduce and illustrate the crowdware design pattern for tackling complex tasks with global constraints by using a shared, collaborative workspace.

Economics of BitTorrent CommunitiesWWW 2012

Ian Kash, John Lai, Haoqi Zhang, and Aviv Zohar

We report on a half-year-long measurement study of DIME – a community for sharing live concert recordings – that sheds light on the economic forces affecting users in private fire-sharing communities.

Supporting Creativity in Web-Based ServicesInternational Journal of Design 2012

Elizabeth Gerber, Caitlin Martin Kennedy

This paper encourages service designers to view the action of customization as a creative process and provides nine theoretically grounded design principles to support customer creativity within Web-based customization self-services.

Innovation Self-Efficacy: Development of a MeasureFIE 2012

Elizabeth Gerber, Caitlin Martin Kennedy, Jennie Braunstein, Elizabeth Kramer, Adam Carberry

Extracurricular Design-Based Learning: Preparing Students for Careers in Innovation, International Journal of Engineering EducationInternational Journal of Engineering Education 2012

Elizabeth Gerber, Jeanne Marie Olson, Rebecca Komarek

The Psychological Experience of PrototypingDesign Studies 2012

Elizabeth Gerber, Maureen Carroll

Priming for Better Performance in Micro-Task Crowdsourcing EnvironmentsIEEE Internet Computing 2012

Robert Morris, Mira Dontcheva, Elizabeth Gerber


Crowdsourcing General ComputationMSR Technical Report 2011

Haoqi Zhang, Eric Horvitz, Rob Miller, and David C. Parkes

We present a direction of research on principles and methods that can enable general problem solving via human computation systems.

An Iterative Dual Pathway Structure for Speech-to-Text TranscriptionHCOMP 2011

Beatrice Liem, Haoqi Zhang, and Yiling Chen

We develop a human computation algorithm for speech-to-text transcription. We introduce an iterative dual pathway structure that eliminates the need to check produced transcripts in a separate process.

PlateMate: Crowdsourcing Nutrition Analysis from Food PhotographsUIST 2011

Jon Noronha, Eric Hysen, Haoqi Zhang, and Krzysztof Z. Gajos

PlateMate allows users to take photos of their meals and receive estimates of food intake and composition from the crowd. PlateMate is nearly as accurate as a trained dietitian and easier to use than traditional self-reporting.

Using tutors to improve educational gamesAIED 2011

Easterday, M. W., Aleven, V., Scheines, R., & Carver, S. M.

Cognitive games that embed tutors in educational games increase both learning and interest.

Towards Large-Scale Collaborative Planning: Answering High-Level Search Queries Using Human ComputationAAAI 2011

Edith Law and Haoqi Zhang

Behind every search query is a high-level mission that the user wants to accomplish. We introduce a human computation system that generates simple plans containing diverse, actionable search results and a roadmap for accomplishing the mission.

Incentive Design for Adaptive AgentsAAMAS 2011

Yiling Chen, Jerry Kung, David Parkes, Ariel Procaccia, and Haoqi Zhang

To explore principles for designing incentives to influence the behavior of learning agents, we consider a setting in which a principle seeks to induce an adaptive agent to select a target action by providing incentives.

Affective Computational Priming and CreativityCHI 2011

Sheena Lewis, Mira Dontcheva, Elizabeth Gerber


Toward Automatic Task Design: A Progress ReportHCOMP 2010

Eric Huang, Haoqi Zhang, David Parkes, Krzysztof Gajos, and Yiling Chen

We provide a method for automating the design of a task to induce good output from crowd workers. We construct and learn models for predicting the rate and quality of work and use them to optimize designs.

An Intelligent Debater for Teaching ArgumentationITS 2010

Easterday, M. W.

people’s choice for best young researchers track poster

Getting and Staying on Topic During a BrainstormCHI 2010

Patti Bao, Elizabeth Gerber, Darren Gergle, David Hoffman

Toward a framework for the analysis and design of educational gamesDIGITEL 2010

Aleven, V., Myers E., Easterday, M. W., & Ogan, A.

The EDGE framework assists educational game design by helping designers (a) align goals, assessment and instruction, (b) considering mechanics, dynamics and aesthetics, and (c) tightlying learning and gameplay.

Learning to Waste – Wasting to Learn? How to Use Cradle to Cradle Principles to Improve the Teaching of DesignInternational Journal of Engineering Education 2010

Elizabeth Gerber, Anne McKenna, Penny Hirsch, Charles Yarnoff

Engineers of the future expected to be knowledgeable about the principles and practices needed to develop eco-effective products and manufacturing processes, yet it is challenging to integrate these principles and practices into undergraduate engineering design education. We present and evaluate an approach that exposing students to cradle to cradle design and help them make progress toward developing adaptive expertise in this area.


A General Approach to Environment Design with One AgentIJCAI 2009

Haoqi Zhang, Yiling Chen, and David C. Parkes

Environment design is the problem of influencing an agent’s decisions by making changes to the agent’s environment. We present a general framework for the formulation and solution of environment design problems with one agent.

Constructing causal diagrams to learn deliberationIJAIED 2009

Easterday, M. W., Aleven, V., Scheines, R. & Carver, S. M.

We present a cognitive framework for policy argumentation and show that causal diagram improve reasoning. Having a correct diagram improves reasoning and so does practice constructing a diagram (in cases where correct diagrams are unavailable).

Will Google destroy western democracy? Bias in policy problem solvingAIED 2009

Easterday, M. W., Aleven, V., Scheines, R., & Carver, S. M.

Confirmation bias in analyzing policy evidence may not arise because of biased search or biased evaluation, but because changes in beliefs after reading contradictory evidence have a very small effect on overall beliefs and because people recall the mass of evidence as supporting their current beliefs. This suggests that a better way to alievate confirmation bias might be to provide external representations of evidence.

Policy Teaching Through Reward Function Learning EC 2009

Haoqi Zhang, David C. Parkes, and Yiling Chen

We study the learning and optimization problem of using rewards to induce an agent to follow a particular policy. We prove that the problem is tractable and that effective rewards can be discovered after few interactions.

Using Improvisation To Enhance the Effectiveness of BrainstormingCHI 2009

Elizabeth Gerber

We explore how designers use techniques from theatrical improvisation to adhere to the rules of brainstorming thereby enhancing group interactions while collaborating. The usefulness of improvisation for brainstorming stems from the similarity of the goals of improvisation and brainstorming, the similarity of the recurrent problems that actors and designers encounter when collaborating, and the distinctness of the ways each have devised to resolve the problems that block the group’s performance.

Design requirements of argument mapping software for teaching deliberationOnline deliberation: Design, research, and practice 2009

Easterday, M. W., Kanarek, J., & Harrell, M.

For argument mapping software to be usuable in education they must: (a) provide appropriate visual represntations, (b) allow flexible consruction, (c) support visualogic, (d) automate nonsemantic operations, (e) allow presentation of multiple diagrams, and (f) allow cross platform campatibility.

Strong Activity Rules for Iterative Combinatorial AuctionsComputers & Operations Research 2009

Pavithra Harsha, Cynthia Barnhart, David C. Parkes, and Haoqi Zhang

We propose strong activity rules that allow for simple, consistent bidding strategies in complex, iterative auctions, for example for spectrum or landing slots at congested airports.


Value-based Policy Teaching with Active Indirect ElicitationAAAI 2008

Haoqi Zhang and David C. Parkes

We study the learning and optimization problem of using rewards to induce an agent making sequential decisions to follow a policy that maximizes the total expected value of an interested party.


‘Tis better to construct than to receive? The effects of diagram tools on causal reasoningAIED 2007

Easterday, M. W., Aleven, V., & Scheines, R.

We show that causal diagrams improve policy reasoning. Having a correct diagram improves reasoning and so does practice constructing a diagram (in cases where correct diagrams are unavailable).

The logic of Babel: Causal reasoning from conflicting sourcesAIED Workshop, 2007

Easterday, M. W., Aleven, V., & Scheines, R.

Improvisation Principles and Techniques for DesignCHI 2007

Elizabeth Gerber

The paper argues that improvisation can build perspectives and skills that are critical for designers, such as creative collaboration, fostering innovation, supporting spontaneity, learning through error, and presenting ideas. The applications are analyzed to demonstrate individual and group level outcomes in design work.