Should We Dangle a Carrot? The Effect of Performance-based Incentives in Visualization Experiments
Authors
Abhraneel Sarma (Graz University of Technology), Matthew Kay (Northwestern University), Sheng Long (Northwestern University), Michael Correll (Northeastern University), Alexander Lex (Graz University of Technology)
Presentation
- Session
- Figuring out how to do good research
- Time
- Tuesday, Nov 10, 15:36 – 15:48 (US/Eastern) · session 15:00 – 16:30
- Location
- Hall America center
Links
Sign in to access the preprint PDF.
Sign in- Download Supplemental Material
Keywords
Perception of Correlation, Uncertainty Visualization, Decision-making, Experiment Design, Incentives
Abstract
A perennial research question in visualization involves identifying which visual encodings for a particular dataset are most effective for users in performing a specific task. The relative effectiveness of the different encodings are commonly identified through controlled experiments. However, designing an experiment involves making many, often ad hoc, decisions about the experimental setup such as whether to include a training module, whether to provide performance-based incentives to participants, etc. Yet, there is limited guidance on how these decisions should be made, and we do not fully understand the impact of these subjective decisions on empirical results. In this paper, we investigate the impact of one such key design decision—monetary rewards. Specifically, we ask: does providing or not providing participants with performance-based financial incentives affect the results and the conclusions that we draw from visualization studies? We conducted two crowdsourced studies investigating the impact of incentives on (i) a low-level, perceptual task (perception of correlations in scatterplots or parallel coordinate plots), and (ii) a task involving reasoning (decision-making based on a weather forecast represented as intervals or density plots). In each of these studies, we manipulate both the visual representation and the presence of incentives as between-subject conditions. We expected to find no effect of incentives on the perceptual task, but to see an effect for the decision-making task. However, we found no effect on task performance in either study. While these are results of only two studies and should be replicated, they suggest that performance-based financial incentives may not always have the intended effect on participants that we presumed, and calls for a reflection of how incentivized studies should be designed. A copy of this paper and all supplemental materials are available at https://osf.io/t8eah.
For Practitioners
The paper provides guidance on how researchers should design experiments