Read, Critique, or Sketch? Investigating Alternative Visualization Literacy Assessment Modalities

Authors

Zach Cutler (University of Utah), Lily W. Ge (Northwestern University), Matthew Kay (Northwestern University), Lane Harrison (Worcester Polytechnic Institute), Andrew M McNutt (University of Utah), Alexander Lex (Graz University of Technology)

Presentation

Session

Kids these days don't learn anything anymore, let's fix that!

Time

Thursday, Nov 12, 15:24 – 15:36 (US/Eastern) · session 15:00 – 16:30

Location

Hall America north

Keywords

Visualization Literacy, Critique, Sketch, Think Aloud

Abstract

Visualization literacy is a multifaceted construct encompassing skills and competencies, such as decoding data, constructing charts, and identifying design flaws. Yet, assessments of these competencies has been primarily constrained to multiple choice assessments that target lower-order skills, such as chart comprehension. As a result, they often exhibit ceiling effects (i.e., even modestly skilled individuals commonly score near the top of the scale), and do not provide enough information about an individual's higher-order skills (e.g., applying external knowledge, formulating critiques, and designing visualizations). To close these gaps, we develop and investigate two web-based qualitative assessments for testing the critique and design aspects of visualization literacy through online think-aloud critique and sketching of visualization designs based on data and a prompt. We compare performance on our assessments to two established visualization literacy assessments, CALVI and Mini-VLAT, by administering them to three groups that represent three experience levels: crowdworkers, students who have taken a relevant course, and researchers. We find that our critique and sketching assessments capture skills distinct from existing measures and that they differentiate between experienced individuals better than multiple choice-based alternatives. Although administering and grading qualitative assessments can be challenging, our findings suggest qualitative, multimodal assessments are a promising complement to existing visualization literacy assessments, in particular when high visualization skills need to be distinguished.

For Practitioners

Any practitioner who needs to teach or train employees could benefit from this work, as the standardized tests may be useful in measuring existing and new skills in employees.