Multimodal Data Comprehension: Understanding How Visual-Textual Chains of Information Influence Data Interpretation

Authors

Arran Zeyu Wang (University of North Carolina-Chapel Hill), Fuling Sun (University of California San Diego), Danielle Albers Szafir (University of North Carolina-Chapel Hill)

Presentation

Session: How you show it makes a difference!
Time: Tuesday, Nov 10, 13:12 – 13:24 (US/Eastern) · session 13:00 – 14:30
Location: Hall Essex north

Keywords

Data comprehension, narrative visualization, framing effect, priming, cognition, multimodal, communication

Abstract

Visualizations and text often work together to support effective data communication. Despite this common paradigm, we know little about how the interplay of these modalities affects people’s data comprehension. We present a novel experimental paradigm to investigate multimodal data comprehension—the process of people comprehending information from multimodal visual and textual data—across both crowdsourced and think-aloud environments. Our methodology employs two sequential chains for presenting multimodal information—a visualization-first chain and a text-first chain—asking people to describe the data presented iteratively. By comparing how people’s data comprehension changes across the chain, we can assess the information contribution of each modality and how they shape subsequent comprehension. We found that the visualization-first chain facilitates exploratory comprehension with hypothesis-driven discovery, whereas the text-first chain yields confirmatory comprehension akin to framing effects where visualizations serve to reinforce and confirm observations drawn from text. Our findings provide empirical insights into multimodal information integration, with implications for designing more effective data-driven communication.

For Practitioners

data journalists, data scientists, designers, visualization psychologists