Similar Ratings, Different Consensus: Visualizing Agreement Uncertainty in Survey Data

Authors

Vrushali Koli (New Jersey Institute of Technology), Yi Meng (New Jersey Institute of Technology), Eugene P. Deess (New Jersey Institute of Technology), Aritra Dasgupta (New Jersey Institute of Technology)

Keywords

survey data, agreement uncertainty, consensus

Abstract

Survey results are routinely summarized with scalar measures of agreement to compare questionnaire items, yet those summaries are hard to interpret on bounded rating scales. Most uncertainty visualizations communicate uncertainty by representing multiple possible values, estimates, or outcomes. We consider a complementary source: the responses and the computed agreement score are fixed, but the meaning of that score depends on where it sits on the scale. Because responses have less room to spread near the endpoints of a five-point Likert item, agreement there is mechanically constrained, so the same score can arise from very different distributions and can imply different consensus at different rating levels. We call this ambiguity agreement uncertainty; collecting more responses does not resolve it, since the score must instead be read against the feasible agreement space at its rating level. We introduce residual consensus, a calibration that expresses agreement relative to what the scale allows, informative only where that space has enough resolution and therefore read alongside the response distribution itself. Drawing on ideas from possibility-framing and meta-uncertainty in uncertainty visualization, we contribute a coordinated view that shows calibrated agreement, its positional informativeness, and the distribution together, demonstrated on real-world survey datasets to support more meaningful comparisons across bounded scales.