Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents
Authors
Srishti Palani (Salesforce), Vidya Setlur (Tableau Research)
Presentation
- Session
- Me, Myself, and AI
- Time
- Wednesday, Nov 11, 13:09 – 13:18 (US/Eastern) · session 13:00 – 14:30
- Location
- Hall America center
Keywords
Tabular Data; Algorithms ; Methodologies; Computational Benchmark Studies; Evaluation Metrics, Reference-Free Metrics; Conversational Visual Analytics; Visualization Design Theory, Cooperative Communication Principles
Abstract
Conversational visual analytics (CVA) agents powered by large language models generate visualizations and natural-language explanations from open-ended queries. Evaluating these multimodal outputs is challenging: curated reference benchmarks are costly to author, cannot comprehensively capture the space of valid responses, and are unavailable in production. Building on the Lexara evaluation framework, we introduce Lexara-RF, a reference-free set of metrics that scores CVA outputs using only the prompt, data, and model response. We reformulate evaluation as verification: 13 metrics operationalize visualization design theory and Gricean cooperative principles as computable consistency, intent-alignment, and design validity checks. On a human-rated corpus of CVA test-cases, Lexara-RF achieves alignment comparable to reference-based formulations, outperforms surface-similarity NLG baselines, and localizes structurally grounded failures with high accuracy.
For Practitioners
We wrote this paper for three main groups working with conversational tools that turn natural language into data visualizations: 1. Developers building these tools (such as Tableau Pulse, Power BI Copilot, or ThoughtSpot Sage). 2. Machine learning and evaluation engineers who handle quality checks, prompt refinement, and pre-release testing. 3. Reliability, trust, and safety teams who monitor these tools in production environments where a perfect "answer key" isn't available. Practitioners can apply LEXARA-RF in three concrete ways: (1) Automated testing: Because our 13 metrics evaluate any combination of user input, data, and system response without needing pre-written answers, practitioners can use them to run automated checks whenever they iterate on prompts or models. (2) Monitor live traffic: When serving real users, there is often no predefined "correct" reference answer to compare against. Our framework can analyze these live sessions to flag issues (such as data pipeline errors, formatting mismatches, or conversational drift) helping practitioners route these problems to the right debugging surface. (3) Diagnostic tool: When troubleshooting, our specific metric scores help practitioners pinpoint exactly where a response failed, which significantly reduces the time needed to diagnose user-reported issues. As discussed in Section 4, the split between structural metrics and rubric-based metrics also helps practitioners decide which tests can be fully automated and which ones still warrant a manual review.