Exploring LLM Agent Designs and Interaction Modalities for Scientific Visualization

Authors

Jackson Vonderhorst (University of Notre Dame), Kuangshi Ai (University of Notre Dame), Haichao Miao (Lawrence Livermore National Laboratory), Shusen Liu (Lawrence Livermore National Laboratory ), Chaoli Wang (University of Notre Dame)

Presentation

Session

Me, Myself, and AI

Time

Wednesday, Nov 11, 13:00 – 13:09 (US/Eastern) · session 13:00 – 14:30

Location

Hall America center

Keywords

Scientific visualization, LLM agents, interaction modalities, evaluation

Abstract

This paper examines how large language model (LLM) agents perform on scientific visualization (SciVis) tasks that require generating visualization workflows from natural-language instructions. We compare three representative agent designs—domain-specific agents with structured tool use, computer-use agents, and general-purpose coding agents—across 15 benchmark tasks, evaluating visualization quality, efficiency, robustness, computational cost, and the impact of persistent memory. We further study interaction modalities, including code scripts, model context protocol (MCP) or API calls, command-line interfaces (CLI), and graphical user interfaces (GUI). Our goal is to characterize the tradeoffs among representative SciVis agent configurations used in practice. The results reveal clear tradeoffs across agent designs and interaction modalities. General-purpose coding agents achieve the highest task success rates but incur greater computational cost, whereas domain-specific agents are more efficient and stable but less flexible. Computer-use agents perform well on individual operations but struggle with multi-step workflows. Across both CLI- and GUI-based settings, persistent memory improves performance over repeated trials, but its effectiveness depends on the interaction mode and the quality of feedback. These findings suggest that future SciVis systems should combine structured tool use, interactive capabilities, and adaptive memory mechanisms to balance performance, robustness, and flexibility.

For Practitioners

This paper is relevant to scientific visualization practitioners, developers of AI-assisted visualization tools, and researchers developing LLM agents and post-training foundation models. Practitioners can use our findings to better understand the strengths and limitations of current LLM-based visualization agents and select appropriate agent configurations for different workflows. The paper also provides design insights for future SciVis systems and may help guide the development of more capable agentic systems for scientific data analysis and visualization.