HSI

Assisting People with Visual Impairments in Understanding Data Visualizations

Automated Alt Text Generation and Evaluation for Charts Using Multimodal Large Language Models (LLMs) such as ChatGPT and LLaMA

Infographic showing the difference between image caption for sighted users and alt text for visually impaired users.
Conceptual illustration of caption vs. alt text

Initial Situation

Alternative text, or "alt text," provides a written description of visual content, such as images or charts. This makes the content accessible to users with visual impairments. Screen readers use alt text to describe visual elements using speech or Braille. Unlike captions, which are brief descriptions placed near a graphic to provide context for all users, alt text is designed for visually impaired users, enabling them to construct a mental model of the graphic. In contrast, alt text is designed for visually impaired users, enabling them to construct a mental model of the graphic.

Despite their importance, both legal and ethical, alt texts are often missing from digital platforms, including charts published by the Swiss newspaper Neue Zürcher Zeitung (NZZ). Recent developments in multimodal LLMs offer new opportunities to generate these texts automatically. This project explores how LLMs can be used to produce high-quality German alt texts for charts on a large scale.

Problem Statement and Goals

Generating alt text for charts is more complex than generating alt text for images. In addition to describing visual elements such as lines, axes, and categories, the alt text must accurately communicate the underlying data trends and insights. This task is further complicated by the variety of chart types, such as line, bar, and stacked bar charts, each of which requires a specific interpretation and potentially a different prompt. Additionally, the absence of human-written reference alt texts (also called gold-summaries) for NZZ charts means there is no existing ground truth for benchmarking. This makes developing and evaluating automated solutions difficult because conventional supervised learning approaches and straightforward performance comparisons are not applicable.

The goal of this project is to investigate the feasibility and quality of automatically generating German alt texts for charts using multimodal LLMs. The project aims to assess whether LLMs can produce meaningful alt texts across different chart types and complexities, and to evaluate the generated texts using both qualitative analysis and metrics.

Impression 1

Infographic showing four layers of an alt text: the short description includes the title, chart type, axis labels, and range; the link points to the long description; the long description covers key messages, trends, comparisons, and extremes; and the data provides numerical values with descriptive column names.

The different elements of alt text for charts include a short description, a link to a long description, the long description itself, and the underlying data values displayed in the chart.

Impression 2

Flowchart visualizing a pipeline for generating and evaluating alt texts for charts. Data from Statista and NZZ are preprocessed and saved in a MySQL database. A multimodal LLM generates alt texts for Statista charts. Outputs, prompts, and metadata are stored and reused to enable knowledge transfer for generating alt texts for NZZ charts.

This flowchart illustrates the data pipeline for alt text generation and evaluation.

Impression 3

An area chart shows the projected growth of electricity generation from low-carbon energy sources (solar, wind, hydro, nuclear, others) from 2010 to 2050 in terawatt-hours. The chart includes both a short and a long German alt text. Color highlights indicate the authors' analysis of specific alt text passages.

This annotated example shows a German alt text with both short and long descriptions for an area chart on global electricity generation. Color highlights indicate the authors’ analysis of the alt text: turquoise for repetition, red for hallucinations, and pink for missing but relevant information.

Solution Developed and Benefits

A key component of the project is the automated pipeline that generates and evaluates alternative text descriptions. The pipeline processes chart data and metadata from Statista and NZZ. This data is preprocessed and stored in a structured MySQL database. The pipeline uses a multimodal large language model (LLM) to generate short and long alt texts. This process includes chart parsing, dynamic prompt rendering, and LLM-based output generation. The outputs are evaluated using a combination of automated metrics (e.g., BLEU, SBERT, and character count) and qualitative assessments, such as case studies and expert reviews. Additionally, an LLM is used as a judge to evaluate criteria such as informativeness, fluency, and coherence. All steps and results are logged in the database to ensure traceability and reproducibility.

Key Findings

  • Multimodal LLMs can reliably generate useful alt texts when guided by well-structured and chart-specific prompts.
  • Simple charts consistently lead to higher-quality results than complex visualizations.
  • The structure and specificity of the prompt have a significant impact on output quality.
  • Among all tested models, GPT-4o produced the most accurate, fluent, and coherent alt texts.
  • Human feedback highlighted aspects like clarity, accuracy, and redundancy — qualities that are not fully captured by automated metrics such as BLEU or SBERT.
  • Feedback from a blind accessibility expert introduced a fundamentally different perspective on what makes an alt text truly helpful.

The project shows that multimodal LLMs can improve accessibility by generating alt texts for charts and offer guidance for future enhancements in prompt design and evaluation.

Key Terms

  • Alt Text A textual alternative to visual content.
  • LLM Multimodal Large Language Model, like GPT-4 or LLaMA.
  • SBERT Semantic textual similarity model.
  • BLEU N-gram precision-based metric for text evaluation.

Project Partner

NZZ logo

Jonas Oesch - Visual Journalist

Team

Julia Locher
Alessia Vannini

Advisors
Prof. Dr. Arzu Çöltekin
Dr. Marianne Santaholma