An automated evaluation pipeline for analyzing how vision-language models recognize, interpret, and explain visual puns.
Visual puns require more than identifying the objects in an image. A model may need to recognize visual elements, understand their spatial arrangement, connect them to a linguistic expression, and determine how those concepts combine to form a punchline.
This project evaluates whether vision-language models can perform that multi-step reasoning reliably.
I developed a Python-based benchmarking pipeline that processes visual-pun examples, collects outputs from multiple model families, standardizes heterogeneous responses, and evaluates both final-answer accuracy and intermediate reasoning quality. The resulting framework was used to analyze dozens of open-source vision-language models across more than 500 curated visual-humor examples.
- Built an automated evaluation pipeline for benchmarking dozens of open-source VLMs.
- Curated and standardized a dataset of 500+ visual-pun examples.
- Designed structured annotations for recognition, linguistic, spatial, and multi-hop reasoning.
- Automated model prompting, response collection, parsing, scoring, and statistical analysis.
- Normalized inconsistent free-form model outputs into machine-readable formats.
- Evaluated model responses using BLEU, ROUGE, ROSCOE, and custom reasoning-hop metrics.
- Developed robust JSON-processing workflows for large, partially malformed experiment outputs.
- Analyzed model trends and failure modes across multiple prompt and model configurations.
Figure 1: Example histogram summary of GPT-4 autoscored VLM evaluation. This open-source model performed very poorly, with most answers varying in terms of sensibility, while nearly all of its attempts missed the punchline.
Figure 2: ROUGE-2 scores on all models tested. Most VLMs did well on OCR (optical character recognition), but did poorly in the other categories.
The project investigates a central multimodal reasoning question:
Can a vision-language model explain how the visual content of an image produces a linguistic pun, rather than simply guessing the punchline?
A model may produce the correct answer for the wrong reason. Because of this, evaluating only the final punchline does not fully measure multimodal understanding.
The evaluation framework separates the task into individual reasoning steps so that model behavior can be examined more precisely.
Each visual pun is represented as a sequence of reasoning hops. Depending on the example, these may include:
Identifying important objects, symbols, letters, actions, or visual features.
Understanding how visual elements are arranged, modified, combined, or placed in an unusual context.
Connecting the recognized visual content to words, phrases, idioms, homophones, or multiple meanings.
Combining the preceding observations to explain the intended joke.
Annotations support both single-hop and multi-hop evaluation. Structured fields include:
subject_id
prompt
test_queries
answers.value
aliases
hop_type
hop_id
singlehop_reasoning
multihop_reasoning
Examples belonging to the same visual-pun subject use a shared identifier prefix, allowing multiple prompts and reasoning questions to be associated with one image.
m00001_p1
m00001_p2
m00001_p3
Here, m00001 identifies the underlying subject, while the suffix identifies the prompt or evaluation variant.
Consider an image containing Scrabble-style tiles arranged as:
G E S G
The intended answer is “scrambled eggs.”
A complete reasoning chain may contain the following steps:
- Recognize the objects as letter tiles.
- Identify the visible letters:
G,E,S, andG. - Rearrange the letters to form
EGGS. - Observe that the letters are currently scrambled.
- Connect the visual arrangement to the phrase “scrambled eggs.”
A model that only responds with “scrambled eggs” may have guessed the answer from the image. A stronger response demonstrates the intermediate visual and linguistic connections that produce the pun.
The project implements an end-to-end workflow for running and comparing multimodal experiments.
Visual-Pun Dataset
│
▼
Prompt and Annotation Loading
│
▼
Model-Specific Request Construction
│
▼
Concurrent VLM Inference
│
▼
Response Parsing and Normalization
│
▼
Answer and Reasoning-Hop Scoring
│
▼
Metric Aggregation
│
▼
Statistical and Failure-Case Analysis
Visual-humor examples collected from online sources were:
- Cleaned and deduplicated.
- Assigned consistent subject and prompt identifiers.
- Converted into a standardized representation.
- Annotated with expected answers and aliases.
- Decomposed into reasoning-hop categories.
- Prepared for reproducible experiments across model families.
The pipeline supports multiple prompt styles, including:
- Final-answer-only questions.
- Explanation-generation questions.
- Explicit step-by-step reasoning questions.
- Questions targeting individual reasoning hops.
- Full multi-hop visual-pun explanations.
This makes it possible to compare whether a model performs differently when asked only for an answer versus when it must explain the visual and linguistic reasoning behind that answer.
The inference workflow was designed to evaluate outputs from dozens of open-source VLMs using a common experiment format.
The pipeline automates:
- Request construction.
- Image and prompt association.
- Concurrent API calls.
- Response collection.
- Experiment metadata tracking.
- Retry and exception handling.
- Output serialization.
Multithreading was used to improve throughput when processing large combinations of models, images, and prompts.
Different models frequently return incompatible outputs. Responses may contain:
- Plain text.
- Markdown code blocks.
- JSON objects.
- Stringified JSON.
- Additional conversational text.
- Missing fields.
- Concatenated JSON arrays.
- Partially completed experiment records.
I developed preprocessing and postprocessing workflows to convert these heterogeneous outputs into a consistent internal representation.
The parsing system includes logic for:
- Extracting structured content from free-form responses.
- Parsing stringified judge responses into dictionaries.
- Reading concatenated JSON arrays with
JSONDecoder.raw_decode. - Matching annotations using subject-ID prefixes.
- Preserving alignment between prompts, responses, and source examples.
- Detecting missing or skipped experiment results.
- Preventing silent truncation caused by mismatched list lengths.
These normalization steps made automated scoring possible across model families with different response conventions.
The framework evaluates both generated language quality and reasoning correctness.
Measures n-gram overlap between generated responses and reference answers. It is useful for detecting direct lexical agreement but can penalize valid paraphrases.
Measures overlap and recall relative to reference explanations. It provides additional information when models generate longer reasoning responses.
Evaluates qualities of model-generated reasoning, including semantic alignment and consistency between reasoning steps and expected explanations.
Standard NLP metrics do not directly determine whether a model completed the correct visual reasoning sequence. Custom scoring therefore evaluates:
- Whether required reasoning hops were present.
- Whether hops appeared in a logically valid order.
- Whether each hop matched the expected category.
- Whether the final punchline followed from the preceding reasoning.
- Whether the model reached the correct answer despite missing or incorrect intermediate steps.
This allows the analysis to distinguish among several outcomes:
Correct answer + correct reasoning
Correct answer + incomplete reasoning
Correct answer + incorrect reasoning
Incorrect answer + partially correct reasoning
Incorrect answer + incorrect reasoning
After scoring, model outputs are aggregated to compare:
- Performance across model families.
- Answer-only versus multi-hop prompting.
- Final-answer accuracy.
- Reasoning-hop completion.
- Performance by hop category.
- Common failure patterns.
- Differences between lexical metrics and reasoning-based evaluation.
- Sensitivity to prompt design.
The analysis emphasizes more than a single aggregate score. It examines where models fail and which stage of visual-linguistic reasoning causes the failure.
Open-source VLMs do not follow a universal response format. The pipeline therefore required a flexible normalization layer rather than model-specific manual analysis.
Some model requests could be skipped, fail, or return incomplete outputs. Directly combining prompts and results with operations such as zip() could silently truncate or misalign records.
The pipeline instead tracks records through stable identifiers and explicit index mappings.
Experiment outputs included standard JSON, stringified JSON fields, and files containing multiple concatenated JSON arrays. A custom decoder workflow was used to recover individual records without assuming that every file contained one valid top-level object.
A model can express the same reasoning using different words. Evaluation therefore combines lexical metrics with structured hop-level analysis instead of relying on exact string matching alone.
Evaluating hundreds of examples across multiple models and prompt variants produces thousands of inference and scoring operations. Automated request generation, multithreading, structured result storage, and reproducible aggregation were necessary to make this analysis practical.
This project demonstrates experience with:
- Vision-language models.
- Large language model evaluation.
- Multimodal reasoning.
- Automated inference pipelines.
- OpenAI API integration.
- Prompt and experiment design.
- Concurrent API processing.
- Dataset curation and annotation.
- JSON and semi-structured data processing.
- NLP evaluation metrics.
- Statistical benchmarking.
- Failure-mode analysis.
- Reproducible machine-learning experimentation.
My work focused on the design and implementation of the experimental pipeline and evaluation framework.
I contributed to:
- Curating, cleaning, and standardizing more than 500 visual-humor examples.
- Designing structured labels for visual and linguistic reasoning.
- Implementing the automated VLM evaluation pipeline.
- Supporting multiple answer-only and multi-hop prompt formats.
- Building concurrent request-processing workflows.
- Developing preprocessing and postprocessing for heterogeneous outputs.
- Implementing automated scoring with BLEU, ROUGE, and ROSCOE.
- Designing hop-level reasoning evaluation methods.
- Creating statistical summaries and model comparisons.
- Investigating model failure cases.
- Adapting prior research methodologies to visual-pun evaluation.
- Contributing to experimental design and interpretation of results.
Automated language metrics cannot fully capture humor understanding. A model may generate a semantically correct explanation that uses different wording from the reference, while another model may reproduce reference keywords without forming a valid reasoning chain.
Visual-pun interpretation can also be subjective. Some images support multiple reasonable explanations, and the expected amount of reasoning may vary by example.
For these reasons, the framework combines automated text metrics, structured reasoning-hop evaluation, and qualitative failure-case inspection.
Potential extensions include:
- Adding human evaluation for explanation quality.
- Measuring agreement between automated judges and human annotators.
- Expanding the benchmark to multilingual and culturally dependent puns.
- Evaluating newer multimodal foundation models.
- Testing robustness to image crops, noise, and misleading visual context.
- Comparing chain-of-thought, structured JSON, and direct-answer prompting.
- Developing learned evaluators specifically for multimodal humor reasoning.
- Creating interactive dashboards for model and failure-case comparison.
This project treats visual-pun understanding as a structured multimodal reasoning problem rather than a simple image-question-answering task.
By combining dataset engineering, automated VLM inference, robust response normalization, NLP metrics, custom reasoning-hop evaluation, and statistical analysis, the pipeline provides a reproducible way to investigate both whether a model reaches the correct answer and how it reasons from visual evidence to language.