Vision-Language Model for Chart Question Answering
A 7B vision-language model is fine-tuned for question answering over scanned financial reports. The training set contains 120,000 image-question-answer triples from 8,000 PDFs. Images are resized to 336×336 pixels to cap visual tokens at 256 per page. Many charts use 6- to 8-point axis labels, dense legends, and thousands separators. Validation loss reaches 0.31, but human evaluation on 500 chart questions shows numeric exact match of 42%, versus 68% for an OCR-plus-LLM baseline. On text-only pages the model scores 81%, beating that baseline by 5 points. Errors include ignoring axis units, misreading 1,200 as 120, and inventing plausible values. An audit finds that 18% of validation page images have near-duplicates in training, created from the same PDF page at different DPI or crop. The team must decide whether to ship next month.
1. Diagnose the chart QA failure. Separate perception, data leakage, and reasoning.
Read the worked answer
The 336×336 resize with a 256-token cap is the first suspect. Six- to eight-point axis labels, legends, and thousands separators become unreadable, while text-only pages remain legible because their content is dominated by larger text. The text-only score of 81% shows language and layout handling are not the main failure. Errors such as 1,200 becoming 120 and units being ignored are consistent with lost visual detail and weak grounding. Hallucinated plausible values suggest the language prior fills gaps when the chart evidence is absent.
Data leakage also matters: 18% of validation pages have near-duplicates in training. Deduplicate by PDF page hash and perceptual hash, then recompute validation loss and accuracy. The 0.31 loss is not trustworthy if the validation set is contaminated. Run an ablation with high-resolution chart crops, with OCR text only, and with a chart-to-table extractor. If high-resolution crops fix the errors, the issue is perception; if not, it is reasoning or label quality. Also audit whether chart answers were derived from captions rather than the plotted values.
2. Propose changes to the data and model input pipeline.
Read the worked answer
Replace the single 336×336 resize with dynamic tiling: render the page at 1024-1536 pixels, split it into 336-pixel tiles, and add a global thumbnail. Cap visual tokens around 1,024-2,048 per page, or crop only the chart region to control cost. Inject OCR tokens with coordinates for axis labels, legends, and units. Consider a tool-augmented pipeline that extracts a table from the chart and passes it to the language model, especially for dense charts.
On the data side, deduplicate by PDF and perceptual hash, stratify by chart type, and add synthetic charts with known values and varied styles. Normalize answers to include units and use consistent thousands separators. Augment with low-quality scans and crops. Add an abstention target when OCR confidence is low. Audit labels for caption-derived errors. The tradeoff is prefill latency: high-resolution tiling increases visual tokens, so measure quality per token and use chart-only crops where possible.
3. Define the evaluation and release gate versus the OCR-plus-LLM baseline.
Read the worked answer
Evaluate on a deduplicated 1,000-question chart set stratified by chart type, scan quality, and question type. Metrics: numeric exact match with tolerance, unit accuracy, axis-label accuracy, answer correctness, abstention rate, hallucination rate, and latency. Compare against the OCR-plus-LLM baseline per stratum, not just overall. Require overall numeric exact match of at least 65%, no chart type more than 5 points below the baseline, and no regression on text-only pages.
Release gate: deduplicated chart numeric exact match at least 65%, unit accuracy at least 90%, hallucination rate below 5%, and appropriate abstention with escalation on low-quality images. Human-review 200 failures before launch and canary at 5%. If the gate is not met, ship the OCR-plus-LLM fallback for charts while keeping the VLM for text pages. Document that the validation set was contaminated and that the new estimate is based on deduplicated data.