Multimodal Data
What this domain tests
This domain tests how text, images, video, and audio are represented, paired, aligned, and fused in multimodal generative systems. Expect questions on tokenization, contrastive alignment, fusion strategies, and data curation.
Core rules and concepts
- Images become tokens via patches (ViT) or discrete VQ tokens; audio is commonly converted to spectrograms or neural-codec tokens; video adds a temporal axis.
- CLIP-style contrastive training pulls matched image-text pairs together and pushes mismatched pairs apart in a shared embedding space.
- Fusion can be early (concatenate features), late (combine predictions), or cross-attention, where queries from one modality attend to another.
- Data quality dominates: captions, resolution, aspect ratio, and licensing all shape what the model learns.
- Alignment matters: audio sample rate and video frame rate must match preprocessing expectations.
Worked example
A batch of N image-text pairs yields an N x N similarity matrix. The diagonal holds the positive pairs. Symmetric InfoNCE applies softmax across rows (image to text) and across columns (text to image), then averages the two losses. If one caption is duplicated across many images, off-diagonal entries become falsely negative, and the model is penalized for correct matches. Deduplicating captions or maintaining a queue of negatives reduces this label noise and stabilizes training.
Traps that cost marks
- Assuming any caption is usable training signal.
- Ignoring aspect-ratio distortion from naive resizing.
- Mixing modalities with mismatched sampling rates or channel layouts.
- Treating fusion architecture as more important than pair quality.
- Evaluating retrieval with a single modality's embeddings only.