LlamaVision: A Tiny Vision-Language Model

I built LlamaVision to answer a fairly narrow question: how does alignment between images and text actually emerge in a small multimodal model? Rather than train something large, I kept everything tiny on purpose so I could watch the mechanism instead of the marketing.

At its core, LlamaVision is a lightweight image projector — around 6.3M parameters — that maps visual features from a Vision Transformer (ViT) into the text embedding space of Llama-3.3-1B.

The one idea I wanted to study

Most of a vision-language model's heavy lifting sits in two components that already work well on their own: a vision encoder and a language model. The projector is the bridge. I wanted to isolate that bridge and study it directly:

• Freeze the strong parts (the ViT and the language model).
• Train only the small connector that has to learn a shared space.
• Watch how quickly — and how well — visual features start "speaking" the language model's embedding language.

By making the projector the only trainable piece, the experiment stayed focused on the alignment question and nothing else.

Why keep it so small

A 6.3M-parameter projector is small by design, and that smallness is the point:

• It's cheap to train and iterate on, so I could run many variations.
• The learned mapping is easier to reason about than a giant fused model.
• It makes the educational takeaway clean — the results are about alignment, not about scale.

Working with Llama-3.3-1B rather than a large model kept the whole system in a range where I could actually inspect behavior instead of just trusting it.

What I took away

A few things stood out from building LlamaVision:

• Alignment is a learnable interface. A small projector can teach ViT features to land meaningfully in a language model's embedding space.
• Freezing the big components is a good research move. It concentrates the learning in one place and makes cause and effect legible.
• Small is a feature for understanding. When the goal is insight rather than a benchmark, a compact setup teaches more per experiment.

This was primarily a research and educational project for me — a way to build intuition about how multimodal models stitch modalities together, rather than to ship a product.

If you want to check out the project directly, here it is: LlamaVision on Hugging Face.