Embedding

Embedding is a dense vector of real numbers that represents the semantic content of a piece of text, an image, or another input, mapping meaning into a high-dimensional space where geometric distance between points corresponds to similarity of meaning. Produced by an encoder model (such as a sentence transformer or a contrastive vision-language model), an embedding typically has a fixed dimensionality—common sizes range from 384 to 1536 dimensions—and is generated by stripping the decoder from a transformer and reading the final hidden state. Two inputs that are semantically close ("cat" and "feline") land near each other in that space, while unrelated inputs land far apart, usually measured by cosine similarity or Euclidean distance. Because the representation is continuous and differentiable, embeddings compose cleanly with downstream math: nearest-neighbor search, clustering, and linear arithmetic. They are the substrate beneath semantic search, retrieval-augmented generation, deduplication pipelines, and recommendation systems, turning fuzzy human meaning into something a database can index and a model can consume.

How it works

An encoder model ingests a tokenized input, runs it through stacked self-attention layers, and emits a hidden state for each token; a pooling step—often mean pooling or taking the CLS token—collapses those into a single fixed-length vector. During training, contrastive or triplet objectives pull semantically matched pairs together and push mismatched pairs apart, shaping the geometry of the space. At inference, the resulting vector is stored as a row in a vector index (HNSW, IVF, or flat brute-force) so that approximate nearest-neighbor queries return the closest stored embeddings in sub-millisecond time. Dimensionality is fixed per model, so switching embedding models requires re-embedding the entire corpus.

Why it matters for AI engineers

Embedding choice directly governs retrieval quality, which in turn governs RAG accuracy, so engineers must benchmark recall on their own domain rather than trust leaderboard numbers. Cost and latency scale with dimensionality and corpus size: a 1536-dim model over ten million rows demands meaningful storage and index-build time, while a 384-dim open-weights model may halve both at a measurable recall cost. Embeddings drift when upstream models are swapped or retrained, so versioning and re-indexing pipelines are a shipping concern, not a nice-to-have. Poorly sanitized source text can also let adversarial or poisoned documents surface in retrieval, making embedding hygiene a quiet security surface.

Embedding vs. alternatives

Related terms