pgvector Distance Functions: Cosine vs L2 vs Inner Product

pgvector's three distance operators look interchangeable at first glance. They all take two vectors, return a number, and work with ORDER BY ... LIMIT. But they measure fundamentally different things, and using the wrong one means your similarity search returns subtly wrong results -- no error, no warning, just worse rankings that quietly degrade your application's quality. Worse still, the most expensive mistake isn't picking the wrong distance function. It's building an index with one operator class while your queries use a different operator, silently forcing sequential scans on every search.

What Each Distance Function Does

<=> Cosine distance measures the angle between two vectors, ignoring magnitude completely. Two vectors pointing in the same direction have cosine distance 0, whether they're length 1 or length 1000. Range: 0 to 2. This is the default choice for normalized embeddings from OpenAI, Cohere, Voyage, and most modern embedding models.

<-> L2 (Euclidean) distance measures the straight-line distance between two points in vector space. It's sensitive to magnitude -- vectors that point the same direction but differ in length will have a nonzero L2 distance. Range: 0 to infinity. Use this for spatial coordinates, sensor data, or any domain where magnitude carries meaning.

<#> Inner product (negative) returns the negated dot product so that ORDER BY returns highest-similarity results first (smallest value = most similar). Sensitive to both angle and magnitude. Used for maximum inner product search in recommendation systems.

The Operator/Index Mismatch

This is where real performance gets destroyed. pgvector indexes are built with a specific operator class:

• vector_cosine_ops accelerates <=> only
• vector_l2_ops accelerates <-> only
• vector_ip_ops accelerates <#> only

If your index uses vector_cosine_ops but your query uses <->, PostgreSQL silently ignores the index and falls back to a sequential scan. No error. No warning. Just every query computing distances row by row across the entire table.

On a table with a million embeddings, that turns a 5ms indexed lookup into a 30-second full table scan.

Detecting the Problem

Find which operators your queries use and compare with your indexes:

-- Find which distance operators your queries use
SELECT
    query,
    calls,
    mean_exec_time
FROM pg_stat_statements
WHERE query LIKE '%<->%'   -- L2 distance
   OR query LIKE '%<=>%'   -- cosine distance
   OR query LIKE '%<#>%'   -- inner product (negative)
ORDER BY calls DESC
LIMIT 20;

-- Check if index operator class matches query operator
SELECT indexname, indexdef
FROM pg_indexes
WHERE indexdef LIKE '%vector%'
ORDER BY indexname;

If the operator in your queries doesn't match the operator class in your index, you have a mismatch.

Verify with EXPLAIN:

EXPLAIN (ANALYZE, BUFFERS)
SELECT id, embedding <=> '[0.1, 0.2, ...]'::vector AS distance
FROM docs
ORDER BY embedding <=> '[0.1, 0.2, ...]'::vector
LIMIT 10;

An Index Scan means the operator matches. A Seq Scan on a table with a vector index means it doesn't.

The Fix

Match every distance operator with its correct index operator class:

-- Cosine distance (<=>): best for normalized embeddings (most LLMs)
CREATE INDEX idx_cosine ON docs USING hnsw (embedding vector_cosine_ops);
SELECT * FROM docs ORDER BY embedding <=> query_vec LIMIT 10;

-- L2 distance (<->): best for spatial/positional data
CREATE INDEX idx_l2 ON docs USING hnsw (embedding vector_l2_ops);
SELECT * FROM docs ORDER BY embedding <-> query_vec LIMIT 10;

-- Inner product (<#>): best for MaxIP retrieval (note: returns negative)
CREATE INDEX idx_ip ON docs USING hnsw (embedding vector_ip_ops);
SELECT * FROM docs ORDER BY embedding <#> query_vec LIMIT 10;

If you find a mismatch, rebuild the index with the correct operator class:

DROP INDEX idx_wrong_ops;
CREATE INDEX idx_correct_ops ON docs USING hnsw (embedding vector_cosine_ops);

When to Use Each

If you're using embeddings from a modern LLM and aren't sure which to pick, start with cosine distance (<=>). It works correctly regardless of whether vectors are normalized and is the ecosystem standard.

Prevention

1. Standardize on one distance function per vector column. Mixing operators means only one gets index acceleration unless you maintain duplicate indexes.

2. Document the convention. The mismatch usually happens when someone copies a query from a tutorial that uses a different operator than your project.

3. Add EXPLAIN validation to your test suite. Critical similarity queries should assert an index scan in the query plan.

4. Run an operator/index alignment check after every migration:

SELECT
    i.indexname,
    i.indexdef,
    CASE
        WHEN i.indexdef LIKE '%cosine_ops%' THEN '<=>'
        WHEN i.indexdef LIKE '%l2_ops%' THEN '<->'
        WHEN i.indexdef LIKE '%ip_ops%' THEN '<#>'
    END AS expected_operator
FROM pg_indexes i
WHERE i.indexdef LIKE '%vector%';

Compare the expected_operator with what your application actually uses. Any discrepancy means you're doing sequential scans on a table that has a perfectly good index sitting unused.

---

Originally published at mydba.dev/blog/pgvector-distance-functions