Scaling The Everything Quiz: Semantic Caching with pgvector

Conversion Economics vs Token Costs

The use of AI is capped by the profit associated with its use: If the token costs for your use case are greater than your revenue, you are only subsidizing AI companies. Truly scaling with AI needs keeping sight of the cost vs the gain. As a product use case, The Everything Quiz, our online quiz game that can generate trivia questions on any topic, backed by LLM generation, is a great example in action.

The Cost: Two-Phase Generative Pipeline

To generate accurate and engaging trivia, you need to separate the pipeline. One prompt focuses solely on a true and engaging question and answer. And like any good test writer, afterwards we generate plausible distractors. We use gemini-3.1-flash-lite for stronger accuracy, and a cheaper gemini-2.5-flash-lite for the distractors. This is however, still costly, in fractions of a cent, that are still a magnitude over what ad impressions would pay for that same users playthrough, effectively losing us money with each game. Furthermore, the two-phase generation introduces a hard 15-second latency floor. While this window can technically be masked by an ad break, it fundamentally disrupts the core multiplayer engagement loop.

So, we had to limit the “Everything” aspect of our quiz to only paid users. And free users were relegated to basically random questions, sorted by upvotes, but still… far from the “Everything” promise. Because we persist every generated question, we built a massive, unstructured data cache. The architectural mandate became clear: intercept user queries with a write-through cache to eliminate redundant LLM compute entirely.

graph TD
    User([Player Submits Topic]) --> API[Node.js Server]
    
    API --> ModeCheck{"Game Mode?"}
    
    %% Paid Flow
    ModeCheck -- "Full Mode (Paid)" --> PaidGen[Inline Generative Pipeline]
    
    %% Free Flow
    ModeCheck -- "Free Mode" --> Embedder[text-multilingual-embedding-002]
    
    subgraph SemanticRoutingLayer [Semantic Routing Layer - Free Mode]
        Embedder --> |"Query Vector"| PG[(Supabase PostgreSQL + pgvector)]
        PG --> |"Cosine Distance <=>"| CacheCheck{"Match > 0.40?"}
    end

    subgraph GenerativePipeline [Generative Pipeline - Gemini]
        PaidGen --> FactGen[Phase 1: gemini-3.1-flash-lite]
        BatchJob[Offline Hydration Worker] --> FactGen
        FactGen --> |"Raw Facts"| DistractorGen[Phase 2: gemini-2.5-flash-lite]
        DistractorGen --> |"Structured Trivia"| SaveDB[(Save to PG)]
    end

    CacheCheck -- No --> Fallback[Serve Random Upvoted Questions]
    CacheCheck -- No --> LogTopic[Log Unfulfilled Topic to DB]
    CacheCheck -- Yes --> ServeCache[Serve Cached Questions]
    
    LogTopic -.-> |"Batch Job Pulls"| BatchJob
    SaveDB --> ServeCache
    ServeCache --> Client([Return to Game Client])
    Fallback --> Client
    SaveDB --> |"Real-time Return"| Client

    style PG fill:#3ecf8e,stroke:#24b47e,stroke-width:2px,color:#fff
    style Embedder fill:#4285f4,stroke:#4285f4,stroke-width:2px,color:#fff
    style FactGen fill:#f4b400,stroke:#f4b400,stroke-width:2px,color:#fff
    style DistractorGen fill:#f4b400,stroke:#f4b400,stroke-width:2px,color:#fff
    style PaidGen fill:#f4b400,stroke:#f4b400,stroke-width:2px,color:#fff

Thousand times cheaper “AI”: Semantic Caching via pgvector

If we try to associate our existing questions with the topic requested by the free user, in a best case effort, we could save the expensive AI calls and still deliver on the player having agency to dictate their quiz’s topic, but how could we know that the questions were relevant to the topic? That’s where the semantic search comes in. If we can calculate a distance (say, a cosine distance) between what the player desires and what we have, this will work.

Embeddings do this. By mapping words into a “vector space”, basically a multidimensional point, from which we now can calculate distances. PostgreSQL enables this via the pgvector extension. For all our cached questions, we add a embedding column. And when a free player requests a topic, for the thousandth of an LLMs cost, we can find the closest matches in our database.

As our quiz supports multiple languages, English, Spanish, Chinese, Japanese, and Korean, we use Google’s multilingual embedding model text-multilingual-embedding-002, which can generate embeddings for text in any of these languages. Embedding with task type RETRIEVAL_DOCUMENT is used for storing the questions, concatenated with their answer and topic to get a complete view of their semantics, meaning. And task type RETRIEVAL_QUERY is used for the player’s requested topic.

As the costs decrease a thousandfold, it is finally economically practical to allow free players to experience the “Everything” in our Quiz.

Vector Search in Practice

To execute this efficiently against our index of 11,050 cached questions, the distance calculation and filtering are handled directly within a PostgreSQL RPC:

BEGIN
  RETURN QUERY
  SELECT * FROM quiz.questions
  WHERE language = input_language
    AND (embedding <=> query_embedding) < (1 - similarity_threshold)
  ORDER BY embedding <=> query_embedding
  LIMIT match_limit;
END;

sequenceDiagram
    actor Player
    participant Server as Node.js Server
    participant EmbedAPI as Google Embed API
    participant DB as Supabase (pgvector)
    participant Batch as Offline Batch Worker

    Player->>Server: Request Topic: "WW2 History"
    
    %% Semantic Cache Phase
    rect rgba(66, 133, 244, 0.15)
        Note right of Server: Semantic Cache Check
        Server->>EmbedAPI: embedContent("Topic: WW2 History")
        EmbedAPI-->>Server: Return [0.012, -0.453, ...] (Vector Array)
        Server->>DB: RPC search_questions(query_vector, threshold: 0.40)
    end

    alt Cache Hit
        DB-->>Server: Return matched questions
    else Cache Miss (or partial hit)
        Note right of Server: Instant Fallback
        DB-->>Server: Return 0 results
        
        Server->>DB: logUnfulfilledTopic("WW2 History")
        Server->>DB: getRandomHighlyUpvotedQuestions()
        DB-->>Server: Return random popular questions
        
        par Asynchronous Backfill
            Batch->>DB: Query unfulfilled topics
            Note over Batch: Generate facts & distractors via Gemini
            Batch->>DB: INSERT new questions with embeddings
        end
    end

    Server-->>Player: Emit Game Event (Start Quiz!)

This drops what used to be a 15-second generative bottleneck to a 297ms database lookup. With a similarity threshold of 0.40, and a hard filter for questions of the same language, we started giving players the “Everything” quiz they always wanted. On cache misses, when we fail to find enough semantically relevant questions for a full round, we fall back to standard random retrieval. However, we now explicitly log those missed user topics to identify semantic gaps in our database. This provides direct telemetry on what our players actually want, allowing us to trigger targeted, automated batch generations to backfill the exact categories we are missing.

You can try the result of all this, now, for free at The Everything Quiz