Search Ranking

Source: 30-System-Design/Framework & Systems

Prompt

• Rank documents for a query with relevance, freshness, and diversity constraints.

Requirements

• Functional: intent understanding, relevance, deduplication.
• Non-functional: low latency, high availability, safety.
• Constraints: tail latency, multilingual queries, fresh indexing.

Success Metrics

• Offline: NDCG@K, MRR, precision@K.
• Online: CTR, reformulation rate, session success rate.
• Guardrails: safe-search violations, latency, null results.

Data

• Sources: query logs, clicks, dwell time, doc metadata.
• Labeling: implicit feedback, human judgments.
• Position bias: log rank positions for debiasing.

Modeling

• Retrieval: lexical (BM25) + semantic (embeddings).
• Ranking: LTR model or transformer re-ranker.
• Re-ranking: diversity, freshness, policy filters.
• Query understanding: spell correction, intent classification.

Serving

• Retrieval: inverted index + ANN vector search.
• Latency budget: retrieval + ranker + re-ranker.
• Caching: popular queries, query suggestions.
• Fallback: lexical-only when vectors unavailable.

Evaluation & Monitoring

• Offline eval: counterfactual evaluation, leakage checks.
• Online eval: A/B tests + guardrails.
• Drift/abuse: query shift, spam detection.
• Monitoring: freshness lag and coverage of new documents.

Risks & Tradeoffs

• Relevance vs freshness.
• Speed vs model complexity.
• Safety filters vs coverage.

Notes