Labeeb Runbook (On-Call)¶

Single page for on-call engineers to restore ${SERVICE} with clarity, accuracy, and safety.

• Read top-to-bottom on first response.
• Use the checklists; copy/paste commands as-is.
• Escalation paths are at the bottom.

At-a-Glance¶

• Environments: [PROD_BADGE] [STAGE_BADGE] [DEV_BADGE]
• Services: api · ai-box · scraper · search
• Quick Links: Dashboards · Alerts · Runbooks

On-call Overview¶

• First checks → health endpoints + traces
• Then → go to the specific runbook (API/AI-Box/Scraper/Search)
• Finally → consult the incident playbooks

curl -s ${API_URL}/health | jq
curl -s ${SCRAPER_URL}/health | jq
curl -s ${AI_BOX_URL}/health | jq
curl -s ${OS_URL}/_cluster/health | jq '.status'

curl -s -D- ${API_URL}/v1/search?q=test -H "X-Request-ID=oncall-$RANDOM" | sed -n '1,40p'

See also: Operational Contracts

First 5 Minutes — Universal Checklist¶

1. Confirm incident scope (user-facing? ingestion-only? search-only?).

   curl -s ${API_URL}/health | jq
   curl -s ${SCRAPER_URL}/health | jq
   curl -s ${AI_BOX_URL}/health | jq
   

2. Check health endpoints across services.

   docker compose ps
   

3. Tail logs with filters.

   docker compose logs -f --tail=200 ${SERVICE} | grep -i "error"
   

4. Capture context (links placeholders).
   • ${TICKET_URL}
   • ${DASHBOARD_URL}

# Start all services
docker compose up -d

# Stop all services
docker compose down

# Tail logs for a service
docker compose logs -f <service>

# Health check
curl -s ${API_URL}/v1/health

# Authenticated request
curl -s -H "Authorization: Bearer ${INGEST_TOKEN}" -X POST ${API_URL}/v1/ingest/articles -d @payload.json

Service Status & Health Commands¶

api¶

• Health: GET ${API_URL}/health → JSON {"status":"ok"}

  curl -s ${API_URL}/health | jq
  docker compose logs api -n 20
  docker compose restart api
  

• Depends on: PostgreSQL, Redis, OpenSearch

ai-box¶

• Health: GET ${AI_BOX_URL}/health → JSON {"status":"ok"}

  curl -s ${AI_BOX_URL}/health | jq
  docker compose logs ai-box -n 20
  docker compose restart ai-box
  

• Depends on: OpenSearch, model cache

scraper¶

• Health: GET ${SCRAPER_URL}/health → JSON {"status":"ok"}

  curl -s ${SCRAPER_URL}/health | jq
  docker compose logs scraper -n 20
  docker compose restart scraper
  

• Depends on: api

search¶

• Health: GET ${OS_URL}/_cluster/health → field status

  curl -s ${OS_URL}/_cluster/health | jq '.status'
  docker compose logs search -n 20
  docker compose restart search
  

• Depends on: disk, JVM heap

Common Runbooks (Pointers)¶

• services/api/runbook.md — for API 5xx or GraphQL failures.
• services/ai-box/runbook.md — for S1/S2 or retrieval errors.
• services/scraper/runbook.md — for ingestion stalls or 429s.
• services/search/troubleshooting.md — for query latency or index issues.

Safety Rails¶

• Purge indices in production without a recent snapshot.
• Reindex with stale mappings.
• Roll ai-box without a warm model cache.

• Verify snapshots.
• Verify .env and compose overrides for the target environment.

Quick Probes (Copy/Paste)¶

curl -s -o /dev/null -w "%{time_total}\n" ${API_URL}/health
curl -s ${API_URL}/v1/search?q=test | jq '.hits | length'
curl -s ${API_URL}/metrics | jq '.http_requests'

curl -s ${AI_BOX_URL}/health | jq
curl -s ${AI_BOX_URL}/s1/check-worthiness -d '{"text":"test"}' | jq '.score'
curl -s ${AI_BOX_URL}/metrics | jq '.inflight'

curl -s ${SCRAPER_URL}/health | jq
curl -s ${SCRAPER_URL}/profiles | jq '. | length'
docker compose logs scraper -n 20

curl -s ${OS_URL}/_cluster/health | jq '.status'
curl -s ${OS_URL}/${INDEX}/_count | jq '.count'
curl -s -H 'Content-Type: application/json' ${OS_URL}/${INDEX}/_search -d '{"query":{"match_all":{}}}' | jq '.hits.total'

Triage Matrix¶

Minimal Architecture (Orientation)¶

flowchart TD
    FE[FE] --> API
    API --> Queue
    Queue --> AIB[AI-Box]
    AIB --> OS[OpenSearch]
    SCR[Scraper] --> API

Escalation¶

• Contacts: ${PRIMARY_ONCALL}, ${BACKUP_ONCALL}, ${ENG_MANAGER}
• Handover checklist:
  • Update incident channel.
  • Link dashboards and logs.
  • Transfer open actions.

Appendices¶

• services/api/runbook.md
• services/ai-box/runbook.md
• services/scraper/runbook.md
• services/search/troubleshooting.md
• architecture/platform.md

Last updated: ${DATE} · Version: ${DOCS_VERSION}