API Service: Standard Operating Procedures¶

This document provides detailed, step-by-step checklists for routine operational and maintenance tasks related to the API service. These procedures focus on the three core processing patterns: ingestion, search, and chat completion.

1. Request Flow Monitoring & Debugging¶

Understanding the request flow is essential for effective troubleshooting. The API service implements three distinct processing patterns.

1.1. Ingestion Request Flow Debugging¶

Pattern: Synchronous validation → Asynchronous processing

flowchart LR
    A[POST /v1/ingest] --> B[Middleware]
    B --> C[Validation]
    C --> D[Idempotency Check]
    D --> E[Queue Job]
    E --> F[202 Response]
    E --> G[Background Processing]

Debug Steps: 1. Check request validation:

docker compose exec api php artisan tinker
# Check recent validation errors

1. Verify idempotency cache:

   docker compose exec redis redis-cli
   KEYS "idempotency:*"
   

2. Monitor queue jobs:

   docker compose exec api php artisan horizon:status
   

1.2. Search Request Flow Debugging¶

Pattern: Request → Multi-service coordination → Response merge

flowchart LR
    A[GET /v1/search] --> B[Mode Detection]
    B --> C[BM25 Query]
    B --> D[Vector Query]
    C --> E[Result Fusion]
    D --> E
    E --> F[Response]

Debug Steps: 1. Test search modes individually:

curl "http://localhost/v1/search?query=test&mode=bm25"
curl "http://localhost/v1/search?query=test&mode=vector"

1. Check AI-Box connectivity:

   docker compose exec api php artisan tinker
   # Test HybridAiClient connection
   

1.3. Chat Stream Flow Debugging¶

Pattern: Request → Stream initiation → Server-sent events

flowchart LR
    A[POST /chat/completions] --> B[Request Validation]
    B --> C[AI Service Call]
    C --> D[Stream Response]
    D --> E[SSE Format]

Debug Steps: 1. Test streaming endpoint:

curl -N -H "Content-Type: application/json" \
     -d '{"messages":[{"role":"user","content":"test"}]}' \
     http://localhost/chat/completions

1. Monitor stream events: Look for data: {"event":"delta"} patterns in response

2. Service Integration Health Checks¶

Objective: Monitor the health of service integrations and data flow patterns.

2.1. Dependency Health Matrix¶

2.2. Request Pipeline Validation¶

Ingestion Pipeline Test:

# Test complete ingestion flow
curl -X POST http://localhost/v1/ingest/articles \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: test-$(date +%s)" \
  -d '{"articles":[{"source_id":"test","external_id":"test-1","url":"http://test.com","title":"Test Article","content":"Test content","lang":"en","published_at":"2025-01-15T10:30:00Z"}]}'

Search Pipeline Test:

# Test hybrid search flow
curl "http://localhost/v1/search?query=test&mode=hybrid&size=5"

Chat Pipeline Test:

# Test streaming chat flow
curl -N -X POST http://localhost/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Hello"}]}'

3. Queue & Background Processing Management¶

These procedures manage the Laravel Horizon queue system that handles asynchronous processing.

3.1. Queue Architecture Overview¶

flowchart TD
    A[API Request] --> B[Immediate Response]
    A --> C[Queue Job]
    C --> D[Horizon Worker]
    D --> E[AI-Box Processing]
    D --> F[Search Indexing]
    D --> G[Data Enrichment]

3.2. Checking Queue Status¶

Objective: To get a real-time snapshot of the queue workers, recent jobs, and any failed jobs.

1. Execute the horizon:status command:

   docker compose exec api php artisan horizon:status
   

2. Analyze Output:
   • Healthy: Horizon is running. and processes count is greater than zero.
   • Unhealthy: Horizon is inactive.

3.3. Viewing Failed Jobs¶

Objective: To list all jobs that have failed and are currently stored in the failed_jobs table.

1. Execute the horizon:failed command:

   docker compose exec api php artisan horizon:failed
   

2. Get Specific Job Details: To see the full exception and stack trace for a single failed job, use its ID from the list.

   docker compose exec api php artisan horizon:failed [FAILED_JOB_ID]
   

3.4. Clearing Failed Jobs¶

Objective: To remove failed jobs from the queue after the root cause has been resolved.

• To forget a single failed job:

  docker compose exec api php artisan horizon:forget [FAILED_JOB_ID]
  

• To forget ALL failed jobs:

  docker compose exec api php artisan horizon:forget-failed
  

4. Cache Management & Performance¶

Objective: Manage caching layers for optimal performance across the three core patterns.

4.1. Cache Layer Architecture¶

flowchart TD
    A[Request] --> B[Application Cache]
    A --> C[Redis Cache]
    A --> D[OpenSearch Cache]
    B --> E[Config/Route Cache]
    C --> F[Idempotency Cache]
    C --> G[Session Cache]
    D --> H[Query Results]

4.2. Application Cache Operations¶

• Clear Application Cache:

  docker compose exec api php artisan cache:clear
  

• Clear and Cache Configuration: This removes the old config cache and creates a new one. Required after any .env change.

  docker compose exec api php artisan config:cache
  

• Clear and Cache Routes: This removes the old route cache and creates a new one. Required after any change to routes/*.php files.

  docker compose exec api php artisan route:cache
  

4.3. Idempotency Cache Management¶

Check idempotency keys:

docker compose exec redis redis-cli
KEYS "idempotency:*"
TTL "idempotency:specific-key"

Clear specific idempotency key:

docker compose exec redis redis-cli DEL "idempotency:specific-key"

5. Data Flow Debugging¶

Objective: Debug data flow through the service architecture layers.

5.1. Request-Response Pattern Analysis¶

Ingestion Data Flow:

# Check article processing status
docker compose exec api php artisan tinker
# Monitor recent articles
App\Models\Article::orderBy('created_at', 'desc')->limit(5)->get();

Search Data Flow:

# Test OpenSearch connectivity
curl http://localhost:9200/_cat/indices
# Check indexed documents
curl "http://localhost:9200/articles/_search?size=1"

Chat Data Flow:

# Test AI-Box connectivity
curl http://ai-box:8000/health
# Check streaming response format
curl -N http://localhost/chat/completions -d '{"messages":[{"role":"user","content":"test"}]}'

5.2. Using the Tinker Shell for Data Analysis¶

Tinker provides an interactive REPL that bootstraps the entire Laravel application, allowing you to use your Eloquent models directly.

1. Open the Tinker Shell:

   docker compose exec api php artisan tinker
   

2. Execute Eloquent Queries: You can now run any PHP code or Eloquent query.

   // Example: Find an article by its ID
   App\Models\Article::find(123);
   
   // Example: Count all articles from a specific source
   App\Models\Source::where('key', 'aljazeera')->first()->articles()->count();
   

5.3. Direct Database Analysis¶

For raw SQL queries, you can connect directly to the database using the psql client.

1. Open a shell in the API container:

   docker compose exec api bash
   

2. Connect to the database: The password will be requested interactively; it is stored in the .env file.

   psql -h db -U app -d appdb
   

3. Execute SQL Queries:

   SELECT * FROM articles WHERE id = 123;