AI API INTEGRATION / LLM INTEGRATION
PROPELOO engineers AI API integrations — from OpenAI, Anthropic and Google to open-source model APIs — with streaming, fallback routing, cost controls, rate limit handling, structured output validation and the monitoring that tells you when your AI integration is degrading. Integrating an AI API is one hour. Building a production AI API integration is weeks.
AI API production failures are different from standard API failures. They are not just downtime and rate limits — they are hallucinations that reach users, latency spikes when the provider's servers are overloaded, silent quality degradations when the provider updates the model, and cost spikes when a single request triggers an unexpectedly long generation. PROPELOO builds AI API integrations with streaming for UX, fallback routing across providers for reliability, structured output validation for correctness, cost controls for budget, and monitoring for quality — because AI APIs require their own category of production engineering.
OpenAI, Anthropic, Google Gemini, Cohere and Groq integrated behind a single interface — route to the best model per task, fall back when a provider is degraded, A/B test between providers.
Server-sent events (SSE) for streaming responses to web/mobile clients. Partial result accumulation, streaming to database, client reconnection handling and streaming through API gateways.
OpenAI function calling / JSON mode, Anthropic tool use, Pydantic validation schemas with retry on validation failure. Transform unstructured LLM output into typed, validated application data.
Per-request token counting, per-user and per-day spend limits, cost attribution by feature/user/team, spend alerts and automatic model downgrade when budget thresholds are approached.
Automatic fallback: if OpenAI is degraded, route to Anthropic. Circuit breaker per provider. Retry with exponential backoff for transient errors. SLA monitoring per provider with alerting.
Centralised LiteLLM proxy or custom gateway — API key management, unified interface across providers, request logging, rate limiting per downstream client, cost allocation.
AI API integration is not different from any other external API integration — except for the hallucination risk, the streaming requirement and the 10-100x cost variance between models.
Applications that call the OpenAI SDK directly throughout the codebase are locked to OpenAI. A thin abstraction layer (LiteLLM or custom) that routes to any provider means: switch providers in one place, A/B test providers without code changes, fall back automatically on degradation. Build the abstraction from day one.
Axiom:
An LLM that generates 500 tokens takes 5-10 seconds. Displayed all at once after generation: perceived as slow. Streamed token-by-token: perceived as fast. Streaming is a 1-2 day implementation decision with significant UX impact.
Axiom:
Structured output validation (Pydantic schemas + retry on failure) catches format errors. LLM-as-judge or RAGAS metrics catch quality degradation when a provider silently updates a model. Both are required in production.
Axiom:
A single prompt that generates 50,000 tokens costs $0.25 at GPT-4o pricing. At 1,000 requests/day: $250/day from one prompt pattern. Per-request token limits, prompt length validation and cost monitoring with alerts are required production safeguards.
Axiom:
Impact: LiteLLM for any application that might ever switch providers or needs multi-provider fallback. Direct SDK only for simple single-provider integrations.
Impact: SSE from backend API route to client for all streaming LLM responses. Never expose provider API keys to client-side code.
Impact: Instructor (Python) for provider-agnostic structured output with Pydantic validation and automatic retry on validation failure. Most reliable approach across providers.
Impact: Layered: per-request token limits (prevent runaway single requests) + per-user daily limits (prevent abuse) + spend alerts (operational visibility).
Impact: Automatic fallback to a different provider (OpenAI → Anthropic) provides real resilience. Same-provider multi-region helps with rate limits but not provider-wide outages.
Impact: Exact match cache for deterministic prompts (classification, extraction with identical inputs). Semantic cache for FAQ-style applications. No cache for creative generation.
Integrate OpenAI/Anthropic into an existing product — streaming API route, cost controls, structured output and quality monitoring.
LiteLLM proxy with provider fallback, cost attribution by team/feature, usage dashboards and automatic model routing by task.
Batch AI processing pipeline — structured extraction from documents, validation, retry on failure, cost-efficient model selection per task complexity.
FastAPI backend for AI features — streaming endpoints, authentication, rate limiting, cost tracking and provider abstraction.
Add AI capabilities to existing application — API integration that does not require rewriting the existing stack, streaming frontend components.
Audit existing AI API usage — identify cost inefficiencies, over-sized models for tasks, missing caching and optimise for 50-70% cost reduction.
Stack: openai (Python/Node), anthropic (Python/Node), google-generativeai, cohere, groq
Stack: LiteLLM, Portkey, Helicone, Custom proxy (FastAPI)
Stack: Instructor (Python), Pydantic v2, OpenAI JSON mode, Zod (TypeScript)
Stack: Server-sent events (SSE), FastAPI StreamingResponse, Vercel AI SDK (Next.js)
Stack: LangSmith, Helicone, Datadog, Custom token counter
Stack: FastAPI, Node.js (Fastify), Redis (cache), PostgreSQL (logs)
Provider API keys server-side only. Never in frontend bundles or environment variables committed to git. AWS Secrets Manager or equivalent. Per-environment separate keys.
OpenAI and Anthropic both offer hard spending limits via the provider dashboard. Set these as a backup to application-level controls. A leaked API key with no spend limit can cost thousands in hours.
Per-user request rate limiting prevents abuse. Without rate limiting, a single determined user can exhaust your AI API budget. Redis-based rate limiting per user/API key.
AI-generated content rendered in HTML must be sanitised — LLMs can generate XSS vectors in certain prompts. Treat all LLM output as untrusted user input before rendering.
User data sent to third-party LLM APIs may violate GDPR/HIPAA. PII detection and redaction before API calls, data processing agreements with providers, self-hosted models for regulated data.
User input included in prompts can override system instructions. Input sanitisation, separate user and system context, and LLM-based injection detection for high-risk applications.
Provider selection, abstraction approach, streaming design, cost model.
Provider SDK integration, abstraction layer, authentication, error handling.
SSE endpoint, client streaming component, reconnection handling.
Schema definition, Instructor/function calling, validation and retry.
Token counting, per-user limits, spend alerts, model routing.
Multi-provider routing, circuit breakers, retry logic.
Cost dashboard, latency tracking, quality monitoring, provider status alerts.
LiteLLM is an open-source proxy that provides a unified OpenAI-compatible API interface to 100+ LLM providers (OpenAI, Anthropic, Google, Cohere, Bedrock, etc.). Your application calls the LiteLLM interface; LiteLLM routes to the configured provider. Benefits: switch providers without code changes, A/B test between models, automatic fallback on provider errors, unified cost tracking.
Automatic fallback: configure primary and secondary providers. If OpenAI returns 5xx errors, route to Anthropic. LiteLLM handles this natively. Circuit breaker: after N consecutive failures from a provider, mark it as degraded for T minutes and route all traffic to the fallback. Monitor provider status pages (OpenAI, Anthropic both have status pages) and alert on degradation.
Four layers: (1) Per-request max tokens (cap individual response length). (2) Per-user daily token budget with Redis counter. (3) Use smaller/cheaper models for simple tasks (GPT-4o-mini for classification, GPT-4o for complex reasoning). (4) Cache responses for identical or semantically similar prompts. Together these can reduce costs by 50-80% vs unoptimised usage.