GENERATIVE AI / LLM ENGINEERING
PROPELOO engineers generative AI applications — from LLM-powered content pipelines and document generation through multi-agent systems, code generation tools and the evaluation infrastructure that separates AI that works reliably from AI that works most of the time. Generative AI in production is an engineering problem, not a prompt engineering problem.
The gap between a generative AI demo and a production generative AI system is an evaluation problem. How do you know when the output is correct? How do you know when it has drifted? How do you catch hallucinations before they reach users? How do you measure quality across thousands of daily outputs? These are engineering questions that require evaluation frameworks, output validation pipelines, human review queues and monitoring infrastructure. PROPELOO builds generative AI systems with production-grade evaluation built in from the start — because a system you cannot measure is a system you cannot trust.
Generative AI in production requires a generation pipeline, an evaluation framework, a human review layer and a monitoring system.
Automated content generation at scale — blog posts, product descriptions, reports, emails, code documentation. Template-controlled generation with brand voice enforcement, factual grounding and human review for high-stakes outputs.
Multi-step AI agents that plan, use tools (web search, code execution, database queries, API calls) and iterate toward a goal. ReAct and plan-and-execute patterns. Agent memory and context management across multi-turn tasks.
LLM-powered data extraction and transformation — extract structured JSON from unstructured text, classify documents, generate SQL from natural language, transform data between formats. Pydantic validation of LLM output.
Automated code generation for boilerplate, test writing, documentation and refactoring. Code review agents that check for security issues, style violations and logical errors. Integration with existing CI/CD pipelines.
Applications combining text, image and document understanding — document OCR and extraction with GPT-4 Vision, image description for accessibility, video transcript processing and mixed-media content generation.
RAGAS for RAG system evaluation, LLM-as-judge for open-ended output quality scoring, regression testing suites for prompt changes, human evaluation workflows and production quality monitoring dashboards.
Generative AI is non-deterministic. You cannot unit test it. You can only evaluate it statistically — and that requires designing evaluation infrastructure before deploying to production.
Defining what good output looks like, building a test set of representative inputs with expected outputs, and measuring quality before launch is not optional for production generative AI. Without evaluation: you do not know if a prompt change made things better or worse, you cannot detect model drift when the LLM provider updates, and you cannot measure the impact of your improvements. LangSmith, Arize Phoenix and RAGAS provide the evaluation infrastructure.
Axiom:
Asking an LLM to return JSON with specific fields is more reliable than parsing JSON from prose output. OpenAI function calling and JSON mode, Anthropic tool use and Pydantic validation schemas constrain the output format and make parsing deterministic. Downstream systems that depend on LLM output should receive validated, typed data — not raw text to parse.
Axiom:
Not every generated output should reach users without review. High-stakes outputs (legal documents, medical information, financial advice) should have human review queues. Lower-stakes outputs can have sampling-based review. Build the human review workflow into the system architecture — not as an afterthought for when things go wrong.
Axiom:
GPT-4o at $5/1M tokens is cheap per query and expensive at scale. 1M daily queries = $5,000/day in LLM costs alone. Cost optimisation: use smaller, cheaper models (GPT-4o-mini, Claude Haiku) for tasks that do not require full capability, cache deterministic outputs, batch non-urgent requests, and set per-user rate limits. Cost monitoring with alerts is required infrastructure for production generative AI.
Axiom:
Each choice affects quality, cost, latency and reliability.
Impact: Match model to task. GPT-4o-mini for classification, extraction and simple generation. GPT-4o / Claude 3.5 for complex reasoning, code generation and nuanced content. Self-hosted for privacy requirements or very high volume.
Impact: Streaming improves perceived UX for interactive applications. Batch is simpler for document generation workflows. Choose based on whether the user is waiting for the response in real time.
Impact: Semantic caching for FAQ-style applications can reduce LLM API costs by 30-60%. For creative or personalised generation, caching is inappropriate. Exact match caching is valuable for deterministic queries like classification.
Impact: Pydantic + function calling / JSON mode for any application that depends on structured LLM output. LLM-as-judge for quality validation of open-ended text generation. Never rely on LLM output format without validation in production.
Impact: Start with the simplest approach that solves the problem. Multi-agent systems are powerful but add orchestration complexity, cost and failure modes. ReAct agents for tasks requiring tool use and iteration. Pipelines for well-defined multi-step workflows.
Impact: LLM-as-judge scoring for open-ended generation quality is the most scalable approach with acceptable accuracy. RAGAS for RAG systems. Always validate automated evaluation against human judgment on a calibration set before trusting it for regression detection.
Automated blog post, product description and marketing copy generation with brand voice enforcement, factual grounding and human review workflow for high-stakes content.
Extract structured data from unstructured documents — contracts, invoices, medical records, financial statements — with Pydantic validation and confidence scoring.
Code generation, review and documentation tool integrated into developer workflow — PR review automation, test generation, documentation writing and boilerplate code generation.
Autonomous research agent that searches the web, reads documents, synthesises findings and produces structured reports with citations — with human review for accuracy-critical outputs.
Personalised customer communication generation at scale — support responses, follow-up emails, offer recommendations — with brand voice, tone matching and human review sampling.
LLM-powered ETL pipeline that understands messy, unstructured data sources and transforms them into clean, structured database records with validation and confidence scoring.
Model APIs, orchestration, evaluation and infrastructure each require specific tooling.
Stack: OpenAI GPT-4o/mini, Anthropic Claude 3.5, Google Gemini 1.5, Groq (fast inference), Together AI (open source)
Stack: LangChain, LlamaIndex, LangGraph (agents), Haystack, Custom Python pipeline
Stack: Pydantic v2, Instructor (structured output), Guardrails AI, OpenAI function calling
Stack: LangSmith, RAGAS, Arize Phoenix, PromptFoo, Custom eval suite
Stack: GPTCache, Redis (semantic cache), LangSmith (cost tracking), Helicone
Stack: FastAPI, Celery (async jobs), AWS Lambda, PostgreSQL, Redis
LLM applications introduce security risks that traditional application security does not cover.
Users can craft inputs that override system instructions. For agentic systems with tool access, prompt injection can trigger unauthorised tool calls, data exfiltration or privilege escalation.
Multi-tenant systems where user inputs are included in prompts alongside other users' data can leak information across tenants. Strict context isolation and per-user data sandboxing in the prompt context are required.
User inputs containing PII sent to third-party LLM APIs may violate GDPR, HIPAA or CCPA. PII detection and redaction before LLM API calls, data processing agreements with LLM providers and self-hosted models for regulated data are mitigation options.
LLM API keys with no spend limits or rate limiting will be exploited if leaked. Server-side key management, per-user rate limiting and spend alerts are required for production systems.
Generated content must be sanitised before rendering in web interfaces — LLM-generated HTML or JavaScript can contain XSS vectors if rendered unsafely. Treat all LLM output as untrusted user input.
Single-provider LLM dependency means provider outages take down your product. Multi-provider fallback (OpenAI → Anthropic → self-hosted) with automatic failover is required for production systems with uptime SLAs.
Define specific generation task, quality criteria, acceptable failure modes and evaluation metrics before any implementation.
LLM selection, prompt strategy, output format, caching approach and evaluation framework design.
Generation pipeline, structured output validation, error handling and retry logic.
Test set construction, LLM-as-judge setup, quality baselines and regression test suite.
Review queue for high-stakes outputs, sampling strategy for quality monitoring, feedback collection.
API development, streaming, rate limiting, cost controls and monitoring dashboard.
Quality drift detection, cost monitoring, model performance comparison and iterative improvement.
Traditional metrics (BLEU, ROUGE) correlate poorly with human judgment for LLM outputs. LLM-as-judge — using GPT-4 or Claude to score outputs against defined criteria — correlates well with human evaluation and scales to thousands of outputs per day. RAGAS provides specialised metrics for RAG systems. Always validate your evaluation methodology against human judgments on a calibration set before using it for production monitoring.
RAG grounding is the primary mechanism — constrain generation to retrieved, verified content. Structured output with source citation. Temperature close to 0 for factual tasks. "I don't know" prompting for low-confidence responses. LLM-as-judge hallucination detection on outputs before delivery. No mechanism eliminates hallucination entirely — monitoring and human review for high-stakes outputs remain necessary.
An AI agent is an LLM that can use tools (search, code execution, database queries, API calls) and iterate over multiple steps to complete a task. Use agents when: the task requires information that is not in the context window (needs search), requires computation (needs code interpreter), or requires multiple sub-tasks that depend on each other. Do not use agents when a single well-designed prompt or RAG call can solve the problem — agents add complexity, latency and failure modes.
At 10,000 daily queries with average 2,000 tokens per query: GPT-4o costs ~$100/day, Claude 3.5 Sonnet ~$90/day, GPT-4o-mini ~$10/day. Self-hosted Llama 3 70B on AWS: ~$50/day for a single A100 instance. For most applications, GPT-4o-mini or Claude Haiku is 80% of GPT-4o quality at 10% of the cost. The right architecture uses tiered models — complex tasks get the expensive model, simple classification and extraction get the cheap model.