AI CLOUD INFRASTRUCTURE / ML PLATFORM
PROPELOO engineers AI cloud infrastructure — from model serving and GPU cluster management through inference optimisation, model registry, feature stores and the MLOps pipeline that automates training, evaluation and deployment. Running an AI model in a Jupyter notebook is not production. Production AI infrastructure is a distinct engineering discipline.
ML model serving has unique infrastructure requirements: GPU memory management, batching to amortise per-request overhead, model warmup to eliminate cold starts, auto-scaling that responds to inference load patterns, and continuous deployment pipelines that can roll back a model version without deploying new application code. These requirements are not met by simply deploying a model in a Docker container. PROPELOO builds AI infrastructure where inference is fast (optimised serving, batching, caching), cheap (right-sized GPU instances, spot instance strategies, model quantisation), reliable (health checks, autoscaling, fallback endpoints) and observable (request latency, GPU utilisation, model performance metrics).
vLLM for LLM serving (10-30x throughput vs naive serving via continuous batching), TorchServe for PyTorch models, NVIDIA Triton for multi-framework serving, FastAPI custom inference server.
Model quantisation (FP16, INT8, INT4 with GPTQ/AWQ), TensorRT compilation for NVIDIA GPUs, ONNX export for cross-platform deployment, KV cache optimisation for LLMs.
MLflow or Weights & Biases for experiment tracking, model registry with versioning, automated evaluation pipelines (accuracy, latency, bias), CI/CD for model deployment with rollback.
EKS with GPU node groups (g4dn, p3, p4), GPU operator for device plugin, KEDA for queue-based autoscaling, spot instances for cost optimisation, Karpenter for fast GPU node provisioning.
Feast or Tecton feature store — offline feature engineering pipeline, online feature serving for low-latency inference, feature versioning and point-in-time correctness for training.
Prediction latency (p50/p95/p99), GPU utilisation and memory, model drift detection (distribution shift), data quality monitoring, business metric correlation.
AI infrastructure is not harder than regular infrastructure — it is differently hard. The unique challenges are GPU management, batching for efficiency, and the model versioning complexity that does not exist for stateless services.
Naive LLM serving (one request at a time) leaves GPU memory largely idle between requests and does not amortise the KV cache overhead across requests. vLLM's continuous batching fills GPU memory with multiple concurrent requests, achieving 10-30x throughput improvement. Any LLM serving infrastructure that is not using continuous batching is leaving GPU capacity on the table.
Axiom:
FP16 inference is ~50% memory savings vs FP32. INT8 is ~75%. INT4 (GPTQ/AWQ) is ~87.5%. A 70B parameter model in FP16 requires ~140GB GPU memory (4x A100 80GB). In INT4: ~35GB (single A100). Quantisation typically introduces 1-3% accuracy degradation. For most production applications, INT8 is the sweet spot — significant memory savings, negligible quality impact.
Axiom:
A GPU instance that was scaled to zero takes 3-5 minutes to start, warmup and serve the first request. For interactive applications, this is unacceptable. Minimum instance count of 1 (never scale to zero), provisioned capacity for baseline load, and scale-out time hidden behind request queuing are the production patterns.
Axiom:
Deploying a new model version should not require a new application deployment. The model registry, model server and application code must be independently versioned and deployable. A model version that degrades accuracy needs to be rolled back in minutes, not hours.
Axiom:
Impact: vLLM for LLM serving — the continuous batching throughput improvement justifies the setup complexity. TorchServe for traditional ML models. FastAPI for simple models where optimised serving is not justified.
Impact: Reserved instances for baseline GPU capacity, spot instances for scale-out. Critical inference endpoints always have at least one reserved instance. Spot for training and batch inference.
Impact: KEDA with queue depth for inference autoscaling — scale GPU workers based on pending request queue length. Responds to demand faster than utilisation-based scaling.
Impact: MLflow for most teams — open source, self-hosted, covers experiment tracking and model registry without per-user cost. W&B for teams that prioritise collaboration features.
Impact: Feast for ML teams that need point-in-time correct features for training and low-latency online serving. No feature store acceptable only for models with no real-time features.
Impact: Kubeflow pipelines for reproducible, orchestrated training workflows. Single GPU for models that fit in memory. Distributed training only when model or data size genuinely requires it.
vLLM-based LLM serving on GPU nodes — OpenAI-compatible API, continuous batching, autoscaling and cost-optimised spot instance strategy.
TorchServe/Triton multi-model serving with versioning, A/B testing, shadow deployment and automated rollback.
End-to-end MLOps with MLflow — training pipeline, experiment tracking, model registry, evaluation gates and automated deployment.
GPU cluster for LLM fine-tuning — LoRA/QLoRA setup, distributed training, checkpoint management and evaluation pipeline.
Audit existing AI infrastructure costs — model quantisation, batching optimisation, spot instance migration and right-sizing. Typically 40-60% cost reduction.
Feast feature store with offline training pipeline, online serving for real-time inference and point-in-time correctness.
Stack: vLLM, TorchServe, NVIDIA Triton, FastAPI (custom), Ray Serve
Stack: TensorRT, ONNX Runtime, GPTQ (quantisation), AWQ (quantisation), Flash Attention
Stack: MLflow, Weights & Biases, DVC (data versioning), Kubeflow Pipelines, ZenML
Stack: EKS with GPU nodes, Karpenter, KEDA (queue autoscaling), NVIDIA GPU Operator, Spot instance configuration
Stack: Feast (feature store), Redis (online features), Apache Spark (offline processing), dbt
Stack: NVIDIA DCGM (GPU metrics), Prometheus + Grafana, Arize Phoenix, Evidently AI (drift)
Model endpoints require authentication — API keys or JWT. Internal model servers behind VPC boundary with no public internet access. GPU nodes in private subnets.
Training data often contains sensitive information. Encryption at rest in S3, access controls per dataset, data lineage tracking and deletion capability for GDPR compliance.
Fine-tuned model weights are valuable intellectual property. Encrypted model storage, access logging for model downloads, and model API serving (never expose weights to clients).
Inference requests may contain sensitive user data. Log at appropriate granularity — metadata yes, PII no. Retention limits. Access controls on inference logs.
ML models and dependencies have supply chain risks. Verify model checksums from Hugging Face, scan Python packages for vulnerabilities, pin dependency versions.
GPU instances have the same security requirements as CPU instances plus: prevent model exfiltration via shared GPU memory, secure model loading process, least-privilege IAM for GPU worker service accounts.
Model serving requirements, GPU sizing, autoscaling strategy, MLOps tooling selection.
EKS GPU node groups, GPU operator, KEDA, Karpenter, spot instance configuration.
vLLM/TorchServe deployment, batching configuration, health checks, OpenAI-compatible API.
Quantisation, TensorRT compilation, latency benchmarking, cost analysis.
MLflow setup, training pipeline, model registry, evaluation gates, deployment automation.
GPU metrics, inference latency, model quality dashboards, drift detection.
Spot instance migration, reserved capacity strategy, quantisation ROI analysis.
vLLM is an LLM inference server that uses PagedAttention — a technique that manages KV cache memory like an OS page table, allowing multiple requests to share GPU memory efficiently. Combined with continuous batching (dynamically grouping in-flight requests), vLLM achieves 10-30x throughput compared to serving one request at a time. It also exposes an OpenAI-compatible API, making migration from the OpenAI API straightforward.
Four main levers: (1) Quantisation (INT8/INT4) reduces memory requirements 2-4x, enabling smaller/cheaper GPU instances. (2) vLLM continuous batching increases GPU utilisation from ~20% to ~80% for LLMs. (3) Spot instances (70-90% cheaper than on-demand) for stateless inference with request retry on interruption. (4) Right-sizing: many teams run inference on oversized GPU instances. A/100 is not always necessary when A10G handles the model.
Model drift occurs when model predictions degrade over time because the input data distribution changes (data drift) or the relationship between inputs and outputs changes (concept drift). Detection: monitor prediction distribution statistics (mean, variance, distribution shape) against training baseline using Evidently AI or Arize Phoenix. Alert when statistical tests indicate significant deviation. For classification models: monitor accuracy against a labelled holdout sample updated monthly.