Leveraging RunPod for Scalable, Cost-Effective AI Inference

We adopted RunPod as the GPU compute layer, using their on-demand and spot GPU instances to run AI inference workloads at a fraction of traditional cloud GPU costs, with a warm-instance architecture to minimize cold starts.

Architecture

• Compute: RunPod GPU pods for inference workloads, with GPU tier selected per workload
• Orchestration: FastAPI orchestrator on primary cloud managing RunPod pods
• Networking: Secure tunnels between primary infrastructure and RunPod instances
• Model Storage: Pre-built Docker images with models baked in for fast startup
• Monitoring: Health checks and auto-restart for pod availability

Infrastructure Design

Pod Configuration

• GPU Selection: Cost-effective GPU tiers selected per workload, achieving ~85-90% cost savings vs. equivalent major cloud provider GPU instances
• Docker Templates: Custom containers with pre-loaded AI models for inference
• Persistent Storage: Network volumes for model weights and configuration files
• Environment Variables: Dynamic configuration for stream endpoints, API keys, and feature flags

Warm Instance Strategy

Instead of cold-starting pods per request, we maintain warm instances during operational hours:

1. Scheduled Scaling — Pods started before peak hours, stopped during off-hours
2. Pre-Loaded Models — Inference engines loaded at container start, ready immediately
3. Health Probes — Orchestrator monitors RunPod pods regularly to verify readiness
4. Auto-Recovery — Unhealthy pods automatically replaced via RunPod API

Cross-Cloud Communication

• Primary Cloud: API servers, databases, recording workers
• GPU Cloud (RunPod): AI inference, object detection, tracking
• Data Flow: Video frames sent from primary cloud to RunPod for inference; detection results returned via WebSocket
• Timestamp Sync: PTS-based synchronization to handle clock skew between clouds

Cost Optimization

RunPod's pricing model delivered significant savings compared to equivalent GPU instances from major cloud providers:

• On-Demand: ~85-90% reduction in hourly GPU compute cost
• Spot Pricing: Additional 50% savings for non-critical batch processing on community cloud
• Scheduled Shutdown: Automated stop/start based on operational hours further reduces costs
• Right-Sizing: Select GPU tier matching actual VRAM needs rather than over-provisioning
• Multi-Pod Distribution: Spread streams across smaller, cheaper GPUs instead of one large instance

Deployment Workflow

1. Build — Docker image with all models, dependencies, and application code
2. Push — Image pushed to container registry
3. Deploy — RunPod API creates pod with specified GPU, image, and volume mounts
4. Configure — Environment variables set for the specific deployment
5. Monitor — Orchestrator verifies pod health and begins routing inference requests
6. Scale — Additional pods launched via API when load increases

Key Features

1. Significant Cost Reduction — 85-90% savings compared to equivalent major cloud GPU instances
2. Pre-Built Containers — Models baked into Docker images for sub-30-second startup
3. API-Driven Scaling — Programmatic pod creation/destruction based on demand
4. Multi-GPU Support — Multiple GPU tiers available depending on workload requirements
5. Spot Instance Fallback — Non-critical workloads run on discounted community cloud
6. Cross-Cloud Architecture — GPU compute decoupled from primary infrastructure