#batching

Inference Cost Optimization: Autoscaling, Batching, Spot

Inference cost is dominated by idle capacity and underused accelerators, not by the per-request price. Autoscaling on the right metric, dynamic batching