Tag
#inference
3 posts tagged inference.
- ops
Inference Cost Optimization: Autoscaling, Batching, Spot
Inference cost is dominated by idle capacity and underused accelerators, not by the per-request price. Autoscaling on the right metric, dynamic batching
- reviews
Model Serving Compared: SageMaker, Vertex AI, Databricks
All three managed platforms will serve a model behind an endpoint. The differences that matter show up in autoscaling behavior, multi-model density, and
- ops
Online Inference Latency: Where the Budget Actually Goes
P99 latency is a product problem as much as an engineering one. Breaking down the inference budget — model compute, preprocessing, retrieval