Terminology· 2 min
Training vs inference
Training builds the model (very expensive, one time). Inference is running it to answer a request (cheap, but constant).
Key idea
Training a frontier model can cost $100M+ and take months on 10k+ GPUs. Inference is what you pay for per API call.
Example
GPT-4-class training cost is estimated at ~$100M. A single ChatGPT reply costs fractions of a cent.
What to remember
- Inference dominates lifetime cost of a deployed model
- Distillation and quantization mostly optimize inference
- Serving = scheduling, batching, KV-cache management