Model Health
What Healthy, Degraded, and Down mean for a Managed Inference endpoint, and how CosmicAC measures them.
Alongside its status, a Managed Inference Job reports its model health, which measures how well its replicas respond to inference requests. CosmicAC determines each replica's health from health probes and recent request success, then combines the replica values into a single health value for the endpoint.
- Healthy: every replica is passing its health checks and serving requests successfully. A replica awaiting its first probe also counts as healthy.
- Degraded: some replicas are unhealthy, but not every replica is down. A replica becomes degraded when it starts failing its health checks, or when its recent success rate falls below the configured threshold even though its health check passes. The endpoint stays up and serves on its healthy replicas.
- Down: every replica is down, or the endpoint has no registered replicas. A replica becomes down when it keeps failing its health checks for longer than the configured threshold, or when its recent success rate reaches zero.
CosmicAC reads both thresholds from the cosmicac-proxy-inference service's configuration. To change them, see Model health settings.
How CosmicAC measures it
CosmicAC measures these metrics itself. The inference proxy times every request it forwards to the model server, records whether it succeeded, and writes the result to a store on disk. The metrics describe how the model server behaved, but CosmicAC doesn't read them from it, so they survive a restart of either one.
The figures count the health probes that CosmicAC sends, not only the requests your users make. An endpoint that serves no user traffic still records traffic.
How it differs from job status
Model health is separate from the job's status. A job can be Running while its endpoint reports Degraded, because the job is up but one replica serves poorly.
Next steps
- Check Managed Inference Endpoint Health to read these values for your endpoints
- Replicas for what CosmicAC is rating
- Model health settings to change the thresholds
- CosmicAC Jobs for the job status values