Information technology / Business software and AI
Gradient Labs: Outages and recovery while running an AI service
- Company
- Gradient Labs
- Country
- Not classified
- Adoption stage
- In operation
- Source published
- Date basis
- The date the source was published. It can differ from the date adoption started.
- How the source was checked
- Read the full source text
The work problem
Gradient Labs' conversational AI containers repeatedly restarted as memory grew. Engineers investigated alerts to avoid interrupting customer conversations; temporary memory increases did not resolve the underlying issue.
Technology and data
The Go agent runs on Cloud Run, with Temporal managing conversation state and response tasks. Engineers used Google Cloud Profiler to trace memory growth to the workflow-history cache and tuned its size. When latency subsequently increased, they investigated task waits and manually raised the minimum instance count.
Results
The company says cache tuning resolved memory problems and minimum-instance adjustments restored latency after scaling down. Identifying and fixing the second problem took under one hour.
Limits and open questions
People diagnosed and repaired the service; AI did not autonomously resolve incidents. Long-term availability figures are undisclosed.
Sources
- Gradient Labs | AI 서비스 운영 장애와 복구blog.gradient-labs.ai, Accessed
- Anatomy of an AI agent incidentGradient Labs, Accessed
Compiled from public sources. These are not results from ATF Works customers.