Inference that never fails
Cloudflare is down again, and I just want to rant a bit about how to think about protecting against these scenarios if you are new to AI inference.
Lessons from consumer inference
AI inference (ie: models running in production), has become a foundation for most AI products today. I personally, have been deploying AI models since as early as 2018... back then we used to have AWS outages every few months. My last company was a consumer product that handled thousands of simultaneous conversations for humans interacting with our AI agents (in 2018!).
Here's a talk about how that system worked.
Fast forward a few years, and our company developed PyTorch Lightning for large-scale, fault-tolerant multi-node training across 4k+ GPUs at Facebook AI. Thus when we decided to launch our inference platform in early 2025, we put a heavy emphasis on how fault-tolerant inference should behave when the clouds failed.
We assumed AWS would fail. We assumed Cloudflare would fail. And we assumed they would do so at the same time at some time. We wanted to plan for the worst, and hope for the best (that's the former military part of me that has this healthy paranoia 😬).
Plan for the worst, hope for the best
If/when you run inference in production, you can't afford any failures. Chatbots could stop working. For many products (like vibe coding tools or ChatGPT), the model IS the product... and these products cannot afford a single second of failure.
As the former CTO of a consumer-facing company that deployed AI in 2018, I can tell you that you have to do a LOT of work as an eng org to truly mitigate failure scenarios. We're not talking about the basics of node failures (that's trivial)... we're talking about real failures.
CDN down? great.
AWS down? great.
Both at the same time?... even better 😏
All these are opportunities to show how you can overcome true adversity.
For startups, that's usually onerous and unrealistic. This includes things like multi-cloud deployments, not relying on single CDN providers, etc... At my last startup we did hacky things like have an exact replica of our platform as the staging environment in a different cloud, so that if a cloud provider failed, we would instantly route traffic to our staging env which was on another cloud... this allowed us to have fault-tolerance without spending more money.
Luckily, at Lightning we built for that world and all our customers are now able to realize the benefits of a platform used by over 350,000 developers from everything from development to consumer-level inference.
Be a prepper
There's probably no simpler way to describe the mentality you need to have, but it's like "be a prepper". If you've never heard that term, a quick Google search will make you laugh a bit. But in times like these, you'll be glad you were this paranoid.
My first job out of college was as a founding engineer for a team called Marquee at Goldman Sachs which incubated products like Kensho, Symphony, Simon, and Strategy Studio which I personally worked on.
One of my most vivid memories on the trading floor was walking into Partner's offices and seeing this photo.
This was the Goldman building during hurricane sandy. The only building in Manhattan that had power because of the multiple layers of fault-tolerance.
Needless to say... in a world where AI IS the product for most new companies, you can't afford a single second of downtime.
Build vs Buy
We believe firmly that the best way to have guaranteed uptime is a mix of build + buy. If you only build, then your whole job becomes about maintaining infrastructure instead of improving your model and product. If you only buy, then you have zero control, and it's the problem with blackbox inference providers.
The approach that we've taken at Lightning is to give you the tools and our expertise to help you build the things that are mission critical for you to own (including your own inference engines with LitServe), while we provide the reliable, distributed, fault-tolerant infrastructure so you can go to sleep at night and not wake up to angry users because "Cloudflare is down".
Read more blogs and case studies:
• Why we merged Lightning AI and Voltage Park to build an AI-native cloud
• Lightning AI Launches First GPU Marketplace Uniting NeoClouds and Hyperscalers, Cutting AI Costs by 70%
• View all