Claude, here is the bug, fix it. This is the new log output, fix the error. Fix the bug. Try a different approach. Reimplement the tests you modified. The bug is still happening, fix it. Fix the error.
We're out of credits, create a new account. We've been API rate limited? When did that start happening? When are we going to get access again?
Wow >31h I am surprised they couldnt rebuild their entire systems in parallel on new infra in that time. Can be hard if data loss is invokved tho (a guess). Would love to see the post mortem so we all can learn.
Hugs to the ones dealing with this and the users of Webflow who invested in them for their clientele. Hoping they'll release a full postmortem once the sky clears up.
More likely that their core database hit some scaling limit and fell over. Their status page talks constantly about them working with their "upstream database provider" (presumably AWS) to find a fix.
My guess. They use AWS hosted Postgresql and autovacuuming fell permanently behind without them noticing, and can't keep up with organic growth, and they can't scale vertically because they already maxed that out before. So they have to do crash migrations of data off their core DB which is why it's taking so long.
Feature factories falling apart always remind me of the stories doctors tell about patients showing up with symptoms that the doctor could have done something about two years ago. Like diabetes so far progressed that amputation is now on the list of possibilities.
Nobody asks for help when the help can still be productive. It's always the death bed conversion.
I have no clue of "webflow" purpose based on it's marketing/buzzword filled landing page, but seems it's just a "no code" abstraction on top of HTML/CSS?
yet another SaaS that really does not need to be online 24/7. It could have been a simple app where you could "no code" on local machine and async state with webflow servers.
It's painful to use, but lets non-technical clients edit copy and create content in a safe environment. There's a runtime CMS types creator and a WYSIWYG html editor with facility for code blocks from global to inline scopes. Also comes with batteries included deploy. It's basically a one or two levels higher Squarespace/Wix
Hugops to the people working on this for the last 31+ hours.
Running incidents of this significance is hard, draining and requires a lot of effort, this going on for so long must be very difficult for all involved.
An outage of this magnitude is almost ALWAYS the direct and immediate fault of senior leaderships priorities and focus. Pushing too hard in some areas, not listening to engineers on needed maintenance tasks etc.
25 comments
[ 6.0 ms ] story [ 70.1 ms ] threadWe're out of credits, create a new account. We've been API rate limited? When did that start happening? When are we going to get access again?
Good luck engineers of the future!
My guess is reason they been down so long is they don’t have good rollback so they attempting to fix forward with limited success.
My guess. They use AWS hosted Postgresql and autovacuuming fell permanently behind without them noticing, and can't keep up with organic growth, and they can't scale vertically because they already maxed that out before. So they have to do crash migrations of data off their core DB which is why it's taking so long.
Nobody asks for help when the help can still be productive. It's always the death bed conversion.
yet another SaaS that really does not need to be online 24/7. It could have been a simple app where you could "no code" on local machine and async state with webflow servers.
Edit, an outage of this length smells of bad systems architecture...
https://www.reddit.com/r/webflow/comments/1mcmxco/comment/n5...