HN Explorer

Netlify or CloudFlare just replaced our production CSS file with “hello”

10 points by frabjoused ↗ HN
We have a few hundred sites serving their nav menus using a shared CSS file located in a private Github repo. This file is hosted via Netlify with CloudFlare in front.

This afternoon we started receiving a number of reports of broken nav styling on the US west coast. On loading these sites, the minified CSS file was replaced with the word "hello".

No commits have been made to this repo since May. I checked the commit history of the minified file and "hello" was never written by accident at any point. Clearing out the CF cache addressed this.

Thoughts?

14 comments

[ 0.26 ms ] story [ 43.8 ms ] thread
blinded
Open a ticket. Could be a bug on their end or an issue on yours.
[deleted]
(comment deleted)
indigomm
Not cool to post with a title slamming Cloudflare when you admit at the bottom of the post it may not be them. Until you are sure where the fault is, best not to start making accusations.
notlify
I'm having this same issue and have a ticket open with cloudflare enterprise support -- started this morning just after cloudflare performed maintenance on their DNS. No response from ENT support yet.

Opening a ticket with netlify too. for now i've simply put a cloudflare page rule up to bypass cache.

frabjoused
Interesting. The fact that we both have CF/Netlify as a stack seems to point more to Netlify and CF is just caching it.

We similarly bypassed the CF cache for now and opened tickets with both.

notlify
Agreed, I can't seem to reproduce with netlify directly though, have a small script checking it now. Will report back if i hear anything
notlify
Got confirmation from cloudflare support that the `hello` response originated from netlify. netlify support predictably blamed cloudflare without any proof.
dknecht
Can you send me your ticket? dane AT cloudflare
jb72123
We had the same issue today. Isolated several variables and confirmed it only happens when caching Netlify sites through Cloudflare. It replaced not only our CSS file, but all cached assets included our homepage, images, videos, .js files... Took down our entire site and the only way to fix was completely disabling caching through Cloudflare. Seems like to me that it is a Netlify issue that is responding incorrectly specifically to CF's servers, but it's also possible it's a Cloudflare issue. We literally just moved our entire site over to a backup server temporarily until we can get it figured out.
frabjoused
Rough. Make sure you file with Netlify, hopefully they see our reports and address. Seems removing CF cache is the universal solution for now.
frabjoused
From Netlify: Hey everyone, this was definitely an issue on our side. We were testing an internal change to our routing mechanism and returned a testing message for some of the requests. Unfortunately, in a subset of CDN locations, for customers on our regular network (not HP Edge) that have a proxy in front of Netlify, those test responses were routed through their proxy. That meant that these test responses were cached there and served instead of the corresponding asset.