194 comments

[ 1.9 ms ] story [ 289 ms ] thread
one other thing i would suggest is to set up a catch-all email for the domain and see what gets sent to it, sometimes you can access accounts associated with the domain, socials etc
I have an interesting 3-letter.net

I set up a catch-all for personal use and wasn't expecting to get flooded with emails.

I was getting business emails, people trying to send money by Zelle, etc.

I was kind of hoping to get something good that I could take action on in the market, so I left it on for a little bit, but then I felt bad that people's emails were not getting answered (at least bouncing), so I turned off the catch-all. Oh well.

I do that and get the occasional account signup. I also ban addresses that fet sent spam, which happens more than the account signups.
I’ll add: and if you lease a VPS, check out its address reputation and reverse DNS record.
How?
Find out the IP address of the machine hosting the domain, then do a reverse lookup on that IP address. It might show the last domain hosted on that IP address.

Using dig:

$>dig yourdomain.tld

1.2.3.4

$>dig -x 1.2.3.4

evilcorp.com

Isn't it pretty safe to just assume that any IP addresses belonging to public clouds, especially cheap ones, have bad reputations?
(comment deleted)
The individual IPs may not all have too bad reputation but you don't control who shares the block with you and don't have any control over new neighbors - and that is enough for some agressive organizations (Microsoft) to block you.
The usual version of this is the popular SEO technique of buying an aged domain with a few backlinks and slapping a wordpress on it.
> Ideally, search engine algorithms would give new domain owners a fresh start.

Sadly, I think this would be instantly gamed by abusers. They would release the domain name and attempt to register as a new owner or start repeatedly doing handoffs. It's difficult to tell who the owner is changing between and whether or not the new one is a better actor than the former.

Google product manager interview question - Write some code with an LLM tool that leverages a LLM to determine if the new owner of a domain is doing (a) same dodgy thing as prior owner that got flagged (b) different dodgy thing as prior owner but should be flagged (c) something completely innocuous (d) needs further review.
Please don't give Google ideas for more ways they can have an algorithm arbitrarily screw you over with no recourse, they're listening.
Follow up interview question. Update the code using your LLM code gen tool of choice that, when someone submits a complaint via an online form, feeds that complaint text back into your LLM to score it again. Points deduction if the candidate ever mentions informing the complainant of anything.
Well, current approach guarantees you’re getting screwed over. Any improvement is beneficial unless it blocks a better approach?
You're looking at this from the perspective of a haunted domain owner. And from that perspective your idea is fine.

A good technique to evaluate ideas though is to try and view it from different perspectives.

In this case from the owner of a non-haunted domain. Can you see any potential problem with your idea when viewed from that perspective?

Now, if there are potential problems, consider the relative sizes of the two groups. Do the benefits to one outweigh harm to the other?

This technique can be used every day with pretty much any idea.

The parents rules seemed to indicate only reevaluating the status of a haunted domain. I see nothing about evaluating a normal domain.
(Therefore, this has a one-way function of improving the status of haunted domains and why I think anything is better than nothing unless it blocks a better strategy.)
Why would they care?
Sadly, the same holds true for IP addresses.
> It's difficult to tell who the owner is changing between and whether or not the new one is a better actor than the former.

This doesn't seem like that hard of a problem to solve, because these are domains with negative reputation, i.e. worse than zero.

So if a) the domain is no longer hosting any of the stuff previously complained about and b) is no longer receiving new complaints over a period of a year, it costs you nothing to reset the domain to zero. Because the bad actors don't have to behave for a year to get back to zero, they can just register a new domain.

All you're doing is giving the new owner the same fresh start that anybody can get by buying a never before registered domain for the same price as a year's renewal on the existing one.

Using a domain every second year in that environment would get it a gradually raising rank where it isn't penalized/sanitized (by accident, on principle, etc) so every restart after a $30 pause year would be much more effective than a new domain.
It gets reset every year so how would it be more effective?
A system gets reset, what happens in obscure places like old HN content?
The search index knows when the first time it saw that old link was. If it was before the reset, regard it as pointing to a different domain than the current one.
Google can take various actions to put pressure but it ultimately doesn't control how the entire world treats archived text.

A google rank at zero and lots of 2 hop routes to your site that google can either penalize for being an accurate historical record or not is better than a rank of zero and a domain that has never been in historical artifacts.

The historical artifacts exist independently of the search ranking. Actual bad guys can get a new domain to get a clean slate without taking the old one down. The reason they care about the cost of domains is their domains get a bad reputation immediately and they have to cycle through far more than one domain a year.

If they were going to consistently use the same domain for links while they churn through hundreds/thousands a year for Google, the extra cost for one extra renewal for the persistent domain would be entirely negligible. And on top of that would make it trivial for Reddit/Facebook/etc. to disable all the historical links because they all go to the same scam site.

How about not even look for a new owner, and just... check the content and complaint levels? If I was hacked and hosted spam, getting blocked/banned for months at a time when... the spam is cleaned and the hole that allowed it is fixed ASAP... that gives folks less incentive to fix/clean/remediate.
3 assumptions that from my read are baked into your comment.

- Any empty domain starts with the same reputation

- Registering a new domain is a 0 cost action

- The eng effort to reset domain reputation is 0

Certain domains are used by abusers more often, usually due to them being cheaper. Forcing them to move domains is extra friction to the abusers which "haunted" domains force more than the proposed new system.

For the last point, I think it's simplifying a complex system change. Even if the new system was marginally better, it could be a large eng effort and not worth pursuing.

edit: styling

> Any empty domain starts with the same reputation

What basis would you have to do otherwise, and if there is something (like TLD), why wouldn't "resetting to zero" in terms of past content just mean resetting to that zero?

> Registering a new domain is a 0 cost action

No, that registering a new domain has a similar cost to renewing an existing domain, which is a valid assumption. In fact, the new domains are often cheaper because registrars often discount the initial registration as a loss leader with the expectation that people will make future renewals at a higher price.

> The eng effort to reset domain reputation is 0

It is the job of the party operating that system to make it operate as correctly as feasible. Needlessly causing collateral damage purely out of laziness and unaccountability is how you get people showing up at government offices demanding for you to be regulated or broken up, if not showing up at your offices with a disposition to cause bodily harm.

> Certain domains are used by abusers more often, usually due to them being cheaper.

Running out of domain names is physically impossible. There are more possible domain names in any given TLD than there are atoms in the observable universe. So the low price is going to be the price set by the registry for that TLD.

Whether the TLD itself has some reputation is orthogonal to the reputation of one domain in that TLD relative to another one in the same TLD. Moreover, you would presumably do the same thing for the TLD -- if one TLD is doing promotion and has $1 registrations this year and then gets used for a lot of scams, and then next year it costs $15 and so do the renewals so the scammers move to a different TLD, the reputation of the TLD should be reset just the same as the individual domains.

> Even if the new system was marginally better, it could be a large eng effort and not worth pursuing.

If the primary goal is to reduce engineering effort then the obvious solution is to delete the entire reputation system so it doesn't have to be maintained anymore. If the primary goal is to make it work well then you have to, well, you know.

> What basis would you have to do otherwise, and if there is something (like TLD), why wouldn't "resetting to zero" in terms of past content just mean resetting to that zero?

Fair enough, but I'm not sure it resolves "haunted" domains as a TLD which is often abused could have a lower "0" reputation and thus by default is "haunted". Perhaps it lessens the impact though by how much is quite opaque to us.

> Whether the TLD itself has some reputation is orthogonal to the reputation of one domain in that TLD relative to another one in the same TLD.

I think this depends on how reputation works and is not so clear. Registrars for these TLD also have a responsibility but have no incentives to stop abusers. If TLD domain reputation is not orthogonal to reputation individual domains on that TLD then that could be an incentive for them to also crack down on abuse as their domains have bad SEO etc.

> If the primary goal is to reduce engineering effort then the obvious solution is to delete the entire reputation system so it doesn't have to be maintained anymore. If the primary goal is to make it work well then you have to, well, you know.

I think this is the most uncharitable interpretation. The eng effort could go to features that improves other customer experiences affecting more people.

If it's instantly released, then yes. But in this thread are reports where the offensive actions happened 15 years ago. After such a long time of "good behavior" it makes no sense for me to still keep the domain blocked/downranked.
Honestly, these days, with domains in general being nearly free compared to the profit potential of a single successful spammer grift, I’m not sure I even see the point of blacklisting domains at all. 25 years ago maybe a spammer would be devastated that he had to “start all over and buy a new domain and build up its reputation.” Now, spammers launch and abandon what, a million new domains a day? Google or anyone spitefully holding onto hard feelings about what a domain “did” years ago is pointless because the spammers will move on anyway. They wouldn’t reuse abcqwertuiop26abc dot xyz anyway because it’s safer to make up a new gibberish domain anyway. Only people who acquire domains legitimately are hurt by this.

I would want to experiment judging them based on what they’ve been seen to do in the past month.

The only reason they go to those new domains is because of the blacklist.

If you remove the blacklist, they’d just stop doing that and it would be even easier for them.

I'm imagining/advocating for blacklisting them for say, 12 months, and re-evaluating them at that point. This imposes the identical cost on the spammer as now (each "detection" costs them a year's domain registration) while allowing a reputation "reset" for innocent people who acquire haunted domains.

Yes, the spammers can sit on their domains once blacklisted, renew them, and redeploy their spam on them 12 months later, but they'd have nothing to gain from the reuse, since the names of their domains are just nonsense anyway.

Fair point.

I’m guessing that would complicate blacklist maintenance quite a bit, which is why we aren’t seeing it work that way.

Most of these blacklists (at least initially) were emergency type measures - ‘block these spammers’, then move on with life.

Blacklist maintainers would need to maintain date first seen/date last seen info, and purge/re-add correctly.

Technically, seems like an ‘append only’ type thing is what they’ve been doing for the most part.

As this evolves and the idea that these do need some kind of expiration or we end up with more maintenance headaches becomes more widely known, maybe eh?

Or if there is some kind of legal rules around it.

Require a deposit then, say 1000$, that is to be refunded after a year of probationary period. You get caught being a scammer/spammer, you lose the deposit.
The deposit would be either too high for normal people to pay, or too low to matter to bad actors
Given that spammers cycle through thousands of domains, they'd run into serious cash flow issues very soon.
Who holds the deposit, and what is to stop them from having someone report your domain as a spammer so they can keep your money?
A tweak to that could be along the lines of "if the DNS lookup of the domain responds with NXDOMAIN for more than x days, give it a fresh start".

I'm not up to date with SEO so unsure whether Google would (or is able to) reset the domain's backlink profile, I'd guess it would be possible. A lot of the value of using expired domains is for backlinks (or at least was)

If it was easy to reset reputation with search engines what's stopping people from saying "under new management" every once in a while for an existing poor reputation domain? Probably better to just cut their losses and find another domain.
For running a mail server every new domain is haunted.
Not really an issue on the same scale because it resolves itself once the domain registration has aged a bit. IP reputation is stickier.
This happened to me and I found this tool super helpful to get my site unblocked: https://dnsblacklist.org/

I purchased a valuable premium domain to host a personal art collection (of anime cels). For some bizarre reason, the site was inaccessible from my work computer and it was de-listed from Google even if I typed the url itself into search.

I hired a square space specialist to figure out why, to no avail. I then begged our company’s CISO to investigate and it turns out we had some firewall setting on UniFi that blocked the domain because it appeared on a list. Once I checked way back, it turns out that it was as an anime porn aggregator years back. I personally reached out to all the web filters out there (Google, Symantec, bing) and one by one filed tickets for them to mark it as art instead of pornography and it worked. I am now properly crawled on Google but still MIA on Bing, search console is giving me some BS error that’s incomprehensible, typical of MSFT.

I'd be somewhat interested in seeing the cels. :)
https://www.neotokyo.com

I have a +100 cel backlog that I need to catalog and photograph. Was planning to do it this holiday season so check back in.

I... actually remember that address floating around and it indeed was hentai.

We're talking like 20 years back. Holy shit, my brain is getting jostled by this sudden tsunami of forgotten memories.

EDIT: Digging around on Wayback Machine (obviously NSFW, for the curious), apparently it was actually still around until somewhere between 2018 and '19 when it finally died. The snapshots from around 2007 are peak Web 1.5 design with stuff like affiliate buttons and table layouts. Man I miss that era.

Great domain name! I can see why you went through the effort of contacting all the web filters.
You have some awesome cells, thanks for sharing them online. Had completely forgotten about Robot Carnival and neat to see you have a few pieces from some of the shorts(episodes?)

Also the resources->galleries was useful, found some new but actually old sites to check out.

I love RC and many of my wishlist items are from it. I regret I was relatively late into collecting it. Glad you appreciate the old galleries, many are internet relics which I love.
Did you get anything from the Heritage auction last week? They had a ton of good stuff.
I watched closely and bid on a few but didn’t pull trigger. I am eyeing a few private pieces and saving my budget.
It is also blocked by the UK ISP porn filter.
Does that still exist? I got a decent ISP (Zen) so they don't block anything.
Ah great! Such nostalgia for that site, they had the -best- porn back in the old days, one of my favorite pron sites.
[flagged]
Turns out humans are not soulless automatons and like to do favours for the people they work with and are friendly with from time to time.
Ooooor, it could be, like, a person helping another person out, or something like that, you know?
> not what work resources are for

Employees are not robots. They are human beings. Sometimes human beings have human problems that need the assistance of other humans. This makes humans happier and more productive.

It's depressing to think that there are people who actually believe that optimal use of work resources is even worth calling out as an issue. In 2024.

>actually believe that optimal use of work resources is even worth calling out as an issue.

Setting aside moral arguments, if it raises to the level of embezzlement, it’s a crime.

I think (hope) the vast majority of people do not believe that asking the CSIO why their website is blocked is embezzlement.
If you want your employees able to deal with emergencies, you can't run them at 100% capacity all the time. You need some slack, so you have capacity when shit hits the fan.

Using a small amount of that slack to keep another employee happy can be a good investment. In addition, it's good for someone like the CISO to poke around the innards of your network (etc) configs from time to time, just to stay up to date with what's going on in the company and to perhaps flag anything that smells suspicious.

You can do these kinds of exploration exercises completely free form, or you can take a little task like 'figure out exactly why this specific site is blocked' as a token of motivation.

I agree that all of this mostly only makes sense, if it doesn't take too much time.

Though if this specific task would take a lot of time, that would also indicate that either the CISO needs to upskill, or the network config is too complicated. In either case, that would be a valuable insight.

> Wait, so you begged your CISO to figure out why your work internet ecosystem was blocking your personal project website from work computers? Man that sounds like a horrible waste of the CISOs time and not what work resources are for.

Sticking to your strict productivity line of thought, this kind of ask would:

1) be a great small teaching task for an intern, and

2) build goodwill elsewhere in a company, something good CISOs won't pass up an opportunity to do when the cost is relatively cheap.

But it's also likely that the CISO just wanted to help.

> I hired a square space specialist

I had no idea such a thing existed.

If you can set up your own domain why would you need someone that specializes in a super limited non technical frontend for customizing prebuilt web templates?

In hindsight I didn’t need him. I am pretty technical but I couldn’t figure out what happened so I hired some squarespace seo guy to make sure I had everything configured properly. It was the first and only time I heard of this happening.
It’s wild how these past associations can stick and haunt a domain, even after it’s changed hands entirely.
I wonder if there’s a market for rehabilitating domain names
Yet another valuable use for the WayBack Machine, glad it got a mention.
I feel like this should be the registrar's responsibility. Least they could do is give a disclaimer and/or a heavy discount.
Interesting. Domain as a unit of trust makes sense until it doesn't. Buying a second hand domain is like a second hand car. But you may not know it is second hand!

I think the mistake here is the redirect old to new. That is always risky so only do it if deseprate. In this case I would have done the redirect from new to old. Then just use the new as a vanity url.

> Buying a second hand domain is like a second hand car.

I have never hear of anyone being denied business because their car has a bad reputation from a previous owner.

Some time ago I noticed that my side project (with a domain that is not haunted) shows up fine on Google but not Bing/DuckDuckGo.

So I checked the Bing Webmaster Tools. URL Inspection says "Discovered but not crawled - The inspected URL is known to Bing but has some issues which are preventing indexation. We recommend you to follow Bing Webmaster Guidelines to increase your chances of indexation."

That's quite unhelpful. What's more, when I open the "Live URL" tab, it says, in green: "URL can be indexed by Bing."

It's a simple static Hugo site hosted on Cloudflare R2 (DNS mapped directly to bucket). https://pagespeed.web.dev gives it a score of 100 in every category.

Anyone else had something like this happen?

Yup. I've regularly had problems with a static site [0]. Sometimes it's a top hit for my name on Bing, sometimes completely unlisted. Seems to flip back and forth - with that same message you get.

It's a handwritten HTML website, enhanced with JS but not reliant on it, hosted on Cloudflare. Not quite a 100 in every PageSpeed category, but just about.

[0] https://jamesmilne.org/

OP here, and yes, I've been getting that same message for musicbox.fun. I thought it just needed some time but I requested a fresh index two weeks ago, and nothing seems to have changed. :/
A side effect of negative seo is that some stuff that hasn't worked on Google for a long time still does on Bing (They, Bing, obviously, not being the real target of the attack).

I've seen a few sites become de-indexed and the 'give away' is the type of results that first appear when the penalty is eventually lifted. For example, just a dozen or so urls with really weird query strings that never existed before. The real stuff does come back after time though and, in my limited experience, it's a one-off incident.

Just to add, not many sites are insignificant enough not to attract negative seo - especially this type of low-level, zero cost malarkey.

Another "haunted domain" check is by trying to post about it on social media. I ran into this with my current project's domain name. After building an MVP and trying to test the social sharing functionality, I found that Facebook was blocking the domain outright. Turns out there was some spamming from it years ago. Getting it unblocked was extra fun, as the page to request manual review was itself broken! Thankfully I knew someone on the inside who alerted the relevant team, but the whole experience was quite the novel speedbump.
I faced the same issue with one of my project. But, as i don't know anybody at Facebook, I left the domain and buy a new one.
So much of the world is still based on who you know. This is a bug in our society I would really, really like to see fixed in my lifetime.
I think with AI it is going to become the opposite. You only trust who you know in real life and ignore everything else.
Huh? Weird. I only trust the AI and ignore everyone in real life life. (/s for the humor impaired)
The fix is called "legal system", or rather, also making it accessible for individuals and small businesses against the large mega corporations without risking getting bankrupt in case of losing. And companies that continuously lose in judgements get fined progressively until they establish enough support infrastructure to not be a burden on society.
Small claims court often works, depending upon jurisdiction.

Where I am there is no forced disclosure, no costs costs assigned, and it is $150 to file.

And while a lawyer can represent a large firm, an employee has to be present, and the lawyer cannot use excessive legalise, the court is carried on in plain language... with the judge expaining things to you if you don't userstand.

That's pretty accessible.

The biggest risk is not knowing about no required discovery, and costs. Lawyers for big corp will ask for things, and hope you work your tail off. I just say no.

They will also elude to how expensive this will be, to which I typically snort.

Said large companies typically spend 50k to 100k on lawyers, and I spend $150 and a dozen or two hours of my personal time.

All very amusing.

Anyhow, a good equalizer.

Is this a bug? I think this is a built in feature since version 1.0.
Depends on the context. Forming a real human connection with someone who has proven they can be trusted is a feature. However, people oftentimes feel they are connected to others based on identity, and then treat those people favorably regardless of merit. The latter is such a major detriment to society that it needs to be actively countered by regulation (and is to some extent).
Sadly, the most likely "fix" would be to remove the "who you know" path and just make things shit for everyone. :(
But would that not introduce pressure for the official paths to become better oiled and working better than before?
Reframe:

It's not that the smooth path you can get via nepotism is the base way things work which people who don't "know a guy" are excluded from. Rather, everything is falling apart and shitty, and if you're lucky, you occasionally get to circumvent that shittyness.

Meritocracy is great and all, but there's a gap between having merit and others seeing the merit.

I don't believe that human society can, practically, get particularly close to the ideal. I question the choice of fatty meat as a substrate for minds.

For my money, I'd suggest that merit will get you further today than in the days of letters of recommendation, but that failures of meritocracy are more visible.

> It's not that the smooth path you can get via nepotism is the base way things work

Well, obviously it isn't if you're not in the 1%. If you're in the 1% then that's the way the world has always worked and you don't know anything differently.

I would really like to see it fixed too, especially as regards these faceless behemoths which nevertheless worm themselves into dictating important parts of real peoples' real lives with absolute authority and no recourse
I have a fairly boring consulting business, blocked by Twitter for being malware. Fortunately FB / LinkedIn / WhatsApp all work.
I had that one happen as well, after launching a project. I could even post in a messages to friends.
Social media platforms can be some of the biggest canaries in the coal mine when it comes to a domain’s “haunted” reputation
Not quite haunted but I've had people report that my website hosted on a .quest domain is blocked on their work computer. My best guess is that their filter thinks it's gaming related (it's not) or maybe they just block all "weird" domains.
unfortunately, blocking newer TLDs altogether seems common
Basic SEO stuff, you have marketplaces that check history, you have domain search engines aggregating data from multiple sources - not only ahrefs.

Checking web archive is a basic operation to test if site was hosting anything fishy - not only pirated stuff or porn - often websites has been hacked and changed into link farms or simply were bought on aftermarket simply to use it's SEO value to pass the strength to other domains.

Anyways good point regarding email filters.

Not always the easiest thing to do. A haunted domain could have been haunted 15 years ago. And Google refuses to tell you why or fix their system.

Just one more place where the web gets screwed by a company too big to have to do basic customer service.

In their defense (and I don’t defend Google often), addressing this really well means:

- knowing all the complexities of every local, state, federal, international jurisdiction that might interfere with the whitelist

- awareness of the content in question which could be millions of subpages

- a customer support team that is definitely not incentivized based on tickets triaged per day, but is somehow incentivized to spend hours on “whale” tickets.

- going through ticket history and solving the problem for everyone now that its policy to solve this

- dealing with the inevitable rush of fraud that follows every tiny change in google systems

"Ideally, search engine algorithms would give new domain owners a fresh start."

I don't think it's possible to fix this problem without also helping bad actors. Maybe it's a problem that just isn't worth fixing. Just don't buy preexisting domains unless it's a project big enough to justify the necessary cost of due diligence.

"Maybe it's a problem that just isn't worth fixing."

There is a finite amount of short, memorisable names.

But also an ever-increasing number of TLDs under which to register them.
The really bad actors just buy and discard new domains daily and silly blacklisting techniques are powerless to prevent that. I don’t think they renew and come back to try to use their domains years later.
Then help them. If a few bad actors is the price of a free internet, so be it. I'd rather deal with those than have a whitelisted internet where you need permission to start a website.
I've had an opposite experience. One domain I bought was used for an entirely different purpose in the past, which got linked on a Wikipedia article in references. This gives me some good link juice and at least matches the geo area of the previous business. Since it's an extremely niche entry and low on the list of references, I decided to be slightly naughty and not touch it for a couple of years. Not sure what's the opposite of haunted in this case, but it was just as surprising.
As someone who knows what active persecution on this site is I relish the opportunity to say what I really know under a pseudonym.
Automattic.com was bought (no idea if it was unregistered / acquired) by Matt Mullenweg when he set up the company. He also bought https://a8c.com.

Here in the UK with EE/BT that correctly redirects to automattic.com, but it might not for you depending on your ISP.

The wayback machine shows adult content links prior to the domain being put on sale, hence the blocking.

see also landslide.com - a domain that should never have been reused imo
Haunted is a weird way to call them, these are stigmatized domains.
Stigmatised would be when it commonly/publicly has a bad rep.
That’s pretty much what happened to those domains.
No, those domains are completely fine, they are just marked as untrustworthy on some obscure google list.
That’s a contradictory statement.
No. There's no general stigma. It's just the one list.
This happens with physical addresses too, for similar reasons. The ABC (Alcoholic Beverages Commision) tracks complaints against physical addresses, and too many violations will get an address banned from permits. Then a new owner comes in with a new business and gets mysteriously denied for a liquor license, even years later.
It is customary to revoke the right of a business to name itself if there were too many violations.

If you've ever gone to a nightclub or bar which has no name, only its street address number, that's what has happened there.

How can a business function without a name? So much tax paperwork requires a name. Is it just a sole proprietor that files everything under the owners name?
It has a name, but that name cannot be different from the address, like "The 1415 Club" on 1415 Main St.
Sounds like a very stupid custom
TLDR: when you rent anything, double check who rented it before you and what they did with it to make sure it’s in good condition.