Like wolco, I'd kinda like a source for that claim, also. Tucows owns Hover, as well as being the back end for domain resellers through OpenSRS. To the best of my knowledge, Hover has never come up as "the preferred registrar of white supremacists"; while it's not at all unlikely that a reseller who uses OpenSRS's infrastructure has provided registration services to unpleasant characters, that would be a rather weaker claim against them.
Had to check. Lycos/tripod is still hosting a stupid website me and some friends made a bit after 2000 (2004?). The cost is probably none (some static files), but I'm impressed they haven't just removed it or lost it during all these years. Stagnant and probably no visitors other than us the weekend we made it.
Edit: the glory: http://moj24.tripod.com/ animated background, marquee and right-click blocking script in case someone tried stealing our content
Aww man, I had a Simpsons fan site up in the 1997/1998 time frame that Tripod eventually took offline due to some sort of abuse or something... which I never figured out. There were a lot of Simpsons fan sites up back then, as I recall.
At that time, FOX was egregious about copyright claim to anything. They worked to take anything offline that was not official, hence licensed. I remember going to fan sites back then. One day they'd be up; the next, they'd 404.
My first website from around 1997 is still on tripod. I check it every so often and I'm impressed it's still there. Tripod is like an internet time capsule.
I used to run an ISP. Our server park was two machines each with 128MB RAM and 120MHz classic Pentiums and 4GB of harddrive space. One of them hosted e-mail and shell access for all our users. The other had web hosting for our business users, our tape drive (we backed up the other machine over the network), and our USENET server...
Man, I would be so pumped if I could find my old tripod site. I've looked before but I forgot the username and could never find it. One night as a young teen I decided to read a book "html for dummies", not knowing that 20+ years later the decision would pay off big!
in seventh grade we had a "web" class that, as an assignment, wanted us to find "pen pals" on the internet (lmao great idea), and to make a personal site about whatever. I made an Escape Velocity site. Lots of under construction gifs even when I submitted it.
That site is cute, but I don't think anyone would seriously claim YouTube is built with "Vanilla JS" in the sense of not requiring a heavy framework to make changes even if the millions of lines od non-vanilla code (guess) they build off of isn't public. And Reddit certainly isn't anymore.
They're referring to loading the scripts right after the opening body tag (rather than at the end of the body), which historically would block content loading until the scripts all load.
When I saw "embedded wav file loading an image", I was mystified and curious what that even meant, so I looked, and I didn't see. The only wav file I noticed in the source was on the Biler page, and I didn't see anything that suggested it was loading an image, but I didn't look very closely and I guess I'm not sure what to look for. I didn't read the JavaScript at all really, but I searched for "bgsound" with no results but the bgsound element itself, and it has no id and I don't know how else it could be referenced.
Don’t forget that <blink> (And maybe <marquee>?) is no longer supported by modern browsers, so you’re definitely missing some major elements of the experience.
Nice. Got a little hit counter at the bottom too. Ah the good old days of the web. I remember when it was all about "hits", "impressions", and "banner ads".
This stuff always makes me nostalgic. I wonder how feasible it would be to build a search engine specifically for 90s era websites. Ignore all modern stuff.
Not all of us are so lucky, one of my first websites was hosted by the little known bamboohost.com. Pretty sure it was http://linkinpark.bamboohost.com, but not even web archive can help me recover it.
Some of us are lucky - my first websites are not available anywhere, including the web archive. All that is left are some lingering sigs in a few usenet posts from 1998ish still archived on google.
Recently, I have been able to find an republish (on my personal site) some of the pages of the mini site I created in 1997: https://bussolon.it/archivio/1997/
When you buy your ad space at Microsoft, you can determine how it shows up at duck-duck-go, which keywords triggers your ad, what region you want it to work etcetera...
(and yes, that is not a typo, you go to microsoft to buy ad space for ddg)
I just searched the exact title of an article... and the result wasn't on the first page.
So... I tried it a few more times and I didn't get a single hit for any thing I tried. Google picked up each one with "I'm feeling lucky". DDG got them all too.
I wouldn't describe this as "pretty good", or even "good" or even "worth the time I spent this last 90 seconds".
Just for grins, try the same search on Bing and on Yandex. Look at the first 20 'organic' results (not always easy I know but its worth trying to figure out).
Search indexes have a reasonably unique fingerprint in the rank order of organic results for a given query. The ads and sponsored content are all over the map and perturbed by a/b tests and other things but the organic results typically come straight from the ranking algorithm.
Most "Search portals" (which is the web page that you type at to get your search engine result page from (SERP)) are either using one of the two English language indexes (Bing and Google) with their own ad network, or "blended" index where the search portal might index somethings like Wikipedia and Stack Overflow and then use Bing or Google for the long tail stuff.
The reason for this is economics, you can stand up a search portal on an AWS instance, feed all the queries to Bing out the back and sell ads on it and make a few bucks.
In contrast building your own index requires many more servers, crawling over the content you want to index (if you want to keep it fresh) and generally much larger network and storage costs. So its harder to pay for that with third party ad networks (not impossible, just harder). Pretty much everyone skips this approach for that reason.
Does anyone know how ddg and co. pay for that? Do they get special rates? Because the normal price for using the Search API of Google or bing seems too expensive to be viable.
I'd suspect so, the sticker price is for the little guy who cannot guarantee them a large volume of organic search queries. With a large volume of course you negotiate.
The most interesting thing here is Startpage. Apparently Google doesn't give access to general Search API (as opposed to custom search limited to some sites) for over a decade now. I idly wonder if folks at Startpage have to maintain an extremely plain and not terribly well marketed site lest they be evicted from their deal, presumably a very very old one. According to Wikipedia they probably use Google since the early 2000s at least.
By credit card? :-) Sillyness aside, it is a regular contract. About 6 years ago you could get Bing results through the Yahoo BOSS API[1]. I'm not sure what they charge these days but they used to charge about 80 cents per thousand queries.
You take the query terms and send it to an ad network (or ideally your own ad infrastructure) and the network sends back 1 - 10 advertisements based on the query terms. You put those and the "organic" results you got back from the API out as your search page.
You hope that in the thousand queries you process, some of the people click on the ad links instead of the organic links. If enough do, you "make" the difference between the money you get paid by the ad network and the money you paid to the search index.
If you can send a lot of queries, then you are more valuable to the search company than they are to you, and they will start to pay you to send them these queries. (this is what Google calls "traffic acquisition costs" in their earnings reports). But we are talking a lot of queries, you'd need a really popular Linux distribution or Web Browser to pull that off. Just a search landing page won't cut it.
The contract will also detail things that you can't advertise when you get results from the search index. For example in California Google can't allow anyone to show a payday loan ad next to Google results.
And there are subtleties within that as well, you typically get a fraction of the revenue on an ad that gets clicked on as the "first" click after showing the search engines results, but you can get more (or all) of the revenue on pages that you show after that. That trick is used by people who crawl all of Wikipedia and then show a version of Wikipedia with ads where they buy an ad on Google's results for Wikipedia type queries, and then send you to their own page / copy of Wikipedia with their much more profitable ads.
The more complex it gets the more sketchy the players involved.
We should have a 20 year ( has it been that long? ) reunion. Get all the cast together. Lycos, Excite, AltaVista, AskJeeves, etc. Google will probably be too cool to attend.
I worked at Go2Net when they acquired Dogpile. I was directly involved in migrating it’s systems to our data centers after the acquisition closed.
I don’t really remember the tech stack, but it was something like 10-15 web servers, probably pentium II or III.
One detail I do remember is the original setup had an Apache log rotation script that would try several times to restart Apache and if it didn’t succeed, it would reboot the server. This ran nightly...
Not sure what bug or issue they were working around. But I thought it was quite an aggressive solution!
You have to do a graceful restart after rotation, according to the docs, which then leads to scripts that try a couple times and then reboot. (It's been a while, but I did have an install where graceful restart would work once, but consistently fail on the second rotation).
If you don’t change the inode then no restart should be required. There’s definitely a few tools out there that solve this problem without needing to restart httpd.
Halt and Catch Fire is criminally underappreciated. It did amazingly well at capturing the quirkier side of the home computer revolution first and foremost, but also the early years of the growth of the net, by not talking about the big, famous survivors.
I learned about competing nets for the first time through that show. And then I learned what a vision the world wide web was.
Amazingly one of the characters said he knows how horrible content becomes when it is open to just anyone. Well we got angelfire and tripod today to do just that.
Appears to be powered by Bing. The search results are nearly identical. More importantly snippets for each site are identical, which is the real giveaway.
Yep. A lot of folks appear to be pretty confused about retail search engines verses the actual index -- Google, Bing, and (at lower quality) Yandex are the only viable English-language search indices right now. Almost every search box you find on the Internets is powered by one of the three.
Doesn't it maintain it's own index on certain terms based on user actions/feedback that changes the bing results resulted? I thought there was an improvement feedback loop.
That doesn't seem to be the case; the only reason they use r.duckduckgo.com is for redirecting without the REFERER according to their docs, so there doesn't seem to be a way for them to do that.
> To do that, DuckDuckGo gets its results from over four hundred sources. These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (our crawler) and crowd-sourced sites (like Wikipedia, stored in our answer indexes). We also of course have more traditional links in the search results, which we also source from multiple partners, though most commonly from Bing (and none from Google).
Yes, as we know nothing about the percentage of traffic which is coming from google, bing or any of the remaining 398 ressources. I can only assume the contribution is abysmal, given the almight of the one trully.
I read that as "Over four hundred sources for instant answers (the big box at the top) etc. What you usually think of as search results ("traditional links") comes from multiple partners, mostly from Bing (and never from Google, which only leaves Yandex and niche providers)".
I've been using your service as my main search engine on desktop and phone for more than one month (I made an xml file to load it as search engine.) It's okish but not very good at dealing with misspelling, with news and with images. Sometimes I touch the Google button at the end of the page to find what I'm looking for. I hope you keep track of the keywords that go to Google and use them to improve your index. Anyway, it's a good service, thank you.
Hey, thanks for using Cliqz. We're working hard on improvements for each one of the things you mentioned (in that order as well) - so things will definitely get much smoother over the next couple of months.
My grandparent was saying "Google, Bing, and (at lower quality) Yandex are the only viable English-language search indices right now", and my parent was asking whether Cliqz counted. I think the GP is correct and Cliqz doesn't.
Hey, I am working on Cliqz Search [0] and would be very thankful if you point me to quality issues. I use it daily myself (in English) and will make sure those issues are addressed.
This is absolutely fantastic! A REAL alternative not just a search collector like ddg which are endorsed by Google or Bing as the traffic and ad opportunities remain unafected.
SwissCows has its own indexer and I believe they don't rely on any external search source like Bing. They should push for better marketing, less nationalism & better UI like duckduckgo and they would go pretty far imo, their results are often better than DDG
edit : my bad, actually they do use Bing in addition to their own crawler
No, not really. Could be a list of elm tree species, or something even more specific (most famous elm trees, species vulnerable to some disease, etc) that's immediately obvious if you're plugged in to the arborist community instead of the hacker community.
A valid but very unlikely interpretation, which is the point. Every query is ambiguous, but if certain interpretations are much more likely than others, the search engine should respond accordingly.
Someone interested in a list of elm species is much more likely to search for "elm species."
It is very unlikely to be the case. Much more likely is that Bing either doesn't have enough search data to properly answer that query or does not have a ranking model that can answer that query correctly.
Exactly, it's just Bing without the image or video snippets in the middle. Other than that... it knows I am in Paris, however shows the current temperature in Fahrenheit. So no, bye-bye Lycos.
Not really. My test is my handle “geuis”. It’s always autocorrected to “genius”. Google and Bing at least have the “did you mean geuis” link. Lycos doesn’t.
In the end it's funny: it shows my location (a borough in London),but when I click on jobs,it shows positions in Nashville... According to Wikipedia, just 3 years ago there were still 500 people working there...
I tried the search which I used a few weeks ago when someone here claimed that DDG was pretty good. Google found what I wanted above the fold. DDG found it at hit #87 or so (click “next” 7 times). Lycos didn’t find it in the first 100 at least.
I used to like Lycos but there’s a reason everyone migrated to Google.
230 comments
[ 0.21 ms ] story [ 320 ms ] threadFrom https://www.searchenginewatch.com/2004/03/15/how-lycos-works...
AllTheWeb === Yahoo now.
The few test queries I tried on Lycos seem to be identical (almost down to the ordering) when I tried them on looksmart.com.
https://www.tucows.com/
http://www.tucows.com/downloads/
That’s also a blast from the past. Look at the number one freeware download:
https://ibb.co/gDZjjBf
It was like running into that old buddy from school and he lets slip he is into some horrible stuff now...
Edit: the glory: http://moj24.tripod.com/ animated background, marquee and right-click blocking script in case someone tried stealing our content
Not sure if much has changed.
My site is so bad I'm not going to link to it.
Also, <marquee> tag and site counter's for life!
(perl's OK, CGI had a lot more server side injection risk from what I remember)
If the former, the injection vulnerability would be in the script talking to the server/database via CGI, rather than in CGI itself.
If the latter I don't remember any major unpatched vulnerabilities in CGI.pm, but it was epically inefficient.
https://i.imgur.com/7mr0vjQ.gif
in seventh grade we had a "web" class that, as an assignment, wanted us to find "pen pals" on the internet (lmao great idea), and to make a personal site about whatever. I made an Escape Velocity site. Lots of under construction gifs even when I submitted it.
Which ones were your favourites?
http://textfiles.com/underconstruction/
they really should put it up for science, so you can look up the shit that's been keyword-buried since then.
IIRC they were not good.
No kidding. Firefox's inspector tool just crashed (went white and unresponsive) while I was making a list of the worst offenses:
- h2, h4, h5 used interchangeably
- embedded wav file loading an image
- loading all JavaScript just below body tag
Fantastic.
Test Test Test Test - Right Now
www.kensaq.com/Test Test Test Test/Here
Welcome to Kensaq.com. Find Test Test Test Test Today!
When you buy your ad space at Microsoft, you can determine how it shows up at duck-duck-go, which keywords triggers your ad, what region you want it to work etcetera...
(and yes, that is not a typo, you go to microsoft to buy ad space for ddg)
So... I tried it a few more times and I didn't get a single hit for any thing I tried. Google picked up each one with "I'm feeling lucky". DDG got them all too.
I wouldn't describe this as "pretty good", or even "good" or even "worth the time I spent this last 90 seconds".
This happens to me on Google a lot. I guess mileage varies.
Search indexes have a reasonably unique fingerprint in the rank order of organic results for a given query. The ads and sponsored content are all over the map and perturbed by a/b tests and other things but the organic results typically come straight from the ranking algorithm.
Most "Search portals" (which is the web page that you type at to get your search engine result page from (SERP)) are either using one of the two English language indexes (Bing and Google) with their own ad network, or "blended" index where the search portal might index somethings like Wikipedia and Stack Overflow and then use Bing or Google for the long tail stuff.
The reason for this is economics, you can stand up a search portal on an AWS instance, feed all the queries to Bing out the back and sell ads on it and make a few bucks.
In contrast building your own index requires many more servers, crawling over the content you want to index (if you want to keep it fresh) and generally much larger network and storage costs. So its harder to pay for that with third party ad networks (not impossible, just harder). Pretty much everyone skips this approach for that reason.
Google:
- dictionary.com
- knowyourmeme
- artsatmichigan.umich.edu
- urbandictionary
- thetab
Bing:
- knowyourmeme
- silvergames
- wikipedia
- urbandictionary
- joke battles wikia
Lycos:
- knowyourmeme
- urbandictionary
- silvergames
- youtube (BIG CHUNGUS | Official Main Theme | Song by Endigo) [this is the sixth result on Bing]
- joke battles wikia
Yandex:
- big chungus wiki
- youtube: big chungus (song)
- knowyourmeme
- memepedia.ru
- urbandictionary
I'd suspect so, the sticker price is for the little guy who cannot guarantee them a large volume of organic search queries. With a large volume of course you negotiate.
The most interesting thing here is Startpage. Apparently Google doesn't give access to general Search API (as opposed to custom search limited to some sites) for over a decade now. I idly wonder if folks at Startpage have to maintain an extremely plain and not terribly well marketed site lest they be evicted from their deal, presumably a very very old one. According to Wikipedia they probably use Google since the early 2000s at least.
You take the query terms and send it to an ad network (or ideally your own ad infrastructure) and the network sends back 1 - 10 advertisements based on the query terms. You put those and the "organic" results you got back from the API out as your search page.
You hope that in the thousand queries you process, some of the people click on the ad links instead of the organic links. If enough do, you "make" the difference between the money you get paid by the ad network and the money you paid to the search index.
If you can send a lot of queries, then you are more valuable to the search company than they are to you, and they will start to pay you to send them these queries. (this is what Google calls "traffic acquisition costs" in their earnings reports). But we are talking a lot of queries, you'd need a really popular Linux distribution or Web Browser to pull that off. Just a search landing page won't cut it.
The contract will also detail things that you can't advertise when you get results from the search index. For example in California Google can't allow anyone to show a payday loan ad next to Google results.
And there are subtleties within that as well, you typically get a fraction of the revenue on an ad that gets clicked on as the "first" click after showing the search engines results, but you can get more (or all) of the revenue on pages that you show after that. That trick is used by people who crawl all of Wikipedia and then show a version of Wikipedia with ads where they buy an ad on Google's results for Wikipedia type queries, and then send you to their own page / copy of Wikipedia with their much more profitable ads.
The more complex it gets the more sketchy the players involved.
[1] https://developer.yahoo.com/search/boss/boss_guide/BOSS_Gett...
https://ibb.co/vLBPYTW
I don’t really remember the tech stack, but it was something like 10-15 web servers, probably pentium II or III.
One detail I do remember is the original setup had an Apache log rotation script that would try several times to restart Apache and if it didn’t succeed, it would reboot the server. This ran nightly...
Not sure what bug or issue they were working around. But I thought it was quite an aggressive solution!
"oh, we forgot the print. lets retry tonight."
"darn, did not happen. lets leave this part in our infra for now until we trigger the edgecase again."
5 weeks (or days...) later: "damn, why did we do that again? better not touch it"
i love it when i find myself cargo culting code like that and finally have the time (and courage) to remove it :)
What’s your log rotation code look like?
https://httpd.apache.org/docs/2.4/logs.html#rotation
What I meant was you should be using pipes and apaches own log rotate code.
Read the “Logging Using Pipes” section at the very end.
https://www.digitalocean.com/community/tutorials/how-to-conf...
(Excuse the DO link, it was just the first result in Duck Duck Go. There’s nothing specific about DO or Ubuntu in this approach)
I’ve used this method on large clusters of very heavily utilised web servers and it works great. No restarts required.
https://en.m.wikipedia.org/wiki/BonziBuddy
The '90s were definitely a time!
Amazingly one of the characters said he knows how horrible content becomes when it is open to just anyone. Well we got angelfire and tripod today to do just that.
And DuckDuckGo.
> To do that, DuckDuckGo gets its results from over four hundred sources. These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (our crawler) and crowd-sourced sites (like Wikipedia, stored in our answer indexes). We also of course have more traditional links in the search results, which we also source from multiple partners, though most commonly from Bing (and none from Google).
Do you view that as misleading/incorrect?
That fits well with what GP claimed.
There is a lot of technical details about how it’s built on the tech blog: https://0x65.dev/
(Disclaimer: I work at Cliqz)
[0] https://beta.cliqz.com/
1. Search for a blog I follow
2. Look at the snippet for the blog's homepage
3. Figure out how long ago the page must have been crawled to get such an old snippet
I only tried a few, but they were months to years out of date.
edit : my bad, actually they do use Bing in addition to their own crawler
https://info.lycos.com/about/contact-us/
- it is only part of the story like with ddg
- or Bing has improved a lot lately
- or for some reason I've been extremely biased against Bing
I just tested it and the results were way better than I would expect from Bing.
- bing has improved and is now usable, maybe even pleasant to use compared to Google.
- the results between Bing, DuckDuckGo an Lycos were kind of similar, but not exactly similar
- autocomplete was much better on Bing
- I actually enjoyed the way certain search results were displayed in Bing
https://search12.lycos.com/web/?q=elm+list
https://www.bing.com/search?q=elm+list
Funny I only ever hear someone call Bing good when it's named something else (DDG, Lycos).
Someone interested in a list of elm species is much more likely to search for "elm species."
Lycos must have hired an archeologist to recover this long lost technology from an ancient data center.
I used to like Lycos but there’s a reason everyone migrated to Google.