Ask HN: Do you use an old or 'unfashionable' programming language?

187 points by open-source-ux ↗ HN
I couldn't think of a better word than 'unfashionable' but what I mean by this is a programming language that is not new, upcoming, or has much traction.

The language may have unique or novel features, it may be a language with a passionate and dedicated band of programmers. But one thing the language does not have is much 'mindshare' amongst programmers: its time in the spotlight has passed. It may still be in active development, or it may be moribund.

Examples of 'unfashionable' languages include: Cobol, Snobol, Icon, Unicon, Forth, Pascal, Eiffel, D, Smalltalk, Basic etc.(Note, I realize this is subjective to a degree.)

If you use an 'unfashionable' language, what keeps you using it? It is a unique feature? Is it familiarity or comfort? Is it speed or performance or some other quality? What do you think we could learn from that language when developing programming languages today?

337 comments

[ 3.1 ms ] story [ 287 ms ] thread
(comment deleted)
(comment deleted)
I use Tcl for AI in Life Sciences and Hedge Funds.
I wanted to say that I still occasionally use TCL, too! :)
I also use Tcl quite a lot for some pretty sophisticated software. (And it's not embedded! It seems most surviving Tcl programmers are using it as an embedded scripting language.)
I've written larger apps in TCL, but it was painful. I must admit to being guilty of using it solely as an embedded language these days.
I use Delphi. The RAD Studio IDE is super featured and it's really easy to create interfaces. A nice feature is that the programs produced contain all dependencies so nothing else needs to be installed when distributing.
Delphi is excellent!
I used to program in Delphi between 1997-1999 and after that I couldn't get even close in terms of productivity with Java. It wasn't the language though, the strongest thing about the Delphi environment was that it was truly comprehensive and you'd just go in and implement your business logic.
Same here. Funny how, 15 years later, there still isn't a competing language/UI builder that can measure up. Xcode + Interface Builder, for example, is a joke compared to Delphi. The first time I used IB, I remember being horrified by how IB requires you to "connect" UI elements to variables, and how it can't even create those variables for you, let alone allow you to add event handlers.
I was so pleased when I could write my Delphi code on iOS/osx and android as well and could abandon the horribleness that is IB.
I'm curious if you're familiar enough with other more modern languages and platforms to give an opinion on the relative efficiency of Delphi compared to some of the others? As someone who used to build lots of apps in MS Access (which is similar to Delphi but not as good), I find it takes far longer to build things with modern platforms, although you have much more capabilities.
I do a lot of Delphi during the day (medical systems), also spent a number of years with other languages - C/C++, Objective C, Java, c#, javascript, typescript, lua, perl etc and I agree whole heartedly. Delphi is still the most productive platform imho, so much so that most side projects I do with delphi. The fact it is now cross platform adds to the list of 'why I think delphi is the best'.

Unfortunately there's no one thing that I can point to that makes Delphi the best, its a lot of little things - the IDE and screen painter, the way you can use pointers, deployment is so easy with a single exe, RAD. The one language I prefer to delphi is typescript, it is Delphi with a lot of functional sugar, but since its written by the same guy probably not a surprise.

Most of my web programming these days is in PHP. It gets the job done, it's easy to use and deploy, and I don't need anything more for the scale and complexity of what I'm doing. It's been around for a long time and doesn't seem to have a risk of going away anytime soon.

SQL databases. All of the above applies. Stored procedures when the database supports them. I do not use an ORM.

I try to use command-line utilites whenever I need to manipulate data. sed, awk, grep, etc. I rarely need to drop into a scripting language for these tasks.

>It gets the job done

This sounds like other similar language will not get the job done. The truth is that there are much better alternatives which are a lot more pleasant to use and will get the job done in a much better way.

> it's easy to use

This is a myth. Php is not easy to use. May be, It is easy to start. But you need to know a lot of useless stuff (outside the context of Php), to get anything to work any where near reliably. Also, be prepared to write a lot more automated tests if you want to keep it that way.

To quote the classical piece on the subject ("PHP: a fractal of bad design"):

    There is a whole lot of action at a distance. 
    Consider this code, taken from the PHP docs somewhere.

    There is a whole lot of action at a distance. 
    Consider this code, taken from the PHP docs somewhere:

    @fopen('http://example.com/not-existing-file', 'r');

    What will it do?

    If PHP was compiled with --disable-url-fopen-wrapper, it won’t work. 
    (Docs don’t say what “won’t work” means; returns null, throws exception?) 
    Note that this flag was removed in PHP 5.2.5.

    If allow_url_fopen is disabled in php.ini, this still won’t work. 
    (How? No idea.)
    Because of the @, the warning about the non-existent file won’t be printed.
    
    But it will be printed if scream.enabled is set in php.ini.
    Or if scream.enabled is set manually with ini_set.
    But not if the right error_reporting level isn’t set.
    If it is printed, exactly where it goes depends on display_errors, again in php.ini. Or ini_set.
etc. :) (Perhaps this particular issue is obsolete now, still it's emblematic of problems underlying the language, the ones I had to encounter when I worked with PHP few years back. Talk about "easy to use")
Some people simply cannot accept that anyone would find PHP useful... or even tolerate its existence, and employ one particular (and increasingly obsolete) rant as if it were an objective argument against someone else's opinion on the matter. And yet, it's all subjective.

If you don't like PHP, don't use PHP. It's that simple.

And that's precisely what I did, of course - I no longer use it.

But it doesn't mean I'm prohibited to share my experience with it.

I never denied it's "useful" (is it supposed to mean much? Obviously, if apps are written in it, apparently it is of use, by definition), however I'm not alone in thinking there are better alternatives and - again, based on my personal experience - the more exposure to other languages and technology stacks a PHP developer has had, the more critical they get about PHP.

The rant may be getting obsolete, but the mentality persists. To quote the creator of PHP himself: "I'm not a real programmer. I throw together things until it works then I move on. The real programmers will say "Yeah it works but you're leaking memory everywhere. Perhaps we should fix that." I’ll just restart Apache every 10 requests". And this attitude seems very common among PHP developers. It's not just about the language, it's also about the culture. And no I'm not saying other languages are surrounded with perfect cultures (like Java's "achitecture astronauts" with mammoth frameworks and overuse of design patterns).

> This sounds like other similar language will not get the job done.

That's not what they're saying. If someone knows a tool and it works for them, then it's more immediately useful than another tool which they don't yet know how to use, or how well it works.

> The truth is that there are much better alternatives which are a lot more pleasant to use and will get the job done in a much better way.

Quantify "better". PHP has its flaws (quite a lot of them), but it also has its advantages. It's a very popular language, so if you want to do something with it, almost certainly someone else has tried to do the same thing, so there will be libraries for it, guides, people who you can ask for help if you have trouble. It has a shared-nothing architecture, which aids scalability. It has extensive documentation. It has a large selection of battle-tested frameworks. There is an almost infinite selection of hosting providers.

> This is a myth. Php is not easy to use. May be, It is easy to start.

PHP's flaws can make it an occasional pain, but it is not really harder to use than any other ALGOL-like language. It is sometimes easier as it tends to expose simple interfaces to things, rather than requiring the user deal with a lot of complexity.

> But you need to know a lot of useless stuff (outside the context of Php), to get anything to work any where near reliably.

This is, to some extent, true for other languages as well. But yes, I'd agree with you here that PHP has more baggage than its contemporaries.

> Also, be prepared to write a lot more automated tests if you want to keep it that way.

This is true of dynamic languages generally. Though PHP has (limited, runtime) type-checking, which can reduce the burden here somewhat.

>Quantify "better"

Better as in more correct. The endless gotchas in the language makes it harder to write correct programs that works as intended by the programmer.

>so if you want to do something with it, almost certainly someone else has tried to do the same thing, so there will be libraries for it, guides, people who you can ask for help if you have trouble.

That is more or less true for the alternatives as well (Python, Ruby, Javascript...)

>This is true of dynamic languages generally.

Yes. that is why I emphasized "a lot more"...

Come on, PHP is hardly old or unfashionable
Perl and C here mostly. C89, that is. Both are widely used languages, just not in the startup space. a lot of our stuff is built on those languages so it wouldn't make sense to rewrite a lot of the stuff at this point unless there was a business case for it.
My personal favourite combination as well! Don't care about if its old-fashioned or what other people think about Perl and C. To me, a programmer should start learning C then work his/her way up!
So I'm not alone :) Perl and C everyday ;)
Perl and C++ here. You are making me feel modern and up to date :-)
I'm mostly a sysadmin so Perl is a big part of my life whether I like it or not, but not much C.
There's a huge difference between languages that are no longer fashionable, call them old-fashioned, and languages that never had wide following, if any, call them niche languages. Most of the languages you mention were never mainstream production languages, for good reasons.
You are surely joking. Do you really mean to say that Cobol is not a mainstream language? Are you claiming that banks and similar institutions are niches? If that is the case then pretty much all programming languages are niche languages.
I am claiming SNOBOL, Icon and Eiffel aren't, for example.
C and Tcl. You have to resort to things like COBOL to have chances to compete.
There's lots of job openings out there for COBOL coders. Nobody wants to learn it, and those who know it are dying off.
As a young developer, do you know where and how I could go about getting into the COBOL industry?

All the job listings seem to require a decade's worth of experience and lists expertise requirements in technologies that are proprietary and only available on a mainframe. COBOL on its own is not very hard to pick up, but I'm lost as to how to learn about the ancillary technologies. I'm not sure it's so much that nobody wants to learn, but that nobody knows where to learn it.

Contact the companies looking for COBOL coders. They're desperate, and will tell you exactly what you need to learn and probably pay for any training you need.

Sure they'd prefer someone with a decade or five of experience, but seriously: COBOL coders are literally dying off at this point. 17 years ago was the last great hurrah for hiring & training COBOL developers, averting the "Y2K crisis" before calendars ticked over to the year 2000; we've got kids born after that who are entering the software workforce now. About 5 years ago I had someone speak in my C++ class begging students to consider a career in "mainframe" technologies; there was virtually no interest shown, and methinks most of the students didn't even know what he was talking about.

But...the need remains. Those systems are so large & profitable they're not going anywhere, and employers must be quite willing to take on new coders willing to learn, directing & funding their education as needed. If you're interested, contact a hiring company directly and ask - even if you have 0 of 10 years required experience.

I wonder, are there any good tools for translating COBOL into something else? There must be at least some demand? Wouldn't that be a good approach in some cases? (I suppose it requires longer-term thinking, and is sometimes simply not possible in a business context, etc.)
Thanks for the advice. I guess it doesn't hurt me to give it a shot. The worse a recruiter can say to me is no. There seems to be no shortage of available COBOL positions at the major banks and telecoms where I live.
Why would you want to?

There's a lot of smoke and noise made about how companies are desperate for COBOL developers because the old ones are retiring or dying blah blah blah, but I've never seen actual good salaries to back it up. And to add insult to injury, you're unlikely to be able to take the skills you've learned to your next job when you inevitably change jobs.

I suppose you're right. After going through some publicly available COBOL salaries in Canada, it seems the salaries for senior COBOL positions were about what I was able to ask for as a junior frontend developer (50-60$/h CAD). I think I would primarily consider a short stint for the heck of it. Maybe some short term contracts to see what it's like.
What kind of salaries have you seen?

I've read that most of the companies still using COBOL are banks, insurance companies, etc. They seem like the companies, if any, that could provide really compelling salaries for something with allegedly high demand and increasingly small supply.

Don't. The job requirements are impossible on purpose because the posts are just there to justify getting guest visas.

You're better off going towards other enterprise technology stacks with more action and rewards.

> There's lots of job openings out there for COBOL coders. Nobody wants to learn it, and those who know it are dying off.

A lot of people say this, but all the COBOL jobs I see advertised (not many) offer a very average salary. Demand can't be that high if COBOL jobs are paying less than the typical "frontend developer" jobs.

The only language I use these days at work is Tcl, as it's the embedded scripting language in the app we use for post-processing finite element analysis [1] results. Thus I use it for small scripts to make various data extraction and processing faster.

It's main strength is also it's main weakness: the whole everything is a string concept, even code. It makes the language both very simple to grasp at first and then also very powerful. It makes lisp-like macros and meta-programming possible. However, it quickly seems to turn into a big ball of mud.

Lack of first-class functions or OO is a major pain point, too. Objects have to be emulated by procs (ie functions), and procs are always in the global scope (there is namespacing but it's quite unwieldy). I wouldn't care that much, except the App's API is an object model, which means even a small script turns into 50 lines of only handle-getting and releasing. Make a small error and forget to release a handle (ie a proc name) and now you have to restart the interpreter (or write a lot of error handling script, which once again for small scripts is a pain). Also, you can't pass objects around by value as they are procs, only by name, which is also a pain.

[1] Numerical method often used by engineers to solve various physical problems modelled by boundary-value PDEs. https://en.wikipedia.org/wiki/Finite_element_method

TCL became the standard embedded language for many EDA tools. I really hate it: it's difficult to use and slow. On the other hand, tcltk made it a convenient GUI language.
TCL/EXPECT was a huge help when I needed tp automate data extraction from a health clinic practice management system. The available interfaces didn't give all the data. Decoding the custom data formats, file layouts, etc... wasn't going well. Scripting the character based UI with TCL/EXPECT was pretty cool including the error handling. If a bit hacky.

I was surprised to find out my temporary solution was still running without a hiccup 5 years after I left. Yes, I was younger then and hadn't learn to appreciate the life expectancy of temporary solutions.

Can you use 8.6? TclOO makes objects a lot more wieldy: they're still commands, but you get automatic cleanup, destructors, some nice options for inheritance/delegation and so on.

"emulating objects with procs" sounds like you're not taking advantage of any of the previous OO extensions (snit, itcl, xotcl ...), which has to be making life harder than it should be :(.

Unfortunately I'm stuck with the embedded 8.5 interpreter.

And, to be entirely honest, most of the gripes above are likely more due to the poorly designed API than to the language itself.

(comment deleted)
Delphi Developer here. Still using it at work and there is still Development in it.
Delphi was the first language I learned and I always thought it was a teaching language. What are you developing in delphi?
Full blown ERP with postgresql. Its super fast, as it is a native windows application and thats what our customers like about it. Also with nice components like devexpress its looking realy good, has high usability and is easy to develop.
I'd love to see some screenshots of a modern Delphi application if that's possible?
I'm pretty sure Altium Designer, a decent and pricey EDA suite, is written mostly in Delphi.
Yeah, I believe it is too. And its primary scripting language is still DelphiScript. We use it daily at my company.
Last I knew Skype (at least for Windows) was written in Delphi. That knowledge is a few years old so may no longer be true post-Microsoft acquisition.
I develop medical systems for windows with Delphi. Also develop a graphical system in my spare time for the Mac with Delphi
(comment deleted)
Though I haven't used it in the last few months, Fortran (2003) is my language of choice for math related stuff. I used it for the first time about 5 years ago and haven't found a single reason why I'd use C/C++ instead. It offers OOP, dynamic memory, easy array manipulation, no need for pointers, ...
Java and XML.

I'm an Android developer.

Me too. Not exactly state-of-art technologies... I really hope Kotlin would catch on.
Not quite "unfashionable" give it's the most used language by many a metric
Popular isn't necessarily the same thing as fashionable.
Though being mainstream, C++ and Java are probably the most "boring" languages you can use LOL.
I really like PHP.
Why? What else have you used?
Why not? It's your fault, if you don't revisit PHP. A lot has changed since PHP3 days. The same goes for MySQL. Yet there are a lot of ill-informed haters. But to each their own.

PHP is the dynamic language cousin of C/C++. PHP7 and HHVM are both great and fast.

While I'm not particularly keen on revisiting PHP, given the bad taste it left me with, I can believe the language has changed for the better, however programming cultures, like all cultures, are way more inertia-prone, and I'm really sceptical about whether the duct tape mentality I've known PHP devs to exhibit has gone away
Duct-tape mentality remains. Just poke around the mailing list.
That's what I suspected. That's why I steer clear of PHP codebases
>It's your fault, if you don't revisit PHP. A lot has changed since PHP3 days.

Wow. You make a lot of assumptions. I have been using Php until quite recently. I left it at 5.6. I know about 7. But that it is getting additions does not compensate for the fact that the core remains rotten, and is still as dangerous and weird as ever.

It does not get much worse than this

https://www.reddit.com/r/lolphp/comments/41fqib/it_is_so_sim...

On a daily basis I work with C and a handful of modern languages. I do reach for Common Lisp and Prolog sometimes, so I'll count those.

Common Lisp: many, many great things about lisp have been incorporated into modern languages, but CL still has a few unique things, and the whole is greater than the sum of the parts. It's less compelling than it was 10 years ago but some problems are expressed more wonderfully in CL than in other languages.

Prolog: when prolog is the right tool then you really can't beat it. You can substitute another logic language if you like, but I'm not aware of major borrowing by modern languages (the odd library doesn't really count).

Things I'm meaning to look into but haven't really used yet:

Forth: I poke at forth every couple of years but always leave off to go do something else. I'm pretty sure I don't want to embrace the whole forth philosophy thing (real forth people are, uh, different), and doing a deep dive to see what I can take with me is something I just haven't made time for.

Snobol: I'm not really interested in Snobol wholesale at this point, but I want to explore how PEGs are used. There are PEG libraries available for other languages and I'd like to see how much benefit there is to be had from working in a modern language with a PEG library.

Although I haven't used Forth much since the time I implemented it on the Atari ST and used it for a commercial product, I still like to think of myself as a Forth person. It continues to influence aspects of my programming style, and can be useful to have known well in surprising circumstances; for example, I once solved a thorny problem in some rather complex XSLT by adapting a technique that's quite normal in Forth, but would be considered downright weird in many other languages. So yes, we are different :-)
> I once solved a thorny problem in some rather complex XSLT by adapting a technique that's quite normal in Forth, but would be considered downright weird in many other languages.

I'd be very curious to know what technique you used here.

I sort of stumbled on Forth recently, and while I'm not sure if I'll do anything with it, I like the philosophy behind it more than any other language I've yet found.

I use Perl for small things. I learned it about two years ago. I like to joke that I was 15 years late learning Perl, and 5 years early learning Go.
(comment deleted)
I use Common Lisp since 2 years now. Here in germany, most of the students I talked to classify CL as an unfashionable language..

I really like the simplicity of the syntax and the stability of the language standard/specification. There will be no "Today we released v2, all your v1 code will be invalid"

I'm curious since you mention students, so you may be one as well, but is Common Lisp just a hobby language for you, or do you have a productive professional use?
I'm a computer science student and will get my master's degree in October. At the moment Common Lisp is just my hobby language, but also the language in which I invest most of my programming/learning time. I'm able to program in a few mainstream languages, but I want to have one language where I really know the tricks and shortcuts.

Getting a job here, where I would program in CL is nearly impossible. So in the next years I'll probably stay in the academic world and then try to do my own startup or freelance stuff with the language(s) I like.

I used to code a lot in Fortran 77, but have been programming in Modern Fortran for the past few years (90, 95... I am not sure which standard I am adhering to exactly). Both of these would also be considered fashionable.

Part of the reason is that it's still the standard in chemical engineering and atmospheric science for large-scale simulation where speed is an issue. There has been a long history of compiler optimization and numerical precision before standardized by ANSI C, and because of the simpler language (than C) it's easier for scientists to code up fast programs. Also because of its 1-indexing and array notation, it's easier to translate math to Fortran code.

With the recent module system there is a feature like Python's 'from module import *' ('use module') which when people use will make the code hard to follow, but when invoked with 'use module, only: var1, var2, foo, bar', you can make what's imported more explicit.

Other than that, I just find it very aesthetically pleasing and makes me very happy to code in it. I have heard they have string processing "as good as C" now, but I haven't used it much.

I've been learning Fortran 90/95 for my computational astrophysics class the language is proving itself to be surprisingly modern and capable, there's a lot of quirks coming from C/C++, but it's mostly syntax preferences and nothing huge. It's also very good at most of the problems that I've thrown at it which require high performance computing of some sort.
I use MATLAB for 90% of my day to day work, for a combination of reasons -

1. Historically it is what has been used at my firm. We have a lot of code already written in MATLAB, interfaces to internal apis, external data providers etc. Everyone at the firm understands MATLAB code.

2. It really is very good for numerical work - both in terms of speed, and clarity of the code (much better than Python and R for clarity - probably on a par with Julia).

I also used KDB+/Q very heavily in a previous job. It is blazing fast in its domain (financial time series) and enables extremely rapid prototyping. The fact that it is a combined query language / programming language is very appealing for data-focused research. I wouldn't want to write a production system in it though (though I know people who have done!)

When I was studying for my PhD I wrote a lot of Fortran 77 and IDL. Essentially because my supervisor used IDL and had a lot of code written in it, and because we were using an external tool which consumed Fortran 77 files as input.

I also dislike R. Strongly disagree with the argument that MATLAB is more clear than python, however. Have you tried numpy/scipy? Provides excellent performance and very powerful APIs.

I've always found that a great deal of the MATLAB syntax does not gel with my expectations. "./" vs. "./" for instance, or using parenthesis to address array elements.

I would much prefer a truely OO capable language, as well.

> "./" vs. "./"

Is this a typo? It seems like two instances of the same thing

Ooops -- './' vs. '/.', e.g. right array vs. left array
It doesn't seem like '/.' is an operator. Do you mean './' vs '.\'? That tends to match my intuition.
The real killer about Matlab is its one-based indexing. Makes arithmetic index operations look ugly (and unintuitive) as hell.
It's one-based indexing combined with length being defined as the number of entries, IIRC. There was a blog post on here a few months ago that explained how you need one or the other to minimize the need for off-by-one corrections, though I can't find it now.

I'd consider switching to numpy (ignoring organizational issues) if it didn't take 10 times as many characters to describe all the operations. If you're spending you're day doing algorithm development having least squares and every linear algebra operation you can think of as first class citizens is a huge plus for readability and code cleanliness.

Pascal - still use it today.

I started with Turbo Pascal 3.0, went through several versions of Borland Pascal (DOS and Windows), several versions of Delphi, and am now using FreePascal. I do most of my work on Linux nowadays. Pascal still gets everything done and after 20+ years I am really fluent in it.

Did a little bit of Javascript with Node.js, also looked at Golang. While I did like some aspects of them, there was always the downside of having to learn a whole new ecosystem. Didn't like the dynamic typing of Javascript. Golang looked better for what I do. But in the end I just wrote a small Pascal library to emulate Golang's channels which I really liked, and that was the end of Golang for me.

My main project in Pascal is a search-engine: https://deusu.org

Sourcecode for that is on GitHub: https://github.com/MichaelSchoebel/DeuSu

So yes, you can (still) do actual stuff in Pascal. Even pretty cutting-edge stuff.

Holy cow! I used Pascal at college about 16 years ago. I never thought I would see it used in the wild, especially in 2016. Nicely done!
I taught Pascal at college about 40 years ago.
Pretty cool! Ever tried Modula-2 or -3?
Did some baby steps in Modula-2 out of curiosity about 20 years back. The IDE and compiler were horrible compared to Turbo Pascal, so I went right back to that.
This is pretty awesome, and the speed is great.

I really like obscure search engines, because their (comparatively :P) tiny indexes mean they'll return significantly different data to what say Google or Bing would return.

If there's one thing I would definitely vote for with projects like this, it would be... an API. In a world where virtually every search engine is built around the idea of controlled, non-open access, a predictable, documented, free-access API would be a breath of fresh air, and you'd get people seriously interested in this even though it's small. (You'd definitely need rate limiting though because there are always the hangers-on with the low IQs. >.>)

You may (or may not, no idea) also be interested in the Common Crawl corpus - http://commoncrawl.org/the-data/get-started/ - which contains a few PB of data collected over the past few years (sorted by date). Access via HTTP is only moderately involved: read the line immediately above "Data Format."

Of potentially much less interest is a 2012 80TB snapshot Archive.org are (or at least were) offering; they seeded their indexer off the Alexa top 1m and boxed up whatever came back. https://blog.archive.org/2012/10/26/80-terabytes-of-archived... It's apparently only available on request (I expect a search engine would definitely qualify, especially considering the data is 4 years old now - although I definitely want to see what's in it if you do decide to chase this - serious :P).

Finally, search for "cheetah project" - the 2nd top-level result is messed up, apparently the Chinese (Unicode) got scrambled. Just a note; you may already be aware.

Non-ASCII characters are indeed messed up. I wrote the HTML parser about 15 years ago. Back then my programming style was so bad, that I'm afraid nothing short of a complete rewrite of the parser will be able to fix it.

The Archive.org crawl is way too old to be of any use. And the Common Crawl corpus is only marginally bigger than what I crawl myself. There are currently just over a billion pages in my search-index and I can recrawl those about every 45 days. Crawling right now at about 650 URLs/s and 200mbit/s. Only about 60% of URLs crawled end up in the index. The rest are errors, timeouts, redirections, etc.

I have been thinking about adding an API. Rate-limiting is a must as you have correctly said. I'm already in a constant fight against SEOs who try to scrape the search-results and who apparently don't see anything wrong in making 10+ queries/second.

The API would definitly not be a free-for-all, but sort of like "send me an email and describe what you want to do with it, and if I like it I'll give you an API-key".Together with a primer on what not to do of course. :) If people do more than a certain amount of queries/month, then it would also have to be paid.

Ah. 2001 was... the era of HTML4, more or less, so it's still seeing the majority of page content, but... no Unicode. I see.

I figured old data was uninteresting, but TIL that crawling data yourself is not impossible. I mean you have to come from a thousand IPs, browser UAs and cookie states to find everything thanks to the sad state of things, but wow, I didn't realize 200Mbps could do so much. :D

As for the errors et al, that makes me think, I wonder what would happen if you saved that data and folded it into the seed list (with maybe 3 or 4 URLs up the chain leading to that link) for next run.

Agh... so you already have leeches :( all I can suggest is maybe mixing up your HTML, or maybe putting your site behind CloudFlare and, if there's an option(?), setting the attack detection sensitivity all the way up. I have no idea if they can do this.

10 queries/sec... that's... not going to be too great on the hardware :( I mean these people can just crawl the info themselves and get more up-to-date content >.< wow

By "free" API, I don't mean free-for-all "here, have at my bandwidth/hardware" xD - I mean more in the sense that curious shy CompSci types can maybe autogenerate themselves a basic-level API key to experiment with ultra-low-rate data requests here and there, and request a threshold upgrade on their key if they think they have an interesting/justifiable use for the data.

And it would be great if this could be a revenue stream! Have you ever featured the site on HN?

PS. Constructive criticism on design: I would personally alter the page styles a little; it currently looks nothing like what I associate "search engine homepage" with based on what I've learned to expect, and the page style, while very nice, makes me feel like the site is a blog, not a dynamic search engine.

Crawler works from a single IP. User-Agent is fixed to the robot's UA. Cookies are totally ignored. The search-engine works with just 2 servers. Crawler/Indexer and Webserver/queries. Crawler is a root-server with 1gbit/s connection hosted in a datacenter. Webserver sits here at home with 200mbit downstream and 20mbit upstream.

I use the Alexa top-1-million sites as seed-list for the crawler. The errors that do appear during the crawl are either sites that have an outage or more likely simply dead-links. Oh, and URLs that turn out to be blocked by robots.txt. There are a lot of sites out there which block anything but Google and Bing from crawling them.

Cloudflare is not an option for me. It would let Cloudflare know what my users are searching for. VERY big no-no. :)

I can filter out 99% of automated queries. Luckily they are still pretty dumb at the moment and give me enough fixed clues to identify them.

I like your idea of keeping the API free with a very low request-rate. That could work. I would have to find a way that they can't just generate many API-keys though. Using captchas for API-key requests won't stop them from doing that.

I posted a "Show HN" about a year ago. Brought in about 1500 extra visitor that day. Got up to 9th place on the HN homepage that day.

New webdesign is already done. I have a German site too. https://deusu.de which actually gets 90% of traffic. That site already has the new design.

1Gbps for the crawler totally explains it: I can see that doing 600 URL/sec. xD

I didn't think of CloudFlare being able to see the traffic... and wow, I never even processed that aspect of their service. But of course...

How good is Google's "[ ] I'm not a robot" checkbox thingy at weeding out bots? And perhaps you could use multiple captcha systems...? (Or are actual people tasked to do signups?!)

I shudder to think of such an idea, but linking API keys to <popular login-with/connect-with-this-site API> may be an alternative. (One thing that comes to mind is that, if someone authenticates using Reddit - which they can do without releasing any account info - is that you could check their (public, but unfakeable) karma counts and use that as a measure of confidence, in addition to the standard account age metric used everywhere.)

The new design is nice :D

And if it's been a year (!), another Show HN sometime would certainly be fine.

The WHATWG had a spec-compliant HTML parser in Pascal, if you need it. Currently only does UTF-8 but that should be fixable.
I use C as my staple language. Old and always unfashionable, but it's never going away in my lifetime.
Unfashionable? Maybe in the sense that tshirts and jeans are unfashionable, but are worn daily by most of the people who make our technological civilization possible.