He's right. I have versions of "cat" in Java, Perl, Python, and C, all source. Reading from stdin and writing to stdout is the skeleton of a lot of tools, so starting from "cat" source is often the best way to get going on a new tool.
This is kind of funny since I taught myself a lot of go by porting a bunch of the coreutils. It still amazes me how much there is to learn from unix. Many many times I find myself going back to some programs src (cron, find, etc.) when I need the design/algorithm/feature in my own programs.
Rodrigo, I thought you should know that when browsing your site on an iPhone 4, the header overlapped the index. That wasn't a great description, so here is a screenshot: http://db.tt/7grTPHsL
You also cannot zoom in, which made it very hard to read.
I always implement "cat" as a first toy application when learning a new langauge - it's a great introduction tool as it deals with dealing with files, dealing with streams (stdin) and so on.
I agree with this and have done it in the past myself. Its a great learning exercise for new languages. The problems are easily defined and its apparent when you've successfully completed. Once you've done with the easier ones try implementing grep and then diff.
This is a brilliant idea, and may I suggest that one takes it slightly further than just implementing the core functionality, and aiming for a complete clone (optional parameters and all) with a few enhancements.
Indeed, for a further ego boost, why not also benchmark the performance of your versions against the performance of the native utilities? You never know, your new clone might end up being the new 'less' to the old 'more'!
Yes. I think I nice approach would be just cloning core functionality of the program initially in your language of choice, then dealing with parameters and more advanced options as you learn that language better.
My favourite way to learn new languages & platforms was implementing a du-like ('disk usage') tool. It was also the exercise I proposed to my students. It doesn't require complex algorithms but touches a lot of the basics: recursion, filesystems, command line parsing, output formatting, etc.
I am taking Operating Systems at Auburn University, and my professor had us work on the Linux kernel source, learn how to define and implement our own system calls using the SYSCALL_DEFINE[0-6] macro and then implement our own memory snapshot tool. It is similar to /proc/<pid>/statm. We had to get other info on major and minor page faults as well. I learn a lot doing things like this.
The example code for cat is only half of cat, which reminds me of Rob Pike's criticism of "cat -v"---Unix programs are often more complicated than they arguably should be, are "reimplementing" Unix programs often means implementing a small subset of the features.
Or, look up the manpage for your locally installed "tree" and see how many non-standard options and features have been bolted on.
I think it would be helpful when reimplementing Unix to try to work in more core features of the language, like its object or module system, which are more important than parsing argv or touching the filesystem.
Even if you ignore all of the bolted on options, his simple implementation of cat doesn't even handle "cat -" so I would say less than half. I think this is still a good exercise if you don't dig deep into the plumbing of all the tools you're reimplementing, but it's even better if you do.
For those who want a very clear, concise set of userland code to try translating for practice, consider using 9base: http://tools.suckless.org/9base
It's not quite Unix, but it's still quite lovely and it's rather succinct:
"It also contains the Plan 9 libc, libbio, libregexp, libfmt and libutf. The overall SLOC is about 66kSLOC, so this userland + all libs is much smaller than, e.g. bash (duh!)."
Windows doesn't have many of the basic Unix tools, but you can still implement them using your programming language of choice (how great). No, you're not limited to these three languages. I only meant in the article that these three are probably the easier to copy Unix with, for their scripting nature.
One of the simplifying assumptions that is built into this idea is that the input and output to the program are simple streams ( a file, a terminal ). Not surprisingly that makes things a lot less complex so you can focus on the programming.
Since a Raspberry Pi is only $35 or $50 with enough stuff to program it, that is one way to get started. Of course taking an older tossed of PC and installing Linux on it works too (which can often be done for free if there are businesses around)
You can download complete Linux VM images that just work from the net. VirtualBox is free and is installed in 5 minutes. It's hardly frustrating. Certainly less so than getting a RPi up and running, not to mention you end up with an odd, underpowered device secondary to your regular machine.
That's not to say the RPi isn't an interesting, rewarding experience on its own merits -- that's why I got one --, but I wouldn't recommend it to anybody interested primarily in getting a *nix environment for experimentation or programming.
The point is to copy the commands, you can copy them on any operating system. It just happens that Mac & Linux have them built in while Windows doesn't.
As for which languages, this actually seems like it would be useful for learning any language that has reasonable ways of interacting with IO streams. Off the top of my head:
I think, almost by definition, for a language to have any interest, it must be able to deal with I/O streams. I can't think of a language that wouldn't work (I'd be very interested if somebody knows one and why it wouldn't work).
What the scripting languages give you is simplicity. No need to muss with compilers.
As far as the Windows and Mac comment goes, there is no reason why you couldn't do this on either. In fact Mac OS X supports the same commands as linux. In Windows case, I see no reason why you couldn't do one of two things:
1. The first would be implementing your own linux commands so they work on Windows. You would probably want something like cygwin, a linux vm, or an actual linux box to test this stuff on though.
2. Pick Windows commands instead and implement them instead. It’s the same idea roughly. I haven't used a Windows machine in almost a decade, so I'm not sure what all would be involved, but seems possible that you could get them to work roughly similar.
As far as specific languages go, there’s no reason that almost any language couldn’t be used as long as it can take arguments from the command line.
On Windows you can install Linux in a virtual machine, or you can install Cygwin (which is basically an almost complete POSIX system on Windows).
You can use your preferred programming language to implement any of these tools: Ruby, Python, Perl, Scheme, Common Lisp, C, C++ ... The original Unix implementation of these tools was done in C.
For a purely functional language like Haskell, this would not be a very good advice. Any kind of I/O would involve monads and other imperative constructs. Better implement an algorithm involving trees or graphs to better appreciate a functional language.
This is a slightly crude implementation of cat in Haskell. As you can see, it is replete with scary monad plumbing that gets in the way of the actual functionality (copying stdin to stdout).
I would explain further and implement more than what the blog post suggested but there's already a book (Real World Haskell) that does this with a number of other Unix commands.
Not at all. You can start doing I/O in Haskell without knowing anything at all about monads by just treating the do-syntax as an imperative DSL.
In fact, Bryan O'Sullivan (who wrote Real World Haskell) just held a tutorial session on Haskell a couple of weeks ago where people completely new to the language implemented simple Unix tools like "wc". I don't think monads were mentioned at all.
On the contrary, any interesting program is going to have to do some I/O. So if you're want to do useful things with a language, I think it's a good thing to learn the basics of opening, reading and writing files.
Also, you should take a look at the top comment in this thread.
I write a unit testing framework in every new language I learn. I find it's a great workout because making it usable for yourself is immediately assessable, and it often forces you into deeper areas of the language, including reflection and meta-programming.
You might want to have a look at QuickCheck. Trying to port it's techniques to other languages will also give you some insight (and will result in valuable tools).
And make you miss the ability to overload functions just on return type. Same thing happens if you use Haskell's regex library or try t port monads to another language.
Indeed. Overloading on return type is one of the few things that you can't do in dynamic languages by design (and in most statically typed ones, neither, but that's an accident).
Please do! It's very interesting; I'm not good enough in PLT to try and find out the answer for myself (in such a short amount of free time I have lately) but I'm very curious if it's true that you cannot have overloading on return type in unityped languages by principle.
This is not hard to understand: under usual semantics for function calls it is clearly impossible to do return type overloading in a dynamically typed language. But, as I pointed out above, it is perfectly practical to cheat and provide an indication of the desired return type as an extra hidden argument to every function call (this is what Perl does with wantarray: it is a "builtin function" but the effect is really that of an extra boolean parameter to every function call that specifies whether the result should be an array or not).
And you can generalize from wantarray's Boolean to a more general parameter indicating required return type.
It gets harder and harder though, to provide information about more and more distant parts of the program in a dynamically typed (or untyped) environment.
From what I've heard, Racket's contract system is another interesting attempt to provide something that's typically done in a static setting---the benefits of dependent typing---in a dynamic one.
Ok, this much I understand, but I wouldn't call this overloading on return type... It's - clearly - overloading on input type, either implicit (wantarray - I checked the docs, it seems to be a variable set automaticaly depending on context by the interpreter) or explicit, but input type nevertheless.
So, the short answer is - as I thought previously - that no, you can't have implicit dispatch on return type in dynamically typed language.
Contracts are probably getting close, but you still need a version of apply that will get expected return type and a bunch of functions to introspect (or a macro that would do the same, but let's not wander there, as lispy macro systems are equivalent with static type systems anyway).
The advantage and disadvantage of this approach: you have to already know Unix commands. Inevitably all programmers learn Unix commands - but probably a rough approach for beginners. Although, I'm guessing this advice isn't really aimed at beginners anyway.
I feel like this approach might get boring after a couple of languages.
I think most people here could skim through the language intro, and then just start working on a new project in that language (picking up more of the language as they go).
This is the approach I usually use, and it seems more fun than porting the same tools. Just my opinion though :)
Your ruby version of cat implements none of the command line switches.
I learned C by going through the FreeBSD code and helping with POSIX compliance. For fun I would implement a lot of the commands in Python. You get the most out of learning both the language and UNIX by implementing all the command line options.
"Your ruby version of cat implements none of the command line switches."
I don't think he was trying to show off what a feature complete cat command he wrote. Just trying to illustrate his idea. No need to be rude about it.
And you get a better grounding in the Unix philosophy if you omit the command line switches and make small utilities to handle those cases, because as the paper said, cat -v is harmful (http://harmful.cat-v.org/cat-v/unix_prog_design.pdf).
The Unix philosophy is somewhat of a myth. http://cm.bell-labs.com/cm/cs/who/dmr/man12.pdf shows that that same first edition already had a -t option on ls. Surely, that should have been some | sort, but AFAICT, they did not bother to write that until some later time.
Sticking to a philosophy like a zealot makes it a religion. Keeping it as a philosophy means it encourages a certain approach, not that it makes it an absolute unbreakable rule, plus it would only results in bikeshed/flamewar arguments.
as PEP20 says:
Special cases aren't special enough to break the rules.
Although practicality beats purity.
I said "somewhat" for a reason, but I think there are just too many special cases to keep calling them exceptions. I use awk and sed, pipe things into bzip2, but I more often use tar with a z or j option to compress stuff an in a pipeline. We also do not have separate tools for parsing python/ruby/perl and executing it. If code reuse using pipes were successful, wouldn't we have a separate 'exec' tool that is backend to all scripting languages? Looking at gcc with its zillions of options, pipes sometimes look more like an implementation detail than as the primary way to compose tools.
Also, other kinds of reuse of the things mentioned as examples of the Unix philosophy are rare. What languages on your system use lex and yacc? 'but thise tools are outdated' is not a good revuttal; if they are, how has that come about? I think the Unix philosophy is perfect for prototyping, but 'real' stuff tends to need polish that 'the unix philosophy' cannot provide. As a final example, consider git. As I understand , it started life as a set of scripts, but eventually became a C program.
I generally use the -j option to tar primarily because it's less typing. I think the ideal option is for these flags to be provided, but implemented using pipes under the hood so tar -cj folder is just syntactic sugar for tar -c | bzip2. Similarly, tar -cf folder.tar folder/ could be implemented as tar -c folder/ > folder.tar. I think this way you get the conciseness of the flags approach, but also each item of functionality is only implemented once. Just a random thought.
Heh. I've been doing something similar. In my free time, I've been actually re-implementing the `tree` command (with most of the switches as well) in Python.
Great advice. I've written a simple version of the `column` tool as an entry to the International Obfuscated Ruby Code Contest: https://github.com/saizai/iorcc/pull/1
Just a bit of fun, the contest doesn't seem to be very 'official'.
It is a good approach and back say 20 years ago the favorite was to redo the du command and add a conversion to MB (now handled by the -h option for human in the du and ls command).
Cat is also a good example and not just to learn a programming language but the OS. Ask somebody to come up with 20 ways to display a file, you can do it with not just the cat command. Now I'm not saying there are twenty ways, but it is one area which some delving and approach will allow people to try and find them.
Examples are for me the best way to learn a programming language and again with the simple Unix commands you have a common base which people are more likely to ifnd an example. Can't recall but great site on the net which has "Hello World" in about every programming language around.
After a while, you will start to add option to your redeveloped commands, then add entirely new commands and from there you can think about writting your own shell. No graphics or the like to distract you too much. Graphics as a rule in programming languages I have found to be like learning an after thought and also you are more controlled in the mentality behind the API. So I do advise not even looking at graphics for a while until you have learned how to flex the language without the distractions graphics and the way they are handled add a overhead to your code. I'm sure somebody will list a programming language that is easier to do graphicly as apposed to text output, though its a safe rule. SO just because you have windows, don't mean you have to jump into GUI's from day one, if even at all.
I remember that hello world site, I think it fell off the internet but there are still many of those types of sites. One of my favs was the 99 bottles song in many languages. There are also larger projects like PLEAC and Rosetta code that are similar.
Earlier this year, I did 21 different implementations of a hex dump program in C (starting here: http://boston.conman.org/2012/01/09.1). I also included implementations in a few other languages just for comparison. Why? It was a neat break from the programming I do for work.
Shouldn't this be doable with simple library wrapping console framebuffer, that you Linux guys have and I suffer lack of on FreeBSD? (I can be very wrong - I'm just a bit jealous, not jealous enough to dive into fb implementation :))
109 comments
[ 8.3 ms ] story [ 328 ms ] threadYou also cannot zoom in, which made it very hard to read.
I'll change it. Thank you very much for reporting the problem. It now allows you to zoom in. I'll take more time to fix the header, tho.
Thanks man.
Indeed, for a further ego boost, why not also benchmark the performance of your versions against the performance of the native utilities? You never know, your new clone might end up being the new 'less' to the old 'more'!
Or, look up the manpage for your locally installed "tree" and see how many non-standard options and features have been bolted on.
I think it would be helpful when reimplementing Unix to try to work in more core features of the language, like its object or module system, which are more important than parsing argv or touching the filesystem.
It's not quite Unix, but it's still quite lovely and it's rather succinct:
"It also contains the Plan 9 libc, libbio, libregexp, libfmt and libutf. The overall SLOC is about 66kSLOC, so this userland + all libs is much smaller than, e.g. bash (duh!)."
For instance, what are your options on Windows or Mac? And are the languages you can do this with limited to Ruby, Python, or Javascript?
Windows doesn't have many of the basic Unix tools, but you can still implement them using your programming language of choice (how great). No, you're not limited to these three languages. I only meant in the article that these three are probably the easier to copy Unix with, for their scripting nature.
Since a Raspberry Pi is only $35 or $50 with enough stuff to program it, that is one way to get started. Of course taking an older tossed of PC and installing Linux on it works too (which can often be done for free if there are businesses around)
That's not to say the RPi isn't an interesting, rewarding experience on its own merits -- that's why I got one --, but I wouldn't recommend it to anybody interested primarily in getting a *nix environment for experimentation or programming.
As for which languages, this actually seems like it would be useful for learning any language that has reasonable ways of interacting with IO streams. Off the top of my head:
Ruby, Python, Javascript (node.js), Perl, Java, C#, Scala, Clojure, C, C++, Go, etc.
What the scripting languages give you is simplicity. No need to muss with compilers.
1. The first would be implementing your own linux commands so they work on Windows. You would probably want something like cygwin, a linux vm, or an actual linux box to test this stuff on though.
2. Pick Windows commands instead and implement them instead. It’s the same idea roughly. I haven't used a Windows machine in almost a decade, so I'm not sure what all would be involved, but seems possible that you could get them to work roughly similar.
As far as specific languages go, there’s no reason that almost any language couldn’t be used as long as it can take arguments from the command line.
On Windows you can install Linux in a virtual machine, or you can install Cygwin (which is basically an almost complete POSIX system on Windows).
You can use your preferred programming language to implement any of these tools: Ruby, Python, Perl, Scheme, Common Lisp, C, C++ ... The original Unix implementation of these tools was done in C.
I would explain further and implement more than what the blog post suggested but there's already a book (Real World Haskell) that does this with a number of other Unix commands.
I just made a few more suggestions. I can't count how many neat, small tools are provided by Unix. It's a large ground to play in!
In fact, Bryan O'Sullivan (who wrote Real World Haskell) just held a tutorial session on Haskell a couple of weeks ago where people completely new to the language implemented simple Unix tools like "wc". I don't think monads were mentioned at all.
With "do" and the coincidental naming of "return", and IORefs if you insist, you can write imperative bottom-up code in Haskell.
http://ocamlunix.forge.ocamlcore.org/
Also, you should take a look at the top comment in this thread.
It gets harder and harder though, to provide information about more and more distant parts of the program in a dynamically typed (or untyped) environment.
From what I've heard, Racket's contract system is another interesting attempt to provide something that's typically done in a static setting---the benefits of dependent typing---in a dynamic one.
So, the short answer is - as I thought previously - that no, you can't have implicit dispatch on return type in dynamically typed language.
Contracts are probably getting close, but you still need a version of apply that will get expected return type and a bunch of functions to introspect (or a macro that would do the same, but let's not wander there, as lispy macro systems are equivalent with static type systems anyway).
I think most people here could skim through the language intro, and then just start working on a new project in that language (picking up more of the language as they go).
This is the approach I usually use, and it seems more fun than porting the same tools. Just my opinion though :)
I learned C by going through the FreeBSD code and helping with POSIX compliance. For fun I would implement a lot of the commands in Python. You get the most out of learning both the language and UNIX by implementing all the command line options.
Facts can be stated in a rude manner; rude statements can be factually correct.
Asshole.
I look at the options for cat on my ubuntu system though, and realise that this problem has only gotten worse.
as PEP20 says:
Also, other kinds of reuse of the things mentioned as examples of the Unix philosophy are rare. What languages on your system use lex and yacc? 'but thise tools are outdated' is not a good revuttal; if they are, how has that come about? I think the Unix philosophy is perfect for prototyping, but 'real' stuff tends to need polish that 'the unix philosophy' cannot provide. As a final example, consider git. As I understand , it started life as a set of scripts, but eventually became a C program.
I'll also be writing some other utilities! :)
Share the code with me if you feel like it. My GitHub is https://github.com/rodrigoalvesvieira and my email is rodrigovieira1994 [at] gmail [dot] com
Just a bit of fun, the contest doesn't seem to be very 'official'.
Cat is also a good example and not just to learn a programming language but the OS. Ask somebody to come up with 20 ways to display a file, you can do it with not just the cat command. Now I'm not saying there are twenty ways, but it is one area which some delving and approach will allow people to try and find them.
Examples are for me the best way to learn a programming language and again with the simple Unix commands you have a common base which people are more likely to ifnd an example. Can't recall but great site on the net which has "Hello World" in about every programming language around.
After a while, you will start to add option to your redeveloped commands, then add entirely new commands and from there you can think about writting your own shell. No graphics or the like to distract you too much. Graphics as a rule in programming languages I have found to be like learning an after thought and also you are more controlled in the mentality behind the API. So I do advise not even looking at graphics for a while until you have learned how to flex the language without the distractions graphics and the way they are handled add a overhead to your code. I'm sure somebody will list a programming language that is easier to do graphicly as apposed to text output, though its a safe rule. SO just because you have windows, don't mean you have to jump into GUI's from day one, if even at all.
http://99-bottles-of-beer.net/ http://pleac.sourceforge.net/ http://rosettacode.org/wiki/Main_Page
I honestly think that plotting per-pixel graphics is a much more fun and rewarding way to learn programming, than a cat program.
It's only a shame Linux has no "mode 13h" screen and a PSET function in the C language :)