Doh, I totally should have mentioned that, cause yeah that was big too. We went from about 80-95mb resident memory (and our cgroups config doesn't allow for much swap) to about 50-60mb on average, which is a metric we weren't even trying to get lower. So, yeah, that's another win we should have mentioned.
Switching to Go though they were able to get 4MB/s. Even though they claimed to do "line-by-line" translation, I wonder how much of that speed-up was language/libraries vs rearchitecting.
I didn't mean to imply they were lying in the post, more that it's probably hard to do a line-by-line translation from Python to Go. The nature of having to convert to things like channels in Go might help to re-architect the program even at a micro-level.
We had about 3 or 4 guys full time. On my silly little i3 laptop, the whole codebase builds in 18 seconds - I suspect most of that is our C extensions.
Yeah, we actually have a few cgo packages. We have OpenSSL bindings (we need to use our hardware accelerated crypto, go's crypto/tls lib doesn't support that (yet)), a faster CRC thing, we needed monotonic clocks and couldn't wait for Go 1.3, etc.
We'll be releasing a bunch of these. Particularly the OpenSSL bindings, which even without hardware acceleration are faster than Go's TLS library, and despite Heartbleed, OpenSSL is not vulnerable to timing attacks like crypto/tls is.
I wonder how many lines of code they ended up with afterward, although admittedly lines of code is a pretty silly metric. I think it is useful in orders of magnitude when describing the size of a project, 100 lines, 1,000 lines, 10,000 lines, 100,000 lines all speak volumes about effort. So I guess what I'm wondering if you got about a roughly similar sized code base or significant change difference?
Actually in this case, comparing two languages doing exactly the same thing, LOC is a relevant measure (its a crap measure of programmer productivity). It could show the expressiveness of a language. Also I am a believer that less lines of code = less likely to have bugs.
Less code in general means fewer bugs. However, there's definitely an inflection point where you're trying to cram too much functionality into too few lines of code, and so the code you're writing is a lot more complex than it really needs to be. This is where Go shines, is in discouraging overly complex single lines of code that don't actually save the programmer any time.
Sure, python has list comprehensions, but I can tell you how often I've seen hugely over-complicated one line list comprehensions that were impossible to understand unless you were absolutely sure what it was supposed to be doing before you read the code. I've refactored some of those in my lifetime just to make them more readable. That's not to say that most simple list comprehensions aren't totally fine.
So, just saying fewer lines of code makes for fewer bugs is not 100% accurate. I'd say, less /functionality/ leads to fewer bugs. But except in extreme circumstances (like writing a whole framework to do one small task), actual lines of code does not correlate to the number of bugs.
We rewrote Twisted Deferred handling to be 30% faster.
We optimized the Twisted Reactor event loop to not only do less constant work, but to prefer incoming I/O.
...We spent long nights poring over profiling readouts and traces, wrote our own monitoring system, and wrote our own benchmarking tools
... We enabled cgroups on our Python process to tightly control memory usage and swap contention without requiring a different Python memory allocator scheme.
This is very interesting for the rest of the Python/Twisted community - now that they are not using this (and so is'nt part of the secret sauce), I wonder if they are willing to push this out as a patch/blog post ?
While we're planning on open sourcing some things, I'm not sure we'll ever open source that one. Even though it was much faster, it didn't see much time in production due to the less than desirable impact it made (go Amdahl's law), and it still has some bugs and was highly customized to our specific protocol buffer definitions.
No, I think we're honestly even more sold on protocol buffers' wire format than before.
That said, if I did want to go looking elsewhere, I think the main serialization format I am really interested in is Cap'n Proto: http://kentonv.github.io/capnproto/
Well it is kind of obvious that a compiled language is going to be faster than an interpreted one, especially the way how these interpreters work (https://wiki.python.org/moin/GlobalInterpreterLock). You can fool yourself with Twisted (or in Ruby with EventMachine, Goliath etc.) but it gets so just a bit ahead. The surprising fact for me is that Go is not as much faster. I was expecting a bigger gap in the performance between Go and Python. Understanding where your bottlenecks are is crucial and it is not super hard in Go. https://www.datadoghq.com/2014/04/go-performance-tales/ I guess the SpaceMonkey guys might further improve the performance just by doing a thorough analysis on their code.
The path they chose was much more predictable than your suggestion. I.e., after their team spent three days at it, their estimate for the total translation was probably within 50% of the month it actually took them. I don't expect they could have been as accurate in estimating an entirely new implementation.
Right on. We immediately had not totally horrible estimates for how long it would take.
Further, we have always really been inspired by Joel Spolsky's article on rewrites: http://www.joelonsoftware.com/articles/fog0000000069.html
In fact, I can't help but wonder if we would have attempted a Go rewrite sooner if not for that article.
I've got a project I'm porting out of Erlang to Go. It's not a transliteration, since it can't be, but by design it's a drop in replacement, modulo a smidge of configuration for each (dozen lines of config, tops). As much as I'd like to GO WILD AND FIX ALL THE THINGS!, it's advantageous to be able to switch back and forth freely during QA and early deployment, because you need a smooth transition.
I was expecting a bigger gap in the performance between Go and Python.
Perhaps we've come a pretty long ways in the interpreted languages department? Or, that the cost of interpreting is vastly less than speed losses due to IO or memory access?
"Interpreted is always slower than compiled lol right guise?" was a tired refrain ten years ago, much less today.
They make it pretty damn fast. But non-JIT runtimes have strict limitations on how fast they can go, unless you design your language specifically to optimize for interpreter speed (like lua).
PyPy is probably your best bet if you want a faster python. The official CPython will never have speed as a primary focus due to a number of self imposed limitations like:
Since it's a reference implementation it should be easy read and learn from.
They're not really willing to accept patches that speed up some things if they at the same time slow down other things (this has been the main problem with all the GIL removal patches that have shown up over the years).
They're not willing to accept patches that break any existing code or libraries.
PyPy on the other hand have non of these limitations and happily break all three, making it great for a subset of python code out there.
Python has benefited from years of throughout optimization of the language for real-world use. Go is still under heavy development and is not as optimized as other languages that have devs focused on optimization (that explains why node.js is a bit faster than Go for handling HTTP requests at the moment).
If I remember correctly the initial versions of node had the ragel-based http parser from mongrel but then ryan rewrote it. http-parser is hand-written.
> it is kind of obvious that a compiled language is going to be faster than an interpreted one
While it may be obvious, the interesting question is why exactly. There's been a good presentation from Alex Gaynor [0] on this topic. TL;DR: Dynamic languages ar slow not because they can't be optimized, but because developers make a worse use of memory and have sucky algorithms, inducing way more mem allocation and copy than needed. If you can be smart about those, then your dynamic language can match static languages.
There are certain things that you can't really avoid using an interpreted language. How would you go around Ruby's GIL for example when you specifically need multiple threads (not talking about green threads). I agree that some of the stuff is on the developers developing libraries and performance is not kept in mind but there are hard limits you can't really do anything about.
Except for the fact there are no 'interpreted' or 'compiled' languages. What we have are implementations.
This is not merely nitpicking. Any turing-complete language can be interpreted or compiled. Language specs tend to make one side easier, but that doesn't mean that Python cannot be compiled (and it is actually compiled to bytecode). You can have something similar to a GIL in a compiled language too.
One of the things that makes it easier to increase performance is having type declarations. This makes it much easier for the compiler to reason about the code, which can lead to increased efficiency. But optional type annotations can accomplish just as much.
That's actually kind of shocking. I thought it would take way more work to port from Python to Go, and a 400% speedup is way less than I'd have expected (though this is string handling, and the Python was already brutally optimised with C modules where it counted).
This article doesn't address one of the biggest benefits I've noticed in switching from Python to Go.
I've been using Go for almost two years now. Like OP, I am/was primarily a Python developer. Like OP, my first Go project was a time-sensitive rewrite[0] of a project from Python (tornado) to Go.
Even though I was an experienced Python developer, the Go rewrite was marginally (~20%) faster[1]. But the real benefit came from the subsequent development - refactoring, rearchitecting, and maintaining the project on an ongoing basis. Go was designed to make it easy to scale the maintenance[2] of software, and on this one axis, it absolutely blows every other language and environment I've used out of the water.
For a fresh project, I'd say Go is about 10% slower to write than the equivalent Python[3] for the average case. But the time/cost savings are very quick to come thereafter.
[0] I would absolutely not recommend doing time-sensitive rewrites in general, but that decision was a separate matter.
[1] Some of this is due to the nature of rewrites in general, but the fact that it wasn't slower to use a language I'd never used before says something about the language.
[2] Scaling software development as teams grow is very different from scaling software as users grow.
[3] Assuming comparable familiarity with both languages, which is rarely the case, of course
- super fast compile times for fast developer iterations
- you can create an interface (set of methods) that your module owns, and apply it to objects created by other modules without recompiling. If you only need one method you can take that one. It encourages decoupling you from your dependencies.
- the code has one correct formatting convention and a tool that will auto-format your code
- many complex rules for numeric conversions that are implicit in C and cause no end of trouble are explicit and much simpler in Go.
- treating concurrency as a series of sequential process connected with channels makes it MUCH easier to reason about.
Don't the first 3 apply just as well to python? It compiles fast, and is duck-typed so interfaces are always there and implicit, and the indent-based blocking ends many arguments about formatting.
If you cut and paste probably the misaligned indentation is the least problem. Cut and paste should be avoided if it's not done during a refactoring (so actual moving of code for better design).
Beginners usually think Python strong indentation is a weakness of the language. I actually find C++ freedom being more error prone:
if (a < b);
a = b
It's a not very frequent bug, but when it happens it takes you hours to spot. ;)
If you ran the code through gofmt, it would remove the () around the cond for you. I code go in SublimeText with GoSublime. On every save it runs the file through gofmt and reformats it for me. Keeps my code looking pretty with very little effort.
it's generous to call the CPython interpreter a VM - the binary encoding of Python isn't some crazy IL bytecode, its' really just python-as-binary. Language constructs converted into opcodes, strings with pre-calculated hashes, and local variables within a scope become a sort of vector... but otherwise, it's a prettymuch 1:1 mapping between Python language constructs and the bytecode form.
I thought you had meant that Go was "easier to maintain" than python - most of these don't apply to Python: Python has faster compilation times (0); interfaces without recompiling; formatting conventions; and numeric conversions is compared explicitly with C. Channels is the only one that applies to python.
Maybe he meant that having a typed language that is checked at compilation time is a huge plus for maintenance, and the usual downsides that come with compilation (mainly slowness) are pretty much inexistant with Go.
As some who generally prefers Python to Go quite strongly, the formatting situation is no where NEAR equivalent. With go it's trivial to setup your editor to perfectly reformat every time you save, or even on carriage return, and it does in a way that is 100% consistent and will never alter the logic of your code.
I'm not a python expert by any measure, but I've written a couple small projects with it. Python surprises for how well documented it is. In this specific case, PEP8 lays down most formatting conventions, so you'd expect some tooling to be available. Is this not the case?
The thing is, while this kind of thing is available for pretty much every language, it is standard for Go. As in, if you don't gofmt your code, everyone who looks at it will bug you to format it. That's a huge difference. There's basically zero code in the wild that you'd ever want to use that /isn't/ gofmted.
Exactly the point. Go is a VERY opinionated language. You see the term "idiomatic Go" used very regularly. The Go team and community push hard for everyone to follow a set way of doing things. This means all Go code must be formatted a certain way, handle errors in a certain way, etc. People even put pre-commit hooks into their VCS system that will reject .go commits unless they are properly formatted before commit.
You are incorrect about compile times in many ways. First- python byte compiles to a weird VM. Then it executes that.
Second- go compilation times are comparable to python byte compilation time but the result is a native executable.
I also rewrote a tornado service in Go. The thing I found most surprising was that the length of the program was about the same as the Python version. I had expected it to be quite a bit longer.
I'm curious, how does it compare to Java with a good IDE (eg. Eclipse or IntelliJ)?
One thing I really missed when I abandoned the Java world and went to C++/Python was the refactoring and formatting tools available in the IDE. I could set it up so it'd auto-format my code on save, so that I could rename methods with a single keystroke, so that I could pull out classes or add getters/setters, so that I could add parameters, etc. I know Go supports some of that through gofmt and go fix, but I'm curious if anyone is both an experienced Java dev and an experienced Go dev and can compare the continued maintenance costs of the two.
BTW, I would put the productivity premium of Go over Python at about 50%-2x, not 10%. I did a mid-size (~6 man-months, though much of that time went into interactions with other teams) green-field prototype in Go at Google, and found myself really missing constructs like list comprehensions, dynamic typing, strong support for literal data, ability to treat user-defined types just like the built-ins, etc. I do think you'll make some of that back in continued maintenance - I actually really enjoyed working with Go - but I didn't experience the claims of "Go is just as good as Python for prototyping", as someone with a fair amount of Python experience.
The availible go ide's are still young but go includes a lot of really nice tooling in the core distribution. The gofmt tool enforces a canonical format for go code and I have sublime text set up (via gosublime) to reformat on save and use the gocode autocomplete/linting daemnon.
Gofmt is actually one of my favorite parts about go and I feel that it, along with the simple documentation system and godoc.org save me a ton of time making my code ready for others to read/use.
The code base behind gofmt and gofix can also be used for other tools, i haven't felt the need for a powerful refactoring tool but I'm sure one will show up eventually.
I do miss list comprehensions and operator overloading but I understand the dev's arguments against those.
missing constructs like list comprehensions, dynamic typing, strong support for literal data, ability to treat user-defined types just like the built-ins, etc.
Of those, I only see comprehensions as something that's severely lacking, though I am not a Python guru by any measure. Dynamic typing: Go has duck typing, which gives some of the same benefits. Different refactoring strategies that take advantage of duck typing over other things you'd do with dynamic languages (like just substituting an object of a different type willy-nilly) will probably make up for the rest. I don't see strong support for literal data as a big productivity gain. Go also does a rather good job of treating user defined types like the built-ins and vice-versa, with a few exceptions like maps.
I didn't experience the claims of "Go is just as good as Python for prototyping", as someone with a fair amount of Python experience.
A lot of your competence with Python for prototyping probably has to do with your competence with Python. A lot of the advantage of Go is what it doesn't have. It's a much simpler system, so it's easier to design around, especially as one needs to get closer to the metal.
The use-case for dynamic typing was that I was decoding a large JSON data structure that could easily have thousands of different fields nested 10+ levels deep. Go's standard json library usually recommends that you define a struct type to hold the result of the decode, tagging field names with the `json` annotation. This obviously doesn't work when you only care about a few dozen of the several thousand possible fields. There is fortunately an escape hatch where you can declare a field as interface{} to say "Any type of data goes here; I'll downcast it later, and just trust me on the type", but then you have to litter your code with downcasts.
Python just hands you back a dict of lists, and you pull out the information you want, no type declarations needed.
The "user-defined types like built-ins" lets you build libraries where you can use the basic language structures on the data structure itself, without having to convert to an intermediate form. For example, the Python bindings to Amazon S3 let you access files through dictionary notation. BeautifulSoup lets you iterate over child nodes with a for loop, or access attributes as a dict. NumPy lets you use standard arithmetic operators on matrices. Judicious use of this in libraries makes the resulting code much briefer.
> I was decoding a large JSON data structure that could
> easily have thousands of different fields nested 10+
> levels deep. Go's standard json library usually
> recommends that you define a struct type to hold the
> result of the decode, tagging field names with the
> `json` annotation. This obviously doesn't work when you
> only care about a few dozen of the several thousand
> possible fields.
Sure it does!
type Response struct {
LevelOne map[string]struct{
LevelTwo map[int]struct{
Value string `json:"eventual_value"`
} `json:"level_two"`
} `json:"level_one"`
}
The response can have way more fields than just the ones you declare in your struct.
I did ignore fields that I wasn't interested in; however, you do at least have to declare all the intermediate structs in the tree. With 10-ish levels of nesting, that's a lot of boilerplate; even in your example with 3 levels, you've still got 6 lines of code for what's a one-liner in Python.
It wouldn't be as much of a problem for a production system where you define your data structures once and then expect to amortize them over many, many changes. But I was prototyping, and the point of prototyping is to not spend time on boilerplate that isn't necessary to prove out your concept. A few hundred lines of type declarations just so I could decode JSON isn't exactly convenient.
Said otherwise, the python oneliner is likely not what you want, in real life you'd have to check for fields existence, purpose defaults or warn, etc. So you must likely end up with a few kings of code anyways.
Context of this thread is prototyping. For prototyping that one-liner is exactly what you want; if the fields don't exist just dump a stack trace and fix your code to use stuff that does exist.
I think the solution posed to you is at least much better than what your comment thread started with: defining the entire JSON structure with Go types. Actually, I think it's a lot better. Personally, I'd probably stop there---even if I were prototyping.
But, at your insistence, we can keep digging. It's easy to write a polymorphic function that arbitrarily picks out values from a Go `interface{}` value.
func lookup(v interface{}, keys ...string) interface{} {
switch len(keys) {
case 0: return v
default:
dict, ok := v.(map[string]interface{})
if !ok {
panic(fmt.Sprintf("%T is not a dictionary", v))
}
return lookup(dict[keys[0]], keys[1:]...)
}
}
This doesn't completely remove the burden of type asserting (unfortunate), but it at least makes nested lookups much easier. Arguably, the nested lookups are probably where the type asserts hurt the most.
The use-case for dynamic typing was that I was decoding a large JSON data structure that could easily have thousands of different fields nested 10+ levels deep.
Yeah, prototyping with JSON is one place where dynamic languages are more nimble.
For example, the Python bindings to Amazon S3 let you access files through dictionary notation. BeautifulSoup lets you iterate over child nodes with a for loop, or access attributes as a dict. NumPy lets you use standard arithmetic operators on matrices. Judicious use of this in libraries makes the resulting code much briefer.
Yes, this is another specific area where dynamic languages do prototyping better and can produce shorter code. I did an exercise where someone implemented a routine in Clojure that I had written in Python. The Python was shorter! In part, this was due to list comprehensions. In part, it was due to an excellent library. What you describe is what many Smalltalkers did as well: create your own control structures and DSLs.
I've sometimes wondered what could be done with a dynamic superset language of Go, such that one could mostly add type definitions and compile to Go.
My biggest pain when programming servers in Go is when handling others' dynamic JSON. Multiple levels of switch-case to handle different types for the same JSON fields at different times. With the current eco systems, Objective C seems the best language, dynamic typing with optional static typing, or mostly static typing in the programs, with dynamic typing for cases like handling dynamic data. Other goods include ARC, Xcode, Apple's frameworks, etc. Concurrency seems to work well with dispatch queues etc. I wonder if Objective C could be as good as Go for servers.
My biggest pain when programming servers in Go is when handling others' dynamic JSON.
It should be simple enough to implement a dynamic JSON parsing DSL in Go. It would be slower and memory inefficient, but much more convenient for rapid development.
>A lot of the advantage of Go is what it doesn't have. It's a much simpler system, so it's easier to design around, especially as one needs to get closer to the metal.
Go doesn't have a REPL and that is objectively a serious dent on effective prototyping.
On formatting, gofmt is awesome. There's an IDE called LiteIDE which automatically calls gofmt for you, or you can just run it yourself whenever. Additionally, because there's deliberately no room to customize gofmt, you don't have to spend time in pointless meetings when "that guy" decides it's time to rewrite your company eclipse templates and argue over all the minutiae.
As far as refactoring, nobody's built an IDE integration that comes close to eclipse/IDEA for that as far as I know. It's totally doable in Go, in a way that it wouldn't be for Python (yay types), but I don't think anyone's done it. On the upside, doing it manually is less painful than it would be in typical Java code, due to type inference and having less code to begin with.
This isn't the same. Go defines a standard format for all code written in Go with nothing left up to programmer opinion. While a lot of great formatting tools exist for other languages, they require choices to be made which inevitably results in disagreement.
If you use C# in Visual Studio, you can press Ctrl+K, Ctrl+D to have the entire document reformatted. Surprisingly few people seem to know about this, judging by how often I've found myself using this on other people's source files and had everything shift around. (Or maybe I'm just unusual and everybody else has just played with the settings.)
The reformatting is actually quite aggressive (it will add/remove blanks and move braces and bits of code around), and so does a good job of keeping everybody's code looking consistent. The Visual Studio text editing functionality only occasionally reaches even the lofty heights of "pretty good", but I do think the document reformatting actually even exceeds that.
(Shame the settings are are per-user rather than per-project, but there you go. It wouldn't be Visual Studio if they didn't manage to fuck something up at the last moment.)
Every decent IDEs have a code formater, but not everyone uses the same IDE so its usefulness is limited. On the other hand, gofmt comes with the platform and make it easy to enforce a common standard between developers.
The default Visual Studio formatting sucks - wasted lines for opening braces when we have limited vertical real estate. Visual Studio should support per-project, or per-solution formatting configurations, which can be placed into the source code repository to ensure everyone on the team is using the same settings - MonoDevelop already does this and it's really useful for switching between projects which use different formats.
That suggests you don't understand what tabs are. Tabs are a single char, not x number of spaces. You can set your text editor to display tabs as any number of spaces you want, I set mine to two.
I wasn't a huge fan of LiteIDE when I tried it. I prefer GoSublime[1] with SublimeCodeIntel[2], with goimports[3] for my gofmt command. The combo of the three is really awesome.
What does SublimeCodeIntel provide that GoSublime doesn't already give you? GoSublime gives you Gocode support, so you have full auto-complete already.
There is actually a decent open source, third party Go plugin for Intellij that works pretty well. Our team has been using it successfully on a fairly large project.
It's slightly annoying to setup, but absolutely worth it.
Sure - I actually gave a talk last year on this exact topic. There's an audio recording somewhere, but here's the repository for the slides (which contains a link to the PDF): https://github.com/ChimeraCoder/go-for-pythonists
I've got a completely data-free feeling that Ruby people hitting performance bottlenecks seem to reach for node whilst Python people have a tendency to look at Go.
Does anyone else think that this rings true? And if so - is it a cultural thing?
I have the feeling that both camps are going for Go instead of Node.
My only data-point is the shop I work for, and folks in the community that I know come from a Ruby background, such as Dominik Honnef, Ben Johnson, Mitchell Hashimoto.
My impression is that people writing Web interfaces reach for Node while people writing CPU-heavy programs reach for Go. It so happens that Ruby people are more often writing simple Web apps while Python is somewhat more likely to be used for more computationally intensive tasks, but I think that's about as far as the correlation goes.
I think that depends on whether they're more of a "Rubyist" or primarily Rails developers who know "just enough Ruby". I think Node (and especially a framework like Meteor or Express) is more approachable for those who spend most of their time in a framework like Rails.
It sounds like the cost of context switching between these very different workloads (crypto, disk I/O, network I/O, protocol buffer parsing and formatting) could be improved in Twisted.
Any idea where the overhead comes from? Twisted, or the Python interpreter itself? Is this a GIL performance issue? Or perhaps even lower -- something here is really hostile to CPU cache?
I realize this is a matter of taste, but my favorite async framework is still the kernel. Write small programs that do one thing well (and thus have pretty uniform workloads) and then let the kernel balance resource usage.
After continuing to hit walls with the standard Python Protocol Buffer library, we wrote our own that was 5x faster than the barely-documented C++ Python Protocol Buffer compiled module support.
Ugh yeah, the standard Python protobuf library is pure python and horribly slow. And it requires code generation -- in a dynamic language! The C++ one is faster, but also requires code generation and is just nasty to work with.
Not that this matters much to you at this point, but I have a small C/Python protobuf library that's 10x faster than the standard Python protobuf: https://github.com/acg/lwpb
PS. I see you're in SLC area -- me too. We should talk tech shop in person sometime!
Hi! Yes, you should totally swing by. Shoot us an email :)
I think the sort of performance issues we were hitting were honestly related to just Python runtime overhead. A Python function call is actually really expensive, and with Twisted Deferred handling, it's really hard to eliminate the massive amount of function calls each I/O event does.
We just had a really slow CPU so we did our best to eliminate as many Python function calls in hotspots as possible, but yeah that was challenging.
A Python function call is actually really expensive, and with Twisted Deferred handling, it's really hard to eliminate the massive amount of function calls each I/O event does.
Ah, makes total sense. Any particular reason you chose to use Twisted in a hardware appliance? Is there a web interface that's supposed to be super responsive?
The main codebase actually started out targeting desktop-class hardware, but the poor uptime of user's work machines (now laptops, soon tablets) actually made our business model not work, hence the dedicated hardware.
Did you ever get a chance to evaluate what it does under PyPy? In my experience, there's quite a few such cases where it eliminates most of the overhead, and it's not even uncommon to see it generate what appears to be optimal x86_64.
Wouldn't java be the natural rewrite, for performance speed up? Java is betwen Python and C, in terms of ease-of-use and performance. They mention a C rewrite, but not a java one...
I guess the concept here is that Go is between Python and Java: (much of) the ease of use of Python + (much of) the performance of Java. The type system literally straddles the dynamic and static.
[ I also detect an enthusiasm for cool, new tech for its own sake (despite risks and cost, or whether it's actually better or not - hell let's find out!) ]
Go has an escape hatch (interface{} and downcasts) that basically give it an embedding of a dynamic type system inside the static one. You end up having to use this a lot if you're doing things like decoding complex JSON data structures or working with containers, since Go doesn't have generics.
> Go has an escape hatch (interface{} and downcasts) that basically give it an embedding of a dynamic type system inside the static one.
No, that gives you a static type system with loopholes, same as Java, you have to use reflect to get a dynamic system in which you can invoke an arbitrary and unchecked method on an object (also same as Java).
Granted, because Go's type system is structural you could define an interface with just the method(s) you want, cast your object to that interface and then call the methods, and you can even define the interface anonymously and inline. Still, go requires your static types to match unless you use reflect.
> [ I also detect an enthusiasm for cool, new tech for its own sake (despite risks and cost, or whether it's actually better or not - hell let's find out!) ]
I'm curious if they tried Cython. I've read that you can achieve up to 35% speedup just by compiling python code, plus you can type pyx file to get near C performance.
I never tried it, this would have been a nice use case.
We did. We used Cython where we could, but Cython doesn't work with Twisted (or at least inline deferreds, which holy we had everywhere), due to different generator semantics. Edit: in fact, our internal protobuf lib was all generated Cython. We had a code-gen inception.
If you did a line for line transliteration, did you use any of Go's concurrency features such goroutines or channels? I ask because I find that when I translate a program from Python to Go that it's beneficial to structure the program differently so that I can use goroutines.
With Twisted (and with our threadpool worker pools), many patterns translated directly into Goroutine usage in a much cleaner way. Where with Python we were using our own helper libraries, Go's stdlib and the language itself were often more than enough.
We didn't end up using channels too much. Deferreds got translated into Futures that we wrote a small library for. Many of our Go-specific utility classes do use channels heavily though.
I think people are rediscovering the benefits of static typing for performance and correctness reasons. The latest generation of statically typed languages[0] all have some level of type inferencing which helps.
If Python is fast enough for you, it’s a fantastic language. The problem is once performance or codebase demands scale, dynamic typing rears its ugly head and there are no simple solutions. At work we sidestep this issue by writing a plethora of tests, but now dynamic typing productivity gains are offset and we spin up a lot of AWS instances for performance.
[0] Go, Rust, Scala. Haskell and OCaml have had it for a while.
While web 2.0. and mobile is reaching maturity with winners are consolidating the market we see the trading of productivity with performance and reduction of operational costs... I m more curious of upcoming booms and which languages and frameworks they are going to bring.
C++ isn't exactly a new language but has also had type inference for a few years. I think people still think of C++ as that dusty old thing in the corner and forget that it is actually capable of a lot of the things users of these new languages get excited about.
I really hope the go compilers will soon support shared object output so we can write python extensions in go instead of having to fully dump one for the other. I know it was a maybe for go 1.3 but I haven't heard much chatter and fear it will get pushed backed.
Can someone tell me how much control GOOG has over the development of Go? I'm interested, but worry that is this another GOOG "open source" project that they have a huge say in.
I'm curious as to how they did a line-by-line port of the twisted program -- twisted's async (plus coroutines-light w/ via generators) seems like a significantly different model than goroutines + channels. I wonder if they actually implemented Deferred in Go?
"So in the bottom of the ninth, we decided to transliterate our 90k lines of Python directly to Go, line by line."
A question here: Is it always a practical approach to transliterate line by line when transitioning to another language or because "Go [is] semantically very similar to Python"?
Both language semantics and language best practices are pretty similar in Python and Go, and I am sure this helps a lot. Transliterating to something like Java would be hard because their interfaces work differently enough from duck typing to force a lot of refactoring. And switching to something even more different, like Haskell or Clojure, would just make transliteration impossible.
I once consulted for a company whose previous consultant had tried to help them do a line-for-line translation (using off-site random-people-in-a-sweatshop online programmers for hire) from ASP to Rails. They pitched it as "transcoding". I will argue that this was not practical (though I can appreciate the idea that at first glance it almost sounds amazingly scalable ;P).
I wrote some Python to run on an OpenWRT (MIPS-based, 400MHz) router that interfaced with an Xbee radio and ran a web interface. I quickly discovered that what I expected to be IO-bound was in fact CPU-bound - reformatting the data and all the related text operations were fast enough after optimization, but only by a hair.
Our solution was to switch to a dedicated ARM board (beaglebone) attached to the router. But I'm definitely going to take a look at using a compiled language now, as the codebase is still very small.
I like seeing people embrace static typing, but it makes me sad that it's a language where you can't even write a container (a map/dictionry/linked-list/etc) without giving up type safety.
I recognize that Golang is filling a gap, letting people get static typing without having to learn much new stuff, and there's value in that, but I think Go enthusiasts should read these great posts by Tikhon Jelvis with an open mind:
Quite frankly, those articles are absolutely terrible. Firstly, it's clear that the author assumes there exists some correct way to design a language. (I reject that.) Secondly, the author never acknowledges any of the trade offs that come from adding additional features to a language. (Hint: one of those trade offs is... there are more features!)
This isn't a revolutionary thought. More compile time safety generally means more complexity in the type system. We, as programmers, must decide when, where and how much we're willing to pay for that type safety.
You can go balls-to-the-wall with Idris or Agda or ATS and encode a whole mess of things directly into your types. Maybe you can completely remove out-of-bounds errors! Surely, this is better than the alternative, which admits less safety.
This isn't a fair comparison because all three of those languages are ridiculously hard to use for practical things. (Well, it's hard for me, anyway.)
Then there are other, more practical languages like Rust, Java, Haskell, OCaml, etc., that all provide a helluva lot more compile time type safety than Go does. (Or, to be precise, a lot more expressive power with respect to compile time safe polymorphism.) Some of the languages go even further; Rust eliminates data races and Haskell isolates "side effects" with monads.
All of these things are wonderful, because we get a lot more guarantees about ways in which our program cannot fail once it's compiled. But they aren't free. They come with a cost. Typically the cost is in greater language complexity. The programmer must write code that the compiler accepts. If you're in that language's wheelhouse, maybe the complexity isn't so bad. But when you want to write a program that isn't easily accepted by the compiler, you run into complexity. Maybe it's not that bad---maybe you just need to figure out how to write a monad transformer. Or maybe you just need to think a bit more carefully about the lifetime of a borrowed pointer. But other times, you need type families and multi-parameter type classes before you can make an unboxed vector with your own type: http://hackage.haskell.org/package/vector-0.10.9.1/docs/Data... --- The safety is great, but dammit, you're going to pay for it.
On some days and for some projects, I'm more than happy to pay the price for safety. It is immensely satisfying to write a program and be able to make all sorts of claims about its safety. (I just wrote a regexp implementation in Rust. Guaranteed no memory errors. No GC. Totally type safe. It's awesome.) But other times, I don't feel like paying the cost.
And that's OK. It's not sad. It's not "ugly." It's a legitimate and reasonable choice given a trade off.
Saying a language is "badly designed" just because it didn't include your preferred set of features is bad juju IMO.
I agree with much of what you said, but an expressive type system isn't just a safety mechanism. It's also a tool to write fast code productively. Without compile time metaprogramming some things will always be either extremely resource hungry or extremely repetitive.
It's absolutely true that there is always a trade-off. Metaprogramming makes code more difficult to understand and hence more prone to certain types of errors. That's true for runtime metaprogramming as well though.
So if the lack of generics leads to an overuse of reflection (or worse to custom code generators) then the complexity you avoided in the language is re-introduced through the backdoor.
At the end of the day I think it's about the culture that gets established around a language. Go's creators are trying really hard to socially engineer a culture of simplicity, and that's laudable. But it reminds me of the early days of Java even though the Go community would rather have me think of C.
The big difference: C had compile time metaprogramming from the beginning, Java did not. Look which one turned out to be the simpler ecosystem in the end.
209 comments
[ 2.8 ms ] story [ 220 ms ] threadOuch. That was a lot of work for 0.2 MB/s. The next 2.8 MB/s was also a lot of work, but it seems conceptually more straightforward.
Python is a joy when it's fast enough, but it's not surprising that it isn't good at throughput on small devices.
It took us about 4 weeks."
How many people worked on the project? What's the build time on a program of this size?
We'll be releasing a bunch of these. Particularly the OpenSSL bindings, which even without hardware acceleration are faster than Go's TLS library, and despite Heartbleed, OpenSSL is not vulnerable to timing attacks like crypto/tls is.
Without tests but with our supporting libraries, our codebase was 36784 lines of Python. Those same lines became 41717 lines of Go.
Counting lines of code is hard.
Sure, python has list comprehensions, but I can tell you how often I've seen hugely over-complicated one line list comprehensions that were impossible to understand unless you were absolutely sure what it was supposed to be doing before you read the code. I've refactored some of those in my lifetime just to make them more readable. That's not to say that most simple list comprehensions aren't totally fine.
So, just saying fewer lines of code makes for fewer bugs is not 100% accurate. I'd say, less /functionality/ leads to fewer bugs. But except in extreme circumstances (like writing a whole framework to do one small task), actual lines of code does not correlate to the number of bugs.
...We spent long nights poring over profiling readouts and traces, wrote our own monitoring system, and wrote our own benchmarking tools
... We enabled cgroups on our Python process to tightly control memory usage and swap contention without requiring a different Python memory allocator scheme.
This is very interesting for the rest of the Python/Twisted community - now that they are not using this (and so is'nt part of the secret sauce), I wonder if they are willing to push this out as a patch/blog post ?
really look forward to this..
Internally, Google has something faster for Python, but they haven't released it yet. You might try putting pressure on them here: https://code.google.com/p/protobuf/issues/detail?id=434
http://yz.mit.edu/wp/fast-native-c-protocol-buffers-from-pyt... talks about the issues some more with some links to other rewrite attempts.
That said, if I did want to go looking elsewhere, I think the main serialization format I am really interested in is Cap'n Proto: http://kentonv.github.io/capnproto/
Further, we have always really been inspired by Joel Spolsky's article on rewrites: http://www.joelonsoftware.com/articles/fog0000000069.html In fact, I can't help but wonder if we would have attempted a Go rewrite sooner if not for that article.
Perhaps we've come a pretty long ways in the interpreted languages department? Or, that the cost of interpreting is vastly less than speed losses due to IO or memory access?
"Interpreted is always slower than compiled lol right guise?" was a tired refrain ten years ago, much less today.
Maybe they'll make it faster in Python 3000, right?
Since it's a reference implementation it should be easy read and learn from.
They're not really willing to accept patches that speed up some things if they at the same time slow down other things (this has been the main problem with all the GIL removal patches that have shown up over the years).
They're not willing to accept patches that break any existing code or libraries.
PyPy on the other hand have non of these limitations and happily break all three, making it great for a subset of python code out there.
Go's http parser is written in Go.
[1]: https://github.com/joyent/http-parser
[1]: https://github.com/joyent/http-parser/blob/master/LICENSE-MI...
http://www.techempower.com/benchmarks/#section=data-r8&hw=i7...
At any rate, "handling HTTP requests" can't possibly be your bottleneck, given network communication, serialization, etc.
While it may be obvious, the interesting question is why exactly. There's been a good presentation from Alex Gaynor [0] on this topic. TL;DR: Dynamic languages ar slow not because they can't be optimized, but because developers make a worse use of memory and have sucky algorithms, inducing way more mem allocation and copy than needed. If you can be smart about those, then your dynamic language can match static languages.
[0] https://speakerdeck.com/alex/why-python-ruby-and-javascript-...
Using any of the Ruby implementations that don't have a GIL. Notably, JRuby, MacRuby/RubyMotion, and (I think) Rubinius.
This is not merely nitpicking. Any turing-complete language can be interpreted or compiled. Language specs tend to make one side easier, but that doesn't mean that Python cannot be compiled (and it is actually compiled to bytecode). You can have something similar to a GIL in a compiled language too.
One of the things that makes it easier to increase performance is having type declarations. This makes it much easier for the compiler to reason about the code, which can lead to increased efficiency. But optional type annotations can accomplish just as much.
I've been using Go for almost two years now. Like OP, I am/was primarily a Python developer. Like OP, my first Go project was a time-sensitive rewrite[0] of a project from Python (tornado) to Go.
Even though I was an experienced Python developer, the Go rewrite was marginally (~20%) faster[1]. But the real benefit came from the subsequent development - refactoring, rearchitecting, and maintaining the project on an ongoing basis. Go was designed to make it easy to scale the maintenance[2] of software, and on this one axis, it absolutely blows every other language and environment I've used out of the water.
For a fresh project, I'd say Go is about 10% slower to write than the equivalent Python[3] for the average case. But the time/cost savings are very quick to come thereafter.
[0] I would absolutely not recommend doing time-sensitive rewrites in general, but that decision was a separate matter.
[1] Some of this is due to the nature of rewrites in general, but the fact that it wasn't slower to use a language I'd never used before says something about the language.
[2] Scaling software development as teams grow is very different from scaling software as users grow.
[3] Assuming comparable familiarity with both languages, which is rarely the case, of course
(disclosure: I haven't fiddled with Go at all yet, so I know nothing about the language itself)
- super fast compile times for fast developer iterations
- you can create an interface (set of methods) that your module owns, and apply it to objects created by other modules without recompiling. If you only need one method you can take that one. It encourages decoupling you from your dependencies.
- the code has one correct formatting convention and a tool that will auto-format your code
- many complex rules for numeric conversions that are implicit in C and cause no end of trouble are explicit and much simpler in Go.
- treating concurrency as a series of sequential process connected with channels makes it MUCH easier to reason about.
Beginners usually think Python strong indentation is a weakness of the language. I actually find C++ freedom being more error prone:
It's a not very frequent bug, but when it happens it takes you hours to spot. ;)EDIT: Whoops, it's actually compiled into bytecode then executed by the VM.
https://en.wikipedia.org/wiki/Python_(programming_language)#...
A quick Google search brought up this, for instance:https://bitbucket.org/StephaneBunel/pythonpep8autoformat
Yes we can also make a single big ass executable out of any python code but it's slower and not supported out of the box.
One thing I really missed when I abandoned the Java world and went to C++/Python was the refactoring and formatting tools available in the IDE. I could set it up so it'd auto-format my code on save, so that I could rename methods with a single keystroke, so that I could pull out classes or add getters/setters, so that I could add parameters, etc. I know Go supports some of that through gofmt and go fix, but I'm curious if anyone is both an experienced Java dev and an experienced Go dev and can compare the continued maintenance costs of the two.
BTW, I would put the productivity premium of Go over Python at about 50%-2x, not 10%. I did a mid-size (~6 man-months, though much of that time went into interactions with other teams) green-field prototype in Go at Google, and found myself really missing constructs like list comprehensions, dynamic typing, strong support for literal data, ability to treat user-defined types just like the built-ins, etc. I do think you'll make some of that back in continued maintenance - I actually really enjoyed working with Go - but I didn't experience the claims of "Go is just as good as Python for prototyping", as someone with a fair amount of Python experience.
Gofmt is actually one of my favorite parts about go and I feel that it, along with the simple documentation system and godoc.org save me a ton of time making my code ready for others to read/use.
The code base behind gofmt and gofix can also be used for other tools, i haven't felt the need for a powerful refactoring tool but I'm sure one will show up eventually.
I do miss list comprehensions and operator overloading but I understand the dev's arguments against those.
Of those, I only see comprehensions as something that's severely lacking, though I am not a Python guru by any measure. Dynamic typing: Go has duck typing, which gives some of the same benefits. Different refactoring strategies that take advantage of duck typing over other things you'd do with dynamic languages (like just substituting an object of a different type willy-nilly) will probably make up for the rest. I don't see strong support for literal data as a big productivity gain. Go also does a rather good job of treating user defined types like the built-ins and vice-versa, with a few exceptions like maps.
I didn't experience the claims of "Go is just as good as Python for prototyping", as someone with a fair amount of Python experience.
A lot of your competence with Python for prototyping probably has to do with your competence with Python. A lot of the advantage of Go is what it doesn't have. It's a much simpler system, so it's easier to design around, especially as one needs to get closer to the metal.
Python just hands you back a dict of lists, and you pull out the information you want, no type declarations needed.
The "user-defined types like built-ins" lets you build libraries where you can use the basic language structures on the data structure itself, without having to convert to an intermediate form. For example, the Python bindings to Amazon S3 let you access files through dictionary notation. BeautifulSoup lets you iterate over child nodes with a for loop, or access attributes as a dict. NumPy lets you use standard arithmetic operators on matrices. Judicious use of this in libraries makes the resulting code much briefer.
I did ignore fields that I wasn't interested in; however, you do at least have to declare all the intermediate structs in the tree. With 10-ish levels of nesting, that's a lot of boilerplate; even in your example with 3 levels, you've still got 6 lines of code for what's a one-liner in Python.
It wouldn't be as much of a problem for a production system where you define your data structures once and then expect to amortize them over many, many changes. But I was prototyping, and the point of prototyping is to not spend time on boilerplate that isn't necessary to prove out your concept. A few hundred lines of type declarations just so I could decode JSON isn't exactly convenient.
But, at your insistence, we can keep digging. It's easy to write a polymorphic function that arbitrarily picks out values from a Go `interface{}` value.
And then we can use it like you would in Python: This doesn't completely remove the burden of type asserting (unfortunate), but it at least makes nested lookups much easier. Arguably, the nested lookups are probably where the type asserts hurt the most.Full example: http://play.golang.org/p/4tWtqGRgU6
Yeah, prototyping with JSON is one place where dynamic languages are more nimble.
For example, the Python bindings to Amazon S3 let you access files through dictionary notation. BeautifulSoup lets you iterate over child nodes with a for loop, or access attributes as a dict. NumPy lets you use standard arithmetic operators on matrices. Judicious use of this in libraries makes the resulting code much briefer.
Yes, this is another specific area where dynamic languages do prototyping better and can produce shorter code. I did an exercise where someone implemented a routine in Clojure that I had written in Python. The Python was shorter! In part, this was due to list comprehensions. In part, it was due to an excellent library. What you describe is what many Smalltalkers did as well: create your own control structures and DSLs.
I've sometimes wondered what could be done with a dynamic superset language of Go, such that one could mostly add type definitions and compile to Go.
It should be simple enough to implement a dynamic JSON parsing DSL in Go. It would be slower and memory inefficient, but much more convenient for rapid development.
Go doesn't have a REPL and that is objectively a serious dent on effective prototyping.
On formatting, gofmt is awesome. There's an IDE called LiteIDE which automatically calls gofmt for you, or you can just run it yourself whenever. Additionally, because there's deliberately no room to customize gofmt, you don't have to spend time in pointless meetings when "that guy" decides it's time to rewrite your company eclipse templates and argue over all the minutiae.
As far as refactoring, nobody's built an IDE integration that comes close to eclipse/IDEA for that as far as I know. It's totally doable in Go, in a way that it wouldn't be for Python (yay types), but I don't think anyone's done it. On the upside, doing it manually is less painful than it would be in typical Java code, due to type inference and having less code to begin with.
https://news.ycombinator.com/item?id=7600130
The reformatting is actually quite aggressive (it will add/remove blanks and move braces and bits of code around), and so does a good job of keeping everybody's code looking consistent. The Visual Studio text editing functionality only occasionally reaches even the lofty heights of "pretty good", but I do think the document reformatting actually even exceeds that.
(Shame the settings are are per-user rather than per-project, but there you go. It wouldn't be Visual Studio if they didn't manage to fuck something up at the last moment.)
See: https://www.youtube.com/watch?v=sln-gJaURzk#t=1735
For example,
https://github.com/zenazn/goji/blob/master/goji.go
[1] https://github.com/DisposaBoy/GoSublime
[2] https://github.com/SublimeCodeIntel/SublimeCodeIntel
[3] https://github.com/bradfitz/goimports
or on emacs, if you use the packaged go-mode.el, you can run gofmt on the "before-save-hook" to do that for you...
http://golang.org/cmd/gofmt/
It's slightly annoying to setup, but absolutely worth it.
http://www.golang-book.com/
http://www.golangbootcamp.com/book
http://talks.golang.org/2012
https://gobyexample.com/
http://blog.repustate.com/migrating-code-from-python-to-gola...
Does anyone else think that this rings true? And if so - is it a cultural thing?
My only data-point is the shop I work for, and folks in the community that I know come from a Ruby background, such as Dominik Honnef, Ben Johnson, Mitchell Hashimoto.
Any idea where the overhead comes from? Twisted, or the Python interpreter itself? Is this a GIL performance issue? Or perhaps even lower -- something here is really hostile to CPU cache?
I realize this is a matter of taste, but my favorite async framework is still the kernel. Write small programs that do one thing well (and thus have pretty uniform workloads) and then let the kernel balance resource usage.
After continuing to hit walls with the standard Python Protocol Buffer library, we wrote our own that was 5x faster than the barely-documented C++ Python Protocol Buffer compiled module support.
Ugh yeah, the standard Python protobuf library is pure python and horribly slow. And it requires code generation -- in a dynamic language! The C++ one is faster, but also requires code generation and is just nasty to work with.
Not that this matters much to you at this point, but I have a small C/Python protobuf library that's 10x faster than the standard Python protobuf: https://github.com/acg/lwpb
PS. I see you're in SLC area -- me too. We should talk tech shop in person sometime!
I think the sort of performance issues we were hitting were honestly related to just Python runtime overhead. A Python function call is actually really expensive, and with Twisted Deferred handling, it's really hard to eliminate the massive amount of function calls each I/O event does.
We just had a really slow CPU so we did our best to eliminate as many Python function calls in hotspots as possible, but yeah that was challenging.
Ah, makes total sense. Any particular reason you chose to use Twisted in a hardware appliance? Is there a web interface that's supposed to be super responsive?
You can find contact information at http://baroquesoftware.com/
Hindsight is 20/20 I guess.
I have my doubts that that would have solved all of our problems though, but that's certainly a different path we could have taken.
I guess the concept here is that Go is between Python and Java: (much of) the ease of use of Python + (much of) the performance of Java. The type system literally straddles the dynamic and static.
[ I also detect an enthusiasm for cool, new tech for its own sake (despite risks and cost, or whether it's actually better or not - hell let's find out!) ]
https://www.aicas.com/cms/en/JamaicaVM
No, that gives you a static type system with loopholes, same as Java, you have to use reflect to get a dynamic system in which you can invoke an arbitrary and unchecked method on an object (also same as Java).
Granted, because Go's type system is structural you could define an interface with just the method(s) you want, cast your object to that interface and then call the methods, and you can even define the interface anonymously and inline. Still, go requires your static types to match unless you use reflect.
Why? Because they did not choose Java?
Opinions vary highly on that claim. In fact, I've never seen anyone claim "ease-of-use" as one of Java's properties.
I never tried it, this would have been a nice use case.
We didn't end up using channels too much. Deferreds got translated into Futures that we wrote a small library for. Many of our Go-specific utility classes do use channels heavily though.
If Python is fast enough for you, it’s a fantastic language. The problem is once performance or codebase demands scale, dynamic typing rears its ugly head and there are no simple solutions. At work we sidestep this issue by writing a plethora of tests, but now dynamic typing productivity gains are offset and we spin up a lot of AWS instances for performance.
[0] Go, Rust, Scala. Haskell and OCaml have had it for a while.
here's the current bleeding edge: https://code.google.com/p/go/source/browse
here's the issue tracker: https://code.google.com/p/go/issues/list
looks like it's being used by the dev team.
If you would like to fork the golang project, run this command:
This is pretty clearly not Android.I suspect we can improve go runtime scheduler performance by starting to go through and replace pieces with channels where appropriate.
A question here: Is it always a practical approach to transliterate line by line when transitioning to another language or because "Go [is] semantically very similar to Python"?
Much easier to convert first (with whatever warts the existing code base has) then do a refactor.
So I doubt their approach had too much to do with Go per se and more with wanting to get something out sooner rather than later.
Our solution was to switch to a dedicated ARM board (beaglebone) attached to the router. But I'm definitely going to take a look at using a compiled language now, as the codebase is still very small.
I recognize that Golang is filling a gap, letting people get static typing without having to learn much new stuff, and there's value in that, but I think Go enthusiasts should read these great posts by Tikhon Jelvis with an open mind:
https://www.quora.com/Googles-programming-language-Go-seems-...
https://www.quora.com/Go-programming-language/Do-you-feel-th...
This isn't a revolutionary thought. More compile time safety generally means more complexity in the type system. We, as programmers, must decide when, where and how much we're willing to pay for that type safety.
You can go balls-to-the-wall with Idris or Agda or ATS and encode a whole mess of things directly into your types. Maybe you can completely remove out-of-bounds errors! Surely, this is better than the alternative, which admits less safety.
This isn't a fair comparison because all three of those languages are ridiculously hard to use for practical things. (Well, it's hard for me, anyway.)
Then there are other, more practical languages like Rust, Java, Haskell, OCaml, etc., that all provide a helluva lot more compile time type safety than Go does. (Or, to be precise, a lot more expressive power with respect to compile time safe polymorphism.) Some of the languages go even further; Rust eliminates data races and Haskell isolates "side effects" with monads.
All of these things are wonderful, because we get a lot more guarantees about ways in which our program cannot fail once it's compiled. But they aren't free. They come with a cost. Typically the cost is in greater language complexity. The programmer must write code that the compiler accepts. If you're in that language's wheelhouse, maybe the complexity isn't so bad. But when you want to write a program that isn't easily accepted by the compiler, you run into complexity. Maybe it's not that bad---maybe you just need to figure out how to write a monad transformer. Or maybe you just need to think a bit more carefully about the lifetime of a borrowed pointer. But other times, you need type families and multi-parameter type classes before you can make an unboxed vector with your own type: http://hackage.haskell.org/package/vector-0.10.9.1/docs/Data... --- The safety is great, but dammit, you're going to pay for it.
On some days and for some projects, I'm more than happy to pay the price for safety. It is immensely satisfying to write a program and be able to make all sorts of claims about its safety. (I just wrote a regexp implementation in Rust. Guaranteed no memory errors. No GC. Totally type safe. It's awesome.) But other times, I don't feel like paying the cost.
And that's OK. It's not sad. It's not "ugly." It's a legitimate and reasonable choice given a trade off.
Saying a language is "badly designed" just because it didn't include your preferred set of features is bad juju IMO.
It's absolutely true that there is always a trade-off. Metaprogramming makes code more difficult to understand and hence more prone to certain types of errors. That's true for runtime metaprogramming as well though.
So if the lack of generics leads to an overuse of reflection (or worse to custom code generators) then the complexity you avoided in the language is re-introduced through the backdoor.
At the end of the day I think it's about the culture that gets established around a language. Go's creators are trying really hard to socially engineer a culture of simplicity, and that's laudable. But it reminds me of the early days of Java even though the Go community would rather have me think of C.
The big difference: C had compile time metaprogramming from the beginning, Java did not. Look which one turned out to be the simpler ecosystem in the end.
Nit: they're typesafe (when done using interfaces), just not statically so.