> I like having a single binary. I usually run code on EC2 machines to give my scripts closer network proximity to S3 and to our database. With Python, I have to ensure all the packages I need are installed on the remote machine, and that one of my coworkers hasn’t installed anything conflicting.
I've written in Python for ~13 years and I'm really enjoying Go. It's fast and its libraries (the few there are) seem to make a lot of sense. I don't think I would use it for daily scripting like the author, but for a large new project I would definitely pick Go over Python now.
I like the expressiveness of Rust in theory, but for the programs I write I don't feel like I get sufficient payoff for fighting the borrow checker, as compared to writing in Go. Would I be correct in saying OCaml is pretty much Rust without the borrow checker? I have no prior experience with the language.
> Would I be correct in saying OCaml is pretty much Rust without the borrow checker?
Yeah, it's very much on those lines, though the syntax is a bit more ML-style (Rust pushes a bit towards a C-like syntax).
(I actually mostly use Scala myself these days, but I know the JVM is a dealbreaker for some people, and the design compromises for the sake of Java compatibility mean Scala is probably not the best choice for a first ML)
Dlang is also very nice if you like Algol style languages. You can get productive in it pretty fast, just like in Go, but it's expressive, it's garbage collected by default and you don't waste time fighting the compiler. Also the code compiles as fast as Go with DMD and LDC. I havent tried GDC.
Plus you get templates and CTFE, which are both awesome. Though if you're doing web stuff I've found D compile times to not be so great anymore, especially when diet templates get into the mix.
Single-process concurrency is great in OCaml. It’s flexible and composable with libraries like Lwt and Async. On the other hand true multicore backend is where the most work is moment. For the latest update on that see: https://discuss.ocaml.org/t/ocaml-multicore-report-on-a-june...
If he(?) wouldn't use it for daily scripting then it isn't meeting all his needs. IME being able to use a single language for both scripting and "real" code is an underrated advantage, since it lets you reuse all your helper functions etc. in your scripts.
Or, because like most professionals, we understand certain tools are better than others at specific tasks, even when there's specific tools that do both things equally well. There can be an option that's preferred even when it's "only" good at one of the things you need.
First let me state that a language is just a brush with which to paint your logic. I really don't subscribe to the X is better than Y. I might concede that x is better than y for certain tasks, but not a sweeping generalisation.
The first point about inconsistent spelling mistakes, I 100% sympathise. I suffer from this terribly. Pylint and syntastic are a silver bullet for me. I have been tempted by a more fancy IDE, but I'm so used to vim, I'm reluctant/scared to change.
Parallelism: I do look over the wall at Go and wish that python 3.x had spent more time on removing the GIL, but that is old news now.
However I enjoy threading with python(not its limitations), because its so trivially easy. The Queues primitive saves 90% of the headache. as for ctrl-c not doing anything set all your threads to daemon (<threadObj>.setDaemon(True)) When the main thread exits, so does everything else. Go shines by having built in true parallelism
If I need horsepower, then I have to run multiple processes (not multiprocessing, that has drawbacks.) which has the annoyance of serializing and message passing. (but that does mean scaling horizontally less painful further on). Again, Go really shines here.
Statically compiling thing: YES. This means that I can 90% of the faff with docker (xyz version not working any more, the build fails) and use a real scheduling system that runs on real steel(or AWS) and not the omnifaff that is kubernetes.
I think when you really need performance it's a great example of a time to start looking at alternative Python implementations, CPython just isn't the best choice for a lot of tasks that actually have processing speed requirements.
After working in the Python speed optimization space as a consultant for a while I have come to see Cython as not really being the right answer for a lot of situations. The main reason is that CPython is slow but Cython really is quite heavily constraining you to the CPython runtime. Cython is significantly different enough to not be Python but it isn't the language I'd want to choose (C/C++/Rust) for going the whole way with foreign function interfacing (I might blog about this again sometime). Sure it's easy to get started with it but I feel like the maintenance costs are higher than people think.
Numba on the other hand has a few nice use cases (where "nice" means close to Pareto-efficient improvement), but I tend to just be using PyPy if I'm going with JIT approach. If for some reason I had to use CPython but had a few bottleneck functions that could be sped up with Numba I'd strongly consider it.
> I have been tempted by a more fancy IDE, but I'm so used to vim, I'm reluctant/scared to change.
I'm happy to report that JetBrains' VIM plugin is almost flawless (Block-based editing, window splitting, save/close, it's all where it's supposed to be).
I used to be in the same camp. But if you add a linter and parser and full blown backend to make lookups for code completion to your VIM configuration you might as well use an IDE. I've switched to IntelliJ for Java.
And they provide a free all pack subscription for open source contributors. Its nice to see all the features at one place though their pipenv integration is still a bit shaky[0]. I haven't got a response yet.
Prospector does a better job and it prevents you from wasting time with what essentially are stylistic choices and BS like a hard 80 chars limit per line (remember, A Foolish Consistency is the Hobgoblin of Little Minds)
And yes, yes, automated stilling tools are better
Intelisense is needed but even the standard vim autocompletion plugin does an ok job
> You're going async by using Golang, why not in python?
In Golang, you're still using a thread-based abstraction.
In Python, "going async" involves switching to a slightly different style of programming and using different I/O libraries. While that has advantages, using threads lets you stick with the standard style.
Pylint solves the undefined variable issue, but it is annoying as hell and always requires configuration. I wish there was a linter for python that didn't care about 80 character limit and was careful not to drown important errors with unimportant stylistic tics.
flake8, like black, shouldn't really require any config on most projects yet annoyingly it always requires at least a little. I really would switch to another tool in a heartbeat if they could just fix that.
Ideally instead of needing to be told whether to only show "serious" errors, for instance, it should just give you a project "score" (sum of the errors x their severity) and show the top 20 or so errors in order of importance and by default it should never exit with error code 1 with just stylistic errors unless explicitly configured to do so.
The real benefit of gofmt is that it's zero-config and used by 100%±1 Go developers. Virtually all Go code is properly formatted and in the same style. It's priceless.
Does someone know a ressource where I can easily learn how to switch from Bash to Go?
To give some context: When it comes to serious programs I use Go already, but often I just want to code a little script to watch a website for changes for example:
#!/usr/bin/env bash
url="https://example.com"
mem="memory.txt"
oldsum="$(cat "$mem")"
newsum="$(curl -sL "$url" | sha512sum)"
if [ "$oldsum" != "$newsum" ]; then
xmpp_notify "$url changed"
echo "$newsum" > "$mem"
fi
I haven't tried it yet, but in my mind it feels like writing such a program would be more time consuming/complex in Go than writing it in bash. Neverthelss, I feel like Go is overall a much better language than Bash (with all those quotes), so I would like to transition to writing more short programs in Go and for that I would like to learn how to do things like reading/writing/grepping files as easily in Go as I do it in Bash. Any idea if someone wrote some hints together already?
The benefit of bash is that it has system binaries as first class citizens; you can leverage the hundreds of thousands of lines of GNU projects with a single keyword. It's hard to beat that expressiveness.
Saying "Go is overall a much better language" is like saying a wrench is a better tool than a screwdriver. You should keep both in your toolbelt and switch out when the need arises.
For your question, start with the regexp and os package, learn how to open a file, read in the contents, and then while iterating over the contents, apply a regex to the text. That should get you to the point that you can search plaintext for character strings. Good luck.
Go is the wrong tool for that. I'm no fan of bash, but it's often the the quick, clear (enough) solution for small tasks. I'm not a python dev - maybe that would be a suitable compromise?
Just a few months ago I replaced an almost 100-line Go "script" with two lines in the Makefile. The fact that Go is great for big projects doesn't automatically make it good for everything. Go is for programs, not scripts.
My problems come from the other end: My Bash scripts tend to become larger. A few weeks ago I wrote a 'cache' utility function (very useful to cache curl requests for example). A few days ago I wanted to use it in a second script was just about to write a package manager for bash modules... I think that is the kind of situation you are getting into when the line count of a bash script increases and you are trying to do proper software development ;-)
So I am not going to build every script in Go from today on, but I would like to learn a bit more on when Go is a viable alternative to Bash.
Don't listen to the naysayers. Go is fantastic for scripts like this - it's really good at network stuff, has a ton of built in functions, etc. And you get actual robustness and error checking that isn't a complete joke. Look how many ways your Bash script could fail! That's fine if you're literally watching the output with your eyes. If not it's going to cause issues when `curl` fails for example.
The only downside really is that you have to compile the Go code, which is fast but not as fast as just interpreting a Bash script (the first time).
Another possible issue is executable size - if you have a lot of Go scripts they may take up a fair bit of space.
I wish Go had an interpreter / JIT mode for scripts.
Edit: Before someone says "but that's 3 times longer than the bash code!" please write the equivalent Bash code that has proper error checking.
That is so repetitive to handle errors. Doesn't go have a cleaner way of handling errors than making the code look far less readable without unified error handling like exceptions?
It is a little repetitive. The best solution is Rust's Result system. But I'd say Ho's error handling system is second to that.
You might be thinking "but... exceptions!". They are a pretty bad solution. The problems are:
* Error handling is hidden - it's impossible in most implementations to know which exceptions can be thrown by a function
* Flow control context is lost. If you wrap lots of lines with try {} you don't know which one caused the error and can't add context information. You end up with shit error messages like "file not found" rather than "error opening config file foo.cfg: file not found". The solution is to wrap every line in a separate try/catch but that is just Go-style error handling but shitter and even more verbose.
Go is a good balance. If you want proper error handling you have to actually write it. Yes it is a little tedious but that's what it takes to do it properly.
If you don't want to do it properly then fine, use Bash and set -e. But accept that it will be shit.
I think for this small size of an example program the error checking in Go is pure overhead, but as soon as the program starts to have a few more functions it will be very useful to have more options than just to abort on fail (Bash trap style / set -e). So I don't have any problem with 'polluting' my source with `if err != nil` checks ;-)
What I am more concerned about are the different ways I have to call functions from different libraries like
That is a lot more 'special' than the other function calls and the bash version and probably requires more documentation reading than '$ curl --help'. But hey, thanks to your example I have a starting point and probably are going to experiment with it in the new future.
For all the others who wrote that Bash is the right tool for the job: I read your warnings, opinions and suggestions and will keep them in mind while exploring the possibilities of using Go for script like jobs.
I love Go and use it for allllmost everything, but I think bash is still better for something like this. The ease with which it can run programs and capture output is the killer feature.
I'd love to see some future bash like language that's strongly typed, has a consistent and simple syntax, but retains these advantages.
for me, I use bash for anything less than 10 lines long, python for anything between 10 and 500 lines, and then I switch to a heavier programming language for anything longer.
Being such a concise language, Haskell could be a good option here. You can do quick scripting while still having a powerful type system (especially compared to Go's) backing you up. I somewhat quickly typed this code up and haven't tested it, but it should replicate that bash script pretty closely:
#!/usr/bin/env stack
-- stack script --resolver=lts-12.2
import System.Process (system)
import Control.Lens
import Crypto.Hash.SHA256
import qualified Data.Bytestring.Lazy as B
import Network.Wreq
url = "https://example.com"
mem = "memory.txt"
main = do
oldsum <- B.readFile mem
newsum <- (hashLazy . view responseBody) <$> get url
when (oldsum /== newsum) $ do
exitCode <- system $ "xmpp_notify " ++ url ++ " changed"
B.writeFile mem newsum
I didn't make use of it, but there's also the Turtle library which aims to provide a fairly solid shell scripting experience, providing a lot of coreutils as simple Haskell functions. You also get a REPL in the form of GHCI, which comes default in Haskell installations.
So much this. No one is writing large production applications without some sort of imported library. Go simply has not been around long enough or had enough users to mature. For all of the pros Go has (not even mentioning the cons) they seem miniscule compared to what we give up switching to it. It has always sounded like the hot new fad and not a stable thing. Especially with daily articles like coaxing people into using it.
> I’m not a fan of full blown IDEs that require their own configuration.
Hmm, but those IDEs actually do solve most of the problems author has had with Python. If one regularly code in Python, spending a few hours configuring the IDE is justifiable. I for example never write scripts outside the IDE, because why?
No they don't. No IDE can detect all typos in Python and do accurate code completion like they can for statically typed languages because the problem is unsolvable in general.
They can use heuristics to get fairly okish. Pycharm is not bad and VSCode's new Python extension seems to work fairly well. But it's never going to be as rock solid as Java code completion for example.
Note: I like Python, so this is me playing devil's advocate here.
If you need an IDE to solve common pain points of a language, isn't it fair to say that another language which lacks those pain points is a better fit for you? Using an IDE to aid you, to smooth things out and allow for more rapid development and better workflow is one thing. Being forced to use one because the language is lacking is a bad look on the language!
You could also say that this attitude holds us back. What if Smalltalk had only been accessible from vi back in the day, would Smalltalk have been as appreciated? I think not.
> I make stupid mistakes in Python constantly. I misname a variable or a function or I pass in the wrong arguments.
> Unit tests would catch most of these, but it’s hard to get 100% code coverage, and I don’t want to spend time writing unit tests for a one-off script.
Sorry, I actually do like golang but this is telling of a bigger issue. Someone else is going to have to delete/fix it
One of the biggest benefits of go vs python is that you only manage your dependencies during build. You end up with a single executable that is easy to deploy and mostly immune to the deployment of other things. In python, you have to also worry about deployment. To be safe, you need to carry the vm and other dependencies with you.
This means you can't just give any python program to someone and have it just work, unless it's pretty trivial.
Depends on your usecase, really. Static linking has been there for a long time and it has its disadvantages. I have more than a dozen programs on my system that share the same Python runtime and libraries. Compare that to every Go program shipping its own copy of the same things.
>every Go program shipping its own copy of the same things.
Is that really an issue? It takes up a bit more space, but nothing that's really any concern on a modern system. People often resort to using Docker to manage dependencies in other programming languages and end up shipping an entire copy of an operating system.
I honestly don't see many downside to shipping a static binary. At least in that case you can chroot the entire thing, something I really wish Python (and most other languages) would make a whole lot easier.
Security fixes require a recompilation, that perhaps the largest downside, but if you use Docker, then you need to rebuild that image as well.
Also, deploying to production is as easy as writing a small shell script to simply copy the new binary and restart your service definition. (or using ansible/puppet/chef/saltstack et al)
Yes, if you can keep things simple. If you've fixed a ux bug and your binary depends on a certain new version of an image/icon etc - it may or may not be that easy.
You can bundle assets, like icon, in your binary. In that case changing an image results in a new build and a new binary. Then deployments are still just one copy via Ansible, or similar.
Go also carries all its dependencies with it; this is why it produces large single static binaries.
Python lacks a tool to painlessly produce the same, that is, a self-contained binary that includes the VM and all the dependencies. Determining Python dependencies may be slightly tricky, because `import` is an executable statement, and dependencies may be native-code libraries.
There are several tools for that, but all have their downsides, and none is in widespread use. Polishing such a tool until it's as simple and fool-proof as Go's build tool could be quite beneficial for Python community.
Right, its not magic to build such a tool. But until it exists, this is a benefit for Go. This is also why I keep using Tcl, with tclkit you can build single-file executables for you tcl programs.
Why does it have to be a self-contained binary? When you pack everything into an virtualenv you get more or less the same result, don't you? If you want to have a single file you can zip it :)
Usability-wise, there's a huge gap between even low-effort and no-effort actions.
Another upside of a single binary is that it's not easy to modify; you can even sign it. Quite an upside if you want to audit what you deploy to your production that handles millions of dollars.
Can't you tar the python venv and sign it as well? Might be more difficult to get reproducible builds tgisbway, but it's sort of the same thing isn't it?
Yea I just do not buy it. How much time is saved? Is it writing an additional SSH command to extract the file or hitting the up-arrow on your keyboard to repeat the previously run command?
Not a lot, but when your test environment is on a remote server (because it replicates the production DB and you can't have that on your dev machine for security reasons), you tend to do it a lot. Add fast compilation to the mix, and you have a very productive workflow. `make && zip && scp && ssh unzip` is way slower than `go build && scp`
Everyone seem to agree the generation of a static binary is easy with go. That's not my experience as soon as CGO and the glib is involved. I would gladly pay 100$ to make it work on my open source project using libvips to anyone who can create the static build: https://github.com/mickael-kerjean/nuage
I do miss editing those with Resource Hacker or XN Resource Editor. I translated tons of Japanese and Chinese software into English that way back in high school and early college.
Speaking from the linux distro perspective, this is not true.
You can develop an application in python that utilizes distro-supplied python libraries. If you package your library in .deb or .rpm, it's trivial to utilize the dependencies from the distro.
Otherwise, you can install from pypi via pip, and especially into a virtualenv. You can declare each dependency's specific version if you so choose. Modern distros support python2.7 and python 3.x.
If you're on some other operating system, yeah, using python sucks for you.
While it's fair to say python is not statically compiled, it's no different than managing dependencies in other dynamically linked software.
Unlike python, go applications have to be cross compiled. For the most part, native python code will run anywhere the interpreter runs.
In the world of containerized applications, vendoring dependencies is a moot point anyway. We're all shipping around big tarballs with everything baked in anyway, so go doesn't offer any advantages in that space other than smaller binary size.
Not to say that go doesn't have some advantages over python, but I think the dependency management problem is vastly overstated, at least when it comes to python.
> You can develop an application in python that utilizes distro-supplied python libraries.
And unless you're aggressively tracking your distro's package releases you'd better hope that the new libdep doesn't introduce any breaking bugs. Also your app is no longer for Linux it's for Ubuntu 18.04 updated roughly on 2018-08-03.
The same goes for the actual python. Distributions apply lots of downstream patches and backport fixes so your python 2.7 is really redhat-python2.7-13 so you might want to test against that -- or you can bundle your own and be done with it.
Unless you have the wall clock time to actually define and test supported distributions you probably want to pretend the system python doesn't exist.
> pypi via pip, and especially into a virtualenv
I would never subject my users to this workflow. What you're describing are source distribution channels which are (ab)used to distribute applications. I have no idea if some dependency's native extension will even compile on their system. Do they even have build tools? Why would they?
> And unless you're aggressively tracking your distro's package releases you'd better hope that the new libdep doesn't introduce any breaking bugs.
Or use a distribution that does not break shit left and right, like Debian or
Red Hat (CentOS).
> Unless you have the wall clock time to actually define and test supported distributions you probably want to pretend the system python doesn't exist.
If you write software that will be run by others (which usually means open
source, probably libraries), yes. If you write software that will only be run
by you (pretty much all dynamic websites a.k.a. webapps land in this
category), you don't want to have three different distributions in half
a dozen different versions anyway, so you can pin yourself to the target
environment just as well.
> Or use a distribution that does not break shit left and right, like Debian or Red Hat (CentOS).
Sadly, there are a community of devs who want to rely on libflakey 0.0.1-beta, released an hour ago, and think that waiting for APIs to stabilise & distros to securely package things is unprofessional (!).
Using libflakey (lol!) is not necessarily a bad thing; The proper course would be to pull out what you need, make sure the interface you care about is sane and works well; you can revisit the upstream in the future.
Raw npm-style clone-from-master seems to be prolific, though.
> If you write software that will be run by others (which usually means open source, probably libraries), yes. If you write software that will only be run by you (pretty much all dynamic websites a.k.a. webapps land in this category), you don't want to have three different distributions in half a dozen different versions anyway, so you can pin yourself to the target environment just as well.
Yeah, my primary software development mode is 1) gather opensource software, 2) build web app or platform-specific app.
> Or use a distribution that does not break shit left and right, like Debian or Red Hat (CentOS).
It's crazy that this even has to be said in this day and age. I suppose distro specific patches to maintain a secure and stable api has gone away in favor of some vendored static dependencies that may or may not ever be upgraded, and rebasing your dependencies introduces the same set of problems.
I foresee dynamically linked go with platform specific binary packaging becoming the future. Recompiling the same bits of software ad infinitum is probably going to get old.
My industry, vfx, isn't as ubiquitous as web development, but it is fairly large with commercial interests to motivate making the "right" decisions and not too odd of a use-case. I've heard some second-hand stories about Guido crashing on the couch of some guys at ILM back in the "early days" helping to spur adoption. However, we're still pretty solidly on Python2 with proposals to start adopting Python3 next year. Most of the commercial applications run Python inside them and the apps are very, very scriptable.
Houdini is a fairly ubiquitous tool for creating dynamics; fire, water, destruction, dust and has been around since 1996. In general, Houdini users tend to prefer Ubuntu and most medium to large vfx houses use CentOS/RHEL (Pixar, DreamWorks, ILM).
For Linux and macOS, Houdini uses your system's Python but it has to monkey patch parts of the standard library in order to work smoothly. You can set a flag (and on Windows this is the default) where it will use the Python shipped with the application.
At my current job we're using slightly out of date versions of everything, which is quite common in production. We're running CentOS 7.2, Houdini 15.5 (released Nov 2017), and Python 2.7.5. If you call "httplib.HTTPSConnection('google.com')" it throws an exception because of ssl library changes across 2.7. I haven't tried any other versions, but the requirements for the current version of Houdini says "CentOS 6+ (64-bit)" (among other OSes).
This isn't unique to Python, but it's very hard to properly support real-world clients and dynamic libraries among operating systems, commercial applications, and less well funded open source or independent efforts.
My M.O. is if a technology is core to your business, you should control it. Which means not using the OS' Python. Upgrading Python and OS independently makes so many things much easier when you have a large codebase.
> I would never subject my users to this workflow. What you're describing are source distribution channels which are (ab)used to distribute applications. I have no idea if some dependency's native extension will even compile on their system. Do they even have build tools? Why would they?
If the users of your software are end-user desktop types, I can see your sentiment. My software is typically for building software (aka, libraries) or users that know how to compile things.
> I make stupid mistakes in Python constantly. I misname a variable or a function or I pass in the wrong arguments. Devtools can catch some of these, but they usually require special setup. I’ve never been able to configure pylint easily, and I’m not a fan of full blown IDEs that require their own configuration. The worst is if you mistype a variable that’s hidden behind conditional logic. Your script might run for hours before triggering the error, and then everything blows up, and you have to restart it.
This this this, a million times this. I work mostly in Clojure where such problems are equally as bad, maybe even more so. Dynamic, elegant languages can feel liberating in so many ways, but there are significant workflow drawbacks that get me nearly every day.
I program mostly in ClojureScript which will "throw" a compiler warning if I misspell a variable. But to get that in an editor is a bit more setup than to just fire up vanilla vim.
Doesn’t stop you from passing in wrong arguments, your function requires an int and you give it a string because you typed the wrong variable name there, or you mistype a keyword, which is never caught by the compiler, and you end up with nil floating through your app.
They don't take care of this issue most of the time, and in Clojurescript they are meaningless. In Clojure, only hints on primitive Java types (integer, etc) do something meaningful, so any custom Clojure types or your own "types" have no such checking, and in particular keywords are always valid if they are mistyped which results in a running program with no errors that creates nil and null pointer exceptions and undefined and other things all over the place.
It's a tradeoff for using an expressive dynamic language, I guess.
First off, I get that it's unfortunate they aren't detected by Python itself in some form of compile step, but that's a cost of using a dynamic language.
But they don't use the tools created to catch these kinds of errors, and complain that they aren't protected against these errors? There's a fairly large disconnect there. Hell, I have protection against this built into my VIM install. Not running these basic tools is tying a hand behind your back.
and run it as an executable if your machine has stack installed. If not -- compile it with a `stack ghc --resolver=lts-12.2 ./file.hs` and you'll have a `./file` executable. No project creation, no need to list the dependencies.
This makes me want to try Go, though I wonder how long it would take to develop the same "fluency" and knowledge of which library to use. It would be exiting to have things run fast though...
One minor plug `pyflakes` is very good for basic checks for mistyped variable and missing imports, etc. It's not "smart" and doesn't check type consistency, but very useful to catch stupid mistakes.
I went from Python to Go; Go is an amazingly simple and quick to learn language and ecosystem. Give it a try -- you'll probably start to feel proficient a lot faster than you'd think.
I've been using Go over python for a couple months now and so far it has been incredibly effective. I never really liked Python's syntax so it's nice to go back to something c-ish.
144 comments
[ 2.5 ms ] story [ 214 ms ] threadpipenv
Yeah, it's very much on those lines, though the syntax is a bit more ML-style (Rust pushes a bit towards a C-like syntax).
(I actually mostly use Scala myself these days, but I know the JVM is a dealbreaker for some people, and the design compromises for the sake of Java compatibility mean Scala is probably not the best choice for a first ML)
https://tour.dlang.org/
I really like OCaml, and you should learn it, but you should also know that concurrency is not a strength of it. It's being worked on...
(Tl;dr: It will be officially merged in 2019.)
I hope that it does get merged soon, people have been waiting quite a while, in my understanding. It’s a tough problem.
https://github.com/facebookincubator/xar/
Then just build your XAR executable with:
$ python setup.py bdist_xar
The first point about inconsistent spelling mistakes, I 100% sympathise. I suffer from this terribly. Pylint and syntastic are a silver bullet for me. I have been tempted by a more fancy IDE, but I'm so used to vim, I'm reluctant/scared to change.
Parallelism: I do look over the wall at Go and wish that python 3.x had spent more time on removing the GIL, but that is old news now.
However I enjoy threading with python(not its limitations), because its so trivially easy. The Queues primitive saves 90% of the headache. as for ctrl-c not doing anything set all your threads to daemon (<threadObj>.setDaemon(True)) When the main thread exits, so does everything else. Go shines by having built in true parallelism
If I need horsepower, then I have to run multiple processes (not multiprocessing, that has drawbacks.) which has the annoyance of serializing and message passing. (but that does mean scaling horizontally less painful further on). Again, Go really shines here.
Statically compiling thing: YES. This means that I can 90% of the faff with docker (xyz version not working any more, the build fails) and use a real scheduling system that runs on real steel(or AWS) and not the omnifaff that is kubernetes.
Numba on the other hand has a few nice use cases (where "nice" means close to Pareto-efficient improvement), but I tend to just be using PyPy if I'm going with JIT approach. If for some reason I had to use CPython but had a few bottleneck functions that could be sped up with Numba I'd strongly consider it.
I'm happy to report that JetBrains' VIM plugin is almost flawless (Block-based editing, window splitting, save/close, it's all where it's supposed to be).
I used to be in the same camp. But if you add a linter and parser and full blown backend to make lookups for code completion to your VIM configuration you might as well use an IDE. I've switched to IntelliJ for Java.
[0] : https://youtrack.jetbrains.com/issue/PY-30855
It dumps numpy arrays in /dev/shm which can loaded by other processes by the filepath, muuuuuch nicer than passing things around
Not even if X is better than Y for every specific task?
- I make stupid mistakes in Python constantly. I misname a variable or a function or I pass in the wrong arguments: Try mypy
- If the task is IO bound: Threads are the wrong the answer to that. You're going async by using Golang, why not in python?
- With Python, I have to ensure all the packages I need are installed on the remote machine: pipenv Like slezyr mentions
- Consistent Styling: Try autopep8 or black
- Intellisense: I would say "use Pycharm" but vscode is getting better I heard
Prospector does a better job and it prevents you from wasting time with what essentially are stylistic choices and BS like a hard 80 chars limit per line (remember, A Foolish Consistency is the Hobgoblin of Little Minds)
And yes, yes, automated stilling tools are better
Intelisense is needed but even the standard vim autocompletion plugin does an ok job
In Golang, you're still using a thread-based abstraction.
In Python, "going async" involves switching to a slightly different style of programming and using different I/O libraries. While that has advantages, using threads lets you stick with the standard style.
Pylint solves the undefined variable issue, but it is annoying as hell and always requires configuration. I wish there was a linter for python that didn't care about 80 character limit and was careful not to drown important errors with unimportant stylistic tics.
flake8 is a linter that doesn't need much config, and you can config it to give you only "serious" errors and not stylistic issues.
I gave a talk on these subjects last year at the Paris Open Source Summit: https://speakerdeck.com/sfermigier/python-quality-engineerin... (NB: Black didn't exist yet at the time).
Ideally instead of needing to be told whether to only show "serious" errors, for instance, it should just give you a project "score" (sum of the errors x their severity) and show the top 20 or so errors in order of importance and by default it should never exit with error code 1 with just stylistic errors unless explicitly configured to do so.
Or rather, I'd vaguely heard the name but I erroneously thought it was just another flake8 type thing.
https://github.com/klen/pylama
To give some context: When it comes to serious programs I use Go already, but often I just want to code a little script to watch a website for changes for example:
I haven't tried it yet, but in my mind it feels like writing such a program would be more time consuming/complex in Go than writing it in bash. Neverthelss, I feel like Go is overall a much better language than Bash (with all those quotes), so I would like to transition to writing more short programs in Go and for that I would like to learn how to do things like reading/writing/grepping files as easily in Go as I do it in Bash. Any idea if someone wrote some hints together already?Saying "Go is overall a much better language" is like saying a wrench is a better tool than a screwdriver. You should keep both in your toolbelt and switch out when the need arises.
For your question, start with the regexp and os package, learn how to open a file, read in the contents, and then while iterating over the contents, apply a regex to the text. That should get you to the point that you can search plaintext for character strings. Good luck.
Just a few months ago I replaced an almost 100-line Go "script" with two lines in the Makefile. The fact that Go is great for big projects doesn't automatically make it good for everything. Go is for programs, not scripts.
My problems come from the other end: My Bash scripts tend to become larger. A few weeks ago I wrote a 'cache' utility function (very useful to cache curl requests for example). A few days ago I wanted to use it in a second script was just about to write a package manager for bash modules... I think that is the kind of situation you are getting into when the line count of a bash script increases and you are trying to do proper software development ;-)
So I am not going to build every script in Go from today on, but I would like to learn a bit more on when Go is a viable alternative to Bash.
The only downside really is that you have to compile the Go code, which is fast but not as fast as just interpreting a Bash script (the first time).
Another possible issue is executable size - if you have a lot of Go scripts they may take up a fair bit of space.
I wish Go had an interpreter / JIT mode for scripts.
Edit: Before someone says "but that's 3 times longer than the bash code!" please write the equivalent Bash code that has proper error checking.
You might be thinking "but... exceptions!". They are a pretty bad solution. The problems are:
* Error handling is hidden - it's impossible in most implementations to know which exceptions can be thrown by a function * Flow control context is lost. If you wrap lots of lines with try {} you don't know which one caused the error and can't add context information. You end up with shit error messages like "file not found" rather than "error opening config file foo.cfg: file not found". The solution is to wrap every line in a separate try/catch but that is just Go-style error handling but shitter and even more verbose.
Go is a good balance. If you want proper error handling you have to actually write it. Yes it is a little tedious but that's what it takes to do it properly.
If you don't want to do it properly then fine, use Bash and set -e. But accept that it will be shit.
I think for this small size of an example program the error checking in Go is pure overhead, but as soon as the program starts to have a few more functions it will be very useful to have more options than just to abort on fail (Bash trap style / set -e). So I don't have any problem with 'polluting' my source with `if err != nil` checks ;-)
What I am more concerned about are the different ways I have to call functions from different libraries like
That is a lot more 'special' than the other function calls and the bash version and probably requires more documentation reading than '$ curl --help'. But hey, thanks to your example I have a starting point and probably are going to experiment with it in the new future.For all the others who wrote that Bash is the right tool for the job: I read your warnings, opinions and suggestions and will keep them in mind while exploring the possibilities of using Go for script like jobs.
I'd love to see some future bash like language that's strongly typed, has a consistent and simple syntax, but retains these advantages.
(Nothing against golang, just wanted to point out it's not that easy for every usecase).
Hmm, but those IDEs actually do solve most of the problems author has had with Python. If one regularly code in Python, spending a few hours configuring the IDE is justifiable. I for example never write scripts outside the IDE, because why?
They can use heuristics to get fairly okish. Pycharm is not bad and VSCode's new Python extension seems to work fairly well. But it's never going to be as rock solid as Java code completion for example.
If you need an IDE to solve common pain points of a language, isn't it fair to say that another language which lacks those pain points is a better fit for you? Using an IDE to aid you, to smooth things out and allow for more rapid development and better workflow is one thing. Being forced to use one because the language is lacking is a bad look on the language!
> Unit tests would catch most of these, but it’s hard to get 100% code coverage, and I don’t want to spend time writing unit tests for a one-off script.
Sorry, I actually do like golang but this is telling of a bigger issue. Someone else is going to have to delete/fix it
This means you can't just give any python program to someone and have it just work, unless it's pretty trivial.
Is that really an issue? It takes up a bit more space, but nothing that's really any concern on a modern system. People often resort to using Docker to manage dependencies in other programming languages and end up shipping an entire copy of an operating system.
I honestly don't see many downside to shipping a static binary. At least in that case you can chroot the entire thing, something I really wish Python (and most other languages) would make a whole lot easier.
Security fixes require a recompilation, that perhaps the largest downside, but if you use Docker, then you need to rebuild that image as well.
Python lacks a tool to painlessly produce the same, that is, a self-contained binary that includes the VM and all the dependencies. Determining Python dependencies may be slightly tricky, because `import` is an executable statement, and dependencies may be native-code libraries.
There are several tools for that, but all have their downsides, and none is in widespread use. Polishing such a tool until it's as simple and fool-proof as Go's build tool could be quite beneficial for Python community.
Another upside of a single binary is that it's not easy to modify; you can even sign it. Quite an upside if you want to audit what you deploy to your production that handles millions of dollars.
Usage would be
python foo/setup.py bdist_xar && scp foo/dist/foo.xar user@server
if my understanding is correct.
[1]: https://github.com/facebookincubator/xar
Personally I'm not a fan but I can totally see the appeal.
1. https://github.com/hashicorp/consul/pull/1543
I also found this comparison of libraries pretty good: https://tech.townsourced.com/post/embedding-static-files-in-...
[1] https://github.com/shurcooL/vfsgen
[2] https://github.com/shurcooL/httpfs/blob/master/html/vfstempl...
Good times.
You can develop an application in python that utilizes distro-supplied python libraries. If you package your library in .deb or .rpm, it's trivial to utilize the dependencies from the distro.
Otherwise, you can install from pypi via pip, and especially into a virtualenv. You can declare each dependency's specific version if you so choose. Modern distros support python2.7 and python 3.x.
If you're on some other operating system, yeah, using python sucks for you.
While it's fair to say python is not statically compiled, it's no different than managing dependencies in other dynamically linked software.
Unlike python, go applications have to be cross compiled. For the most part, native python code will run anywhere the interpreter runs.
In the world of containerized applications, vendoring dependencies is a moot point anyway. We're all shipping around big tarballs with everything baked in anyway, so go doesn't offer any advantages in that space other than smaller binary size.
Not to say that go doesn't have some advantages over python, but I think the dependency management problem is vastly overstated, at least when it comes to python.
And unless you're aggressively tracking your distro's package releases you'd better hope that the new libdep doesn't introduce any breaking bugs. Also your app is no longer for Linux it's for Ubuntu 18.04 updated roughly on 2018-08-03.
The same goes for the actual python. Distributions apply lots of downstream patches and backport fixes so your python 2.7 is really redhat-python2.7-13 so you might want to test against that -- or you can bundle your own and be done with it.
Unless you have the wall clock time to actually define and test supported distributions you probably want to pretend the system python doesn't exist.
> pypi via pip, and especially into a virtualenv
I would never subject my users to this workflow. What you're describing are source distribution channels which are (ab)used to distribute applications. I have no idea if some dependency's native extension will even compile on their system. Do they even have build tools? Why would they?
Or use a distribution that does not break shit left and right, like Debian or Red Hat (CentOS).
> Unless you have the wall clock time to actually define and test supported distributions you probably want to pretend the system python doesn't exist.
If you write software that will be run by others (which usually means open source, probably libraries), yes. If you write software that will only be run by you (pretty much all dynamic websites a.k.a. webapps land in this category), you don't want to have three different distributions in half a dozen different versions anyway, so you can pin yourself to the target environment just as well.
Sadly, there are a community of devs who want to rely on libflakey 0.0.1-beta, released an hour ago, and think that waiting for APIs to stabilise & distros to securely package things is unprofessional (!).
Raw npm-style clone-from-master seems to be prolific, though.
Yeah, my primary software development mode is 1) gather opensource software, 2) build web app or platform-specific app.
> Or use a distribution that does not break shit left and right, like Debian or Red Hat (CentOS).
It's crazy that this even has to be said in this day and age. I suppose distro specific patches to maintain a secure and stable api has gone away in favor of some vendored static dependencies that may or may not ever be upgraded, and rebasing your dependencies introduces the same set of problems.
I foresee dynamically linked go with platform specific binary packaging becoming the future. Recompiling the same bits of software ad infinitum is probably going to get old.
Houdini is a fairly ubiquitous tool for creating dynamics; fire, water, destruction, dust and has been around since 1996. In general, Houdini users tend to prefer Ubuntu and most medium to large vfx houses use CentOS/RHEL (Pixar, DreamWorks, ILM).
For Linux and macOS, Houdini uses your system's Python but it has to monkey patch parts of the standard library in order to work smoothly. You can set a flag (and on Windows this is the default) where it will use the Python shipped with the application.
At my current job we're using slightly out of date versions of everything, which is quite common in production. We're running CentOS 7.2, Houdini 15.5 (released Nov 2017), and Python 2.7.5. If you call "httplib.HTTPSConnection('google.com')" it throws an exception because of ssl library changes across 2.7. I haven't tried any other versions, but the requirements for the current version of Houdini says "CentOS 6+ (64-bit)" (among other OSes).
This isn't unique to Python, but it's very hard to properly support real-world clients and dynamic libraries among operating systems, commercial applications, and less well funded open source or independent efforts.
My M.O. is if a technology is core to your business, you should control it. Which means not using the OS' Python. Upgrading Python and OS independently makes so many things much easier when you have a large codebase.
If the users of your software are end-user desktop types, I can see your sentiment. My software is typically for building software (aka, libraries) or users that know how to compile things.
Nuitka (nuitka.net) has been able to compile python programs to a single stand alone executable reliably, robustly and easily for a while now.
It's a real gem and deserves to be more popular. We are very far from the quirks of pyinstaller or cxfreeze : it just works, even with numpy or qt.
Yes, go is still superior for deployment :
- it can cross compile
- executables are smaller
- it's built in so it's just an easier experience
- it doesn't depend on libc
But it's important to spread the word: the time were your python program was hard to ship is over.
Just use nuitka.
> To be safe, you need to carry the vm and other dependencies with you.
You should make it easy as possible for users of your software .
This this this, a million times this. I work mostly in Clojure where such problems are equally as bad, maybe even more so. Dynamic, elegant languages can feel liberating in so many ways, but there are significant workflow drawbacks that get me nearly every day.
It's a tradeoff for using an expressive dynamic language, I guess.
But they don't use the tools created to catch these kinds of errors, and complain that they aren't protected against these errors? There's a fairly large disconnect there. Hell, I have protection against this built into my VIM install. Not running these basic tools is tying a hand behind your back.
https://github.com/python-mode/python-mode
One minor plug `pyflakes` is very good for basic checks for mistyped variable and missing imports, etc. It's not "smart" and doesn't check type consistency, but very useful to catch stupid mistakes.
I recently started using python as the primary language, the more I use, the more I like it.
Go's put everything in a static binary has its pros and cons.
And the HN discussion https://news.ycombinator.com/item?id=10402307
numpy - https://github.com/gonum/gonum matplotlib - https://github.com/gonum/plot pandas - https://github.com/kniren/gota (API not finalized)