No, because by default it only logs what's happening directly in your function, not what happens deeper in the stack. (i.e. functions that your function calls.)
You can set depth=2 or depth=3 and then you'll get a huge chunk of log. The deeper levels will be indented.
Interesting project, though I'd suggest losing the line about "can't be bothered to set one up right now" regarding a full debugger. (i)pdb is built in and is simple to use. Perhaps focus on what this can add rather than framing the project as something like a lazy alternative (especially when this may actually be harder to set up than throwing in "import pdb; pdb.set_trace()")?
pdb and its variants (my favorite is PuDB) are generally difficult to use in complex, corporate projects. If you've got a multi-process, multi-thread Python project running on a remote host, you'll need a really full-featured debugger to work with it effectively. I recommend Wing IDE or PyCharm for that.
Certainly not a counter, as I'm less familiar with Wing and Pycharm's debugging features, but both of these have been helpful to me in multi-process, multi-thread python environments.
> You can use it in your shitty, sprawling enterprise codebase without having to do any setup.
Debugging is a sensitive subject, particularly given how frustrating it can be. There’s a place for vulgarity somewhere, but I’d rather see your README provide authoritative info than crack jokes.
I appreciated the README as written. I immediately understood what kind of environment the author intended this to be used in. A more formal explanation would have degraded the message. Vulgarity doesn't have to taken as judgemental, it can also be a good way to express empathy. The way I read it, the author's intent was the latter.
One-word quips sound good on paper, but this is an open source project with engineers as the target audience. It's not seeking (at this time) to bring revenue or make sales, so I'd argue that speaking truth to the problem is more likely to drive up adoption.
> May I ask how the word "shitty" more truthful than "poorly written" or "complex"?
Evokes emotions people affected by a situation can sympathize with more effectively, which by virtue of establishing a shared emotional bond over a topic helps the developer convey not just the situation but the frustrations of the situation more effectively than one might expect "poorly written" or "complex" to do alone.
> Citing professionalism isn't a quip, it's shorthand for a code of conduct and long accepted practices of interaction with others.
The code of conduct isn't uniform, so it can't be used effectively as shorthand for such. But at this point we're in the weeds.
Again, no short-term revenue prospects, just a tool OP wants to socialize to make a few lives easier. If you have an objection over verbiage, that's fine, but it's an exhibition of professionalism from yourself to the OP to build a sound defense of your position as to how it would help the engineer to self-censor the description of a tool where the audience by-and-large may not care.
Up to you. My point is the engineer doesn't need to suppress who they are in this specific context, and my point to you is it shouldn't impact your usage of what looks to be an effective short-term debugging tool.
> exhibition of professionalism from yourself to the OP to build a sound defense of your position
It's well understood that some words are offensive to some people. Just like I wouldn't advertise my utility as being exceptionally useful in a nigger-rigged or woman written code base, I don't see any advantage to using shitty over poorly written or complex. I recognize that I'm using terms that are a lot more inflammatory, I am trying to illustrate the point with an easily understood example. You might not care about shitty, someone else may not care about nigger-rigged, and someone else may care about both. Since the idea can be expressed without offending any of the group, why offend unnecessarily?
> where the audience by-and-large may not care.
Objectively the audience does care. Otherwise the initial comment would never have been made. We all can let it roll off our backs, but it does affect the author's image among his potential user base.
"Any time somebody tells you that you shouldn’t do something because it’s “unprofessional,” you know that they’ve run out of real arguments." -- Joel Spolsky
Or there is a whole suite of behavior and conduct with well thought out reasons that have been comprehensively debated over the years that have been put under the rubric of professionalism and there is zero reason to rehash the same arguments over and over and over ad infinitum.
It isn't always that easy. For example, where I work, bringing up a full stack locally involves multiple Python processes (servers) with the frontend accessed via the browser.
Using this tool will work immediately, while using pdb would be a bother (comparatively).
> It isn't always that easy. For example, where I work, bringing up a full stack locally involves multiple Python processes (servers) with the frontend accessed via the browser.
I'd add that discovering where to put the debugger and how to gate it when e.g. the issue needs warmup isn't trivial in large projects. Even if you know where to put the debugger (possibly a quest in and of itself) the exact callsite might get hit tens or hundreds of times before the issue shows itself.
And then, Python doesn't have a reverse / time traveling debugger, so hitting the callsite isn't always sufficient to understand the issue.
In all honesty, I think a more useful way to do this would be defining high-level tooling based on ebpf or dtrace, such that you can printf or snoop from the outside without having to perform any source edition.
And possibly combine that with a method to attach a debugger from the outside to a running program, using either PyDev.Debugger or a signal handler triggering a set_trace as a poor man's PDD.
I'm currently learning Python and a little bit of numerical analysis with jupiter notebooks. I currently don't know what I'm doing. Pysnooper looks like it could drop in nicely to give some much needed help.
See also: the unofficial Jupyter extension variable inspector. It shows current values of variables in your scope integrated in a notebook [1]. Pretty slick!
Sometimes using a debugger isn’t always the best way. Sometimes you want to print out and examine lots of data taken from multiple runs/loop iterations and it’s simply easier to consume that information when it’s sitting in front of you all at once.
Maybe it’s just me but I’ve never been happy with what pdb provides. My primary mode of debugging is to jump into a i python console and go line by line. If it’s django as mentioned above, I have to say I have learned so much about Django through the console that I don’t think I’d be anywhere at this point if I’d used a debugger.
If the function is launched in a spawned process, it'll work, though you might have trouble getting the stderr, so you better include a log file as the first argument, like this `@snoop('/var/log/snoop.log')`
If the function launches new processes internally... I'm not sure.
There is a race condition when writting on the same log file from several processes at the same time, which is a typical use case for wsgi frameworks such as django or flask.
I have to say sometimes I do find it much easier to read logs than to muck about in an interactive interpreter. That said I did just add `export PYTHONBREAKPOINT=pudb.set_trace` to my bashrc and am slowly going through and removing all my old `from IPython import embed` lines. A much simpler workflow that doesn't incur major runtime costs.
This looks great, good job! It strikes a great balance between PuDB (my favorite, but can't easily run in Docker/remotely) and q (very simple to use but you need too many print() calls everywhere). PySnooper seems great to just drop in and get the full context of what I want to debug.
Can it be used as a context manager (`with pysnooper.snoop():`) for even finer targeting?
Using print is sometimes superior to pdb. If you want to quickly see how the program runs without stepping through each line print is justified. This looks like an evolved print. Nice tool!
As a very mediocre ruby programmer, is there an equivalent in ruby to this? If you have a little pattern that is more clever than 'puts' everywhere, please share, it would be well received by at least one person out there. Thanks!
Something that prints out the state after running each line of code? No, I don't think so, but I believe it wouldn't be too hard to rig up something similar with TracePoint - https://ruby-doc.org/core-2.2.0/TracePoint.html - which is a built in Ruby way to trace code execution.
For example, dump this into a file and run it:
TracePoint.trace(:line) do |tp|
STDERR.puts "Ran: #{File.readlines(tp.path)[tp.lineno - 1]}"
end
a = 10
b = 5
puts a
a += b
puts a
puts b
It'll print out each line of code as it runs it. You could then parse each line for identifiers and print them out, etc. It'd be a project, but it's doable.
This is great, thanks! I've been wanting this from the q library for ages, but the author hasn't responded in a few years. I'll switch to this right away, thanks again!
I come from the php world now coding in python. There used to be a very nice library from Symfony called VarDumper: https://symfony.com/doc/current/components/var_dumper.html that would just pretty print variables you dumped to the browser in an easy to consume form. Is there anything like this in python?
I think there is a difference because python doesnt require a web browser to run. People are just running programs with no web UI or webserver etc... If you really wanted, you could pipe your debug prints to a file and load in a browser if that is what you want :p
There is pretty-print [0], which you could dump inside pre tags, or print to stdout. But honestly, depending on the framework, quicker options exist than (basically) printf debugging.
Pretty much the stuff already mentioned elsewhere on HN for this article. I often find Werkzeug's excellent debugger is enough: https://news.ycombinator.com/item?id=19718869 . pdb/ipdb/pudb et al (pick your fav) can help for really tricky stuff. And sufficient logging, so you know what's going on at all times even without a debugger attached.
(occasionally, the low effort of print-debugging works, but if you keep having to print in more/different locations... it's a blunt tool IMO)
I try to have "debug calls" that I can call from the browser or Postman or Swagger, etc. and return all sorts of debug information, etc.
I have the impression that local program-specific debugging tools quickly evolve into something like functional tests that uncover issues that functional tests proper might not cover. For example, if I'm serving a ML model but I have a debug call that runs sanity checks on the data that are too expensive to run each time.
We have some fairly intense functions that needs to be profiled with realistic data and io conditions. I've been doing that by hand, line by line in the shell.
I just tried wrapping this around one of these functions in the django shell (actually shell_plus, but same thing). Just imported pysnooper, created a new function using `new_f = pysnooper.snoop()(original_f)` and called new_f on the realistic data and got a nice printout that included values and times.
Is there an equivalent of Common Lisp's TRACE in the Python world?
IMO it is the most value for money (i.e. time and convenience) debugging tool I've used till now. Simply write (TRACE function1 function2 ...) in the REPL and you will get a nicely formatted output of arguments passed and value(s) returned for each invocation of the given functions. Another nice feature is that a deeper an invocation is in the stack the more it is indented -- so recursive functions are fairly easy to debug too.
You can't use it for everything but its sufficient most of the time.
PySnooper looks good, but it is inconvenient in a couple of ways:
1. It prints every line of the traced function -- most of the time this is overkill and not what one needs.
2. To snoop on a function you need to modify the source file. Not a deal breaker but you still have to remember to revert this.
It drops me into IPython, an interactive Python shell, with the interactive Python debugger. You can type variables and use `pdb` primitives like (c)ontinue, (u)p call stack, (n)ext line, etc. I really like it, but ofc YMMV.
Try pdb++ (pdbpp) instead — it's like a much improved ipdb. It monkeypatches itself into pdb rather than defining a new package, so it works from any context that drops into the debugger. Its “sticky mode” alone is worth the switch.
Yeah but the idea is that, with TRACE, most of the time you shouldn't need to drop into the debugger (all CLs come with an interactive debugger too). If TRACE is not sufficient, only then do I go down the debugger route (or strategic print statements).
I've done something similar (but not exact) for Lua [1]. I was testing code to trace Lua execution and found some code that was executed more than I expected [2]. Adding a trace like you described should be easy as well in Lua.
Super neat project. I am a print debugger myself and will definitely use this at some point in the future. For scenarios where PySnooper might be overkill and you just want to see the value of specific variables, I wrote a port of the Rust `dbg` macro for both Python and Go that are pretty nifty when looking at values really quickly:
For my Django projects the development server is werkzeug [1] and anywhere a break point is needed I'll add 1/0 in the code then hit refresh in the browser which pulls up an interactive debugger [2].
thanks for showing me this! I use the more I learn about django-extensions the more impressed I am with its features. Being able to interact with the django shell with a jupyter notebook is awesome too.
The visual of the entire traceback in the browser and the ability to drop into an interactive shell within a specific frame is (for me) a way better experience than pdb.
Before clicking I was like: why would one not use pdb. With vim, python-mode, and 'find . -name "*.py"| entr python testcases.py', setting break point and re-running is painless.
I was wrong. Upon skimming, it seems a huge plus PySnooper have over pdb is auto inspecting states, sparing whole lot of manual typing
Haha. I don't know if this is satire or not, but in case it's not: You can pass any writable stream as the first argument and PySnooper will use it. So it should be easy to integrate with Slack or anything else.
Yes, I was serious. I'm curious why you thought I was joking? cool-RR thought I was joking, as well, so I must be missing something... ¯\_(ツ)_/¯ Thanks for the tip on Sentry -- I'll take a look!
Oh, just that "chat ops" is one of those goofy fads that seemed to flare up everywhere and burn itself out really quickly, so any mention of it immediately triggers my satire meter.
Quite apart from that, the tool described looks like something you'd use in an intensive debugging session, so it's hard for me to imagine how it would fit in with an alerting workflow.
While I sometimes use one debug tool or another, I have never understood the aversion to print statements reflected in the title. Sometimes, often even, a print statement is just fine, and anything else is overabstracting it. Not saying other options aren't nice to have available, just that there is nothing wrong with using a simple print statement in many situations.
Once you go down the path, it's easy to start adding more and more, and it's easy to forget about them. They sometimes blend in well with patches you generate, since it's just a 'print()' and nothing obvious like 'import pdb; pdb.set_trace()'
That's a matter of being organized and competent, not a fault of print().
People create bugs by forgetting things all the time. If you believe their spin, FB snarfed millions of contact lists by accident because of that. I've troubleshot countless things that happened because of code that fell between the cracks. And on the other side of that, I could tell you about the time we went a month without logs in production because someone didn't properly test a library change, so everyone's carefully manicured logging broke.
There's nothing wrong with print(), at least that isn't wrong with a lot of other things.
Because the observability you get with a print statement is limited. You can't do further investigation without modifying the code and running it again. With a debugger you can explore the program state interactively.
My experience with python is limited but in my day job it's not uncommon to be tracking down stuff that happens infrequently. Debug cycles get brutally long.
You also can't be confident that a debugged program is in a natural state without restarting it, at which point the re-setting of the breakpoints and scripting you did in the last invocation - or saving and loading what you've already done - is a pain point. If all of this is quicker than recompiling/relaunching, then IMO there is a second bug in a lack of effective logging.
Most bugs I create are for simple reasons and can be found by scanning the first error logs. If I add print statement debugging because I couldn't then they'll often be adapted into additional logging. If I use a debugger for this as my first tool and don't add logs, I'll have to do it again next time, too[1].
If the bug is not a simple one and is not a structural bug, there's a decent chance it's something debuggers deal with poorly: data races, program boundaries, non-determinism, memory errors. If it's something that can be found by calling a function with certain parameters, it's a missing test case.
So the times I find debuggers to be worth it are after I've already decided it's a difficult yet uncommon bug. So I use them with despair.
[1] If I fix it with a debugger and then add the logs, I still have to prove it gives the right output when it fails.
I agree. Print is the most fundamental tool available to you: debuggers are kernel and hardware spanning monsters that introduce an external logic dependency. We don't advocate for more complexity than is necessary in other places.
174 comments
[ 2.9 ms ] story [ 211 ms ] threadYou can set depth=2 or depth=3 and then you'll get a huge chunk of log. The deeper levels will be indented.
edit: spelling
pdb and its variants (my favorite is PuDB) are generally difficult to use in complex, corporate projects. If you've got a multi-process, multi-thread Python project running on a remote host, you'll need a really full-featured debugger to work with it effectively. I recommend Wing IDE or PyCharm for that.
Certainly not a counter, as I'm less familiar with Wing and Pycharm's debugging features, but both of these have been helpful to me in multi-process, multi-thread python environments.
> You can use it in your shitty, sprawling enterprise codebase without having to do any setup.
Debugging is a sensitive subject, particularly given how frustrating it can be. There’s a place for vulgarity somewhere, but I’d rather see your README provide authoritative info than crack jokes.
May I ask how the word "shitty" more truthful than "poorly written" or "complex"?
Evokes emotions people affected by a situation can sympathize with more effectively, which by virtue of establishing a shared emotional bond over a topic helps the developer convey not just the situation but the frustrations of the situation more effectively than one might expect "poorly written" or "complex" to do alone.
> Citing professionalism isn't a quip, it's shorthand for a code of conduct and long accepted practices of interaction with others.
The code of conduct isn't uniform, so it can't be used effectively as shorthand for such. But at this point we're in the weeds.
Again, no short-term revenue prospects, just a tool OP wants to socialize to make a few lives easier. If you have an objection over verbiage, that's fine, but it's an exhibition of professionalism from yourself to the OP to build a sound defense of your position as to how it would help the engineer to self-censor the description of a tool where the audience by-and-large may not care.
Up to you. My point is the engineer doesn't need to suppress who they are in this specific context, and my point to you is it shouldn't impact your usage of what looks to be an effective short-term debugging tool.
It's well understood that some words are offensive to some people. Just like I wouldn't advertise my utility as being exceptionally useful in a nigger-rigged or woman written code base, I don't see any advantage to using shitty over poorly written or complex. I recognize that I'm using terms that are a lot more inflammatory, I am trying to illustrate the point with an easily understood example. You might not care about shitty, someone else may not care about nigger-rigged, and someone else may care about both. Since the idea can be expressed without offending any of the group, why offend unnecessarily?
> where the audience by-and-large may not care.
Objectively the audience does care. Otherwise the initial comment would never have been made. We all can let it roll off our backs, but it does affect the author's image among his potential user base.
Your comment comes off as out of touch and old fashioned.
Using this tool will work immediately, while using pdb would be a bother (comparatively).
I'd add that discovering where to put the debugger and how to gate it when e.g. the issue needs warmup isn't trivial in large projects. Even if you know where to put the debugger (possibly a quest in and of itself) the exact callsite might get hit tens or hundreds of times before the issue shows itself.
And then, Python doesn't have a reverse / time traveling debugger, so hitting the callsite isn't always sufficient to understand the issue.
In all honesty, I think a more useful way to do this would be defining high-level tooling based on ebpf or dtrace, such that you can printf or snoop from the outside without having to perform any source edition.
And possibly combine that with a method to attach a debugger from the outside to a running program, using either PyDev.Debugger or a signal handler triggering a set_trace as a poor man's PDD.
[1] https://jupyter-contrib-nbextensions.readthedocs.io/en/lates...
If the function is launched in a spawned process, it'll work, though you might have trouble getting the stderr, so you better include a log file as the first argument, like this `@snoop('/var/log/snoop.log')`
If the function launches new processes internally... I'm not sure.
If you try it and it doesn't work, open an issue: https://github.com/cool-RR/PySnooper/issues
https://hackernoon.com/python-3-7s-new-builtin-breakpoint-a-...
Presumably it works with Django? If yes then thrice thanks.
This project would justify an additional dedicated screen just for dumping function debug logs.
Can it be used as a context manager (`with pysnooper.snoop():`) for even finer targeting?
For example, dump this into a file and run it:
It'll print out each line of code as it runs it. You could then parse each line for identifiers and print them out, etc. It'd be a project, but it's doable.https://github.com/deivid-rodriguez/byebug
But just in case if you from print cult, you will like https://github.com/gruns/icecream
[0] https://docs.python.org/3/library/pprint.html#example
(occasionally, the low effort of print-debugging works, but if you keep having to print in more/different locations... it's a blunt tool IMO)
I have the impression that local program-specific debugging tools quickly evolve into something like functional tests that uncover issues that functional tests proper might not cover. For example, if I'm serving a ML model but I have a debug call that runs sanity checks on the data that are too expensive to run each time.
I just tried wrapping this around one of these functions in the django shell (actually shell_plus, but same thing). Just imported pysnooper, created a new function using `new_f = pysnooper.snoop()(original_f)` and called new_f on the realistic data and got a nice printout that included values and times.
Very useful.
IMO it is the most value for money (i.e. time and convenience) debugging tool I've used till now. Simply write (TRACE function1 function2 ...) in the REPL and you will get a nicely formatted output of arguments passed and value(s) returned for each invocation of the given functions. Another nice feature is that a deeper an invocation is in the stack the more it is indented -- so recursive functions are fairly easy to debug too.
You can't use it for everything but its sufficient most of the time.
PySnooper looks good, but it is inconvenient in a couple of ways:
1. It prints every line of the traced function -- most of the time this is overkill and not what one needs. 2. To snoop on a function you need to modify the source file. Not a deal breaker but you still have to remember to revert this.
``` import ipdb ipdb.set_trace() ```
It drops me into IPython, an interactive Python shell, with the interactive Python debugger. You can type variables and use `pdb` primitives like (c)ontinue, (u)p call stack, (n)ext line, etc. I really like it, but ofc YMMV.
[1] http://boston.conman.org/2015/02/12.1
[2] http://boston.conman.org/2015/02/13.1
https://ipython.readthedocs.io/en/stable/interactive/magics....
https://github.com/tylerwince/pydbg
https://github.com/tylerwince/godbg
[1] https://django-extensions.readthedocs.io/en/latest/runserver...
[2] https://werkzeug.palletsprojects.com/en/0.15.x/debug/#using-...
https://docs.python.org/3/library/functions.html#breakpoint
I was wrong. Upon skimming, it seems a huge plus PySnooper have over pdb is auto inspecting states, sparing whole lot of manual typing
Quite apart from that, the tool described looks like something you'd use in an intensive debugging session, so it's hard for me to imagine how it would fit in with an alerting workflow.
People create bugs by forgetting things all the time. If you believe their spin, FB snarfed millions of contact lists by accident because of that. I've troubleshot countless things that happened because of code that fell between the cracks. And on the other side of that, I could tell you about the time we went a month without logs in production because someone didn't properly test a library change, so everyone's carefully manicured logging broke.
There's nothing wrong with print(), at least that isn't wrong with a lot of other things.
Or you can be organized and competent at a higher level and just use tools. We're using computers for a reason :)
They're very visible in diffs then. And even if accidentally committed they're still easy to spot and eliminate later.
My experience with python is limited but in my day job it's not uncommon to be tracking down stuff that happens infrequently. Debug cycles get brutally long.
Most bugs I create are for simple reasons and can be found by scanning the first error logs. If I add print statement debugging because I couldn't then they'll often be adapted into additional logging. If I use a debugger for this as my first tool and don't add logs, I'll have to do it again next time, too[1].
If the bug is not a simple one and is not a structural bug, there's a decent chance it's something debuggers deal with poorly: data races, program boundaries, non-determinism, memory errors. If it's something that can be found by calling a function with certain parameters, it's a missing test case.
So the times I find debuggers to be worth it are after I've already decided it's a difficult yet uncommon bug. So I use them with despair.
[1] If I fix it with a debugger and then add the logs, I still have to prove it gives the right output when it fails.