Ask HN: Good Python codebases to read?

Thanks! I am also looking into using Flask soon too. I dabbled in django for a while, but too much magic was happening and I am trying to learn.

  # ignore south migrations, and external libraries.
  # You can alter this if you care about reading libs' codebase
  alias pygrep='grep -rin --include=*.py --exclude=*.pyc --exclude-dir=lib --exclude-dir=migrations'

Also MarkupSafe - https://github.com/mitsuhiko/markupsafe

It's a little bit meta, but I had lots of "wow" moments. Also, it's a nice example of using C to speed up certain operations.

Armin Ronacher often writes pretty and compact Python — https://github.com/mitsuhiko

Flask was the framework that finally made web frameworks "click" for me - the biggest advantage of learning about Flask is you can build up your knowledge of web services piece by piece, allowing you to take time and fully understand each component before moving on to the next. That way you're not trying to comprehend all the "magic" that's going on behind the scenes all at once.

Thanks, I've been hearing a lot about this package, I think it's one I will check out.

Cheers!

requests is very useful, but it always gets mentioned as a good python codebase and I'm not sure I agree. One example:

The first thing many users will do is

  requests.get?

Which tells them that it takes some kwargs, but doesn't tell them what those kwargs are. It's easy, especially for a newcomer, to read "optional arguments that `request` takes" and fail to understand that they should look up the docs on (not-really-encouraged-as-part-of-public-API) function `request`. That's pretty bad; those kwargs are important! (The reason is because requests.get is implemented as a call to request(method, ..., kwargs) but the user doesn't care what the implementation-level reason is.)

Beyond that I did look into the codebase once to investigate a possible bug and there were a few python style things I wanted to fix, but I don't remember them so this comment probably sounds kind of annoying (it would annoy me if I were reading it not writing it...). It didn't strike me a really clean codebase. But yes the library is very useful and I'm sure it's a pretty decent python codebase.

  >>> requests.get?
  Signature: requests.get(url, params=None, **kwargs)
  Docstring:
  Sends a GET request.
  
  :param url: URL for the new :class:`Request` object.
  :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
  :param \*\*kwargs: Optional arguments that ``request`` takes.

Here are all the kwargs the user probably wanted to know but failed to find out and was forced to either browse online docs or the source code:

  def request(method, url, **kwargs):
      """Constructs and sends a :class:`Request <Request>`.
      :param method: method for the new :class:`Request` object.
      :param url: URL for the new :class:`Request` object.
      :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
      :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
      :param json: (optional) json data to send in the body of the :class:`Request`.
      :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
      :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
      :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': ('filename', fileobj)}``) for multipart encoding upload.
      :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
      :param timeout: (optional) How long to wait for the server to send data
          before giving up, as a float, or a (`connect timeout, read timeout
          <user/advanced.html#timeouts>`_) tuple.
      :type timeout: float or tuple
      :param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.
      :type allow_redirects: bool
      :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
      :param verify: (optional) if ``True``, the SSL cert will be verified. A CA_BUNDLE path can also be provided.
      :param stream: (optional) if ``False``, the response content will be immediately downloaded.
      :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
      :return: :class:`Response <Response>` object
      :rtype: requests.Response

  __shared_docstring_content = """
  bar
  baz
  """

  def f():
      "f docstring"
      pass

  f.__doc__ += __shared_docstring_content


  def g():
      "g docstring"
      pass

  g.__doc__ += __shared_docstring_content

Thanks for the recommendations, I think they may be a bit above me at the moment (I don't have enough knowledge of Python to be able to identify dust under the carpet :)

But you did teach me one thing, I didn't know that the """ could be used for commenting. I thought it was only for multi-line print statements.

These are exactly the type of things I hope to learn while code-reading.

Cheers!

I have been debating whether or not to read the standard library, pro - it's the standard library, if I learn it I may save myself innumerable hours from reinventing wheels, con - it may not be the best example of code (thinking about the 400 line function from vim to see if there is input from the keyboard. Justified in context, but would be a horrible way to learn some code)

Bazaar I may look into, as I know it's a very capable piece of software, and now I know it's well documented with tests makes it very relevant to my interests :)

Edit: I'm on windows, but I'm using vagrant with Ubuntu 14lts image for my development work. Specifically the data science vagrant box...

This one could be very interesting as I have decades of experience with DBs and SQL, so I was always wondering about ORMs, and sqlalchemy seems to be best in field at the moment.

Thanks for the recommendation.

Cheers!

You might actually find Dogpile (also by Mike Bayer) more approachable and useful in this context. Even the separation of the project into logically distinct packages is instructive.

- https://bitbucket.org/zzzeek/dogpile.core - https://bitbucket.org/zzzeek/dogpile.cache

Thanks! Not sure if I am ready to take on OOP and mixins in Python yet, but I do like the idea of module organization so it's definitely on my list now.

Cheers!

Cool, thanks, didn't realize openstack was Python. It may be too much for me to take on at my level, but it is definitely on my 'to look at' list.

Cheers!

Openstack is so large it also has a plethora of examples of bad code. Especially in vendor-contributed drivers and such. I wouldn't pay attention to the code so much as the incredible infrastructure and process that manages it and improves it over time.

Seriously, I've seen "sychronization threads" that use no synchronization primitives that have a single bare exception handler trap the critical section. The handler just restarts the computation... leading to fun times.

Cinder also has an interesting problem in the explosion of ABC "mixins" used for constructing a backend driver. It went from 5 in the previous release to something like 25 presently. A patch to fix it: https://review.openstack.org/#/c/201812/

But even the process isn't perfect. I've seen patches that change a couple of log lines get dog-piled with -1 nit-picks about inconsequential wording and take months to get through.

Openstack is an interesting beast but it's not a good example if you're just learning, IMO.

I know it is a good project, but would you recommend it for someone to read through the codebase?

(Honestly, looking at the repo, I don't even know where to start if I wanted to do a read through. Has anyone created a map? :)

I don't know what would be of my life right now without Django and I can't praise it enough... but "good example of readable python-code", it is not.

Holy crap! I don't know if this helps or hinders! ;)

Thank you for this resource, much appreciated!

+1 to anything written by Kenneth Reitz, he's also a fantastic OSS maintainer and extremely welcoming of newbies and their PR's if you want to put some of your learning into practice

I would add the Django project to that list as it's a very large, mature, and successful open source python project - https://github.com/django/django

SQLAlchemy is one of my favourite larger python codebases: https://github.com/zzzeek/sqlalchemy

I've had to read the source of boto since there were some boto exceptions I was seeing in stack traces in the fabric deploy process - and ugh, I found the code to be not very intuitive and the documentation poor. Anyone else run into the same issues with boto?

Boto3 might also be worth a read. https://github.com/boto/boto3

His coding style is not common to most writers of Python.

He's using overly terse, poorly descriptive variable names, not using multiline strings for docstrings, not indenting where he should, and one-lining if/elif statements and function definitions. This style does not contribute to readability.

This is not how I would want someone just learning Python to learn it.

  absolute_path = lambda path: path if path.startswith('/') else '/' + path

  def absolute_path(path):
      if not path.startswith('/'):
          path = '/' + path
      return path

  >>> absolute_0 = lambda path: path if path.startswith('/') else '/' + path
  >>> def absolute_1(path):
  ...     '''Return the absolute unix path from a given path name'''
  ...     if not path.startswith('/'):
  ...         path = '/' + path
  ...     return path
  >>> import dis
  >>> dis.dis(absolute_0)
    2           0 LOAD_FAST                0 (path)
                3 LOAD_ATTR                0 (startswith)
                6 LOAD_CONST               1 ('/')
                9 CALL_FUNCTION            1
               12 POP_JUMP_IF_FALSE       19
               15 LOAD_FAST                0 (path)
               18 RETURN_VALUE        
          >>   19 LOAD_CONST               1 ('/')
               22 LOAD_FAST                0 (path)
               25 BINARY_ADD          
               26 RETURN_VALUE        
  >>> dis.dis(absolute_1)
    4           0 LOAD_FAST                0 (path)
                3 LOAD_ATTR                0 (startswith)
                6 LOAD_CONST               1 ('/')
                9 CALL_FUNCTION            1
               12 POP_JUMP_IF_TRUE        28
    5          15 LOAD_CONST               1 ('/')
               18 LOAD_FAST                0 (path)
               21 BINARY_ADD          
               22 STORE_FAST               0 (path)
               25 JUMP_FORWARD             0 (to 28)
    6     >>   28 LOAD_FAST                0 (path)
               31 RETURN_VALUE        

  >>> absolute_0.__name__
  '<lambda>'
  >>> absolute_0.__name__ = '<lambda "absolute_0">'
  >>> absolute_0.__name__
  '<lambda "absolute_0">'

  >>> absolute_0.__doc__
  >>> absolute_0.__doc__ = 'Return the absolute unix path from a given path name'
  >>> help(absolute_0)
  Help on function <lambda "absolute_0">:
  <lambda "absolute_0">(path)
    Return the absolute unix path from a given path name

  # Mandelbrot set
  print (lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+y,map(lambda y,
  Iu=Iu,Io=Io,Ru=Ru,Ro=Ro,Sy=Sy,L=lambda yc,Iu=Iu,Io=Io,Ru=Ru,Ro=Ro,i=IM,
  Sx=Sx,Sy=Sy:reduce(lambda x,y:x+y,map(lambda x,xc=Ru,yc=yc,Ru=Ru,Ro=Ro,
  i=i,Sx=Sx,F=lambda xc,yc,x,y,k,f=lambda xc,yc,x,y,k,f:(k<=0)or (x*x+y*y
  >=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr(
  64+F(Ru+x*(Ro-Ru)/Sx,yc,0,0,i)),range(Sx))):L(Iu+y*(Io-Iu)/Sy),range(Sy
  ))))(-2.1, 0.7, -1.2, 1.2, 30, 80, 24)
  #    \___ ___/  \___ ___/  |   |   |__ lines on screen
  #        V          V      |   |______ columns on screen
  #        |          |      |__________ maximum of "iterations"
  #        |          |_________________ range on y axis
  #        |____________________________ range on x axis

  >>> sorted(((x, -(x**2)) for x in xrange(10) if 0 == x % 2), key=lambda item: item[1])
  [(8, -64), (6, -36), (4, -16), (2, -4), (0, 0)]

  >>> sorted(((x, -(x**2)) for x in xrange(10) if 0 == x % 2), key=operator.itemgetter(1))
  [(8, -64), (6, -36), (4, -16), (2, -4), (0, 0)]

  >>> len('lambda item: item[1]')
  20
  >>> len('operator.itemgetter(1)')
  22

  >>> def named_lambda(procedure, name, documentation=''):
  ...     procedure.__name__ = '<lambda {}>'.format(name)
  ...     procedure.__doc__ = documentation
  ...     return procedure
  ... 
  >>> absolute_path = named_lambda(lambda path: path if path.startswith('/') else '/' + path, 'absolute_path', 'Return the absolute unix path from a given path name')
  >>> absolute_path.__name__
  '<lambda absolute_path>'
  >>> absolute_path.__doc__
  'Return the absolute unix path from a given path name'

  >>> named_lambda.func_name
  'named_lambda'
  >>> named_lambda.__name__
  'named_lambda'
  >>> named_lambda.__name__ = 'lol'
  >>> named_lambda.func_name
  'lol'
  >>> named_lambda.func_name = 'named_lambda'
  >>> named_lambda.func_name
  'named_lambda'

  iterable = xrange(10)

  ge = (x*x for x in iterable if not x % 2)

  mf = map(lambda x: x*x, filter(lambda x: not x % 2, iterable))

  def absolute_path(path):
      if not path.startswith('/'):
          return '/' + path
      return path

  def absolute_path(path):
      return path if path.startswith('/') else '/' + path

    if True:
        a = 0 if True else 1
    else:
        a = 2

    print(a)

Norvig's code have several qualities to praise

Adherence to Python's best practices / pep-8 are not one of them

Putting on my "Python teacher" hat for a moment (I also work at Nylas), our codebase is interesting for learning purposes because it's a fairly homogenous, 'live' product managed by a commercial engineering team. Definitely on the complex side of things, though -- let me know if you want any help figuring it out (we also have a community Slack at http://slack-invite.nylas.com/).

On the testing side, I was chatting to a fellow Pyladies attendee about how we do test fixture setup and teardown (especially around databases) -- you might find that interesting to look at too.

I clicked through on this and browsed a few directories, none of which seemed likely. I do not see any `.py` files aside from an empty `__init__.py` and the `setup.py`. Are you sure a beginner just learning Python should see this?

Don't let it turn you into a fucking pedant, though.

agreed: I also learned a lot from nltk too, esp. freqdist/probdist as well as the classifiers in nltk and sklearn

Agreed. NLTK's value is not that it should be your go-to source for NLP algorithms -- when there's a specific task you need to accomplish, there'll be a specific solution by now that works better than NLTK.

NLTK's value is that it shows you how to write NLP algorithms, and gives you an understandable starting point when you need to do something that nobody has implemented yet.