Python's splitlines does more than just newlines (yossarian.net)

Given that encoded characters must have one and only one General_Category value, it might be too imprecise or arbitrary in some cases. If you ever need more power, it's worth browsing the other character properties Unicode exposes. For example, `Lu` (Uppercase_Letter) only covers some uppercase letters, whereas the `Uppercase` property covers all of them.

---

For anyone that wants to learn more about specific Unicode stuff, the three big data sources are The Core Spec, the Unicode Technical Annexes (UAXs), and the Unicode Character Database itself (the database is a bunch of text files. There's an XML version now as well).

For further reading on this specifically, it might be worth looking at:

[Unicode Core Spec - Chapter 4: Character Properties] https://www.unicode.org/versions/Unicode17.0.0/core-spec/cha...

├ [General Category] https://www.unicode.org/versions/Unicode17.0.0/core-spec/cha...

└ [Properties for Text Boundaries] https://www.unicode.org/versions/Unicode17.0.0/core-spec/cha...

[UAX #44 - Unicode Character Database (Technical Report)] https://www.unicode.org/reports/tr44/

├ [General Category Values] https://www.unicode.org/reports/tr44/#General_Category_Value...

└ [Property Definitions] https://www.unicode.org/reports/tr44/#Property_Definitions

And, if you're brave and want to see the data itself (skim through UAX #44 first):

[Unicode Character Database] https://www.unicode.org/Public/17.0.0/ucd/

There's so many super useful things in the Python docs that you never see in the wild. For example, I recently learned that the sqlite3 module has a set_authorizer function that lets you limit the types of statements that can be run / tables that can be accessed.

https://www.sqlite.org/c3ref/set_authorizer.html

https://docs.python.org/3/library/sqlite3.html#sqlite3.Conne...

It has similar (but not identical) behaviour though:

  >>> for line in StringIO("foo\x85bar\vquux\u2028zoot"): print(line)
  ... 
  foo
  bar
   quux zoot

  >>> from io import StringIO
  >>> for line in StringIO("foo\x85bar\vquux\u2028zoot"): print(repr(line))
  ... 
  'foo\x85bar\x0bquux\u2028zoot'

If it's reading from a file, you wouldn't be using splitlines() anyway; you'd use readlines().

For string you’d need to

  import io

  for line in io.StringIO(str):
    pass

splitlines() is sometimes nice for adhoc parsing (of well behaved stuff...) because it throws out whitespace-only lines from the resulting list of strings.

#1 use-case of that for me is probably just avoiding the cases where there's a trailing newline character in the output of a command I ran by subprocess.

In that example str.split() has the same result as str.splitlines(), but it's not in general the same, even without custom delimiter.

str.split() splits on runs of consecutive whitespace, any type of whitespace, including tabs and spaces which splitlines() doesn't do.

    >>> 'one two'.split()
    ['one', 'two']
    >>> 'one two'.splitlines()
    ['one two']

split() without custom delimiter also splits on runs of whitespace, which splitline() also doesn't do (except for \r\n because that combination counts as one line ending):

    >>> 'one\n\ntwo'.split()
    ['one', 'two']
    >>> 'one\n\ntwo'.splitlines()
    ['one', '', 'two']

One thing to note tho is that they take character sets, as long as they encounter characters in the specified set they will keep stripping. Lots of people think if you give it a string it will remove that string.

That feature was added in 3.9 with the addition of `removeprefix` and `removesuffix`.

Sadly,

1. unlike Rust's version they provide no way of knowing whether they stripped things out

2. unlike startswith/endswith they do not take tuples of prefixes/suffixes