HN Explorer

69 comments

[ 3.1 ms ] story [ 29.6 ms ] thread
fulafel
Switching the default encoding from ASCII to utf-8 sounds like a pretty big change.
akx
Not really, imho. It makes more things work out of the box, but input and output for things that used to be ASCII anyway remain the same.
[dead] user5994461
Are you American? Americans always think that changing the encoding is a minor detail.

Guess what, changing the default encoding breaks every language that is not English.

noselasd
If it was changed from ascii to utf-8, how does it break everyting else ?
user5994461
ASCII doesn't exist as such.

Characters are encoded in one byte and ascii leaves the highest 128 characters undefined. They are interpreted system-wise, depends on the locale and user settings.

warbiscuit
"ascii" the codec does exist under python. It's strictly defined as byte values 0-127, anything in the 128-255 range causes a decoding error...

    >>> b"abc\xf0".decode("ascii")
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 3
Thus, moving the default from "ascii" to an ASCII-superset should have no decoding issues for previously valid files, since those bytes were never valid to start with.
noselasd
ASCII very much exists - but what you describe is an entierly different thing. Sure - if the change is from default-user-locale to UTF-8, things would break.

But if the changelog is truthful, and the change is from ASCII to UTF-8, things should not break - at least not to any great extent.

[deleted]
(comment deleted)
akx
I'm Finnish, thank you for asking, so I'm quite aware of different encodings.
JupiterMoon
Is Finnish representable in Latin1? If so you might be missing some of the suitabilities of converting from "asci" (most things around Europe weren't asci but latin1) to utf.
coldtea
Isn't ASCII a strict subset of UTF-8?

If anything this should make sure programs don't break because they read some utf-8 extended character when expecting just ascii -- which is the most common case.

fulafel
I didn't say it's a bad change!

But since you raise the issue, there are downsides: Many programs developed on 3.7 will inadvertently break on pre-3.7 Python versions. As we know, this is a big thing on parts of the Python ecosystem because users frequently rely on the Python version that come with their OS or framework (eg AWS Lambda), and many developers want to stay compatible with the most common platform-shipped Python versions.

ericfrederich
It's a good assumption for 3.7. In isolation this is a pragmatic decision and makes sense.

You are correct though. A .py file which uses non-ascii and doesn't declare the encoding should be considered an error though. Hopefully everybody's dev tools will pick this up (even on 3.7 where they wouldn't need to)

VHRanger
The only real question is:

When am I going to be able to name my variable as emojis?

agumonkey
When Apple releases a MacBook Pro emoji keys only
dbcurtis
You've been able to do that for a while.

edit: I take that back. Apparently thebunicode allowed in identifers is not unrestricted..... in my defense I don't use unicode identifiers as a rule....

masklinn
You can use non-ascii characters but not emoji as arbitrary symbols are not generally classified as either XID_Start or XID_Continue.
masklinn
Well Python 3 broadly follows UAX 31 "UNICODE IDENTIFIER AND PATTERN SYNTAX", and emoji don't have the xid_start or xid_continue properties, so… never unless you decide to be the change to believe in and get the Unicode consortium to change its ways?
dorfsmay
"More than 255 arguments can now be passed to a function, and a function can now have more than 255 parameters."

There are human devs who needed this? Or is this a sign that AI bots are now involved in language design?

akx
Could be useful for code autogeneration of some ilk.
orf
Perhaps it could trigger if you do function(*iterable), with a huge iterable?
drizze
I think *iterable would just pass one argument that's a list, not pass each argument individually
fiddlerwoaroof 

    >>> print([1,2,3])
    [1,2,3]
    >>> print(*[1,2,3])
    1 2 3
aisofteng
That doesn't tell you anything about when the list is expanded.
dragonwriter
The asterisk at the call site (where it turns an iterable into multiple passed arguments) not the function definition site (where it collects otherwise unconsumed arguments into a single list.)
orf
There may be some optimization at the interpreter level, but it appears to Python that each element of 'iterable' is passed as an individual argument
winterlight
Nope, that works just fine.

  >>> def f(*x): print(x[-1])
  >>> f(range(300))
  299
And surprisingly, in python 2.7, I was able to define a function that takes more than 255 arguments. But perhaps this only worked because I cheated and used exec.

  >>> exec("def f(" + ",".join("f" + str(x) for x in range(300)) + "): print(f299)"
  >>> f(*range(300))
  299
qquark
Interestingly enough, the function def (the exec part) does not work in 3.6, as one could expect, but in 2.7 even though the function definition works, you can make the call fail by generating actual parameters:

  >>> exec("def f(" + ",".join("f" + str(x) for x in range(300)) + "): print(f299)")
  >>> f(*range(300))
  299
  >>> exec("f(" + ", ".join(str(i) for i in range(300)) + ")")
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<string>", line 1
  SyntaxError: more than 255 arguments
When reducing the number of parameters to be invalid for f, the following error is shown:

  >>> exec("f(" + ", ".join(str(i) for i in range(30)) + ")")
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<string>", line 1, in <module>
  TypeError: f() takes exactly 300 arguments (30 given)
Which seems a bit paradoxical - "You need 300 arguments" - "No, wait, actually, you can't have more than 255" ... ;)

It looks like they either had changed the internals of python 3, making it fail at the definition as a side effect instead of when calling, or used the same underlying logic but purposely chose to issue the exception to potentially catch the problem earlier...

Waterluvian
I believe args and *kwargs have special opcodes already.
twoodfin
First thing that pops to mind is ORM for extremely column-wide tables, which are not unusual in some domains.
userbinator
I'm not too familiar with Python but something like that would typically be handled by passing a pointer to a data structure containing the column data, and not each column itself (or does/did it have a 255-item limit on its other data structures too...?)
twoodfin
OK, but then you need to populate the data structure. You can do it with individual member assignments of course, but some quick Googling (I am not a Python expert either) suggests that the default row/object constructor for SQLAlchemy is an arg-per-field function.
nerdponx
On python-dev a few month ago, Guido agreed with you that this is an arbitrary limitation that should be removed at some point. In particular, he worried that programmatically generated code would tend to run into this limitation.[0]

[0]: https://bugs.python.org/msg142998

TCM
They have issued tagged with the bug report and adding more arguments to named tuple and to general functions seems to be the issue. The motivation for some of these changes were due to automatic code generation.

See https://bugs.python.org/issue18896 and https://bugs.python.org/issue12844 for more information.

Also, Django 1.3 had issues with 255 arguments for named tuples.

To my knowledge there are no "AI Bots" that are involved in language design. Code generation is possible. Also solver aided languages See https://emina.github.io/rosette/

sbergot
Saw one of those in a vb.net shop. It went that way:

- someone creates a function to do something on a business concept that was not materialized by a class. Think of a bunch (~ 8) of properties that belong together but for some bad reason are moved in separate variables all over the place

- another function has to work on a set of those objects. The dev who implements it thinks it would be a good idea to keep all parameters separate. No we have nproperties * nobjects parameters. The dev use a rigorous naming convention for readability (creatorFirstName, creatorLastName, validatorFirstName, etc)

- now we have multiple sets (creatorFirstName1, 2, 3 ...)

I don't remember the details but there was around three hundred parameters.

edit: formatting

raymondh
Guido had stated that he didn't want the implementation to place arbitrary limits on the language specification.

This particular limit was not present in the early Python 2 releases.

When the limit was added, limiting arguments wasn't the goal; instead, it was just an artifact of a patch regarding how keyword arguments and positional arguments we encoded in Python's bytecode scheme.

Kpourdeilami
Unrelated to python 3.7, but they recently added this feature that you can add two dictionaries together in 3.5 by doing:

  new_dict = {**dict1, **dict2}
It is so handy and nice
agumonkey
I didn't know about that. It's also unbounded:

    >>> d = {**{1:2,2:2,3:2}
            ,**{4:2,5:2,6:2}
            ,**{7:2,8:2,9:2}}
    >>> d
    { 1: 2, 2: 2, 3: 2
    , 4: 2, 5: 2, 6: 2
    , 7: 2, 8: 2, 9: 2}
JupiterMoon
What happens to duplicate keys?
Kpourdeilami
They get overwritten by the value of the dictionary that came last.

  >>> dict1 = {"key1": 1, "key2": "hello", "key3": [1, 2, 3]}
  >>> dict2 = {key1": 1, "key2": "world", "key3": [1, 4], "key4": "4"}

  >>> {**dict1, **dict2}
  {'key1': 1, 'key2': 'world', 'key3': [1, 4], 'key4': '4'}
ericfrederich
Lots of code to show that. How 'bout this: ;-)

  >>> {**{'a': 'first'}, **{'a': 'second'}}
  {'a': 'second'}
masklinn
Last pair wins, same as setting an existing key or doing an update() with an existing key.
joelthelion
Ah, great. This used to be rather clunky to do.
saurik
Really? This is how that was done in Python 2.7 and it doesn't seem that much clunkier to me...

    dict(dict1, **dict2)
snthpy
Interesting. I never thought of that. I also asked a question about this on StackOver flow once and no one suggested that. Very useful.
heinrich5991
That's only allowed if the keys in the second dictionary are valid Python identifiers (don't contain spaces etc.). I think CPython is OK with this, but PyPy is not.
mixmastamyk
Why doesn't the + symbol work?
Kpourdeilami
+ is not a supported operand for dict. You can probably subclass dict and add the plus operand
mixmastamyk
Why add that crazy syntax when a + would be a lot clearer?
akvadrako
It makes sense in context. {} is a dict literal, and * * dict is already used other places to expand a dict into key-value pairs.

Given

  a = {1: 1}
  b = {2: 2}
This

  c = {**a, **b}
is equivalent to

  c = {1: 1, 2: 2}
namelost
a|b would be the obvious choice, because it already works for set:

    {1,2}|{2,3} == {1,2,3}
I guess the reason they went for the weird {∗∗a, ∗∗b} syntax is that it is not a commutative operation.
akvadrako
I guess I didn't explain it clearly enough. ∗∗a is not new syntax, it's just applicable in a new context.
ericfrederich
Correct. This is called orthogonality. I just used it now to discover that this same syntax works for both sets and lists (using * instead of )

  >>> s1 = {'a', 'b', 'c'}
  >>> s2 = {1, 2, 3}
  >>> {*s1, *s2}
  {'b', 1, 2, 3, 'c', 'a'}

  >>> l1 = ['a', 'b', 'c']
  >>> l2 = [1, 2, 3]
  >>> [*l1, *l2]
  ['a', 'b', 'c', 1, 2, 3]
Twirrim
Some nice optimisations included:

    Added two new opcodes: LOAD_METHOD and CALL_METHOD to avoid instantiation of bound method objects for method calls, which results in method calls being faster up to 20%. (Contributed by Yury Selivanov and INADA Naoki in bpo-26110.)
    Searching some unlucky Unicode characters (like Ukrainian capital “Є”) in a string was to 25 times slower than searching other characters. Now it is slower only by 3 times in worst case. (Contributed by Serhiy Storchaka in bpo-24821.)
    Fast implementation from standard C library is now used for functions erf() and erfc() in the math module. (Contributed by Serhiy Storchaka in bpo-26121.)
    The os.fwalk() function has been sped up by 2 times. This was done using the os.scandir() function. (Contributed by Serhiy Storchaka in bpo-25996.)
    Optimized case-insensitive matching and searching of regular expressions. Searching some patterns can now be up to 20 times faster. (Contributed by Serhiy Storchaka in bpo-30285.)
    selectors.EpollSelector.modify(), selectors.PollSelector.modify() and selectors.DevpollSelector.modify() may be around 10% faster under heavy loads. (Contributed by Giampaolo Rodola’ in bpo-30014)
edit: That one around the unlucky Unicode characters is fascinating. https://bugs.python.org/issue24821

Due to the way that regex searching is optimised, it trips up because: ".. the lowest byte of the code of Ukrainian capital letter Є (U+0404) matches the highest byte of codes of most Cyrillic letters (U+04xx). There are similar issues with some other scripts ..."

CaliforniaKarl
> Support for building --without-threads is removed. (Contributed by Antoine Pitrou in bpo-31370.).

Woooo! That means, at some future time, I'll be able to remove code checks for non-threading Pythons.

viraptor
What kind of checks did you use?
raymondh
This patch was a little aggressive and may get reverted (it caused immediate code breakage in downstream distros and projects). Changes like this typically need a deprecation period (that said, to the extent the current feature didn't really work in the first place, we can immediately take it out).
Dowwie
What version will contain dataclasses?
ericfrederich
What do you mean by this? Do namedtuples not give you this?

  >>> class MyClass(namedtuple('_MyClass', ['x', 'y', 'z'])):
  ...   def foo(self):
  ...     return self.x * self.y + self.z
  ... 
  >>> x = MyClass(4, 5, 6)
  >>> x.foo()
  26
... or an attrs decorated class with frozen=True?

  >>> import attr
  >>> @attr.s(frozen=True)
  ... class MyClass:
  ...   x = attr.ib()
  ...   y = attr.ib()
  ...   z = attr.ib()
  ...   def foo(self):
  ...     return self.x * self.y + self.z
  ... 
  >>> x = MyClass(4, 5, 6)
  >>> x.foo()
  26