HN Explorer

Ask HN: What do you think about type coverage in Python?

2 points by stefanhoelzl ↗ HN
I was thinking about a concept lately I would refer to as type coverage.

The idea behind it is to measure how well a python function is tested by recording all inputs/outputs during a complete test run and comparing the values with the annotated types.

e.g. a function `foo(a: Optional[float])` which is only tested with `foo(5)` gets a low coverage because `foo(None)` is not being tested. Or it could be hinted that a more feasible type annotation would be `foo(a: int)`.

As a use case I was thinking of testing APIs, to make sure you covered all use cases of your API which you advertise by the annotated types.

An extension of this concept could be that you check how extensive you tested a type. Like did you test `foo(a: int)` with negative, positive and zero values? If not it could be a hint that your test coverage is too low or you have a wrong type, maybe a enum would be suited better.

I am curious to hear about your thoughts about this concept.

4 comments

[ 3.0 ms ] story [ 22.9 ms ] thread
Uberphallus
While not exactly the same, you can obtain the same benefits with mypy.[0] It will perform all of what you say staticaly, without any test, though obviously it will not be smart enough when you use libraries that aren't using annotated types. But if you test for those, then you obtain 100% of the functionality that you described, unless I misunderstood something.

[0] https://mypy.readthedocs.io/en/stable/

stefanhoelzl
mypy accepts much broader types then the concept I was proposing. Lets take this example:

  def foo(bar: Optional[float]) -> Optional[float]:
      return bar

  def test_foo():
      assert foo(5) == 5
It has 100% code coverage and mypy will accept it.

What I proposed will complain in this case, that you never tested `foo` with `None` or `float` as input parameters and that it never has seen `foo` returning `None` or `float`.

So even with 100% code coverage and mypy not complaining 'type coverage' could give you some hints that a) your tests are not extensive enough or b) your annotated types are too broad

Uberphallus
Yeah, but in practice you rarely (if ever) call foo() with a literal. In normal code you'll use variables or return values, and if they're annotated, mypy will catch mismatching types.

What you propose will certainly solve the issue when dealing with unannotated code, though.

stefanhoelzl
Yes exactly, mypy will catch mismatching types. But in my example I do not have mismatching types, but still can detect an issue when testing a function which expects `Optional[float]` only with an `int`.

Regarding literals or annotated variables: mypy will detect mismatching types in both cases.