'typing' talk poster

typing module, Gradual typing and mypy

Python with Graz Uhrturm

2019-09-03, Lukas Prokop, pygraz

Python Enhancement proposals

PEPs

But that was just the beginning …

A short reminder (via PEP 0001)

Python Enhancement Proposal Flow

PEPs, part 2

PEPs, part 3

PEPs, part 4

## Pre-PEP 3107 [PyPI: accepts module](https://pypi.org/project/accepts/) ```python >>> @accepts(int) ... def inc(value): ... return value+1 >>> inc(1) # ok >>> inc(1.5) # exception TypeError: .... >>> # multiple types ... @accepts((int,float)) >>> # None ... @accepts((int,float,type(None))) ```
## PEP 3107 (2006) > This PEP introduces a syntax for adding arbitrary metadata annotations to Python functions 1. Function annotations are completely optional. 2. Function annotations only associate arbitrary Python expressions with various parts of a function at compile-time. 3. No attempt to introduce any standard semantics. Left to third-party libraries.
## PEP 3107 (2006) Available since Py3k. Documentation? Type checking? ```python def compile(source: "something compilable", filename: "where the compilable thing comes from", mode: "is this a single statement or a suite?"): ... def haul(item: Haulable, *vargs: PackAnimal) -> Distance: ... ```
## PEP 3107 (2006) ```python def foo(a: expression, b: expression = 5): ... ``` 1. annotations always precede a parameter's default value 2. annotations and default values are optional 3. all annotation expressions are evaluated when the function definition is executed, just like default values
## PEP 3107 (2006) Return values: ```python def sum() -> expression: ... ``` * U+002D HYPHEN-MINUS * U+003E GREATER-THAN SIGN
## PEP 3107 (2006) Annotations for nested parameters always follow the name of the parameter, not the last parenthesis. Annotating all parameters of a nested parameter is not required: ```python def foo((x1, y1: expression), (x2: expression, y2: expression)=(None, None)): ... ```
## Nested parameters ```python Python 2.7.15+ (default, Nov 27 2018, 23:36:35) [GCC 7.3.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> def f((a, b), c): return a**b + c ... >>> f((2, 4), 26) 42``` → [PEP 3113 “Removal of Tuple Parameter Unpacking”](https://www.python.org/dev/peps/pep-3113/) ```python Python 3.6.8 (default, Jan 14 2019, 11:02:34) [GCC 8.0.1 20180414] on linux Type "help", "copyright", "credits" or "license" for more information. >>> def f((a, b), c): return a**b + c File "<stdin>", line 1 def f((a, b), c): return a**b + c ^ SyntaxError: invalid syntax ```
## PEP 3107 (2006) ```python >>> # valid >>> def foo(a: ... "parameter a", b: ... "parameter b" = 5): ... return a, b ``` ```python >>> # invalid >>> def foo() ... -> bool: ... return True ``` ```python >>> # invalid >>> nand = lambda (a: bool, b: bool): not (a and b) ``` No lambda support due to ambiguity in colon suffix.
## PEP 3107 (2006) Accessing annotations programmatically: ```python >>> def foo(msg: "log message", **kwargs: "format placeholder values") -> str: ... return msg.format(**kwargs) ... >>> foo.__annotations__ {'msg': 'log message', 'kwargs': 'format placeholder values', 'return': <class 'str'>} ``` * dict ⇒ order of parameters is lost. * `return` cannot be a parameter name.
## PEP 3107 (2006) ```python >>> def nothing(): pass ... >>> nothing.__annotations__ {} >>> no = lambda: None >>> no.__annotations__ {} ```
## PEP 3107 (2006) ```python >>> def foo(a: 'x', b: 5 + 6, c: list) -> max(2, 9): ... ... ... >>> foo.__annotations__ ``` What is the value of the `return` key?
## PEP 3107 (2006) ```python >>> def foo(a: 'x', b: 5 + 6, c: list) -> max(2, 9): ... ... ... >>> foo.__annotations__ ``` What is the value of the `return` key? ```python {'a': 'x', 'b': 11, 'c': <class 'list'>, 'return': 9} ```
## PEP 3107 (2006) Support by `inspect` suggested. How? ```python >>> def foo(msg: "log message", **kwargs: "format placeholder values") -> str: ... return msg.format(**kwargs) ... >>> import inspect >>> inspect.getfullargspec(foo) FullArgSpec(args=['msg'], varargs=None, varkw='kwargs', defaults=None, kwonlyargs=[], kwonlydefaults=None, annotations={'return': <class 'str'>, 'msg': 'log message', 'kwargs': 'format placeholder values'}) ```
## BTW: python2 `inspect` and nested parameters: ```python >>> def f((a, b), c): return a**b + c ... >>> inspect.getargspec(f) ArgSpec(args=[['a', 'b'], 'c'], varargs=None, keywords=None, defaults=None) ```
## `inspect` and function annotations More sophisticated data: ```python >>> import inspect >>> def foo(a, *, b: int, **kwargs): ... pass ... >>> inspect.signature(foo).parameters mappingproxy(OrderedDict([('a', <Parameter "a">), ('b', <Parameter "b:int">), ('kwargs', <Parameter "**kwargs">)])) >>> inspect.signature(foo).parameters['b'].annotation <class 'int'> >>> inspect.formatargspec(['a', 'b'], annotations={'a': 4, 'b': 7}) '(a: 4, b: 7)' ```
## PEP 0483 (2014) What is a type? A type checking concept. Different ways of definition: * List all values: `bool = {True, False}` * Specify functions required to implement: `Sized =` has function `__len__` * Simple class definition: all instances of a class form a type * Complex types like `FancyList =` lists containing only instances of `int`, `str` or their subclasses
## PEP 0483 (2014): subtyping OOP ⇒ subtyping relationship (“If `v1` has type `t1`, and `v2` has type `t2`, is it safe to assign `v1 = v2`?”). Strong criteria: * every value from `t2` is also in the set of values of `t1`; and * every function from `t1` is also in the set of functions of `t2`.
## PEP 0483 (2014): subtyping Thus * Every type is a subtype of itself. * The set of values becomes smaller in the process of subtyping, while the set of functions becomes larger. “Every Dog is an Animal, also Dog has more functions, for example it can bark, therefore Dog is a subtype of Animal. Conversely, Animal is not a subtype of Dog.”
## PEP 0483 (2014): subtyping ``` lucky_number = 3.14 # type: float lucky_number = 42 # Safe lucky_number * 2 # This works lucky_number << 5 # Fails unlucky_number = 13 # type: int unlucky_number << 5 # This works unlucky_number = 2.72 # Unsafe ``` * `int` is a subtype of `float` * “If `v1` has type `float`, and `v2` has type `int`, it is __safe__ to assign `v1 = v2`“
## Related: [co | contra]variance * subtyping ⇒ `Cat` is a subtype of `Animal` * Thus, any expression of type `Cat` can be substituted with expression of type `Animal` Questions: * Is a list of `Cat` a list of `Animal`? * Is a list of `Animal` a list of `Cat`? * Is there any relation between lists of `Cat` and lists of `Animal`?
## Related: [co | contra]variance “Depending on the variance of the type constructor, the subtyping relation of the simple types may be either preserved, reversed, or ignored for the respective complex types” ([Wikipedia](https://en.wikipedia.org/wiki/Covariance_and_contravariance_(computer_science))) * covariance ⇒ preserved type relation * contravariance ⇒ reversed type relation * invariance ⇒ ignored type relation
## Related: [co | contra]variance OCaml: `Cat` is a subtype of `Animal`. * covariance w.r.t. lists ⇒ list of Cat is a subtype of list of Animal * contravariance w.r.t. functions ⇒ `function from Animal to String` is a subtype of `function from Cat to String`
## [co | contra]variance C#: `Cat` is a subtype of `Animal`. Thus, * `IEnumerable<Cat>` is a subtype of `IEnumerable<Animal>`. `IEnumerable<T>` is __covariant__ on `T` (preserved). * `Action<Animal>` is a subtype of `Action<Cat>`. `Action<T>` is __contravariant__ on `T` (reversed). * Neither `IList<Cat>` nor `IList<Animal>` is a subtype of the other as `IList<T>` is invariant on `T`.
## [co | contra]variance “The variance of a C# generic interface is declared by placing the **out** (covariant) or **in** (contravariant) attribute on (zero or more of) its type parameters.” ([Wikipedia](https://en.wikipedia.org/wiki/Covariance_and_contravariance_(computer_science))) For example, the delegate type `Func<in T, out TResult>` represents a function with a __contravariant__ input parameter of type `T` and a __covariant__ return value of type `TResult`.
## PEP 0483 (2014): subtyping Python? `int` is a subtype of `float`. Thus, “If `List[int]` denotes the type formed by all lists containing only integers, then it is not a subtype of `List[float]`, formed by all lists that contain only real numbers. The 1st condition of subtyping holds, but appending a real number only works with `List[float]` so that the 2nd condition fails.“ ```python def append_pi(lst: List[float]) -> None: lst += [3.14] my_list = [1, 3, 5] # type: List[int] append_pi(my_list) # Naively, this should be safe... my_list[-1] << 5 # ... but this fails ```
## PEP 0483 (2014): subtyping Python? `int` is a subtype of `float`. * `List[int]` is a subtype of `List[float]` * `List[float]` is **not** a subtype of `List[int]` * There is some relationship `List[T]` is covariant to `T` (like in case of OCaml) ?
## PEP 0483 (2014): subtyping via [mypy readthedocs](https://mypy.readthedocs.io/en/latest/generics.html#variance-of-generics) ```python class Shape: pass class Circle(Shape): def rotate(self): ... def add_one(things: List[Shape]) -> None: things.append(Shape()) my_things: List[Circle] = [] add_one(my_things) # should be fine? my_things[0].rotate() # no, because this fails ```
## PEP 0483 (2014): subtyping via [mypy readthedocs](https://mypy.readthedocs.io/en/latest/common_issues.html#variance) ```python class A: ... class B(A): ... lst = [A(), A()] # Inferred type is List[A] new_lst = [B(), B()] # Inferred type is List[B] lst = new_lst # mypy will complain about this # because List is invariant ``` * `List[T]` is invariant to `T` * `Sequence[T]` is an immutable type and covariant to `T`
## PEP 0483 (2014): subtyping * `T` is a subtype of `T` (⇒ reflexive) * If `T` is a subtype of `S` and `S` is a subtype of `T`, then `T = S` (⇒ anti-symmetric) [AFAIK] * `T` as subtype of `S` then `S` is __not necessarily__ subtype of `T` (⇒ not symmetric) * If `T` is a subtype of `S` and `S` is a subtype of `U`, then `T` is a subtype of `U` (⇒ transitive)
## PEP 0483 (2014): consistency Similar to `is-subtype-of`. * Not transitive when the new type `Any` is involved. * Not symmetric. “Assigning `a_value` to `a_variable` is OK if the type of `a_value` __is consistent__ with the type of `a_variable`”
## PEP 0483 (2014): consistency Definition: 1. A type `t1` is consistent with a type `t2` if `t1` is a subtype of `t2`. (But not the other way around.) 2. `Any` is consistent with every type. (But `Any` is not a subtype of every type.) 3. Every type is consistent with `Any`. (But every type is not a subtype of `Any`.) → [“What is Gradual Typing” by Jeremy Siek](http://wphomes.soic.indiana.edu/jsiek/what-is-gradual-typing/)
→ Thatte’s [Quasi-static Typing](https://dl.acm.org/citation.cfm?doid=96709.96747), `Dynamic` and top/bottom types
## PEP 0483 (2014): consistency ```python class Employee: ... class Manager(Employee): ... worker = Employee() # type: Employee worker = Manager() # subtype ✓ boss = Manager() # type: Manager boss = Employee() # fails static check ❌ ``` ⇒ rule 1 (`t1` is consistent with a type `t2` if `t1` is a subtype of `t2`)
## PEP 0483 (2014): consistency ```python anything: Any = "string" something: int = anything ``` ⇒ rule 2 (`Any` is consistent with every type) ```python the_answer: int = 42 anythin: Any = the_answer ``` ⇒ rule 3 (Every type is consistent with `Any`)
## PEP 0483 (2014): types and classes * class = `class` statement or returned by `type(obj)` built-in function (object factories) (dynamic, runtime concept) * type = concept from above (appears in variable and function type annotations) (used by static type checkers) Every class is a type as discussed above.

PEP 0483 (2014): Union type

Union type
## PEP 0483 (2014): Union type ```python x: str = "foobar" x = 42 # ❌ a: Union[str, int] = "foobar" a = 42 # ✓ ```
## PEP 0483 (2014): Union type ```python >>> from typing import Union >>> Union[str, str]() '' >>> Union[str, int]() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.6/typing.py", line 194, in __call__ raise TypeError("Cannot instantiate %r" % type(self)) TypeError: Cannot instantiate typing.Union >>> ``` * `str` and `int` are types and classes * `Union[str, int]` is a type but not a proper class
## PEP 0483 (2014): Union type ```python >>> class Employee: ... pass ... >>> class Manager(Employee): ... pass ... >>> Union[Employee, Manager] <class '__main__.Employee'> ```
## PEP 0483 (2014): Building blocks * `Any` * `Union[t1, t2, …]` * `Optional[t1] = Union[None, t1]` * `Tuple[t1, t2, …, tn]` (represents a tuple) (is a subtype of `Tuple[u1, u2, …, um]` if `n=m` and for all i, `ti` is a subtype of `ti`) (`Tuple[()]`) (variadic homogeneous tuple: `Tuple[t1, ...]`)
## PEP 0483 (2014): Building blocks * `Callable` * `Callable[[t1, t2, ..., tn], tr]` is a callable with argument type `t1, t2, …, tn` and return type `tr` * `Callable[[], tr]` is no argument is taken * `Callable[..., tr]` → unchecked args * no way to indicate optional or keyword arguments, nor varargs * `Intersection` * type `t` is consistent with `Intersection[t1, t2, ...]` if `t` is subtype of each `ti` for all i * Not part of Python 3.7 (might be added)
## PEP 0483 (2014): Type variables What if we want to have a variable of dynamic type? ```python from typing import TypeVar T = TypeVar('T', str, float) def add_them(x: T, y: T) -> T: return x + y def percent_representation(z: float): print('{:02f} %'.format(z)) sum = add_them(4, 5) concat = add_them("foo", "bar") percent_representation(sum) # ✓ # percent_representation(concat) # ❌ ```
## PEP 0483 (2014): Type variables ```python S = TypeVar('S') def longest(first: S, second: S) -> S: return first if len(first) >= len(second) else second class MyStr(str): ... result = longest(MyStr('a'), MyStr('abc')) # result is of type MyStr longest_of_them_all = longest(MyStr('a'), 'spam_and_eggs') # longest_of_them_all is of type str ```
## PEP 0483 (2014): Generic types ```python from typing import TypeVar, Generic T = TypeVar('T') class Stack(Generic[T]): def __init__(self) -> None: self.items: List[T] = [] def push(self, item: T) -> None: self.items.append(item) def pop(self) -> T: return self.items.pop() ```
## PEP 0483 (2014): Generic types ```python Table = Dict[int, T] # Table is generic Messages = Table[bytes] # Same as Dict[int, bytes] ```
## PEP 0483 (2014): variance ```python T_co = TypeVar('T_co', covariant=True) class Box(Generic[T_co]): # this type is declared covariant def __init__(self, content: T_co) -> None: self._content = content def get_content(self) -> T_co: return self._content ``` Invariant (default), covariant (by keyword) or contravariant (by keyword)
## PEP 0483 (2014): Pragmatics * Use `None`, not `NoneType` (`type(None)`) * Type aliases: `Url = str` * Forward reference via strings: ```python class MyComparable: def compare(self, other: 'MyComparable') -> int: ... ``` * Default value None ⇒ Optional[T] ```python def get(key: KT, default: VT = None) -> VT: ... ``` * python2: `lst = [] # type: Sequence[int]` * type casting: `new = cast(new_type, old)`
## PEP 0483 (2014): bytes versus str ```python AnyStr = TypeVar('AnyStr', str, bytes) ``` ```python from typing import TypeVar, AnyStr def bar(t: AnyStr) -> None: if isinstance(t, bytes): print(t.decode('utf-8')) else: print(t) bar("Hello") bar(b"Hello") ``` ⇒ `error: "str" has no attribute "decode"; maybe "encode"?`
## PEP 0483 (2014): bytes versus str ```python from typing import TypeVar, AnyStr def foo(t: AnyStr) -> None: try: print(t.decode('utf-8')) except AttributeError: print(t) foo("Hello") foo(b"Hello") ```
## PEP 0484 (2014) “Type Hints” This PEP define many semantics that were implied in my code samples before
## PEP 0526 (2016) “Syntax for Variable Annotations” ```python a: int (x): int # Annotates x with int (y): int = 0 # Same situation here. d = {} d['a']: int = 0 # Annotates d['a'] with int. d['b']: int # Annotates d['b'] with int. ```
## PEP 0544 (2017) “Protocols: Structural subtyping (static duck typing)“ ```python from typing import Protocol class SupportsClose(Protocol): def close(self) -> None: ... class Socket: def close(self): pass x: SupportsClose = Socket() y: SupportsClose = str() ```
## PEP 0593 (2019) ```python UnsignedShort = Annotated[int, struct2.ctype('H')] SignedChar = Annotated[int, struct2.ctype('b')] class Student(struct2.Packed): # mypy typechecks 'name' field as 'str' name: Annotated[str, struct2.ctype("<10s")] serialnum: UnsignedShort school: SignedChar # 'unpack' only uses the metadata within the type annotations Student.unpack(record) # Student(name=b'raymond ', serialnum=4658, school=264) ```
## PEP 0589 (2019) “TypedDict: Type Hints for Dictionaries with a Fixed Set of Keys” ```python from typing import TypedDict class Movie(TypedDict): name: str year: int movie: Movie = {'name': 'Blade Runner', 'year': 1982} ```
## Standard library Type system related: * [https://docs.python.org/dev/library/typing.html](typing – Support for type hints) * [https://docs.python.org/dev/library/types.html](types – Define names for built-in types that aren't directly accessible as a builtin)
## `dir()` ```python Python 3.6.5 (default, Apr 1 2018, 05:46:30) [GCC 7.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import typing >>> dir(typing) ['AbstractSet', 'Any', 'AnyStr', 'AsyncContextManager', 'AsyncGenerator', 'AsyncIterable', 'AsyncIterator', 'Awaitable', 'BinaryIO', 'ByteString', 'CT_co', 'Callable', 'CallableMeta', 'ChainMap', 'ClassVar', 'Collection', 'Container', 'ContextManager', 'Coroutine', 'Counter', 'DefaultDict', 'Deque', 'Dict', 'FrozenSet', 'Generator', 'Generic', 'GenericMeta', 'Hashable', 'IO', 'ItemsView', 'Iterable', 'Iterator', 'KT', 'KeysView', 'List', 'Mapping', 'MappingView', 'Match', 'MethodDescriptorType', 'MethodWrapperType', 'MutableMapping', 'MutableSequence', 'MutableSet', 'NamedTuple', 'NamedTupleMeta', 'NewType', 'NoReturn', 'Optional', 'Pattern', 'Reversible', 'Sequence', 'Set', 'Sized', 'SupportsAbs', 'SupportsBytes', 'SupportsComplex', 'SupportsFloat', 'SupportsInt', 'SupportsRound', 'T', 'TYPE_CHECKING', 'T_co', 'T_contra', 'Text', 'TextIO', 'Tuple', 'TupleMeta', 'Type', 'TypeVar', 'TypingMeta', 'Union', 'VT', 'VT_co', 'V_co', 'ValuesView', 'WrapperDescriptorType', '_Any', '_ClassVar', '_FinalTypingBase', '_ForwardRef', '_G_base', '_NoReturn', '_Optional', '_PY36', '_Protocol', '_ProtocolMeta', '_TypeAlias', '_TypingBase', '_TypingEllipsis', '_TypingEmpty', '_Union', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_allowed_types', '_check_generic', '_cleanups', '_collections_abc', '_eval_type', '_generic_new', '_get_defaults', '_get_type_vars', '_make_nmtuple', '_make_subclasshook', '_next_in_mro', '_no_slots_copy', '_overload_dummy', '_prohibited', '_qualname', '_remove_dups_flatten', '_replace_arg', '_special', '_subs_tree', '_tp_cache', '_trim_name', '_type_check', '_type_repr', '_type_vars', 'abc', 'abstractmethod', 'abstractproperty', 'cast', 'collections', 'collections_abc', 'contextlib', 'functools', 'get_type_hints', 'io', 'no_type_check', 'no_type_check_decorator', 'overload', 're', 'stdlib_re', 'sys', 'types'] ```
## `dir()` ```python >>> [p for p in dir(typing) if not p.startswith('_')] ['AbstractSet', 'Any', 'AnyStr', 'AsyncContextManager', 'AsyncGenerator', 'AsyncIterable', 'AsyncIterator', 'Awaitable', 'BinaryIO', 'ByteString', 'CT_co', 'Callable', 'CallableMeta', 'ChainMap', 'ClassVar', 'Collection', 'Container', 'ContextManager', 'Coroutine', 'Counter', 'DefaultDict', 'Deque', 'Dict', 'FrozenSet', 'Generator', 'Generic', 'GenericMeta', 'Hashable', 'IO', 'ItemsView', 'Iterable', 'Iterator', 'KT', 'KeysView', 'List', 'Mapping', 'MappingView', 'Match', 'MethodDescriptorType', 'MethodWrapperType', 'MutableMapping', 'MutableSequence', 'MutableSet', 'NamedTuple', 'NamedTupleMeta', 'NewType', 'NoReturn', 'Optional', 'Pattern', 'Reversible', 'Sequence', 'Set', 'Sized', 'SupportsAbs', 'SupportsBytes', 'SupportsComplex', 'SupportsFloat', 'SupportsInt', 'SupportsRound', 'T', 'TYPE_CHECKING', 'T_co', 'T_contra', 'Text', 'TextIO', 'Tuple', 'TupleMeta', 'Type', 'TypeVar', 'TypingMeta', 'Union', 'VT', 'VT_co', 'V_co', 'ValuesView', 'WrapperDescriptorType', 'abc', 'abstractmethod', 'abstractproperty', 'cast', 'collections', 'collections_abc', 'contextlib', 'functools', 'get_type_hints', 'io', 'no_type_check', 'no_type_check_decorator', 'overload', 're', 'stdlib_re', 'sys', 'types']```
## mypy * http://mypy-lang.org/examples.html * heavily support by Python community * [https://mypy.readthedocs.io/en/latest/cheat_sheet_py3.html](mypy cheat sheet) * [https://mypy.readthedocs.io/en/latest/index.html#overview-type-system-reference](mypy Type System Reference) * [https://stackoverflow.com/questions/tagged/mypy](Stackoverflow tag mypy)
## mypy usage ``` usage: mypy [-h] [-v] [-V] [more options; see below] [-m MODULE] [-p PACKAGE] [-c PROGRAM_TEXT] [files ...] Mypy is a program that will type check your Python code. Pass in any files or folders you want to type check. Mypy will recursively traverse any provided folders to find .py files: $ mypy my_program.py my_src_folder … -m MODULE, --module MODULE Type-check module; can repeat for more modules -p PACKAGE, --package PACKAGE Type-check package recursively; can be repeated -c PROGRAM_TEXT, --command PROGRAM_TEXT Type-check program passed in as string files Type-check given files or directories ```
## Cool tip > `typing.TYPE_CHECKING` is only `True` while type checking. > This allows you to conditionally import files that would otherwise cause a circular dependency so that you can use the symbols for annotations. via [news.ycombinator.com](https://news.ycombinator.com/item?id=20159827)

typing backport module on PyPI

downloadsperiod
351105201909 (3 days)
5805834201908
5653431201907
5238428201906
5601651201905
4066071201904
4217391201903

Thanks!