1
0
mirror of https://github.com/satwikkansal/wtfpython synced 2024-11-22 02:54:25 +01:00

Minor updates to slowinig down dict lookups example

This commit is contained in:
Satwik 2020-07-10 22:28:44 +05:30
parent 098d71f348
commit f97cbdd919

34
README.md vendored
View File

@ -92,7 +92,7 @@ So, here we go...
* [Section: Miscellaneous](#section-miscellaneous) * [Section: Miscellaneous](#section-miscellaneous)
+ [ `+=` is faster](#--is-faster) + [ `+=` is faster](#--is-faster)
+ [ Let's make a giant string!](#-lets-make-a-giant-string) + [ Let's make a giant string!](#-lets-make-a-giant-string)
+ [ `dict` lookup performance](#-dict-lookup-performance) + [ `dict` lookup performance](#-slowing-down-dict-lookups)
+ [ Minor Ones *](#-minor-ones-) + [ Minor Ones *](#-minor-ones-)
- [Contributing](#contributing) - [Contributing](#contributing)
- [Acknowledgements](#acknowledgements) - [Acknowledgements](#acknowledgements)
@ -3349,36 +3349,38 @@ Let's increase the number of iterations by a factor of 10.
--- ---
### ▶ `dict` lookup performance ### ▶ Slowing down `dict` lookups
```py ```py
>>> some_dict = {str(i): 1 for i in range(1_000_000)} some_dict = {str(i): 1 for i in range(1_000_000)}
another_dict = {str(i): 1 for i in range(1_000_000)}
```
**Output:**
```py
>>> %timeit some_dict['5'] >>> %timeit some_dict['5']
28.6 ns ± 0.115 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) 28.6 ns ± 0.115 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
>>> some_dict[1] = 1 >>> some_dict[1] = 1
>>> %timeit some_dict['5'] >>> %timeit some_dict['5']
37.2 ns ± 0.265 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) 37.2 ns ± 0.265 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
# why did it become much slower?
```
#### 💡 Explanation: >>> %timeit another_dict['5']
+ CPython has a generic dictionary lookup function that handles all types of keys (`str`, `int`, any object ...), and a specialized one for the common case of dictionaries composed of `str`-only keys.
+ The specialized function (named `lookdict_unicode` in CPython's sources) knows all existing keys (including the looked-up key) are strings, and uses the faster & simpler string comparison to compare keys, instead of calling the `__eq__` method.
+ The first time a `dict` instance is accessed with a non-`str` key, it's modified so future lookups use the generic function.
+ This process is not reversible for the particular `dict` instance, and the key doesn't even have to exist in the dictionary - attempting a failed lookup has the same effect:
```py
>>> some_dict = {str(i): 1 for i in range(1_000_000)}
>>> %timeit some_dict['5']
28.5 ns ± 0.142 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) 28.5 ns ± 0.142 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
>>> some_dict[1] >>> another_dict[1] # Trying to access a key that doesn't exist
Traceback (most recent call last): Traceback (most recent call last):
File "<stdin>", line 1, in <module> File "<stdin>", line 1, in <module>
KeyError: 1 KeyError: 1
>>> %timeit some_dict['5'] >>> %timeit another_dict['5']
38.5 ns ± 0.0913 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) 38.5 ns ± 0.0913 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
``` ```
Why are same lookups becoming slower?
#### 💡 Explanation:
+ CPython has a generic dictionary lookup function that handles all types of keys (`str`, `int`, any object ...), and a specialized one for the common case of dictionaries composed of `str`-only keys.
+ The specialized function (named `lookdict_unicode` in CPython's [source](https://github.com/python/cpython/blob/522691c46e2ae51faaad5bbbce7d959dd61770df/Objects/dictobject.c#L841)) knows all existing keys (including the looked-up key) are strings, and uses the faster & simpler string comparison to compare keys, instead of calling the `__eq__` method.
+ The first time a `dict` instance is accessed with a non-`str` key, it's modified so future lookups use the generic function.
+ This process is not reversible for the particular `dict` instance, and the key doesn't even have to exist in the dictionary. That's why attempting a failed lookup has the same effect.
---
### ▶ Minor Ones * ### ▶ Minor Ones *
<!-- Example ID: f885cb82-f1e4-4daa-9ff3-972b14cb1324 ---> <!-- Example ID: f885cb82-f1e4-4daa-9ff3-972b14cb1324 --->