reynir
/
reynir.dk
1
0
Fork 0
Code Issues 5 Pull Requests Packages Projects Releases Wiki Activity

Add python str.__repr__() post

This commit is contained in:
Reynir Björnsson 2024-02-04 17:00:07 +01:00
parent 297a40ebf5
commit 5b25460da7
2 changed files with 294 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

293
posts/2024-02-03-python-str-repr.md Normal file
View File

@ -0,0 +1,293 @@
---
title: Python's `str.__repr__()`
date: 2024-02-03
---
Sometimes software is written using whatever built-ins you find in your programming language of choice.
This is usually great!
However, it can happen that you depend on the precise semantics of those built-ins.
This can be a problem if those semantics become important to your software and you need to port it to another programming language.
This story is about Python and its `str.__repr()__` function.
The piece of software I was helping port to [OCaml][ocaml] was constructing a hash from the string representation of a tuple.
The gist of it was basically this:
```python
def get_id(x):
id = (x.get_unique_string(), x.path, x.name)
return myhash(str(id))
```
In other words it's a Python tuple consisting of mostly strings but also a `PosixPath` object.
The way `str()` works is it calls the `__str__()` method on the argument objects (or otherwise `repr(x)`).
For Python tuples the `__str__()` method seems to print the result of `repr()` on each elemenet separated by a comma and a space and surrounded by parenthesis.
So good so far.
If we can precisely emulate `repr()` on strings and `PosixPath` it's easy.
In the case of `PosixPath` it's really just `'PosixPath('+repr(str(path))+')'`;
so in that case it's down to `repr()` on strings - which is `str.__repr__()`,
There had been a previous attempt at this that would use OCaml's string escape functions and surround the string with single quotes (`'`).
This works for some cases, but not if the string has a double quote (`"`).
In that case OCaml would escape the double quote with a backslash (`\"`) while python would not escape it.
So a regular expression substitution was added to replace the escape sequence with just a double quote.
This pattern of finding small differences between Python and OCaml escaping had been repeated,
and eventually I decided to take a more rigorous approach to it.
## What is a string?
First of all, what is a string? In Python? And in OCaml?
In OCaml a string is just a sequence of bytes.
Any bytes, even `NUL` bytes.
There is no concept of unicode in OCaml strings.
In Python there is the `str` type which is a sequence of Unicode code points[^python-bytes].
I can recommend reading Daniel Bünzli's [minimal introduction to Unicode][unicode-minimal].
Already here there is a significant gap in semantics between Python and OCaml.
For many practical purposes we can get away with using the OCaml `string` type and treating it as a UTF-8 encoded Unicode string.
This is what I will do as in both the Python code and the OCaml code the data being read is a UTF-8 (or often only the US ASCII subset) encoded string.
## What does a string literal look like?
### OCaml
I will not dive too deep into the details of OCaml string literals, and focus mostly on how they are escaped by the language built-ins (`String.escaped`, `Printf.printf "%S"`).
Normal printable ASCII is printed as-is.
That is, letters, numbers and other symbols except for backslash and double quote.
There are the usual escape sequences `\n`, `\t`, `\r`, `\"` and `\\`.
Any byte value can be represented with decimal notation `\032` or octal notation '\o040' or hexadecimal notation `\x20`.
The escape functions in OCaml has a preference for the decimal notation over the hexadecimal notation.
Finally I also want to mention the Unicode code point escape sequence `\u{3bb}` which represents the UTF-8 encoding of U+3BB.
While the escape functions do not use it, it will become handy later on.
Illegal escape sequences (escape sequences that are not recognized) will emit a warning but otherwise result in the escape sequence as-is.
It is common to compile OCaml programs with warnings-as-errors, however.
### Python
Python has a number of different string literals and string-like literals.
They all use single quote or double quote to delimit the string (or string-like) literals.
There is a preference towards single quotes in `str.__repr__()`.
You can also triple the quotes if you like to write a string that uses a lot of both quote characters.
This format is not used by `str.__repr__()` so I will not cover it further, but you can read about it in the [Python reference manual](https://docs.python.org/3/reference/lexical_analysis.html#strings).
The string literal can optionally have a prefix character that modifies what type the string literal is and how its content is interpreted.
The `r`-prefixed strings are called *raw strings*.
That means backslash escape sequences are not interpreted.
In my experiments they seem to be quasi-interpreted, however!
The string `r"\"` is considered unterminated!
But `r"\""` is fine as is interpreted as `'\\"'`[^raw-escape-example].
Why this is the case I have not found a good explanation for.
The `b`-prefixed strings are `bytes` literals.
This is close to OCaml strings.
Finally there are the unprefixed strings which are `str` literals.
These are the ones we are most interested in.
They use the usual escape `\[ntr"]` we know from OCaml as well as `\'`.
`\032` is **octal** notation and `\x20` is hexadecimal notation.
There is as far as I know **no** decimal notation.
The output of `str.__repr__()` uses the hexadecimal notation over the octal notation.
As Python strings are Unicode code point sequences we need more than two hexadecimal digits to be able to represent all valid "characters".
Thus there are the longer `\u0032` and the longest `\U00000032`.
## Intermezzo
While studying Python string literals I discovered several odd corners of the syntax and semantics besides the raw string quasi-escape sequence mentioned earlier.
One fact is that Python doesn't have a separate character or Unicode code point type.
Instead, a character is a one element string.
This leads to some interesting indexing shenanigans: `"a"[0][0][0] == "a"`.
Furthermore, strings separated by spaces only are treated as one single concatenated string: `"a" "b" "c" == "abc"`.
These two combined makes it possible to write this unusual snippet: `"a" "b" "c"[0] == "a"`!
For byte sequences, or `b`-prefixed strings, things are different.
Indexing a bytes object returns the integer value of that byte (or character):
```python
>>> b"a"[0]
97
>>> b"a"[0][0]
<stdin>:1: SyntaxWarning: 'int' object is not subscriptable; perhaps you missed a comma?
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'int' object is not subscriptable
```
For strings `\x32` can be said to be shorthand for `"\u0032"` (or `"\u00000032"`).
But for bytes `"\x32" != "\u0032"`!
Why is this?!
Well, bytes is a byte sequence and `b"\u0032"` is not interpreted as an escape sequence and is instead **silently** treated as `b"\\u0032"`!
Writing `"\xff".encode()` which encodes the string `"\xff"` to UTF-8 is **not** the same as `b"\xff"`.
The bytes `"\xff"` consist of a single byte with decimal value 255,
and the Unicode wizards reading will know that the Unicode code point 255 (or U+FF) is encoded in two bytes in UTF-8.
## Where is the Python code?
Finding the implementation of `str.__repr__()` turned out to not be so easy.
In the end I asked on the Internet and got a link to [cpython's `Objects/unicodeobject.c`][unicodeobject.c].
And holy cow!
That's some 160 lines of C code with two loops, a switch statement and I don't know how many chained and nested if statements!
Meanwhile the OCaml implementation is a much less daunting 52 lines of which about a fifth is a long comment.
It also has two loops which each contain one much more tame match expression (roughly a C switch statement).
In both cases they first loop over the string to compute the size of the output string.
The Python implementation also counts the number of double quotes and single quotes as well as the highest code point value.
The latter I'm not sure why they do, but my guess it's so they can choose an efficient internal representation.
Then the Python code decides what quote character to use with the following algorithm:
Does the string contain single quotes but no double quotes? Then use double quotes. Otherwise use single quotes.
Then the output size estimate is adjusted with the number of backslashes to escape the quote character chosen and the two quotes surrounding the string.
Already here it's clear that a regular expression substitution is not enough by itself to fix OCaml escaping to be Python escaping.
My first step then was to implement the algorithm only for US ASCII.
This is simpler as we don't have to worry much about Unicode, and I could implement it relatively quickly.
The first 32 characters and the last US ASCII character (DEL or `\x7f`) are considered non-printable and must be escaped.
I then wrote some simple tests by hand.
Then I discovered the OCaml [py][ocaml-py] library which provides bindings to Python from OCaml.
Great! This I can use to test my implementation against Python!
## How about Unicode?
For the non-ascii characters (or code points rather) they are either considered *printable* or *non-printable*.
For now let's look at what that means for the output.
A printable character is copied as-is.
That is, there is no escaping done.
Non-printable characters must be escaped, and python wil use `\xHH`, `\uHHHH` or `\UHHHHHHHH` depending on how many hexadecimal digits are necessary to represent the code point.
That is, the latin-1 subset of ASCII (`0x80`-`0xff`) can be represented using `\xHH` and neither `\u00HH` nor `\U000000HH` will be used etc.
### What is a printable Unicode character?
In the cpython [function][unicodeobject.c] mentioned earlier they use the function `Py_UNICODE_ISPRINTABLE`.
I had a local clone of the cpython git repository where I ran `git grep Py_UNICODE_ISPRINTABLE` to find information about it.
In [unicode.rst][unicode.rst-isprintable] I found a documentation string for the function that describes it to return false if the character is nonprintable with the definition of nonprintable as the code point being in the categories "Other" or "Separator" in the Unicode character database **with the exception of ASCII space** (U+20 or ` `).
What are those "Other" and "Separator" categories?
Further searching for the function definition we find in [`Include/cpython/unicodeobject.h`][unicodeobject.h-isprintable] the definition.
Well, we find `#define Py_UNICODE_ISPRINTABLE(ch) _PyUnicode_IsPrintable(ch)`.
On to `git grep _PyUnicode_IsPrintable` then.
That function is defined in [`Objects/unicodectype.c`][unicodectype.c-isprintable].
```C
/* Returns 1 for Unicode characters to be hex-escaped when repr()ed,
0 otherwise.
All characters except those characters defined in the Unicode character
database as following categories are considered printable.
* Cc (Other, Control)
* Cf (Other, Format)
* Cs (Other, Surrogate)
* Co (Other, Private Use)
* Cn (Other, Not Assigned)
* Zl Separator, Line ('\u2028', LINE SEPARATOR)
* Zp Separator, Paragraph ('\u2029', PARAGRAPH SEPARATOR)
* Zs (Separator, Space) other than ASCII space('\x20').
*/
int _PyUnicode_IsPrintable(Py_UCS4 ch)
{
const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);
return (ctype->flags & PRINTABLE_MASK) != 0;
}
```
Ok, now we're getting close to something.
Searching for `PRINTABLE_MASK` we find in [`Tools/unicode/makeunicodedata.py`][makeunicodedata.py-printable-mask] the following line of code:
```Python
if char == ord(" ") or category[0] not in ("C", "Z"):
flags |= PRINTABLE_MASK
```
So the algorithm is really if the character is a space character or if its Unicode general category doesn't start with a `C` or `Z`.
This can be implemented in OCaml using the uucp library as follows:
```OCaml
let py_unicode_isprintable uchar =
(* {[if char == ord(" ") or category[0] not in ("C", "Z"):
flags |= PRINTABLE_MASK]} *)
Uchar.equal uchar (Uchar.of_char ' ')
||
let gc = Uucp.Gc.general_category uchar in
(* Not those categories starting with 'C' or 'Z' *)
match gc with
| `Cc | `Cf | `Cn | `Co | `Cs | `Zl | `Zp | `Zs -> false
| `Ll | `Lm | `Lo | `Lt | `Lu | `Mc | `Me | `Mn | `Nd | `Nl | `No | `Pc | `Pd
| `Pe | `Pf | `Pi | `Po | `Ps | `Sc | `Sk | `Sm | `So ->
true
```
After implementing unicode I expanded the tests to generate arbitrary OCaml strings and compare the results of calling my function and Python's `str.__repr__()` on the string.
Well, that didn't go quite well.
OCaml strings are just any byte sequence, and ocaml-py expects it to be a UTF-8 encoded string and fails on invalid UTF-8.
Then in qcheck you can "assume" a predicate which means if a predicate doesn't hold on the generated value then the test is skipped for that input.
So I implement a simple verification of UTF-8.
This is far from optimal because qcheck will generate a lot of invalid utf-8 strings.
The next test failure is some unassigned code point.
So I add to `py_unicode_isprintable` a check that the code point is assigned using ``Uucp.Age.age uchar <> `Unassigned``.
Still, qcheck found a case I hadn't considered: U+61D.
My python version (Python 3.9.2 (default, Feb 28 2021, 17:03:44)) renders this as `'\u061'` while my OCaml function prints it as-is.
In other words my implementation considers it printable while python does not.
I try to enter this Unicode character in my terminal, but nothing shows up.
Then I look it up and its name is `ARABIC END OF TEXT MARKER`.
The general category according to uucp is `` `Po ``.
So this **should** be a printable character‽
After being stumped by this for a while I get the suspicion it may be dependent on the Python version.
I am still on Debian 11 and my Python version is far from being the latest and greatest.
I ask someone with a newer Python version to write `'\u061d'` in a python session.
And 'lo! It prints something that looks like `''`!
Online I figure out how to get the unicode version compiled into Python:
```Python
>>> import unicodedata
>>> unicodedata.unidata_version
'13.0.0'
```
Aha! And with uucp we find that the unicode version that introduced U+61D to be 14.0:
```OCaml
# Uucp.Age.age (Uchar.of_int 0x61D);;
- : Uucp.Age.t = `Version (14, 0)
```
My reaction is this is seriously some ungodly mess we are in.
Not only is the code that instigated this journey highly dependent on Python-specifics - it's also dependent on the specific version of unicode and thus the version of Python!
I modify our `py_unicode_isprintable` function to take an optional `?unicode_version` argument and replace the "is this unassigned?" check with the following snippet:
```OCaml
let age = Uucp.Age.age uchar in
(match (age, unicode_version) with
| `Unassigned, _ -> false
| `Version _, None -> true
| `Version (major, minor), Some (major', minor') ->
major < major' || (major = major' && minor <= minor'))
```
Great! I modify the test suite to first detect the unicode version python uses and then pass that version to the OCaml function.
Now I can't find anymore failing test cases!
## Epilogue
If you are curious to read the resulting code you may find it on github at [github.com/reynir/python-str-repr](https://github.com/reynir/python-str-repr).
I have documented the code to make it more approachable and maintainable by others.
Hopefully it is not something that you need, but in case it is useful to you it is licensed under a permissive license.
If you have a project in OCaml or want to port something to OCaml and would like help from me and my colleagues at [Robur](https://robur.coop/) please [get in touch](https://robur.coop/Contact) with us and we will figure something out.
[ocaml]: https://ocaml.org/
[unicode-minimal]: https://ocaml.org/p/uucp/13.0.0/doc/unicode.html#minimal
[unicodeobject.c]: https://github.com/python/cpython/blob/963904335e579bfe39101adf3fd6a0cf705975ff/Objects/unicodeobject.c#L12245-L12405
[escaped]: https://github.com/ocaml/ocaml/blob/a51089215d5ae1187688a5b130e9f62bf50adfeb/stdlib/bytes.ml#L170-L222
[ocaml-py]: https://github.com/zshipko/ocaml-py
[unicode.rst-isprintable]: https://github.com/python/cpython/blob/963904335e579bfe39101adf3fd6a0cf705975ff/Doc/c-api/unicode.rst?plain=1#L257-L265
[makeunicodedata.py-printable-mask]: https://github.com/python/cpython/blob/963904335e579bfe39101adf3fd6a0cf705975ff/Tools/unicode/makeunicodedata.py#L450-L451
[unicodectype.c-isprintable]: https://github.com/python/cpython/blob/963904335e579bfe39101adf3fd6a0cf705975ff/Objects/unicodectype.c#L158-L163
[unicodeobject.h-isprintable]: https://github.com/python/cpython/blob/963904335e579bfe39101adf3fd6a0cf705975ff/Include/cpython/unicodeobject.h#L683
[^python-bytes]: There is as well the `bytes` type which is a byte sequence like OCaml's `string`.
The Python code in question is using `str` however.
[^raw-escape-example]: Note I use single quotes for the output. This is what Python would do. It would be equivalent to `"\\\""`.

1
src/dune
View File

@ -1,4 +1,5 @@
(executable
(public_name reynir-www)
(name reynir_www)
(libraries
logs