valberg.dk/content/django-server-sent-events.md

Title: Server-Sent Events and PostgreSQL LISTEN/NOTIFY using Djangos StreamingHttpRequest
Date: 2023-05-17
Status: hidden
Tags: django, sse, postgresql
Slug: django-sse-postgresql-listen-notify
Authors: Víðir Valberg Guðmundsson
Summary: A write-up of how I implemented server-sent events using Django 4.2 and PostgreSQL LISTEN/NOTIFY

---



With the release of Django 4.2 we got the following [0]:

> [`StreamingHttpResponse`](https://docs.djangoproject.com/en/4.2/ref/request-response/#django.http.StreamingHttpResponse "django.http.StreamingHttpResponse") now supports async iterators when Django is served via ASGI.

And the documentation has been expanded with the following [1]:

> When serving under ASGI, however, a [`StreamingHttpResponse`](https://docs.djangoproject.com/en/4.2/ref/request-response/#django.http.StreamingHttpResponse "django.http.StreamingHttpResponse") need not stop other requests from being served whilst waiting for I/O. This opens up the possibility of long-lived requests for streaming content and implementing patterns such as long-polling, and server-sent events.

Being a sucker for simplicity I got quite intrigued by the possibility to serve
server-sent events (also known as SSE) from Django in an asynchronous manner.

So I set out to write a small, drumroll please, chat application!

The code for the chat application can be found at
[github.com/valberg/django-sse](https://github.com/valberg/django-sse).

### What are server-sent events and why do we want to use them?

Server-sent events is "old tech", as in that is has been supported in major
browser since around 2010-2011 [2]. The idea is that the client "subscribes" to
an HTTP endpoint, and the server can then issue data to the client as long as
the connection is open. This is a great performance boost compared to other
techniques as for instance polling the server.

_But wait, isn't websockets "shinier"?_

It depends. In many situations when it comes to developing web applications, we
just want a way to push data to the client, and here a bidirectional
connection like websockets feel like an overkill. Also, I would argue that using
POST/PUT requests from the client and SSE to the client might be "just enough"
compared to websockets.

SSE also has the added benefit of having a built-in reconnection mechanism, 
which is something we would have to implement ourselves with websockets.

All in all SSE is a much simpler solution than websockets, and in many (most?) 
cases that is all we need.

### A simple implementation

So lets get to some code! 

First we need our model for storing the chat messages:

    :::python
    class ChatMessage(models.Model):
        user = models.CharField(max_length=255)
        text = models.CharField(max_length=255)

With the model defined we can write our view to stream the messages.

The following is something along the lines of my initial attempt. First we have
to define the view, which in fact will not change for the remainder of this
blog post. The juicy bits are in the  `stream_messages()` function. Note that
the view is an async view, denoted by the `async` keyword.

    :::python
    async def stream_messages_view(request: HttpRequest) -> StreamingHttpResponse:
        return StreamingHttpResponse(
            streaming_content=stream_messages(),
            content_type="text/event-stream",
        )

We tell the `StreamingHttpResponse` class to get its streaming content from the
`stream_messages` function. I implemented this as follows initially:

    ::python
    async def stream_messages() -> AsyncGenerator[str, None]:
        latest_message = None

        while True:
            current_message = await ChatMessage.objects.order_by("-id").afirst()

            # If we have a new foo yield that
            if latest_message != current_message:
                yield "data: {current_message.text}\n\n"
                latest_message = current_message

            await asyncio.sleep(5)

So we've gotten rid of the HTTP overhead of polling by not having to do a
request from the client every 5 seconds. But we are still doing a query to the
database every 5 seconds, and that for each client. This is not ideal and is 
probably something we could have done with a synchronous view.

Let's see if we can do better. But first we'll have to talk about how to run
this code.

#### Aside: Use an ASGI server for development

One thing that took me some time to realise is that the Django runserver is not
capable of running async views returning `StreamingHttpResponse`.

Running the above view with the runserver results in the following error:

    :::text
    .../django/http/response.py:514: Warning: StreamingHttpResponse must
    consume asynchronous iterators in order to serve them synchronously. 
    Use a synchronous iterator instead.

So I had to result to installing uvicorn and run my project as so:

    :::bash
    $ uvicorn --log-level debug --reload --timeout-graceful-shutdown 0 project.asgi:application`

The `--reload` part is particulary important when doing development, but it
does not work well when working with open connections since the server will
wait for the connection to close before reloading. This is why
`--timeout-graceful-shutdown 0` is needed.

### More old tech to the rescue: PostgreSQL LISTEN/NOTIFY

This is where we could reach for more infrastructure which could help us giving
the database a break. This could be listening for data in Redis (this is what
django-channels does), or even having a queue in RabbitMQ. No matter what, it
is more infrastructure.

But I use PostgreSQL - and PostgreSQL is, like Django, "batteries included".

PostgreSQL has this mechanism called "LISTEN/NOTIFY" where one client can
LISTEN to a channel and then anyone can NOTIFY on that same channel.

This seems like something we can use - but psycopg2 isn't async, so I'm not
even sure if `sync_to_async` would help us here.

### Enter psycopg 3

I had put the whole thing on ice until I realized that another big thing (maybe
a bit bigger than StreamingHttpResponse) in Django 4.2 is the support for
psycopg 3 - and psycopg 3 is very much async!

So I went for a stroll in the psycopg 3 documentation and found this gold[3]:

    ::python
    import psycopg
    conn = psycopg.connect("", autocommit=True)
    conn.execute("LISTEN mychan")
    gen = conn.notifies()
    for notify in gen:
        print(notify)
        if notify.payload == "stop":
            gen.close()
    print("there, I stopped")

This does almost what we want! It just isn't async and isn't getting connection
info from Django.

So by combining the snippet from the psycopg 3 documentation and my previous
`stream_foos` I came up with this:

    :::python
    from collections.abc import AsyncGenerator
    import psycopg
    from django.db import connection

    async def stream_foos() -> AsyncGenerator[str, None]:
        connection_params = connection.get_connection_params()
        connection_params.pop('cursor_factory')
        aconnection = await psycopg.AsyncConnection.connect(
            **connection_params,
            autocommit=True,
        )
        channel_name = "new_foo"

        async with aconnection.cursor() as acursor:
            await acursor.execute(f"LISTEN {channel_name}")
            gen = aconnection.notifies()
            async for notify in gen:
                yield f"data: {notify.payload}\n\n"

I was almost about to give up again, since this approach didn't work initially.
All because I for some reason had removed the `autocommit=True` in my attempts
to async-ify the snippet from the psycopg 3 documentation.

#### Aside: Difference between 4.2 and 4.2.1

the code worked initially in 4.2, but 4.2.1 fixed a regression regarding
setting a custom cursor in the database configuration.

In 4.2 we get this from `connection.get_connection_params()`:

    :::javascript
    {
        'dbname': 'postgres',
        'user': 'postgres',
        'password': 'postgres',
        'host': 'localhost',
        'port': 5432,
        'context': <psycopg.adapt.AdaptersMap object at 0x7f019cda7a60>,
        'prepare_threshold': None
    }

in 4.2.1 we get this:

    :::javascript
    {
        'dbname': 'postgres',
        'client_encoding': 'UTF8',
        'cursor_factory': <class 'django.db.backends.postgresql.base.Cursor'>,
        'user': 'postgres',
        'password': 'postgres',
        'host': 'localhost',
        'port': '5432',
        'context': <psycopg.adapt.AdaptersMap object at 0x7f56464bcdd0>,
        'prepare_threshold': None
    }

`django.db.backends.postgresql.base.Cursor` is not async iterable.

So we can probably try to set our own `cursor_factory` in settings:

    :::python
    from psycopg import AsyncCursor

    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql',
            'NAME': 'postgres',
            'USER': 'postgres',
            'PASSWORD': 'postgres',
            'HOST': 'localhost',
            'PORT': '5432',
            'OPTIONS': {
                "cursor_factory": AsyncCursor
            }
        }
    }

But alas. For some reason this does not work. I guess that Django does some
wrapping of the cursor - or maybe I've just encountered a bug. The cursor is at
least not treated as an async cursor and thus we get the following error:

    :::pytb

    .../django-sse/venv/lib/python3.11/site-packages/django/db/backends/utils.py:41: 
    RuntimeWarning: coroutine 'AsyncCursor.close' was never awaited

      self.close()

    RuntimeWarning: Enable tracemalloc to get the object allocation traceback

    .../django-sse/venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py:1560:
    RuntimeWarning: coroutine 'AsyncCursor.execute' was never awaited

      cursor.execute(sql, params)

    RuntimeWarning: Enable tracemalloc to get the object allocation traceback

So instead I opted for removing the `cursor_factory` in the streaming function.
So that now looks like so:

    :::python
    async def stream_messages() -> AsyncGenerator[str, None]:
        connection_params = connection.get_connection_params()
        connection_params.pop('cursor_factory')
        aconnection = await psycopg.AsyncConnection.connect(
            **connection_params,
            autocommit=True,
        )
        channel_name = "new_message"
        async with aconnection.cursor() as acursor:
            print(type(acursor))
            await acursor.execute(f"LISTEN {channel_name}")
            gen = aconnection.notifies()
            async for notify in gen:
                yield f"data: {notify.payload}\n\n"

### Test the endpoint with curl

So now we've got the `LISTEN` part in place.

If we connect to the endpoint using curl (`-N` disables buffering and is a way to consume streming content with curl):

    :::console
    $ curl -N http://localhost:8000/messages/

And connect to our database and run:

    :::sql
    NOTIFY new_message, 'Hello, world!';


We, excitingly, get the following result :

    :::text
    data: Hello, world!

Amazing!

### Issuing the NOTIFY

But we want the `NOTIFY` command to be issued when a new chat message is submitted.

For this we'll have a small utility function which does the heavy lifting. Note
that this is just a very simple synchronous function since everything is just
happening within a single request-response cycle.

    :::python
    from django.db import connection


    def notify(*, channel: str, event: str, payload: str) -> None:
        payload = json.dumps({
            "event": event,
            "content": payload,
        })
        with connection.cursor() as cursor:
            cursor.execute(
                f"NOTIFY {channel}, '{payload}'",
            )

And then we can use this in our view (I'm using `@csrf_exempt` here since this is just a quick proof of concept):

    :::python
    @csrf_exempt
    @require_POST
    def post_message_view(request: HttpRequest) -> HttpResponse:
        message = request.POST.get("message")
        user = request.POST.get("user")
        message = ChatMessage.objects.create(user=user, text=message)
        notify(
            channel="lobby",
            event="message_created",
            content=json.dumps({
                "text": message.text,
                "user": message.user,
            })
        )
        return HttpResponse("OK")

The keen observer will notice that we are storing the payload content as a JSON string within a JSON string.

This is because we have two recipients of the payload. The first is the `stream_messages` function which is going to
send the payload to the client with a `event`, and the second is the browser which is going to parse the payload and use
the `event` to determine what to do with the payload.

For this we'll have to update our `stream_messages` function as follows:

    :::python
    async def stream_messages() -> AsyncGenerator[str, None]:
    connection_params = connection.get_connection_params()

        # Remove the cursor_factory parameter since I can't get
        # the default from Django 4.2.1 to work.
        # Django 4.2 didn't have the parameter and that worked.
        connection_params.pop('cursor_factory')

        aconnection = await psycopg.AsyncConnection.connect(
            **connection_params,
            autocommit=True,
        )
        channel_name = "lobby"
        async with aconnection.cursor() as acursor:
            await acursor.execute(f"LISTEN {channel_name}")
            gen = aconnection.notifies()
            async for notify in gen:
                payload = json.loads(notify.payload)
                event = payload.pop("event")
                data = payload.pop("data")
                yield f"event: {event}\ndata: {data}\n\n"

Everything is the same except that we now parse the payload from the `NOTIFY` command and construct the SSE payload with
an `event` and a `data` field. This will come in handy when dealing with the frontend.

Another way to do this would be to use Django's
[signals](https://docs.djangoproject.com/en/4.2/topics/signals/) or event
writing a PostgreSQL
[trigger](https://www.postgresql.org/docs/15/plpgsql-trigger.html) which issues
the `NOTIFY` command.

### Frontend stuff

Now that we've got the backend in place, we can get something up and running on
the frontend.

We could use HTMX's [SSE
extension](https://htmx.org/extensions/server-sent-events/) but for this
example we'll just use the
[EventSource](https://developer.mozilla.org/en-US/docs/Web/API/EventSource) API
directly.

    :::html
    <template id="message">
        <div style="border: 1px solid black; margin: 5px; padding: 5px;">
            <strong class="user"></strong>: <span class="message"></span>
        </div>
    </template>

    <div id="messages"></div>

    <script>
        const source = new EventSource("/messages/");

        // Note that the event we gave our notify utility function is called "message_created"
        // so that's what we listen for here.
        source.addEventListener("message_created", function(evt) {
            // Parse the payload
            let payload = JSON.parse(evt.data);

            // Get and clone our template
            let template = document.getElementById('message');
            let clone = template.content.cloneNode(true);

            // Update our cloned template
            clone.querySelector('.user').innerText = payload.user;
            clone.querySelector('.message').innerText = payload.text;

            // Append the cloned template to our list of messages
            document.getElementById('messages').appendChild(clone);
        });
    </script>


And that's it! We can now open two browser windows and see the messages appear in real time.

Check out the repo for the full code where I've also added a simple form for submitting new messages.

### Conclusion

Django might not be the shiniest framework out there, but it is solid and boring - which is a good thing! 

And with the continued work on async support, it is becoming a viable option for doing real time stuff, especially when paired with other solid and boring tech like PostgreSQL and SSE! 

[0]: https://docs.djangoproject.com/en/4.2/releases/4.2/#requests-and-responses
[1]: https://docs.djangoproject.com/en/4.2/ref/request-response/#django.http.StreamingHttpResponse
[2]: https://caniuse.com/eventsource
[3]:https://www.psycopg.org/psycopg3/docs/advanced/async.html#index-4