diff --git a/content/django-server-sent-events.md b/content/django-server-sent-events.md index 3bb6706..9db9bb1 100644 --- a/content/django-server-sent-events.md +++ b/content/django-server-sent-events.md @@ -8,13 +8,11 @@ Summary: A write-up of how I implemented server-sent events using Django 4.2 and --- - - -With the release of Django 4.2 we got the following [0]: +With the release of Django 4.2 we got the following [^0]: > [`StreamingHttpResponse`](https://docs.djangoproject.com/en/4.2/ref/request-response/#django.http.StreamingHttpResponse "django.http.StreamingHttpResponse") now supports async iterators when Django is served via ASGI. -And the documentation has been expanded with the following [1]: +And the documentation has been expanded with the following [^1]: > When serving under ASGI, however, a [`StreamingHttpResponse`](https://docs.djangoproject.com/en/4.2/ref/request-response/#django.http.StreamingHttpResponse "django.http.StreamingHttpResponse") need not stop other requests from being served whilst waiting for I/O. This opens up the possibility of long-lived requests for streaming content and implementing patterns such as long-polling, and server-sent events. @@ -26,10 +24,14 @@ So I set out to write a small, drumroll please, chat application! The code for the chat application can be found at [github.com/valberg/django-sse](https://github.com/valberg/django-sse). +**Table of contents** + +[TOC] + ### What are server-sent events and why do we want to use them? Server-sent events is "old tech", as in that is has been supported in major -browser since around 2010-2011 [2]. The idea is that the client "subscribes" to +browser since around 2010-2011 [^2]. The idea is that the client "subscribes" to an HTTP endpoint, and the server can then issue data to the client as long as the connection is open. This is a great performance boost compared to other techniques as for instance polling the server. @@ -98,34 +100,6 @@ probably something we could have done with a synchronous view. Let's see if we can do better. But first we'll have to talk about how to run this code. -#### Aside: Use an ASGI server for development - -One thing that took me some time to realise is that the Django runserver is not -capable of running async views returning `StreamingHttpResponse`. - -Running the above view with the runserver results in the following error: - - :::text - .../django/http/response.py:514: Warning: StreamingHttpResponse must - consume asynchronous iterators in order to serve them synchronously. - Use a synchronous iterator instead. - -Fortunately Daphne, the ASGI server which was developed to power Django Channels, has an async runserver which we can use: - -To set this up we'll have to install the `daphne` package, add `daphne` to the top of our installed apps, and set -the `ASGI_APPLICATION` setting to point to our ASGI application. - - :::python - INSTALLED_APPS = [ - "daphne", - ... - "chat", # Our app - ] - - ASGI_APPLICATION = "project.asgi.application" - -Now we can just run `./manage.py runserver` as before and we are async ready! - ### More old tech to the rescue: PostgreSQL LISTEN/NOTIFY This is where we could reach for more infrastructure which could help us giving @@ -141,13 +115,13 @@ LISTEN to a channel and then anyone can NOTIFY on that same channel. This seems like something we can use - but psycopg2 isn't async, so I'm not even sure if `sync_to_async` would help us here. -### Enter psycopg 3 +#### Enter psycopg 3 I had put the whole thing on ice until I realized that another big thing (maybe a bit bigger than StreamingHttpResponse) in Django 4.2 is the support for psycopg 3 - and psycopg 3 is very much async! -So I went for a stroll in the psycopg 3 documentation and found this gold[3]: +So I went for a stroll in the psycopg 3 documentation and found this gold[^3]: ::python import psycopg @@ -172,8 +146,17 @@ So by combining the snippet from the psycopg 3 documentation and my previous from django.db import connection async def stream_foos() -> AsyncGenerator[str, None]: + + # Get the connection params from Django connection_params = connection.get_connection_params() + + # Somehow Django 4.2.1 sets the cursor_factory to + # django.db.backends.postgresql.base.Cursor + # which causes problems. Read more about it in the + # "Differences between 4.2 and 4.2.1" section in the Appendix. + # Removing it from the connection parameters works around this. connection_params.pop('cursor_factory') + aconnection = await psycopg.AsyncConnection.connect( **connection_params, autocommit=True, @@ -186,13 +169,201 @@ So by combining the snippet from the psycopg 3 documentation and my previous async for notify in gen: yield f"data: {notify.payload}\n\n" -I was almost about to give up again, since this approach didn't work initially. -All because I for some reason had removed the `autocommit=True` in my attempts -to async-ify the snippet from the psycopg 3 documentation. +Appart from problems with the `cursor_factory` (which I'll get back to in the +[appendix](#difference-between-42-and-421)), this code is pretty straight forward and, most importantly, works! -#### Aside: Difference between 4.2 and 4.2.1 -the code worked initially in 4.2, but 4.2.1 fixed a regression regarding +### Test the endpoint with curl + +So now we've got the `LISTEN` part in place. + +If we connect to the endpoint using curl (`-N` disables buffering and is a way to consume streming content with curl): + + :::console + $ curl -N http://localhost:8000/messages/ + +And connect to our database and run: + + :::sql + NOTIFY new_message, 'Hello, world!'; + + +We, excitingly, get the following result : + + :::text + data: Hello, world! + +Amazing! + +### Issuing the NOTIFY command from Django + +But we want the `NOTIFY` command to be issued when a new chat message is submitted. + +For this we'll have a small utility function which does the heavy lifting. Note +that this is just a very simple synchronous function since everything is just +happening within a single request-response cycle. + + :::python + from django.db import connection + + + def notify(*, channel: str, event: str, payload: str) -> None: + payload = json.dumps({ + "event": event, + "content": payload, + }) + with connection.cursor() as cursor: + cursor.execute( + f"NOTIFY {channel}, '{payload}'", + ) + +And then we can use this in our view (I'm using `@csrf_exempt` here since this is just a quick proof of concept): + + :::python + @csrf_exempt + @require_POST + def post_message_view(request: HttpRequest) -> HttpResponse: + message = request.POST.get("message") + user = request.POST.get("user") + message = ChatMessage.objects.create(user=user, text=message) + notify( + channel="lobby", + event="message_created", + content=json.dumps({ + "text": message.text, + "user": message.user, + }) + ) + return HttpResponse("OK") + +The keen observer will notice that we are storing the payload content as a JSON string within a JSON string. + +This is because we have two recipients of the payload. The first is the `stream_messages` function which is going to +send the payload to the client with a `event`, and the second is the browser which is going to parse the payload and use +the `event` to determine what to do with the payload. + +For this we'll have to update our `stream_messages` function as follows: + + :::python + async def stream_messages() -> AsyncGenerator[str, None]: + connection_params = connection.get_connection_params() + + # Remove the cursor_factory parameter since I can't get + # the default from Django 4.2.1 to work. + # Django 4.2 didn't have the parameter and that worked. + connection_params.pop('cursor_factory') + + aconnection = await psycopg.AsyncConnection.connect( + **connection_params, + autocommit=True, + ) + channel_name = "lobby" + async with aconnection.cursor() as acursor: + await acursor.execute(f"LISTEN {channel_name}") + gen = aconnection.notifies() + async for notify in gen: + payload = json.loads(notify.payload) + event = payload.pop("event") + data = payload.pop("data") + yield f"event: {event}\ndata: {data}\n\n" + +Everything is the same except that we now parse the payload from the `NOTIFY` command and construct the SSE payload with +an `event` and a `data` field. This will come in handy when dealing with the frontend. + +Another way to do this would be to use Django's +[signals](https://docs.djangoproject.com/en/4.2/topics/signals/) or event +writing a PostgreSQL +[trigger](https://www.postgresql.org/docs/15/plpgsql-trigger.html) which issues +the `NOTIFY` command. + +### Hooking up the frontend + +Now that we've got the backend in place, we can get something up and running on +the frontend. + +We could use HTMX's [SSE +extension](https://htmx.org/extensions/server-sent-events/) but for this +example we'll just use the +[EventSource](https://developer.mozilla.org/en-US/docs/Web/API/EventSource) API +directly. + + :::html + + +
+ + + + +And that's it! We can now open two browser windows and see the messages appear in real time. + +Check out the repo for the full code where I've also added a simple form for submitting new messages. + +### Conclusion + +Django is boring, which is a good thing, to the degree where it is always the safe option. But with the advances in +async support it is becoming a viable, and shiny, option for doing real time stuff. Mix in some other solid and boring +tech like PostgreSQL and SSE, and you end up with a very solid foundation for building real time applications. + + +### Appendix + +#### How to run ASGI applications in development + +One thing that took me some time to realise is that the Django runserver is not +capable of running async views returning `StreamingHttpResponse`. + +Running the view with the builtin runserver results in the following error: + + :::text + .../django/http/response.py:514: Warning: StreamingHttpResponse must + consume asynchronous iterators in order to serve them synchronously. + Use a synchronous iterator instead. + +Fortunately Daphne, the ASGI server which was developed to power Django Channels, has an async runserver which we can use: + +To set this up we'll have to install the `daphne` package, add `daphne` to the top of our installed apps, and set +the `ASGI_APPLICATION` setting to point to our ASGI application. + + :::python + INSTALLED_APPS = [ + "daphne", + ... + "chat", # Our app + ] + + ASGI_APPLICATION = "project.asgi.application" + +Now we can just run `./manage.py runserver` as before and we are async ready! + + +#### Difference between 4.2 and 4.2.1 + +The code worked initially in 4.2, but 4.2.1 fixed a regression regarding setting a custom cursor in the database configuration. In 4.2 we get this from `connection.get_connection_params()`: @@ -283,163 +454,8 @@ So that now looks like so: async for notify in gen: yield f"data: {notify.payload}\n\n" -### Test the endpoint with curl -So now we've got the `LISTEN` part in place. - -If we connect to the endpoint using curl (`-N` disables buffering and is a way to consume streming content with curl): - - :::console - $ curl -N http://localhost:8000/messages/ - -And connect to our database and run: - - :::sql - NOTIFY new_message, 'Hello, world!'; - - -We, excitingly, get the following result : - - :::text - data: Hello, world! - -Amazing! - -### Issuing the NOTIFY - -But we want the `NOTIFY` command to be issued when a new chat message is submitted. - -For this we'll have a small utility function which does the heavy lifting. Note -that this is just a very simple synchronous function since everything is just -happening within a single request-response cycle. - - :::python - from django.db import connection - - - def notify(*, channel: str, event: str, payload: str) -> None: - payload = json.dumps({ - "event": event, - "content": payload, - }) - with connection.cursor() as cursor: - cursor.execute( - f"NOTIFY {channel}, '{payload}'", - ) - -And then we can use this in our view (I'm using `@csrf_exempt` here since this is just a quick proof of concept): - - :::python - @csrf_exempt - @require_POST - def post_message_view(request: HttpRequest) -> HttpResponse: - message = request.POST.get("message") - user = request.POST.get("user") - message = ChatMessage.objects.create(user=user, text=message) - notify( - channel="lobby", - event="message_created", - content=json.dumps({ - "text": message.text, - "user": message.user, - }) - ) - return HttpResponse("OK") - -The keen observer will notice that we are storing the payload content as a JSON string within a JSON string. - -This is because we have two recipients of the payload. The first is the `stream_messages` function which is going to -send the payload to the client with a `event`, and the second is the browser which is going to parse the payload and use -the `event` to determine what to do with the payload. - -For this we'll have to update our `stream_messages` function as follows: - - :::python - async def stream_messages() -> AsyncGenerator[str, None]: - connection_params = connection.get_connection_params() - - # Remove the cursor_factory parameter since I can't get - # the default from Django 4.2.1 to work. - # Django 4.2 didn't have the parameter and that worked. - connection_params.pop('cursor_factory') - - aconnection = await psycopg.AsyncConnection.connect( - **connection_params, - autocommit=True, - ) - channel_name = "lobby" - async with aconnection.cursor() as acursor: - await acursor.execute(f"LISTEN {channel_name}") - gen = aconnection.notifies() - async for notify in gen: - payload = json.loads(notify.payload) - event = payload.pop("event") - data = payload.pop("data") - yield f"event: {event}\ndata: {data}\n\n" - -Everything is the same except that we now parse the payload from the `NOTIFY` command and construct the SSE payload with -an `event` and a `data` field. This will come in handy when dealing with the frontend. - -Another way to do this would be to use Django's -[signals](https://docs.djangoproject.com/en/4.2/topics/signals/) or event -writing a PostgreSQL -[trigger](https://www.postgresql.org/docs/15/plpgsql-trigger.html) which issues -the `NOTIFY` command. - -### Frontend stuff - -Now that we've got the backend in place, we can get something up and running on -the frontend. - -We could use HTMX's [SSE -extension](https://htmx.org/extensions/server-sent-events/) but for this -example we'll just use the -[EventSource](https://developer.mozilla.org/en-US/docs/Web/API/EventSource) API -directly. - - :::html - - -
- - - - -And that's it! We can now open two browser windows and see the messages appear in real time. - -Check out the repo for the full code where I've also added a simple form for submitting new messages. - -### Conclusion - -Django is boring, which is a good thing, to the degree where it is always the safe option. But with the advances in -async support it is becoming a viable, and shiny, option for doing real time stuff. Mix in some other solid and boring -tech like PostgreSQL and SSE, and you end up with a very solid foundation for building real time applications. - -[0]: https://docs.djangoproject.com/en/4.2/releases/4.2/#requests-and-responses -[1]: https://docs.djangoproject.com/en/4.2/ref/request-response/#django.http.StreamingHttpResponse -[2]: https://caniuse.com/eventsource -[3]:https://www.psycopg.org/psycopg3/docs/advanced/async.html#index-4 +[^0]: [https://docs.djangoproject.com/en/4.2/releases/4.2/#requests-and-responses]() +[^1]: [https://docs.djangoproject.com/en/4.2/ref/request-response/#django.http.StreamingHttpResponse]() +[^2]: [https://caniuse.com/eventsource]() +[^3]: [https://www.psycopg.org/psycopg3/docs/advanced/async.html#index-4]() diff --git a/pelicanconf.py b/pelicanconf.py index 4e43deb..078769c 100644 --- a/pelicanconf.py +++ b/pelicanconf.py @@ -25,6 +25,16 @@ LINKS = (('Pelican', 'https://getpelican.com/'), SOCIAL = (('You can add links in your config file', '#'), ('Another social link', '#'),) +MARKDOWN = { + 'extension_configs': { + 'markdown.extensions.codehilite': {'css_class': 'highlight'}, + 'markdown.extensions.extra': {}, + 'markdown.extensions.meta': {}, + 'markdown.extensions.toc': {}, + }, + 'output_format': 'html5', +} + DEFAULT_PAGINATION = 10 # Uncomment following line if you want document-relative URLs when developing