valberg/valberg.dk
1
0
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity

Upd.

Browse source
This commit is contained in:
Víðir Valberg Guðmundsson 2023-05-17 17:27:22 +02:00
parent bb5d8c9019
commit 1660f4a529
2 changed files with 225 additions and 199 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

414
content/django-server-sent-events.md
View file

@ -8,13 +8,11 @@ Summary: A write-up of how I implemented server-sent events using Django 4.2 and
---
With the release of Django 4.2 we got the following [0]:
With the release of Django 4.2 we got the following [^0]:
> [`StreamingHttpResponse`](https://docs.djangoproject.com/en/4.2/ref/request-response/#django.http.StreamingHttpResponse "django.http.StreamingHttpResponse") now supports async iterators when Django is served via ASGI.
And the documentation has been expanded with the following [1]:
And the documentation has been expanded with the following [^1]:
> When serving under ASGI, however, a [`StreamingHttpResponse`](https://docs.djangoproject.com/en/4.2/ref/request-response/#django.http.StreamingHttpResponse "django.http.StreamingHttpResponse") need not stop other requests from being served whilst waiting for I/O. This opens up the possibility of long-lived requests for streaming content and implementing patterns such as long-polling, and server-sent events.
@ -26,10 +24,14 @@ So I set out to write a small, drumroll please, chat application!
The code for the chat application can be found at
[github.com/valberg/django-sse](https://github.com/valberg/django-sse).
**Table of contents**
[TOC]
### What are server-sent events and why do we want to use them?
Server-sent events is "old tech", as in that is has been supported in major
browser since around 2010-2011 [2]. The idea is that the client "subscribes" to
browser since around 2010-2011 [^2]. The idea is that the client "subscribes" to
an HTTP endpoint, and the server can then issue data to the client as long as
the connection is open. This is a great performance boost compared to other
techniques as for instance polling the server.
@ -98,34 +100,6 @@ probably something we could have done with a synchronous view.
Let's see if we can do better. But first we'll have to talk about how to run
this code.
#### Aside: Use an ASGI server for development
One thing that took me some time to realise is that the Django runserver is not
capable of running async views returning `StreamingHttpResponse`.
Running the above view with the runserver results in the following error:
:::text
.../django/http/response.py:514: Warning: StreamingHttpResponse must
consume asynchronous iterators in order to serve them synchronously.
Use a synchronous iterator instead.
Fortunately Daphne, the ASGI server which was developed to power Django Channels, has an async runserver which we can use:
To set this up we'll have to install the `daphne` package, add `daphne` to the top of our installed apps, and set
the `ASGI_APPLICATION` setting to point to our ASGI application.
:::python
INSTALLED_APPS = [
"daphne",
...
"chat", # Our app
]
ASGI_APPLICATION = "project.asgi.application"
Now we can just run `./manage.py runserver` as before and we are async ready!
### More old tech to the rescue: PostgreSQL LISTEN/NOTIFY
This is where we could reach for more infrastructure which could help us giving
@ -141,13 +115,13 @@ LISTEN to a channel and then anyone can NOTIFY on that same channel.
This seems like something we can use - but psycopg2 isn't async, so I'm not
even sure if `sync_to_async` would help us here.
### Enter psycopg 3
#### Enter psycopg 3
I had put the whole thing on ice until I realized that another big thing (maybe
a bit bigger than StreamingHttpResponse) in Django 4.2 is the support for
psycopg 3 - and psycopg 3 is very much async!
So I went for a stroll in the psycopg 3 documentation and found this gold[3]:
So I went for a stroll in the psycopg 3 documentation and found this gold[^3]:
::python
import psycopg
@ -172,8 +146,17 @@ So by combining the snippet from the psycopg 3 documentation and my previous
from django.db import connection
async def stream_foos() -> AsyncGenerator[str, None]:
# Get the connection params from Django
connection_params = connection.get_connection_params()
# Somehow Django 4.2.1 sets the cursor_factory to
# django.db.backends.postgresql.base.Cursor
# which causes problems. Read more about it in the
# "Differences between 4.2 and 4.2.1" section in the Appendix.
# Removing it from the connection parameters works around this.
connection_params.pop('cursor_factory')
aconnection = await psycopg.AsyncConnection.connect(
**connection_params,
autocommit=True,
@ -186,13 +169,201 @@ So by combining the snippet from the psycopg 3 documentation and my previous
async for notify in gen:
yield f"data: {notify.payload}\n\n"
I was almost about to give up again, since this approach didn't work initially.
All because I for some reason had removed the `autocommit=True` in my attempts
to async-ify the snippet from the psycopg 3 documentation.
Appart from problems with the `cursor_factory` (which I'll get back to in the
[appendix](#difference-between-42-and-421)), this code is pretty straight forward and, most importantly, works!
#### Aside: Difference between 4.2 and 4.2.1
the code worked initially in 4.2, but 4.2.1 fixed a regression regarding
### Test the endpoint with curl
So now we've got the `LISTEN` part in place.
If we connect to the endpoint using curl (`-N` disables buffering and is a way to consume streming content with curl):
:::console
$ curl -N http://localhost:8000/messages/
And connect to our database and run:
:::sql
NOTIFY new_message, 'Hello, world!';
We, excitingly, get the following result :
:::text
data: Hello, world!
Amazing!
### Issuing the NOTIFY command from Django
But we want the `NOTIFY` command to be issued when a new chat message is submitted.
For this we'll have a small utility function which does the heavy lifting. Note
that this is just a very simple synchronous function since everything is just
happening within a single request-response cycle.
:::python
from django.db import connection
def notify(*, channel: str, event: str, payload: str) -> None:
payload = json.dumps({
"event": event,
"content": payload,
})
with connection.cursor() as cursor:
cursor.execute(
f"NOTIFY {channel}, '{payload}'",
)
And then we can use this in our view (I'm using `@csrf_exempt` here since this is just a quick proof of concept):
:::python
@csrf_exempt
@require_POST
def post_message_view(request: HttpRequest) -> HttpResponse:
message = request.POST.get("message")
user = request.POST.get("user")
message = ChatMessage.objects.create(user=user, text=message)
notify(
channel="lobby",
event="message_created",
content=json.dumps({
"text": message.text,
"user": message.user,
})
)
return HttpResponse("OK")
The keen observer will notice that we are storing the payload content as a JSON string within a JSON string.
This is because we have two recipients of the payload. The first is the `stream_messages` function which is going to
send the payload to the client with a `event`, and the second is the browser which is going to parse the payload and use
the `event` to determine what to do with the payload.
For this we'll have to update our `stream_messages` function as follows:
:::python
async def stream_messages() -> AsyncGenerator[str, None]:
connection_params = connection.get_connection_params()
# Remove the cursor_factory parameter since I can't get
# the default from Django 4.2.1 to work.
# Django 4.2 didn't have the parameter and that worked.
connection_params.pop('cursor_factory')
aconnection = await psycopg.AsyncConnection.connect(
**connection_params,
autocommit=True,
)
channel_name = "lobby"
async with aconnection.cursor() as acursor:
await acursor.execute(f"LISTEN {channel_name}")
gen = aconnection.notifies()
async for notify in gen:
payload = json.loads(notify.payload)
event = payload.pop("event")
data = payload.pop("data")
yield f"event: {event}\ndata: {data}\n\n"
Everything is the same except that we now parse the payload from the `NOTIFY` command and construct the SSE payload with
an `event` and a `data` field. This will come in handy when dealing with the frontend.
Another way to do this would be to use Django's
[signals](https://docs.djangoproject.com/en/4.2/topics/signals/) or event
writing a PostgreSQL
[trigger](https://www.postgresql.org/docs/15/plpgsql-trigger.html) which issues
the `NOTIFY` command.
### Hooking up the frontend
Now that we've got the backend in place, we can get something up and running on
the frontend.
We could use HTMX's [SSE
extension](https://htmx.org/extensions/server-sent-events/) but for this
example we'll just use the
[EventSource](https://developer.mozilla.org/en-US/docs/Web/API/EventSource) API
directly.
:::html
<template id="message">
<div style="border: 1px solid black; margin: 5px; padding: 5px;">
<strong class="user"></strong>: <span class="message"></span>
</div>
</template>
<div id="messages"></div>
<script>
const source = new EventSource("/messages/");
// Note that the event we gave our notify utility function is called "message_created"
// so that's what we listen for here.
source.addEventListener("message_created", function(evt) {
// Parse the payload
let payload = JSON.parse(evt.data);
// Get and clone our template
let template = document.getElementById('message');
let clone = template.content.cloneNode(true);
// Update our cloned template
clone.querySelector('.user').innerText = payload.user;
clone.querySelector('.message').innerText = payload.text;
// Append the cloned template to our list of messages
document.getElementById('messages').appendChild(clone);
});
</script>
And that's it! We can now open two browser windows and see the messages appear in real time.
Check out the repo for the full code where I've also added a simple form for submitting new messages.
### Conclusion
Django is boring, which is a good thing, to the degree where it is always the safe option. But with the advances in
async support it is becoming a viable, and shiny, option for doing real time stuff. Mix in some other solid and boring
tech like PostgreSQL and SSE, and you end up with a very solid foundation for building real time applications.
### Appendix
#### How to run ASGI applications in development
One thing that took me some time to realise is that the Django runserver is not
capable of running async views returning `StreamingHttpResponse`.
Running the view with the builtin runserver results in the following error:
:::text
.../django/http/response.py:514: Warning: StreamingHttpResponse must
consume asynchronous iterators in order to serve them synchronously.
Use a synchronous iterator instead.
Fortunately Daphne, the ASGI server which was developed to power Django Channels, has an async runserver which we can use:
To set this up we'll have to install the `daphne` package, add `daphne` to the top of our installed apps, and set
the `ASGI_APPLICATION` setting to point to our ASGI application.
:::python
INSTALLED_APPS = [
"daphne",
...
"chat", # Our app
]
ASGI_APPLICATION = "project.asgi.application"
Now we can just run `./manage.py runserver` as before and we are async ready!
#### Difference between 4.2 and 4.2.1
The code worked initially in 4.2, but 4.2.1 fixed a regression regarding
setting a custom cursor in the database configuration.
In 4.2 we get this from `connection.get_connection_params()`:
@ -283,163 +454,8 @@ So that now looks like so:
async for notify in gen:
yield f"data: {notify.payload}\n\n"
### Test the endpoint with curl
So now we've got the `LISTEN` part in place.
If we connect to the endpoint using curl (`-N` disables buffering and is a way to consume streming content with curl):
:::console
$ curl -N http://localhost:8000/messages/
And connect to our database and run:
:::sql
NOTIFY new_message, 'Hello, world!';
We, excitingly, get the following result :
:::text
data: Hello, world!
Amazing!
### Issuing the NOTIFY
But we want the `NOTIFY` command to be issued when a new chat message is submitted.
For this we'll have a small utility function which does the heavy lifting. Note
that this is just a very simple synchronous function since everything is just
happening within a single request-response cycle.
:::python
from django.db import connection
def notify(*, channel: str, event: str, payload: str) -> None:
payload = json.dumps({
"event": event,
"content": payload,
})
with connection.cursor() as cursor:
cursor.execute(
f"NOTIFY {channel}, '{payload}'",
)
And then we can use this in our view (I'm using `@csrf_exempt` here since this is just a quick proof of concept):
:::python
@csrf_exempt
@require_POST
def post_message_view(request: HttpRequest) -> HttpResponse:
message = request.POST.get("message")
user = request.POST.get("user")
message = ChatMessage.objects.create(user=user, text=message)
notify(
channel="lobby",
event="message_created",
content=json.dumps({
"text": message.text,
"user": message.user,
})
)
return HttpResponse("OK")
The keen observer will notice that we are storing the payload content as a JSON string within a JSON string.
This is because we have two recipients of the payload. The first is the `stream_messages` function which is going to
send the payload to the client with a `event`, and the second is the browser which is going to parse the payload and use
the `event` to determine what to do with the payload.
For this we'll have to update our `stream_messages` function as follows:
:::python
async def stream_messages() -> AsyncGenerator[str, None]:
connection_params = connection.get_connection_params()
# Remove the cursor_factory parameter since I can't get
# the default from Django 4.2.1 to work.
# Django 4.2 didn't have the parameter and that worked.
connection_params.pop('cursor_factory')
aconnection = await psycopg.AsyncConnection.connect(
**connection_params,
autocommit=True,
)
channel_name = "lobby"
async with aconnection.cursor() as acursor:
await acursor.execute(f"LISTEN {channel_name}")
gen = aconnection.notifies()
async for notify in gen:
payload = json.loads(notify.payload)
event = payload.pop("event")
data = payload.pop("data")
yield f"event: {event}\ndata: {data}\n\n"
Everything is the same except that we now parse the payload from the `NOTIFY` command and construct the SSE payload with
an `event` and a `data` field. This will come in handy when dealing with the frontend.
Another way to do this would be to use Django's
[signals](https://docs.djangoproject.com/en/4.2/topics/signals/) or event
writing a PostgreSQL
[trigger](https://www.postgresql.org/docs/15/plpgsql-trigger.html) which issues
the `NOTIFY` command.
### Frontend stuff
Now that we've got the backend in place, we can get something up and running on
the frontend.
We could use HTMX's [SSE
extension](https://htmx.org/extensions/server-sent-events/) but for this
example we'll just use the
[EventSource](https://developer.mozilla.org/en-US/docs/Web/API/EventSource) API
directly.
:::html
<template id="message">
<div style="border: 1px solid black; margin: 5px; padding: 5px;">
<strong class="user"></strong>: <span class="message"></span>
</div>
</template>
<div id="messages"></div>
<script>
const source = new EventSource("/messages/");
// Note that the event we gave our notify utility function is called "message_created"
// so that's what we listen for here.
source.addEventListener("message_created", function(evt) {
// Parse the payload
let payload = JSON.parse(evt.data);
// Get and clone our template
let template = document.getElementById('message');
let clone = template.content.cloneNode(true);
// Update our cloned template
clone.querySelector('.user').innerText = payload.user;
clone.querySelector('.message').innerText = payload.text;
// Append the cloned template to our list of messages
document.getElementById('messages').appendChild(clone);
});
</script>
And that's it! We can now open two browser windows and see the messages appear in real time.
Check out the repo for the full code where I've also added a simple form for submitting new messages.
### Conclusion
Django is boring, which is a good thing, to the degree where it is always the safe option. But with the advances in
async support it is becoming a viable, and shiny, option for doing real time stuff. Mix in some other solid and boring
tech like PostgreSQL and SSE, and you end up with a very solid foundation for building real time applications.
[0]: https://docs.djangoproject.com/en/4.2/releases/4.2/#requests-and-responses
[1]: https://docs.djangoproject.com/en/4.2/ref/request-response/#django.http.StreamingHttpResponse
[2]: https://caniuse.com/eventsource
[3]:https://www.psycopg.org/psycopg3/docs/advanced/async.html#index-4
[^0]: [https://docs.djangoproject.com/en/4.2/releases/4.2/#requests-and-responses]()
[^1]: [https://docs.djangoproject.com/en/4.2/ref/request-response/#django.http.StreamingHttpResponse]()
[^2]: [https://caniuse.com/eventsource]()
[^3]: [https://www.psycopg.org/psycopg3/docs/advanced/async.html#index-4]()

10
pelicanconf.py
View file

@ -25,6 +25,16 @@ LINKS = (('Pelican', 'https://getpelican.com/'),
SOCIAL = (('You can add links in your config file', '#'),
('Another social link', '#'),)
MARKDOWN = {
'extension_configs': {
'markdown.extensions.codehilite': {'css_class': 'highlight'},
'markdown.extensions.extra': {},
'markdown.extensions.meta': {},
'markdown.extensions.toc': {},
},
'output_format': 'html5',
}
DEFAULT_PAGINATION = 10
# Uncomment following line if you want document-relative URLs when developing