SDC Decomposition Sprint 2

common-content/en/module/decomposition/adding-like-dislike/index.md

@@ -0,0 +1,64 @@
++++
+title = "Adding like/dislike"
+headless = true
+time = 120
+facilitation = false
+emoji= "👍👎"
+objectives = [
+    "Identify what data needs to be stored and exchanged between a client and server.",
+    "Devise a scheme for differentiating messages with different meanings (e.g. a new message vs a new like).",
+    "Contrast giving updated values as absolute values or relative changes.",
+    "Implement an end-to-end feature involving data updates and reconciliation across a client and server.",
+]
++++
+
+The last requirement we have for our chat application is the ability to like/dislike a message (and see what messages have been liked/disliked).
+
+{{<note type="Exercise">}}
+Think about what information a client would need to provide to a server in order to like/dislike a message.
+
+Think about what information a server would need to provide to a client in order to display how many likes/dislikes a message has.
+
+Think about what information a server would need to provide to a client in order to _update_ how many likes/dislikes a message has.
+
+Write these things down.
+{{</note>}}
+
+### Identifiers
+
+One of the key new requirements to add liking/disliking a message is knowing _which_ message is being liked/disliked.
+
+When a client wants to like a message, it needs some way of saying _this_ is the message I want to like.
+
+This suggests we need a unique identifier for each message:
+* When the server tells a client about a message, it needs to tell it what the identifier is for that message.
+* When a client tells the server it wants to like a message, it needs to tell it the identifier for the message it wants to like.
+* When the server tells a client a message has been liked, it needs to tell the client which message was liked, and the client needs to know enough about that message to be able to update the UI.
+
+### Message formats
+
+Now that your server will be sending multiple kinds of updates ("Here's a new message", or "Here's an update to the number of likes of an existing message"), you'll need to make sure the client knows the difference between these messages. The client will need to know how to act when it receives each kind of message.
+
+### Changes or absolutes?
+
+When new likes happen, a choice we need to make is whether the server should tell a client "this message was liked again" or should tell the client "this message now has 10 likes". Both of these can work.
+
+{{<note type="Exercise">}}
+Write down some advantages and disadvantages of a server -> client update being "+1 compared to before" or "now =10".
+
+Choose which approach you want to take.
+{{</note>}}
+
+{{<note type="Exercise">}}
+Implement liking and disliking messages.
+
+If a message has a non-zero number of likes or dislikes, the frontend needs to show this.
+
+The frontend needs to expose some way for a user to like or dislike any message.
+
+When a user likes or dislikes a message, the frontend needs to tell the backend about this, and the backend needs to notify all clients of this.
+
+When a frontend is notified by a backend about a new like or dislike, it needs to update the UI to show this.
+
+You may do this in your polling implementation, WebSockets implementation, or both.
+{{</note>}}

common-content/en/module/decomposition/chat-application-requirements/index.md

@@ -0,0 +1,40 @@
++++
+title = "Chat application requirements"
+headless = true
+time = 30
+facilitation = false
+emoji= "📝"
+objectives = [
+    "Explain the requirements of our chat application.",
+    "Explain what requirements are out of scope for our chat application.",
+]
++++
+
+We are going to make a chat application which lets multiple users exchange messages.
+
+Let's think about the core requirements for our application:
+* As a user, I can send add a message to the chat.
+* As a user, when I open the chat I see the messages that have been sent by any user.
+* As a user, when someone sends a message, it gets added to what I see.
+* As a user, I can "like" or "dislike" someone's message.
+* When messages are liked or disliked, a count of the likes and dislikes is displayed next to the message.
+
+We can imagine other requirements too (e.g. replying to specific messages, reacting with emojis, being able to edit or delete messages, registering exclusive use of a username, ...). We will stick just to the requirements we've listed.
+
+Because users want to see things, we know we'll need a frontend.
+
+Because multiple users want to be able to share information (messages), we know we'll need a backend for them to communicate via.
+
+### What we already know and what's new
+
+Some of these requirements are similar to the quote server we've already implemented:
+* Adding messages is like adding quotes.
+* Seeing messages is like seeing quotes.
+
+Others are new:
+* Live updates
+* Interacting with a message
+
+First let's make a backend and a frontend to do what we already know. This shouldn't take us very long (we know how to do it, and have done it recently). It will give us a useful framework to experiment with the things that are new to us.
+
+If we thought it would take us a long time to do what we already know, we may approach this differently. We would probably try to work out the new things first. Because they may change how we want to do everything. But because it should be quick to do what we do know, we'll start there.

common-content/en/module/decomposition/deploy-chat-application/index.md

@@ -1,11 +1,14 @@
 +++
 title = "Deploying the chat application"
 headless = true
-time = 30
+time = 60
 facilitation = false
-emoji= "📖"
-[objectives]
-    1="Deploy all components of a frontend/backend/database application so it can be used on the internet"
+emoji= "➡️"
+objectives = [
+    "Deploy a frontend and backend so it can be used on the internet.",
+]
 +++
 
-### Deploying the chat application
+{{<note type="Exercise">}}
+Deploy your chat application so that both the frontend and backend are on the Internet, can talk to each other, and can be used by people.
+{{</note>}}

common-content/en/module/decomposition/designing-streaming-api/index.md

@@ -0,0 +1,46 @@
++++
+title = "Designing a streaming API"
+headless = true
+time = 30
+facilitation = false
+emoji= "🧑‍🔬"
+objectives = [
+    "Describe the trade-offs of different streaming APIs.",
+]
++++
+
+So far we have a frontend that asks a backend for all of its chat messages, and shows them.
+
+We want the frontend to find out about new messages.
+
+One way we could do this is for our frontend to frequently ask our backend for all of its messages. We could then either re-render the page with all of the messages, or we could work out which messages are new and add them to the UI.
+
+This approach can work, but it is {{<tooltip title="expensive">}}When we use the word expensive, we don't always mean in terms of money. Sometimes we mean something is slow, something will use a lot of memory, or will use a lot of some other resource{{</tooltip>}}. In this case, the reason it's expensive is that it involves sending a lot of data over the network. Often the _same_ data _repeatedly_.
+
+A cheaper approach would be for the client to be able to ask only for the new messages. But this is a more complicated question than it seems. What does "new" mean? New since when?
+
+We can divide approaches to identifying new messages based on whose job it is to answer the question "since when?".
+
+### Remembering "since when" on the client
+
+If the server tells the client, in its response, some identifier for its newest message (e.g. the timestamp, or some ID), the client could ask in its next request "Please give me all messages since this timestamp".
+
+The client would need to remember the last timestamp/ID it has seen.
+
+The server will need to be able to answer that question: "What messages have been sent since this time?".
+
+There are a few ways we can support this on the server:
+1. If we're storing messages in an array, in all our responses we could tell the client what the last index we returned to it was. Then the server could just `slice` the array to get the values to return.
+2. Or if we don't want to expose array indexes (e.g. because perhaps messages can be deleted), we could put a timestamp in every message object, and `filter` our array based on the timestamp we're searching from.
+
+### Remembering "since when" on the server
+
+The server could remember, for each client, what the last message it sent to that client was.
+
+The client would no longer need to remember the last timestamp/ID it had been sent. It would just ask for new messages.
+
+The server would need a way of identifying which client is making a request, and would itself need to remember which messages it had already sent that client.
+
+This simplifies things for the client, but means that the server needs to remember information for each client it has. If there are a lot of clients, this may be a lot of work, and take up a lot of memory. One challenge with this approach is that the server needs to know when it can forget about a client - if it never forgets about clients, it will keep using more and more memory to track this information for every client it's ever had. And we probably can't rely on a client to tell our backend "you can forget about me now".
+
+We will try out both of these approaches in two different frontends to the same backend.

common-content/en/module/decomposition/long-polling/index.md

@@ -0,0 +1,85 @@
++++
+title = "Long-polling"
+headless = true
+time = 90
+facilitation = false
+emoji= "⏳"
+objectives = [
+    "Stream live updates from a server by using long-polling.",
+    "Compare the trade-offs involved with frequent small requests vs long-polling.",
+]
++++
+
+We noticed when we were repeatedly polling for updates that we were making lots of requests, and a lot of their responses were empty.
+
+Another approach we could take is to make an HTTP request, but have the server not respond to it until there is a non-empty response.
+
+One request gets one response, but the server can take some time to make that response.
+
+This is _also_ expensive, but in a different way.
+
+### Different kinds of expensive
+
+Making lots of requests was expensive because establishing connections takes time, CPU, and network resource. And on the server calculating that there were no new messages to respond with took a bit of time.
+
+Keeping requests open for a long time is expensive because it means the server needs to keep all of the resources associated with a request open. The whole time the request is waiting, the server needs to keep the TCP connection open, needs to keep the memory associated with the request, needs to remember that it should respond to it (and how to). It also means that we need to be careful about restarting the server (e.g. if we change its code and want to run the new code) - stopping the server will break any connections that are waiting.
+
+When we were making lots of small quick connections, we could do things like start a new server, shift over all new traffic to it, and turn off the old server. This was easy for us to do because each request was quick, so we knew when we switched to the new server, any requests to the old server would be done quickly, and it wouldn't get any more.
+
+Now that we're using long connections, if we turn on a new server and switch new requests to it, it may still be a while before the old server's requests are finished.
+
+In different contexts, it may be worse to have lots of short requests, or fewer longer requests.
+
+### Changing our code to support long-polling
+
+On the client side, we probably don't need to change anything about our code. Just the requests it makes may take longer to resolve.
+
+On the server side, we will need to change things around. Right now, our `GET` handler _always_ responds based on what it currently knows.
+
+We want to change this.
+
+If we don't have any messages to respond with, we want to wait until we _do_ have something to respond with.
+
+We know `res.send` is a callback function - it's a function we can call when we have a response to send.
+
+Now we don't always want to call `res.send` in our handler. Sometimes we want to say: "When we next get a message, call `res.send` with that message".
+
+This means we need to share our callback between the `GET` handler for asking about new messages, and the `POST` handler for sending new messages.
+
+When the `POST` handler saves a new message, we want to check if there are any `GET` handlers waiting for a message, and call those `GET` handlers' `res.send` callbacks with the new message.
+
+To achieve this, we probably want to make a new piece of state on our server where we can remember which callbacks still need to be called with responses:
+
+```js
+const callbacksForNewMessages = [];
+```
+
+In our `GET` handler, we want to write some code to remember that someone of our callbacks aren't being instantly called, and will need calling in the future:
+
+```js
+if (messagesToSend.length === 0) {
+    // Note: We need to use an arrow function here, rather than just pushing `res.send` directly.
+    // This is because of handling of "this".
+    // You can read about "this" at https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/this
+    callbacksForNewMessages.push((value) => res.send(value));
+} else {
+    res.send(messagesToSend);
+}
+```
+
+and in our `POST` handler, when we are saving a new message we want to run some code to call any callbacks which are still waiting for a response when we get a new message:
+
+```js
+while (callbacksForNewMessages.length > 0) {
+    const callback = callbacksForNewMessages.pop();
+    callback([newMessage]);
+}
+```
+
+Note how we're wrapping `newMessage` in an array - we need to return the same type of information whether we're immediately calling our callback, or doing it later on.
+
+{{<note type="Exercise">}}
+Update your backend to support long-polling.
+
+You may want to make this an optional behaviour (e.g. by adding a command line flag, or allowing a client to opt in to fast- or long-polling based on a query parameter) so that you can easily experiment with both.
+{{</note>}}

common-content/en/module/decomposition/non-live-chat-application/index.md

@@ -1,13 +1,36 @@
 +++
 title = "Making a non-live chat application"
 headless = true
-time = 30
+time = 60
 facilitation = false
-emoji= "📖"
-[objectives]
-    1="Write and run a frontend, backend, and database"
-    2="Persist data passed from a user into a database"
-    3="Display data from a database to a user"
+emoji= "🚧"
+objectives = [
+    "Write and run a frontend and backend",
+    "Store data passed from a user into a backend",
+    "Display data from a backend to a user",
+]
 +++
 
-### Making a non-live chat application
+{{<note type="Exercise">}}
+Write a frontend and backend for our chat application.
+
+Remember, the two requirements we have to start with are:
+1. As a user, I can send add a message to the chat.
+2. As a user, when I open the chat I see the messages that have been sent by any user.
+
+But soon we'll be adding:
+
+3. As a user, when someone sends a message, it gets added to what I see.
+4. As a user, I can "like" or "dislike" someone's message.
+5. When messages are liked or disliked, a count of the likes and dislikes is displayed next to the message.
+
+It's useful to remember what we're going to do next - it may impact how we do things now.
+
+But for now, our _main_ focus should be our two first requirements.
+{{</note>}}
+
+{{<note type="Exercise">}}
+Run your frontend and backend together locally.
+
+Test that it works as we expect. Send several messages. Refresh the page. Make sure you see all of the message (in the correct order).
+{{</note>}}

common-content/en/module/decomposition/repeated-client-requests-for-updates/index.md

@@ -0,0 +1,45 @@
++++
+title = "Repeated client requests for updates"
+headless = true
+time = 90
+facilitation = false
+emoji= "🔄"
+objectives = [
+    "Stream live updates from a server by using repeated HTTP polling.",
+]
++++
+
+First let's explore how the client can remember "since when".
+
+{{<note type="Exercise">}}
+Teach your backend how to answer "since when" queries.
+
+You can choose whether you want to use array indexes, timestamps in objects, or some other "since when" tracker.
+
+You can also choose whether you want this to be the same endpoint as your "get all messages" endpoint (with some query parameter), or a separate endpoint.
+{{</note>}}
+
+Now that our backend can answer the question "show me messages since", we can update our frontend to repeatedly ask for new messages, and add them to the UI.
+
+Here is an example function we could write to do this:
+
+```js
+const keepFetchingMessages = async () => {
+    const lastMessageTime = state.messages.length > 0 ? state.messages[state.messages.length - 1].timestamp : null;
+    const queryString = lastMessageTime ? `?since=${lastMessageTime}` : "";
+    const url = `${server}/messages${queryString}`;
+    const rawResponse = await fetch(url);
+    const response = await rawResponse.json();
+    state.messages.push(...response);
+    render();
+    setTimeout(keepFetchingMessages, 100);
+}
+```
+
+It asks for messages since the last seen message (or all messages, if we don't know about any), updates the known messages state, re-renders the page, and then calls itself in 100ms.
+
+This means that within about 100ms of a message being sent, we should know about it and display it.
+
+We only transfer new messages, we're not sending all messages over the network every 100ms. That would be a lot of data.
+
+But we _are_ making a lot of requests. Probably most of those requests get an empty response. And this is in its own way expensive - making a new HTTP connection over and over again takes some compute and network resources. And each time we make a request, our server has to do some work to work out how it should answer.