Add a REPL webserver

This adds a server that acts as a function

  (exp, env) -> (result, env')

meaning the server has to hold no state. The client maintains and
transmits the state.

This has some bugs right now, such as not being able to serialize
recursive functions. Those will be fixed soon.

---

This commit was comprised of the following commits:

Add initial ScrapMonad

Add a notion of relocations

We'll need to link native functions, for example, if we take in a
serialized environment from the outside.

Add small server with JSON API and little HTML page

Provide a networked REPL, perhaps as a demo.

Add bdecode

This is a little more stateful than bencode so add a class and call into
the class from a `bdecode` function.

Begin deserialization

Use some Python nonsense to automatically add deserializer functions to
a dict if Object subclasses have them.

Add some more deserializers, including first recursive one

This is a little painful and I wonder if there is a better way to do it.

Some niceties

Add deserialization for EnvObject

Deserialize env and wrap it in EnvObject

We need to pass around entire scrap objects.

Persist env in client

Serialize/deserialize functions, closures

Next up: NativeFunctionRelocation.

Type ScrapMonad.__init__

Drop the _

It's cleaner.

Fix infinite recursion bug

All Objects have deserialize (defined or inheritecd), but all of them
*define* it. Check if they define it by looking in `__dict__`.

Add support for deserializing NativeFunctionRelocation

With this, the web REPL can ship an env back and forth to the idempotent
eval server!

Fix Python 3.8

Unwrap classmethod and store inner function. Kind of surprised mypy
didn't catch this, since I was never storing functions before--always
classmethod.

Persist env in LocalStorage

Persist history in LocalStorage

Focus input field on page load

Add TODO

Throw some errors that do not normally raise

Don't double fetch

Add title and note

Remove begin/end newlines

Bound output window height and scroll to bottom

Add doctype

Embed empty favicon in page

Save a request that will always 404.

Add a little favicon

Thanks, antifavicon.

Set 1 hour cache control on eval requests

As the server is stateless, we can have some intermediate server cache
eval(env, exp) requests. If everyone loads the page and types `123`, for
example, we only need receive that request once (per hour).

It's not clear if fly.io (for example) does this middle cache thing
automatically.

Try building image in CI (#19)

* Try building image in CI

* wip

* Add quick check post-build

* Don't be interactive

* Use tags from meta step to identify image

Oops, remove target branch

Remove excess env

Scroll to bottom after loading history

Add space in favicon

Add button to clear LocalStorage

Add TODO

Add multiprocessing server

Serialize/deserialize Apply

Check in fly.toml

Make web repl return result, not _

Report eval errors somewhat opaquely in console

Add updateHistory function

Use try/catch in client

Display server error message in output

Move all explicit exception handling into do_index

Link to repl page

Add charset and viewport

Allow reuse of address

This means we can C-c the server and re-start it on the same port
immediately.

Check in scrapscript.org style

Adjust color and use <main> for padding

Add fonts

Render REPL with code/result a little better

Bundle style.css with Docker image

Fix input background

Use prompt as label

Remove TODO

Don't highlight input box

Deserialize bool and record

Move html to separate file

Expand input into textarea with Shift-Enter

Add goatcounter

Remove console.log

Color textareas too

Make the webserver fork

Bring Chris's code up to date

Print host in debug output

Dockerfile

@@ -12,6 +12,8 @@ RUN bin/ape.elf bin/python -m compileall Lib
 RUN mv Lib/__pycache__/scrapscript*.pyc Lib/scrapscript.pyc
 RUN rm Lib/scrapscript.py
 RUN cp bin/python bin/scrapscript.com
+COPY style.css Lib
+COPY repl.html Lib
 RUN sh bin/zip -A -r bin/scrapscript.com Lib .args
 RUN bin/ape.elf bin/assimilate bin/scrapscript.com
 

build-com

@@ -4,14 +4,17 @@ PREV="$(pwd)"
 DIR="$(mktemp -d)"
 cp scrapscript.py "$DIR"
 cp .args "$DIR"
+cp repl.html "$DIR"
+cp style.css "$DIR"
 cd "$DIR"
 wget https://cosmo.zip/pub/cosmos/bin/python
 wget https://cosmo.zip/pub/cosmos/bin/zip
 chmod +x python
 chmod +x zip
 ./python -m compileall scrapscript.py
 mkdir Lib
-cp __pycache__/scrapscript.cpython-311.pyc Lib/scrapscript.pyc
+cp __pycache__/scrapscript.*.pyc Lib/scrapscript.pyc
+cp style.css repl.html Lib
 cp python scrapscript.com
 ./zip -r scrapscript.com Lib .args
 echo "Testing..."

fly.toml

@@ -0,0 +1,25 @@
+# fly.toml app configuration file generated for scrapscript on 2023-11-29T02:06:16-05:00
+#
+# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
+#
+
+app = "scrapscript"
+primary_region = "ewr"
+
+[build]
+
+[http_service]
+  internal_port = 8000
+  force_https = true
+  auto_stop_machines = true
+  auto_start_machines = true
+  min_machines_running = 0
+  processes = ["app"]
+
+[[vm]]
+  cpu_kind = "shared"
+  cpus = 1
+  memory_mb = 256
+
+[experimental]
+cmd = ["serve", "--port", "8000", "--debug", "--fork"]

repl.html

@@ -0,0 +1,166 @@
+<!DOCTYPE html>
+<html>
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>Scrapscript Web REPL</title>
+<link rel="shortcut icon" type="image/x-icon" href="data:image/vnd.microsoft.icon;base64,AAABAAEAEBAQAAAAAAAoAQAAFgAAACgAAAAQAAAAIAAAAAEABAAAAAAAgAAAAAAAAAAAAAAAEAAAAAAAAAD/AAAA////AAAA/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAREBEQAAAAAQAQEAEAAAABABAQAQAAAAAREBEQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACIiIiIiIiIiIiIiIiIiIiIiERIiESEiIiIiESEiISIiIhEiISIhEiIiIREiESEhIiIiIiIiIiIiIiIiIiIiIiIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA">
+<link rel="preconnect" href="https://fonts.googleapis.com" />
+<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
+<link
+  href="https://fonts.googleapis.com/css2?family=Fira+Code&display=swap"
+  rel="stylesheet"
+/>
+<link
+  href="https://fonts.googleapis.com/css2?family=Rubik:ital,wght@0,500;0,700&display=swap"
+  rel="stylesheet"
+/>
+<link
+  href="https://fonts.googleapis.com/css2?family=Nunito+Sans:ital,wght@0,400;0,700;0,900;1,400&display=swap"
+  rel="stylesheet"
+/>
+<link rel="stylesheet" href="/style.css" />
+</head>
+<body>
+<main>
+<div>
+<p>See <a href="https://scrapscript.org/">scrapscript.org</a> for a slightly
+out of date language reference.</p>
+<p>This REPL sends programs to a server to be evaluated, but the server is
+completely stateless. All persistence is in the browser.</p>
+</div>
+<div>
+<!-- TODO(max): Add button to save to/load from disk. -->
+<button id="clear-local-storage">Clear LocalStorage</button>
+</div>
+<div id="output" style="height: 400px; overflow: auto;">
+Output:
+</div>
+<div>
+<code>>>> </code><input id="input" type="text" />
+</div>
+<script type="module">
+"use strict";
+
+function updateHistory(inp, out) {
+    const pre_inp = document.createElement("pre");
+    pre_inp.setAttribute("class", "language-text");
+    const code_inp = document.createElement("code");
+    code_inp.setAttribute("class", "language-text");
+    code_inp.append(`>>> ${inp}`);
+    pre_inp.append(code_inp);
+
+    const pre_out = document.createElement("pre");
+    pre_out.setAttribute("class", "result language-text");
+    const code_out = document.createElement("code");
+    code_out.setAttribute("class", "language-text");
+    code_out.append(`${out}`);
+    pre_out.append(code_out);
+
+    output.append(pre_inp);
+    output.append(pre_out);
+    output.scrollTop = output.scrollHeight;
+}
+
+async function sendRequest(env, exp) {
+    const params = env === null ? {exp} : {exp, env};
+    return fetch("/eval?" + new URLSearchParams(params));
+}
+
+let input = document.getElementById("input");
+const output = document.getElementById("output");
+const history = [];
+const button = document.getElementById("clear-local-storage");
+
+function renderHistory() {
+    for (const [inp, out] of history) {
+        updateHistory(inp, out);
+    }
+}
+
+function loadFromLocalStorage() {
+    history.length = 0;
+    output.innerHTML = "";
+    const hist = window.localStorage.getItem('history');
+    if (hist !== null) {
+        history.push(...JSON.parse(hist));
+    }
+    renderHistory();
+    document.env = window.localStorage.getItem('env');
+}
+
+function expandInput() {
+    if (input.tagName === "TEXTAREA") {
+        return;
+    }
+    const textarea = document.createElement("textarea");
+    textarea.setAttribute("id", "input");
+    textarea.setAttribute("rows", "1");
+    textarea.setAttribute("style", "height: 1.5em;");
+    textarea.value = input.value;
+    input.replaceWith(textarea);
+    input = textarea;
+    input.focus();
+    input.addEventListener("keyup", e => inputHandler(e));
+}
+
+function shrinkInput() {
+    if (input.tagName === "INPUT") {
+        return;
+    }
+    const newInput = document.createElement("input");
+    newInput.setAttribute("id", "input");
+    newInput.setAttribute("type", "text");
+    newInput.value = input.value;
+    input.replaceWith(newInput);
+    input = newInput;
+    input.focus();
+    input.addEventListener("keyup", e => inputHandler(e));
+}
+
+loadFromLocalStorage();
+input.focus();
+async function inputHandler(e) {
+    // TODO(max): Make up/down arrow keys navigate history
+    if (e.key === "Enter") {
+        if (e.shiftKey && input.tagName === "INPUT") {
+            // Shift-Enter expands the input to a textarea and does not submit
+            // input.
+            expandInput();
+            return;
+        }
+        if (!e.shiftKey && input.tagName === "TEXTAREA") {
+            // Normal Enter in a textarea should not submit input.
+            return;
+        }
+        const response = await sendRequest(document.env, input.value);
+        if (response.ok) {
+            const {env, result} = await response.json();
+            updateHistory(input.value, result);
+            history.push([input.value, result]);
+            input.value = "";
+            document.env = env;
+            window.localStorage.setItem('env', env)
+            window.localStorage.setItem('history', JSON.stringify(history));
+        } else {
+            const msg = await response.text();
+            updateHistory(input.value, msg);
+            input.value = "";
+        }
+        if (input.tagName === "TEXTAREA") {
+            shrinkInput();
+        }
+    }
+}
+input.addEventListener("keyup", e => inputHandler(e));
+button.addEventListener("click", () => {
+    window.localStorage.clear();
+    loadFromLocalStorage();
+    input.focus();
+});
+</script>
+</main>
+<script data-goatcounter="https://scrapscript.goatcounter.com/count"
+        async src="//gc.zgo.at/count.js"></script>
+</body>
+</html>

scrapscript.py

@@ -4,18 +4,21 @@
 import code
 import dataclasses
 import enum
+import http.server
 import json
 import logging
 import os
 import re
+import socketserver
 import sys
 import typing
 import unittest
 import urllib.request
+import urllib.parse
 from dataclasses import dataclass
 from enum import auto
 from types import FunctionType, ModuleType
-from typing import Any, Callable, Dict, Mapping, Optional, Union
+from typing import Any, Callable, Dict, Mapping, Optional, Tuple, Union
 
 readline: Optional[ModuleType]
 try:
@@ -1320,6 +1323,96 @@ def deserialize(msg: str) -> Object:
     return Object.deserialize(decoded)
 
 
+class ScrapMonad:
+    def __init__(self, env: Env) -> None:
+        assert isinstance(env, dict)  # for .copy()
+        self.env: Env = env.copy()
+
+    def bind(self, exp: Object) -> Tuple[Object, "ScrapMonad"]:
+        env = self.env
+        result = eval_exp(env, exp)
+        if isinstance(result, EnvObject):
+            return result, ScrapMonad({**env, **result.env})
+        return result, ScrapMonad({**env, "_": result})
+
+
+ASSET_DIR = os.path.dirname(__file__)
+
+
+class ScrapReplServer(http.server.SimpleHTTPRequestHandler):
+    def do_GET(self) -> None:
+        logger.debug("GET %s", self.path)
+        parsed_path = urllib.parse.urlsplit(self.path)
+        query = urllib.parse.parse_qs(parsed_path.query)
+        logging.debug("PATH %s", parsed_path)
+        logging.debug("QUERY %s", query)
+        if parsed_path.path == "/repl":
+            return self.do_repl()
+        if parsed_path.path == "/eval":
+            try:
+                return self.do_eval(query)
+            except Exception as e:
+                self.send_response(400)
+                self.send_header("Content-type", "text/plain")
+                self.end_headers()
+                self.wfile.write(str(e).encode("utf-8"))
+                return
+        if parsed_path.path == "/style.css":
+            self.send_response(200)
+            self.send_header("Content-type", "text/css")
+            self.end_headers()
+            with open(os.path.join(ASSET_DIR, "style.css"), "rb") as f:
+                self.wfile.write(f.read())
+            return
+        return self.do_404()
+
+    def do_repl(self) -> None:
+        self.send_response(200)
+        self.send_header("Content-type", "text/html")
+        self.end_headers()
+        with open(os.path.join(ASSET_DIR, "repl.html"), "rb") as f:
+            self.wfile.write(f.read())
+        return
+
+    def do_404(self) -> None:
+        self.send_response(404)
+        self.send_header("Content-type", "text/html")
+        self.end_headers()
+        self.wfile.write(b"""try hitting <a href="/repl">/repl</a>""")
+        return
+
+    def do_eval(self, query: Dict[str, Any]) -> None:
+        exp = query.get("exp")
+        if exp is None:
+            raise TypeError("Need expression to evaluate")
+        if len(exp) != 1:
+            raise TypeError("Need exactly one expression to evaluate")
+        exp = exp[0]
+        tokens = tokenize(exp)
+        ast = parse(tokens)
+        env = query.get("env")
+        if env is None:
+            env = STDLIB
+        else:
+            if len(env) != 1:
+                raise TypeError("Need exactly one env")
+            env_object = deserialize(env[0])
+            assert isinstance(env_object, EnvObject)
+            env = env_object.env
+        logging.debug("env is %s", env)
+        monad = ScrapMonad(env)
+        result, next_monad = monad.bind(ast)
+        serialized = EnvObject(next_monad.env).serialize()
+        encoded = bencode(serialized)
+        response = {"env": encoded.decode("utf-8"), "result": str(result)}
+        self.send_response(200)
+        self.send_header("Content-type", "text/plain")
+        self.send_header("Cache-Control", "max-age=3600")
+        self.end_headers()
+        self.wfile.write(json.dumps(response).encode("utf-8"))
+        return
+
+
 class TokenizerTests(unittest.TestCase):
     def test_tokenize_digit(self) -> None:
         self.assertEqual(tokenize("1"), [IntLit(1)])
@@ -3918,6 +4011,21 @@ def test_serialize_function(self) -> None:
         )
 
 
+class ScrapMonadTests(unittest.TestCase):
+    def test_create_copies_env(self) -> None:
+        env = {"a": Int(123)}
+        result = ScrapMonad(env)
+        self.assertEqual(result.env, env)
+        self.assertIsNot(result.env, env)
+
+    def test_bind_returns_new_monad(self) -> None:
+        env = {"a": Int(123)}
+        orig = ScrapMonad(env)
+        result, next_monad = orig.bind(Assign(Var("b"), Int(456)))
+        self.assertEqual(orig.env, {"a": Int(123)})
+        self.assertEqual(next_monad.env, {"a": Int(123), "b": Int(456)})
+
+
 class PrettyPrintTests(unittest.TestCase):
     def test_pretty_print_int(self) -> None:
         obj = Int(1)
@@ -4254,6 +4362,21 @@ def test_command(args: argparse.Namespace) -> None:
     unittest.main(argv=[__file__, *args.unittest_args])
 
 
+def serve_command(args: argparse.Namespace) -> None:
+    if args.debug:
+        logging.basicConfig(level=logging.DEBUG)
+    server: Union[type[socketserver.TCPServer], type[socketserver.ForkingTCPServer]]
+    if args.fork:
+        server = socketserver.ForkingTCPServer
+    else:
+        server = socketserver.TCPServer
+    server.allow_reuse_address = True
+    with server(("", args.port), ScrapReplServer) as httpd:
+        host, port = httpd.server_address
+        print(f"serving at http://{host!s}:{port}")
+        httpd.serve_forever()
+
+
 def main() -> None:
     parser = argparse.ArgumentParser(prog="scrapscript")
     subparsers = parser.add_subparsers(dest="command")
@@ -4277,6 +4400,12 @@ def main() -> None:
     apply.add_argument("program")
     apply.add_argument("--debug", action="store_true")
 
+    serve = subparsers.add_parser("serve")
+    serve.set_defaults(func=serve_command)
+    serve.add_argument("--port", type=int, default=8000)
+    serve.add_argument("--debug", action="store_true")
+    serve.add_argument("--fork", action="store_true")
+
     args = parser.parse_args()
     if not args.command:
         args.debug = False