From d409f13ef210cdc438242218bb59af1cfb565fde Mon Sep 17 00:00:00 2001 From: Max Bernstein Date: Tue, 28 Nov 2023 18:21:34 -0500 Subject: [PATCH] Add a REPL webserver This adds a server that acts as a function (exp, env) -> (result, env') meaning the server has to hold no state. The client maintains and transmits the state. This has some bugs right now, such as not being able to serialize recursive functions. Those will be fixed soon. --- This commit was comprised of the following commits: Add initial ScrapMonad Add a notion of relocations We'll need to link native functions, for example, if we take in a serialized environment from the outside. Add small server with JSON API and little HTML page Provide a networked REPL, perhaps as a demo. Add bdecode This is a little more stateful than bencode so add a class and call into the class from a `bdecode` function. Begin deserialization Use some Python nonsense to automatically add deserializer functions to a dict if Object subclasses have them. Add some more deserializers, including first recursive one This is a little painful and I wonder if there is a better way to do it. Some niceties Add deserialization for EnvObject Deserialize env and wrap it in EnvObject We need to pass around entire scrap objects. Persist env in client Serialize/deserialize functions, closures Next up: NativeFunctionRelocation. Type ScrapMonad.__init__ Drop the _ It's cleaner. Fix infinite recursion bug All Objects have deserialize (defined or inheritecd), but all of them *define* it. Check if they define it by looking in `__dict__`. Add support for deserializing NativeFunctionRelocation With this, the web REPL can ship an env back and forth to the idempotent eval server! Fix Python 3.8 Unwrap classmethod and store inner function. Kind of surprised mypy didn't catch this, since I was never storing functions before--always classmethod. Persist env in LocalStorage Persist history in LocalStorage Focus input field on page load Add TODO Throw some errors that do not normally raise Don't double fetch Add title and note Remove begin/end newlines Bound output window height and scroll to bottom Add doctype Embed empty favicon in page Save a request that will always 404. Add a little favicon Thanks, antifavicon. Set 1 hour cache control on eval requests As the server is stateless, we can have some intermediate server cache eval(env, exp) requests. If everyone loads the page and types `123`, for example, we only need receive that request once (per hour). It's not clear if fly.io (for example) does this middle cache thing automatically. Try building image in CI (#19) * Try building image in CI * wip * Add quick check post-build * Don't be interactive * Use tags from meta step to identify image Oops, remove target branch Remove excess env Scroll to bottom after loading history Add space in favicon Add button to clear LocalStorage Add TODO Add multiprocessing server Serialize/deserialize Apply Check in fly.toml Make web repl return result, not _ Report eval errors somewhat opaquely in console Add updateHistory function Use try/catch in client Display server error message in output Move all explicit exception handling into do_index Link to repl page Add charset and viewport Allow reuse of address This means we can C-c the server and re-start it on the same port immediately. Check in scrapscript.org style Adjust color and use
for padding Add fonts Render REPL with code/result a little better Bundle style.css with Docker image Fix input background Use prompt as label Remove TODO Don't highlight input box Deserialize bool and record Move html to separate file Expand input into textarea with Shift-Enter Add goatcounter Remove console.log Color textareas too Make the webserver fork Bring Chris's code up to date Print host in debug output --- Dockerfile | 2 + build-com | 5 +- fly.toml | 25 +++++ repl.html | 166 +++++++++++++++++++++++++++++ scrapscript.py | 131 ++++++++++++++++++++++- style.css | 284 +++++++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 611 insertions(+), 2 deletions(-) create mode 100644 fly.toml create mode 100644 repl.html create mode 100644 style.css diff --git a/Dockerfile b/Dockerfile index 90df097d..b44e9f5a 100644 --- a/Dockerfile +++ b/Dockerfile @@ -12,6 +12,8 @@ RUN bin/ape.elf bin/python -m compileall Lib RUN mv Lib/__pycache__/scrapscript*.pyc Lib/scrapscript.pyc RUN rm Lib/scrapscript.py RUN cp bin/python bin/scrapscript.com +COPY style.css Lib +COPY repl.html Lib RUN sh bin/zip -A -r bin/scrapscript.com Lib .args RUN bin/ape.elf bin/assimilate bin/scrapscript.com diff --git a/build-com b/build-com index 064b4547..be879fd5 100755 --- a/build-com +++ b/build-com @@ -4,6 +4,8 @@ PREV="$(pwd)" DIR="$(mktemp -d)" cp scrapscript.py "$DIR" cp .args "$DIR" +cp repl.html "$DIR" +cp style.css "$DIR" cd "$DIR" wget https://cosmo.zip/pub/cosmos/bin/python wget https://cosmo.zip/pub/cosmos/bin/zip @@ -11,7 +13,8 @@ chmod +x python chmod +x zip ./python -m compileall scrapscript.py mkdir Lib -cp __pycache__/scrapscript.cpython-311.pyc Lib/scrapscript.pyc +cp __pycache__/scrapscript.*.pyc Lib/scrapscript.pyc +cp style.css repl.html Lib cp python scrapscript.com ./zip -r scrapscript.com Lib .args echo "Testing..." diff --git a/fly.toml b/fly.toml new file mode 100644 index 00000000..0943ae52 --- /dev/null +++ b/fly.toml @@ -0,0 +1,25 @@ +# fly.toml app configuration file generated for scrapscript on 2023-11-29T02:06:16-05:00 +# +# See https://fly.io/docs/reference/configuration/ for information about how to use this file. +# + +app = "scrapscript" +primary_region = "ewr" + +[build] + +[http_service] + internal_port = 8000 + force_https = true + auto_stop_machines = true + auto_start_machines = true + min_machines_running = 0 + processes = ["app"] + +[[vm]] + cpu_kind = "shared" + cpus = 1 + memory_mb = 256 + +[experimental] +cmd = ["serve", "--port", "8000", "--debug", "--fork"] diff --git a/repl.html b/repl.html new file mode 100644 index 00000000..e9066b81 --- /dev/null +++ b/repl.html @@ -0,0 +1,166 @@ + + + + + +Scrapscript Web REPL + + + + + + + + + +
+
+

See scrapscript.org for a slightly +out of date language reference.

+

This REPL sends programs to a server to be evaluated, but the server is +completely stateless. All persistence is in the browser.

+
+
+ + +
+
+Output: +
+
+>>> +
+ +
+ + + diff --git a/scrapscript.py b/scrapscript.py index 101e2320..2e396989 100755 --- a/scrapscript.py +++ b/scrapscript.py @@ -4,18 +4,21 @@ import code import dataclasses import enum +import http.server import json import logging import os import re +import socketserver import sys import typing import unittest import urllib.request +import urllib.parse from dataclasses import dataclass from enum import auto from types import FunctionType, ModuleType -from typing import Any, Callable, Dict, Mapping, Optional, Union +from typing import Any, Callable, Dict, Mapping, Optional, Tuple, Union readline: Optional[ModuleType] try: @@ -1320,6 +1323,96 @@ def deserialize(msg: str) -> Object: return Object.deserialize(decoded) +class ScrapMonad: + def __init__(self, env: Env) -> None: + assert isinstance(env, dict) # for .copy() + self.env: Env = env.copy() + + def bind(self, exp: Object) -> Tuple[Object, "ScrapMonad"]: + env = self.env + result = eval_exp(env, exp) + if isinstance(result, EnvObject): + return result, ScrapMonad({**env, **result.env}) + return result, ScrapMonad({**env, "_": result}) + + +ASSET_DIR = os.path.dirname(__file__) + + +class ScrapReplServer(http.server.SimpleHTTPRequestHandler): + def do_GET(self) -> None: + logger.debug("GET %s", self.path) + parsed_path = urllib.parse.urlsplit(self.path) + query = urllib.parse.parse_qs(parsed_path.query) + logging.debug("PATH %s", parsed_path) + logging.debug("QUERY %s", query) + if parsed_path.path == "/repl": + return self.do_repl() + if parsed_path.path == "/eval": + try: + return self.do_eval(query) + except Exception as e: + self.send_response(400) + self.send_header("Content-type", "text/plain") + self.end_headers() + self.wfile.write(str(e).encode("utf-8")) + return + if parsed_path.path == "/style.css": + self.send_response(200) + self.send_header("Content-type", "text/css") + self.end_headers() + with open(os.path.join(ASSET_DIR, "style.css"), "rb") as f: + self.wfile.write(f.read()) + return + return self.do_404() + + def do_repl(self) -> None: + self.send_response(200) + self.send_header("Content-type", "text/html") + self.end_headers() + with open(os.path.join(ASSET_DIR, "repl.html"), "rb") as f: + self.wfile.write(f.read()) + return + + def do_404(self) -> None: + self.send_response(404) + self.send_header("Content-type", "text/html") + self.end_headers() + self.wfile.write(b"""try hitting /repl""") + return + + def do_eval(self, query: Dict[str, Any]) -> None: + exp = query.get("exp") + if exp is None: + raise TypeError("Need expression to evaluate") + if len(exp) != 1: + raise TypeError("Need exactly one expression to evaluate") + exp = exp[0] + tokens = tokenize(exp) + ast = parse(tokens) + env = query.get("env") + if env is None: + env = STDLIB + else: + if len(env) != 1: + raise TypeError("Need exactly one env") + env_object = deserialize(env[0]) + assert isinstance(env_object, EnvObject) + env = env_object.env + logging.debug("env is %s", env) + monad = ScrapMonad(env) + result, next_monad = monad.bind(ast) + serialized = EnvObject(next_monad.env).serialize() + encoded = bencode(serialized) + response = {"env": encoded.decode("utf-8"), "result": str(result)} + self.send_response(200) + self.send_header("Content-type", "text/plain") + self.send_header("Cache-Control", "max-age=3600") + self.end_headers() + self.wfile.write(json.dumps(response).encode("utf-8")) + return + + class TokenizerTests(unittest.TestCase): def test_tokenize_digit(self) -> None: self.assertEqual(tokenize("1"), [IntLit(1)]) @@ -3918,6 +4011,21 @@ def test_serialize_function(self) -> None: ) +class ScrapMonadTests(unittest.TestCase): + def test_create_copies_env(self) -> None: + env = {"a": Int(123)} + result = ScrapMonad(env) + self.assertEqual(result.env, env) + self.assertIsNot(result.env, env) + + def test_bind_returns_new_monad(self) -> None: + env = {"a": Int(123)} + orig = ScrapMonad(env) + result, next_monad = orig.bind(Assign(Var("b"), Int(456))) + self.assertEqual(orig.env, {"a": Int(123)}) + self.assertEqual(next_monad.env, {"a": Int(123), "b": Int(456)}) + + class PrettyPrintTests(unittest.TestCase): def test_pretty_print_int(self) -> None: obj = Int(1) @@ -4254,6 +4362,21 @@ def test_command(args: argparse.Namespace) -> None: unittest.main(argv=[__file__, *args.unittest_args]) +def serve_command(args: argparse.Namespace) -> None: + if args.debug: + logging.basicConfig(level=logging.DEBUG) + server: Union[type[socketserver.TCPServer], type[socketserver.ForkingTCPServer]] + if args.fork: + server = socketserver.ForkingTCPServer + else: + server = socketserver.TCPServer + server.allow_reuse_address = True + with server(("", args.port), ScrapReplServer) as httpd: + host, port = httpd.server_address + print(f"serving at http://{host!s}:{port}") + httpd.serve_forever() + + def main() -> None: parser = argparse.ArgumentParser(prog="scrapscript") subparsers = parser.add_subparsers(dest="command") @@ -4277,6 +4400,12 @@ def main() -> None: apply.add_argument("program") apply.add_argument("--debug", action="store_true") + serve = subparsers.add_parser("serve") + serve.set_defaults(func=serve_command) + serve.add_argument("--port", type=int, default=8000) + serve.add_argument("--debug", action="store_true") + serve.add_argument("--fork", action="store_true") + args = parser.parse_args() if not args.command: args.debug = False diff --git a/style.css b/style.css new file mode 100644 index 00000000..ef1f33be --- /dev/null +++ b/style.css @@ -0,0 +1,284 @@ +html { + font-size: 100%; + background-color: black; +} +body { + font-family: "Nunito Sans", sans-serif; + font-weight: 400; + line-height: 1.75; + margin: 0; + max-width: 55rem; +} +p { + margin-bottom: 1rem; +} +div { + color: #d4d4d4; +} + +a:link { + text-decoration: none; + text-decoration-thickness: 1px; + text-underline-offset: 0.15rem; + text-decoration-style: dotted; +} +a { + color: #7ec699; +} + +ul, +ol { + padding-left: 1.5rem; +} + +blockquote { + color: #d4d4d4; + border-left: 2px dotted #666; + margin-left: 0.5rem; + padding-left: 1.5rem; + margin-bottom: 2rem; +} + +h1, +h2, +h3, +h4, +h5 { + margin: 3rem 0 1.38rem; + font-family: "Rubik", sans-serif; + font-weight: 500; + line-height: 1.3; + text-transform: capitalize; +} +h1 { + font-weight: 700; + text-transform: uppercase; + margin-top: 0; + letter-spacing: 1px; +} +h2 { + margin-top: 5rem; + padding-top: 2rem; + border-top: 2px dotted #666; +} +h3 { + margin-top: 3rem; +} + +h1 { + font-size: 4.209rem; +} +h2 { + font-size: 3.157rem; +} +h3 { + font-size: 2.369rem; +} +h4 { + font-size: 1.777rem; +} +h5 { + font-size: 1.333rem; +} +small, +.text_small { + font-size: 0.75rem; +} + +header > div, +footer > div { + padding: 2rem 2rem; + display: flex; + align-items: center; + gap: 1rem; + text-transform: uppercase; + font-weight: 600; + line-height: 2; + font-size: 0.875rem; + max-width: 35rem; + margin: 0 auto; +} +header a, +footer a { + color: #cc99cd; + text-decoration: none; + max-height: 1.75rem; +} +header img, +footer img { + height: 1.75rem; + -webkit-filter: invert(1); + filter: invert(1); + padding-right: 0.5rem; +} +footer { + margin-bottom: 6rem; +} + +main { + max-width: 35rem; + margin: 0 auto; + background-color: black; +} + +a:hover { + opacity: 0.8; +} + +pre { + overflow-x: scroll; +} +code, +pre, +code[class*="language-"], +pre[class*="language-"] { + font-family: "Fira Code", Monaco, Menlo, Consolas, "Courier New", monospace; + position: relative; + font-size: 0.8rem; +} +pre[class*="language-"] { + border-radius: 5px; + padding: 0.5em 0.5em; + margin: 0em; + z-index: 2; +} +code, +pre > code { + font-size: 0.8rem; + line-height: 1.42; + -webkit-text-size-adjust: 100%; +} +pre > code { + font-size: 0.8rem; + margin-left: 2.5%; + display: block; +} + +:not(pre) > code[class*="language-"], pre[class*="language-"] { + background: #2d2d2d; +} +pre[class*="language-"] + pre[class*="language-"].result { + background-color: #202020; + position: relative; + opacity: 0.9; +} + +pre[class*="language-"].result { + z-index: 1; +} +input[type="text"], textarea { + background-color: #2d2d2d; + outline: none; + color: #ccc; +} + +/* +main { + margin: 2rem; + max-width: 650px; +} +img { + max-width: 600px; + width: 95vw; +} +ul { + list-style-type: circle; + padding-left: 20px; +} + +p, dl, ol, ul { + line-height: 1.8rem; + max-width: 50ch; + color: #CCC; +} + +html { + background-color: #151515; + color: #eee; +} +a:link { + text-decoration: none; + text-decoration-thickness: 1px; + text-underline-offset: 0.15rem; + text-decoration-style: dotted; +} +a { + color: #A0D2A2; +} +a:visited { +} +a:active { +} + +h1 { + font-weight: 900; + font-size: 3rem; + letter-spacing: 0.1rem; + text-transform: uppercase; +} +h1 a, h1 a:visited, h1 a:active { + color: white; +} +h2 { + font-size: 1.8rem; + margin-top: 4.5rem; + padding-top: 2rem; + border-top: 1px solid #666; +} +h3 { + font-size: 1.4rem; + margin-top: 3rem; + font-weight: 600; +} + +@media (min-width: 760px) { + main { + margin: 3rem auto; + } + h1, h2 { + margin-left: -1rem; + } +} + +h1 { + display: flex; + flex-wrap: wrap; + gap: 1rem; + align-items: center; + justify-items: left; + justify-content: left; + line-height: 1.5; +} +h1 a { + line-height: 0; +} +h1 img { + width: 2.5rem; + height: 2.5rem; + -webkit-filter: invert(1); + filter: invert(1); +} +#coming { + font-weight: 500; + font-size: 0.5rem; + padding: 0.5rem; + border: 1px solid white; + border-radius: 5px; + display: inline-block; + order: 1; +} +@media (max-width: 505px) { + #title { + display: none; + } +} +@media (max-width: 760px) { + h1 { + font-size: 2rem; + } +} + +#table-of-contents, h2, h3 { + text-transform: capitalize; +} +*/