Skip to content

Commit ecd644b

Browse files
whitphxwhitphx
authored
feature/original webserver replacing tornado (#492)
* [WIP] Create stlite_webserver package and its Server class with unit tests copied from tornado-e2e package * Use the new stlite server impl at ./py/stlite-webserver/stlite_webserver/__init__.py in worker.ts * Check the websocket endpoint path * Make stlite-web-server package and import its wheel from worker.ts like other packages * Rename stlite-webserver to stlite-server * Move the Server class to stlite_server.server * Copy make_url_path_regex from streamlit.web.server_util * Implement Server.receive_http() * Implement UploadFileRequestHandler * Add pytest, mypy and code formatters as dev dependencies * Apply formatters * Configure mypy to ignore pyodide * Fix some typing errors * Introduce Response class * Refactoring * Remove tornado * Replace test-tornado-e2e with test-stlite-server CI job * Add stlite-server-wheel Make rule * Remove unused imports * Fix typing * Fix test * Remove packages/kernel/py/e2etests * Revert kernel/package.json * Set necessary Streamlit config overrides * Create stlite_server.bootstrap.prepare() to run necessary patching and mocking that streamlit.web.bootstrap.run() does on the upstream Streamlit * Apply code formatter to bootstrap.py * Extend the test case accessing the media endpoint to cover the case where the file is registered without its file name * Create ComponentRequestHandler so custom components work * Use SNOWPARK_CONDA_BUILD flag to exclude unnecessary requirements for the Streamlit package * Update the desktop package to use stlite_server package and remove ssl packages * Call runtime.stop() from server.stop() * Update packages/kernel/py/README.md * Rename tests/test_httpserver.py -> tests/test_server.py * Fix * Add bootstrap_test.py * Fix .github/workflows/main.yml * Fix typing settings * Update packages/kernel/py/README.md * Fix worker.ts * Fix worker.ts * Fix worker.ts * Fix test_server.py * Remove unused patches * Remove Union * Fix typing * Rename logger object to match the original * Fix httputil.py * Fix typing * Fix typing on tests * Fix media_file_handler.py * Add matplotlib for tests * Fix media_file_handler.py * Fix server.py * Fix upload_file_request_handler.py
1 parent 961fc23 commit ecd644b

30 files changed

+3133
-2621
lines changed

‎.github/workflows/main.yml

+18-20
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,8 @@ jobs:
77
runs-on: ubuntu-latest
88
outputs:
99
kernel: ${{ steps.filter.outputs.kernel }}
10-
# tornado-e2e: ${{ steps.filter.outputs.tornado-e2e }}
11-
tornado-e2e: true # This step does not detect changes in the `streamlit` submodule that is needed to trigger the test-tornado-e2e job (https://github.com/dorny/paths-filter/issues/143), so skip checking and make it always return true as a workaround.
10+
# stlite-server: ${{ steps.filter.outputs.stlite-server }}
11+
stlite-server: true # This step does not detect changes in the `streamlit` submodule that is needed to trigger the test-stlite-server job (https://github.com/dorny/paths-filter/issues/143), so skip checking and make it always return true as a workaround.
1212
mountable: ${{ steps.filter.outputs.mountable }}
1313
sharing-editor: ${{ steps.filter.outputs.sharing-editor }}
1414
sharing-common: ${{ steps.filter.outputs.sharing-common }}
@@ -22,9 +22,8 @@ jobs:
2222
kernel:
2323
- 'packages/kernel/**/*'
2424
# - '!packages/kernel/py/**/*' # Not supported by paths-filter now: https://github.com/dorny/paths-filter/issues/106
25-
# tornado-e2e: # We run this job anytime. See above.
26-
# - 'packages/kernel/py/tornado'
27-
# - 'packages/kernel/py/e2etests/**/*'
25+
# stlite-server: # We run this job anytime. See above.
26+
# - 'packages/kernel/py/stlite-server/**/*'
2827
# - 'streamlit/**/*'
2928
mountable:
3029
- 'packages/mountable/**/*'
@@ -60,15 +59,14 @@ jobs:
6059
yarn check:prettier
6160
- run: yarn test
6261

63-
test-tornado-e2e:
62+
test-stlite-server:
6463
needs: changes
65-
if: ${{ needs.changes.outputs.tornado-e2e == 'true' }}
64+
if: ${{ needs.changes.outputs.stlite-server == 'true' }}
65+
66+
env:
67+
python-version: "3.10"
6668

6769
runs-on: ubuntu-latest
68-
strategy:
69-
fail-fast: false
70-
matrix:
71-
python-version: ["3.10"]
7270

7371
steps:
7472
- uses: actions/checkout@v3
@@ -112,7 +110,7 @@ jobs:
112110
shell: bash
113111
run: |
114112
. .venv/bin/activate
115-
cd packages/kernel/py/e2etests
113+
cd packages/kernel/py/stlite-server
116114
poetry install
117115
118116
# Set up the /streamlit submodule
@@ -128,22 +126,22 @@ jobs:
128126
- name: Run linter and code formatter to the test code module
129127
run: |
130128
. .venv/bin/activate
131-
cd packages/kernel/py/e2etests
129+
cd packages/kernel/py/stlite-server
132130
poetry run black . --check
133131
poetry run isort . --check
134132
poetry run flake8
135133
136134
- name: Run mypy
137135
run: |
138136
. .venv/bin/activate
139-
cd packages/kernel/py/e2etests
137+
cd packages/kernel/py/stlite-server
140138
poetry run mypy .
141139
142140
- name: Run pytest
143141
shell: bash
144142
run: |
145143
. .venv/bin/activate
146-
cd packages/kernel/py/e2etests
144+
cd packages/kernel/py/stlite-server
147145
poetry run python -m pytest -v tests
148146
149147
test-mountable:
@@ -257,7 +255,7 @@ jobs:
257255

258256
build-kernel:
259257
if: ${{ ! failure() }} # This job should run even if the depending jobs are skipped, but not when those jobs failed: https://qiita.com/abetomo/items/d9ede7dbeeb24f723fc5#%E8%A8%AD%E5%AE%9A%E4%BE%8B4
260-
needs: [test-kernel, test-tornado-e2e]
258+
needs: [test-kernel, test-stlite-server]
261259

262260
env:
263261
python-version: "3.10"
@@ -378,7 +376,7 @@ jobs:
378376

379377
build-mountable:
380378
if: ${{ ! failure() }} # This job should run even if the depending jobs are skipped, but not when those jobs failed: https://qiita.com/abetomo/items/d9ede7dbeeb24f723fc5#%E8%A8%AD%E5%AE%9A%E4%BE%8B4
381-
needs: [test-kernel, test-tornado-e2e, test-mountable]
379+
needs: [test-kernel, test-stlite-server, test-mountable]
382380

383381
env:
384382
python-version: "3.10"
@@ -499,7 +497,7 @@ jobs:
499497

500498
build-sharing:
501499
if: ${{ ! failure() }} # This job should run even if the depending jobs are skipped, but not when those jobs failed: https://qiita.com/abetomo/items/d9ede7dbeeb24f723fc5#%E8%A8%AD%E5%AE%9A%E4%BE%8B4
502-
needs: [test-kernel, test-tornado-e2e, test-sharing-common]
500+
needs: [test-kernel, test-stlite-server, test-sharing-common]
503501

504502
env:
505503
python-version: "3.10"
@@ -605,7 +603,7 @@ jobs:
605603

606604
build-sharing-editor:
607605
if: ${{ ! failure() }} # This job should run even if the depending jobs are skipped, but not when those jobs failed: https://qiita.com/abetomo/items/d9ede7dbeeb24f723fc5#%E8%A8%AD%E5%AE%9A%E4%BE%8B4
608-
needs: [test-kernel, test-tornado-e2e, test-sharing-editor, test-sharing-common, deploy-sharing]
606+
needs: [test-kernel, test-stlite-server, test-sharing-editor, test-sharing-common, deploy-sharing]
609607

610608
runs-on: ubuntu-latest
611609

@@ -665,7 +663,7 @@ jobs:
665663

666664
build-desktop:
667665
if: ${{ ! failure() }} # This job should run even if the depending jobs are skipped, but not when those jobs failed: https://qiita.com/abetomo/items/d9ede7dbeeb24f723fc5#%E8%A8%AD%E5%AE%9A%E4%BE%8B4
668-
needs: [test-kernel, test-tornado-e2e, test-desktop]
666+
needs: [test-kernel, test-stlite-server, test-desktop]
669667

670668
env:
671669
python-version: "3.10"

‎.gitmodules

-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,3 @@
11
[submodule "streamlit"]
22
path = streamlit
33
url = https://github.com/whitphx/streamlit.git
4-
[submodule "packages/kernel/py/tornado"]
5-
path = packages/kernel/py/tornado
6-
url = https://github.com/whitphx/tornado.git

‎Makefile

+7-7
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sharing-editor := packages/sharing-editor/build/*
77
desktop := packages/desktop/build/*
88
kernel := packages/kernel/dist/*
99
pyarrow_wheel := packages/kernel/py/stlite-pyarrow/dist/stlite_pyarrow-0.1.0-py3-none-any.whl
10-
tornado_wheel := packages/kernel/py/tornado/dist/tornado-6.2-py3-none-any.whl
10+
stlite-server-wheel := packages/kernel/py/stlite-server/dist/stlite_server-0.1.0-py3-none-any.whl
1111
streamlit_proto := streamlit/frontend/src/autogen
1212
streamlit_wheel := packages/kernel/py/streamlit/lib/dist/streamlit-1.18.1-py2.py3-none-any.whl
1313

@@ -95,7 +95,7 @@ $(desktop): packages/desktop/src/*.ts packages/desktop/src/*.tsx packages/deskto
9595

9696
.PHONY: kernel
9797
kernel: $(kernel)
98-
$(kernel): packages/kernel/src/*.ts $(pyarrow_wheel) $(tornado_wheel) $(streamlit_wheel) $(streamlit_proto)
98+
$(kernel): packages/kernel/src/*.ts $(pyarrow_wheel) $(stlite-server-wheel) $(streamlit_wheel) $(streamlit_proto)
9999
cd packages/kernel; \
100100
yarn build
101101
@touch $@
@@ -105,11 +105,11 @@ $(pyarrow_wheel): $(VENV) packages/kernel/py/stlite-pyarrow/pyarrow/*.py
105105
cd packages/kernel/py/stlite-pyarrow && \
106106
poetry build
107107

108-
$(tornado_wheel): $(VENV) packages/kernel/py/tornado/tornado/*.py
108+
.PHONY: stlite-server-wheel
109+
$(stlite-server-wheel): $(VENV) packages/kernel/py/stlite-server/stlite_server/*.py
109110
. $(VENV)/bin/activate && \
110-
cd packages/kernel/py/tornado && \
111-
pip install -U build && \
112-
TORNADO_EXTENSION=0 python -m build --wheel
111+
cd packages/kernel/py/stlite-server && \
112+
poetry build
113113
@touch $@
114114

115115
$(streamlit_proto): $(VENV) streamlit/proto/streamlit/proto/*.proto
@@ -123,6 +123,6 @@ streamlit-wheel: $(streamlit_wheel)
123123
$(streamlit_wheel): $(VENV) $(streamlit_proto) streamlit/lib/streamlit/**/*.py streamlit/lib/Pipfile streamlit/lib/setup.py streamlit/lib/bin/* streamlit/lib/MANIFEST.in
124124
. $(VENV)/bin/activate && \
125125
cd streamlit && \
126-
$(MAKE) distribution
126+
SNOWPARK_CONDA_BUILD=true $(MAKE) distribution
127127
mkdir -p `dirname $(streamlit_wheel)`
128128
cp streamlit/lib/dist/streamlit-1.18.1-py2.py3-none-any.whl $(streamlit_wheel)

‎packages/desktop/bin/dump_artifacts.ts

+4-1
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,10 @@ async function createSitePackagesSnapshot(
8787
// TODO: Set the wheel file names dynamically
8888
await installLocalWheel(
8989
pyodide,
90-
path.join(stliteKernelPyDir, "tornado/dist/tornado-6.2-py3-none-any.whl")
90+
path.join(
91+
stliteKernelPyDir,
92+
"stlite-server/dist/stlite_server-0.1.0-py3-none-any.whl"
93+
)
9194
);
9295
await installLocalWheel(
9396
pyodide,

‎packages/desktop/package.json

+1-1
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919
"eject": "craco eject",
2020
"start:electron": "tsc -p electron -w",
2121
"build:electron": "tsc -p electron",
22-
"build:pyodide": "curl -L https://github.com/pyodide/pyodide/releases/download/0.22.1/pyodide-0.22.1.tar.bz2 | tar xj -C ./build --files-from=./pyodide-files.txt",
22+
"build:pyodide": "curl -L https://github.com/pyodide/pyodide/releases/download/0.22.1/pyodide-core-0.22.1.tar.bz2 | tar xj -C ./build --files-from=./pyodide-files.txt",
2323
"build:bin": "./scripts/build_bin.js && sed -i'' -e '1 s/^#!.*$/#!\\/usr\\/bin\\/env node/' ./bin/*.js",
2424
"typecheck": "yarn tsc --noEmit -p electron",
2525
"start": "concurrently \"cross-env BROWSER=none yarn start:web\" \"wait-on http://localhost:3000 && yarn start:electron\" \"wait-on http://localhost:3000 && tsc -p electron && cross-env NODE_ENV=\"development\" electron .\"",

‎packages/desktop/pyodide-files.txt

-2
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,6 @@
1-
pyodide/openssl-1.1.1n.zip
21
pyodide/pyodide.asm.data
32
pyodide/pyodide.asm.js
43
pyodide/pyodide.asm.wasm
54
pyodide/pyodide.js
65
pyodide/pyodide_py.tar
76
pyodide/repodata.json
8-
pyodide/ssl-1.0.0.zip

‎packages/kernel/py/README.md

+12-7
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,15 @@
1-
## `tornado`
1+
## `stlite-server`
22

3-
A customized version of `tornado`.
3+
Streamlit has 2 separate submodules, `web` and `runtime`.
4+
We use `runtime` only, which contains all necessary core features of Streamlit, but we omit `web` because it relies on `tornado`, which cannot be installed on Pyodide, and even if we hack it to be installable, we don't need its full HTTP-compatible implementation. In contrast, we only need a small subset (or imitation) of HTTP for our original communication layer used in `worker.ts`, and we prioritize a smaller bundle size.
45

5-
The original Tornado cannot be installed and run on Pyodide environment and stlite kernel needs a customized communication protocol, so we forked and customized it.
6+
For that purpose, we created this package, `stlite-server`, including `server.Server` class.
7+
It exposes methods for the WebSocket- and HTTP-like connections that are handled in `worker.ts`, and it also implements request handlers similar to the ones defined in the original server implementation, `streamlit.web.server.server.Server`.
68

7-
### Implementation principles
8-
* Simplicity and readability is the top priority.
9-
* Neither reproducing the original behavior nor implementing the correct HTTP protocol is needed.
10-
* We do not pay efforts to keep the original code. We do it if it supports reading and implementing the code, and we don't if it does not.
9+
See the following links for the details.
10+
* A PR where this package is created.
11+
* https://github.com/whitphx/stlite/pull/492
12+
* PRs for the upstream Streamlit repo where the `web` and `runtime` sub-packages are decoupled.
13+
* https://github.com/streamlit/streamlit/pull/4956
14+
* https://github.com/streamlit/streamlit/pull/5072
15+
* https://github.com/streamlit/streamlit/pull/5136

0 commit comments

Comments
 (0)