dev/upgrade streamlit 1.27.0 (#671)

* Copy .nvmrc from [email protected]

* Update the streamlit submodule, fix some package resolutions for it, and fix @stlite/kernel for streamlit 1.27.0

* Update @stlite/common-react and mountable

* Set webpack version in @stlite/mountable as same as [email protected]

* Set the resolved webpack version to avoid an error "The 'compilation' argument must be an instance of Compilation", which is caused by multiple versions of webpack. Ref: https://qiita.com/Hanpen3019/items/e6ae5d29b334bc3d7af3

* Update @stlire/sharing

* Update @stlite/desktop

* Update Makefile

* Remove blob-polyfill

* Update Makefile

* Update the streamlit submodule

* Fix .github/workflows/main.yml

* Add streamlit/frontend as a workspace

* Add a make rule for the streamlit frontend lib

* Update wheel versions

* Update the streamlit submodule

* Set the immutable's version as same as the locked version of the upstream Streamlit frontend to avoid type errors

* Add stlite modifications (#672)

* Add stlite modifications

* Add back connection manager

* Move methods

* Some updates

* Remove header

* More cleanup

* More changes

* More cleanup

* More modifications

* Don't update pyiodide

* More cleanup

* Don't Update pyodide

* Don't update pyiodide

* More cleanup

* More cleanup

* Rename of option

* Remove new line

* Minor cleanup

* Fix wrong option name

* Minor renaming

* Fix typo

Co-authored-by: Yuichiro Tachibana (Tsuchiya) <[email protected]>

* Update comment

Co-authored-by: Yuichiro Tachibana (Tsuchiya) <[email protected]>

* Remove paths

* Change to put

* Update URL

* Add gatherUsageStats flag

---------

Co-authored-by: Yuichiro Tachibana (Tsuchiya) <[email protected]>

* Update the streamlit submodule

* Add the PUT type to the HttpRequest interface

* Fix the imports in ConnectionManager.ts

* Fix upload_file_request_handler.py to refer to `kwargs` of the handler method

* Fix stlite_server/upload_file_request_handler.py to catch up with the upstream implementation

* Remove `form-data-encoder` from `@stlite/kernel` as it's moved to `@streamlit/lib` in whitphx/streamlit#7

* Fix the mountable option parser to distinguish the toast callback options from the kernel options

* Remove ctx.disableToast() and ctx.enableToast() which were introduced in #636 because the `disableProgressToasts` and `disableErrorToasts` options play the same role

* Update CHANGELOG.md

* Apply Prettier to kernel

* Fix stlite-server tests

* Fix the CI workflows

* Fix format and types in stlite-server

* Update main.yml fixing the build jobs to initialize with venv

* Add the streamlit lib package to the deps of the kernel make rule

* Remove the TS dependency from the kernel module to the build artifact of `@streamlit/lib`

* Fix main.yml

* Fix .github/workflows/gh-pages.yml

* Update the streamlit submodule

* Fix the streamlit-proto make rule to call the python-init-dev-only rule

* Fix the venv make rule to execute it every time

* Update DEVELOPMENT.md

* Fix desktop/bin/dump_artifacts.ts not to pin the altair version

* Fix CI files

* Fix tsconfig

* Refresh package.json and yarn.lock

* Fix stlite_server/server.py making it look more similar to the original server.py

* Fix dump_artifacts.ts

* Add a test for the file delete endpoint

---------

Co-authored-by: Lukas Masuch <[email protected]>

Author: whitphx

.github/workflows/gh-pages.yml

@@ -13,7 +13,7 @@ concurrency:
 
 env:
   GITHUB_PAGES_SUBDIR_MOUNTABLE: lib/mountable
-  USE_CONSTRAINT_FILE: false  # https://github.com/streamlit/streamlit/blob/1.23.1/.github/workflows/release.yml#L61-L62
+  USE_CONSTRAINTS_FILE: false  # https://github.com/streamlit/streamlit/blob/1.27.0/.github/workflows/release.yml#L67-L68
 
 jobs:
   build:
@@ -33,6 +33,8 @@ jobs:
 
     steps:
     - uses: actions/checkout@v3
+      with:
+        submodules: true
 
     ## Set up Python and Poetry environment
     - name: Set up Python ${{ env.python-version }}
@@ -53,7 +55,7 @@ jobs:
       run: echo "$HOME/.local/bin" >> $GITHUB_PATH
 
     - name: Configure poetry
-      run: poetry config virtualenvs.in-project true
+      run: poetry config virtualenvs.in-project false
 
     - name: Set up cache
       uses: actions/cache@v3
@@ -66,11 +68,15 @@ jobs:
       if: steps.cache.outputs.cache-hit == 'true'
       run: timeout 10s pip --version || rm -rf .venv
 
+    - name: Create virtualenv
+      run: python -m venv .venv
+
     ## Set up Node environment
     - uses: actions/setup-node@v3
       with:
         node-version-file: .nvmrc
         cache: 'yarn'
+    - run: yarn install --frozen-lockfile
 
     # We require protoc >= 3.20, but Ubuntu 22.04 - the OS that these Github
     # Actions are running as of 2023.05.03 - doesn't have recent versions
@@ -96,7 +102,9 @@ jobs:
       run: echo "PUBLIC_URL=." >> $GITHUB_ENV
     - if: matrix.target == 'mountable'
       name: Build @stlite/mountable
-      run: make mountable
+      run: |
+        . .venv/bin/activate
+        make mountable
 
     - if: matrix.target == 'mountable'
       name: Upload the built files as an artifact

.github/workflows/main.yml

@@ -15,7 +15,7 @@ env:
   # so we set the max memory size as 6.5 GiB like https://nodejs.org/api/cli.html#--max-old-space-sizesize-in-megabytes
   NODE_OPTIONS: "--max-old-space-size=6656"
   RUN_ALL_TESTS: ${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v') }}
-  USE_CONSTRAINT_FILE: false  # https://github.com/streamlit/streamlit/blob/1.23.1/.github/workflows/release.yml#L61-L62
+  USE_CONSTRAINTS_FILE: false  # https://github.com/streamlit/streamlit/blob/1.27.0/.github/workflows/release.yml#L67-L68
 
 jobs:
   changes:  # See https://github.com/dorny/paths-filter#examples
@@ -128,6 +128,13 @@ jobs:
         cd packages/kernel/py/stlite-server
         poetry install
 
+    # Node packages are needed to build proto
+    - uses: actions/setup-node@v3
+      with:
+        node-version-file: .nvmrc
+        cache: 'yarn'
+    - run: yarn install --frozen-lockfile
+
     # We require protoc >= 3.20, but Ubuntu 22.04 - the OS that these Github
     # Actions are running as of 2023.05.03 - doesn't have recent versions
     # of protoc in its package repository.
@@ -144,8 +151,7 @@ jobs:
     - name: Set up Streamlit and build proto
       run: |
         . .venv/bin/activate
-        cd streamlit
-        make mini-init
+        make streamlit-proto
 
     - name: Run linter and code formatter to the test code module
       run: |
@@ -285,6 +291,8 @@ jobs:
 
     steps:
     - uses: actions/checkout@v3
+      with:
+        submodules: true
 
     ## Set up Python and Poetry environment
     - name: Set up Python ${{ env.python-version }}
@@ -305,7 +313,7 @@ jobs:
       run: echo "$HOME/.local/bin" >> $GITHUB_PATH
 
     - name: Configure poetry
-      run: poetry config virtualenvs.in-project true
+      run: poetry config virtualenvs.in-project false
 
     - name: Set up cache
       uses: actions/cache@v3
@@ -318,6 +326,9 @@ jobs:
       if: steps.cache.outputs.cache-hit == 'true'
       run: timeout 10s pip --version || rm -rf .venv
 
+    - name: Create virtualenv
+      run: python -m venv .venv
+
     ## Set up Node environment
     - uses: actions/setup-node@v3
       with:
@@ -346,7 +357,9 @@ jobs:
     - name: Set PUBLIC_URL
       run: echo "PUBLIC_URL=." >> $GITHUB_ENV
     - name: Build @stlite/kernel
-      run: make kernel
+      run: |
+        . .venv/bin/activate
+        make kernel
 
     - name: Package
       working-directory: packages/kernel
@@ -408,6 +421,8 @@ jobs:
 
     steps:
     - uses: actions/checkout@v3
+      with:
+        submodules: true
 
     ## Set up Python and Poetry environment
     - name: Set up Python ${{ env.python-version }}
@@ -428,7 +443,7 @@ jobs:
       run: echo "$HOME/.local/bin" >> $GITHUB_PATH
 
     - name: Configure poetry
-      run: poetry config virtualenvs.in-project true
+      run: poetry config virtualenvs.in-project false
 
     - name: Set up cache
       uses: actions/cache@v3
@@ -441,6 +456,9 @@ jobs:
       if: steps.cache.outputs.cache-hit == 'true'
       run: timeout 10s pip --version || rm -rf .venv
 
+    - name: Create virtualenv
+      run: python -m venv .venv
+
     ## Set up Node environment
     - uses: actions/setup-node@v3
       with:
@@ -469,7 +487,9 @@ jobs:
     - name: Set PUBLIC_URL
       run: echo "PUBLIC_URL=." >> $GITHUB_ENV
     - name: Build @stlite/mountable
-      run: make mountable
+      run: |
+        . .venv/bin/activate
+        make mountable
 
     - name: Package
       working-directory: packages/mountable
@@ -531,6 +551,8 @@ jobs:
 
     steps:
     - uses: actions/checkout@v3
+      with:
+        submodules: true
 
     ## Set up Python and Poetry environment
     - name: Set up Python ${{ env.python-version }}
@@ -551,7 +573,7 @@ jobs:
       run: echo "$HOME/.local/bin" >> $GITHUB_PATH
 
     - name: Configure poetry
-      run: poetry config virtualenvs.in-project true
+      run: poetry config virtualenvs.in-project false
 
     - name: Set up cache
       uses: actions/cache@v3
@@ -564,6 +586,9 @@ jobs:
       if: steps.cache.outputs.cache-hit == 'true'
       run: timeout 10s pip --version || rm -rf .venv
 
+    - name: Create virtualenv
+      run: python -m venv .venv
+
     ## Set up Node environment
     - uses: actions/setup-node@v3
       with:
@@ -595,7 +620,9 @@ jobs:
       if: github.ref_name == github.event.repository.default_branch
       run: echo "REACT_APP_EDITOR_APP_ORIGIN=https://edit.share.stlite.net" >> $GITHUB_ENV
     - name: Build @stlite/sharing
-      run: make sharing
+      run: |
+        . .venv/bin/activate
+        make sharing
 
     - name: Upload the built directory as an artifact
       uses: actions/upload-artifact@v3
@@ -692,6 +719,8 @@ jobs:
 
     steps:
     - uses: actions/checkout@v3
+      with:
+        submodules: true
 
     ## Set up Python and Poetry environment
     - name: Set up Python ${{ env.python-version }}
@@ -712,7 +741,7 @@ jobs:
       run: echo "$HOME/.local/bin" >> $GITHUB_PATH
 
     - name: Configure poetry
-      run: poetry config virtualenvs.in-project true
+      run: poetry config virtualenvs.in-project false
 
     - name: Set up cache
       uses: actions/cache@v3
@@ -725,6 +754,9 @@ jobs:
       if: steps.cache.outputs.cache-hit == 'true'
       run: timeout 10s pip --version || rm -rf .venv
 
+    - name: Create virtualenv
+      run: python -m venv .venv
+
     ## Set up Node environment
     - uses: actions/setup-node@v3
       with:
@@ -750,7 +782,9 @@ jobs:
 
     ## Build and deploy @stlite/desktop
     - name: Build @stlite/desktop
-      run: make desktop
+      run: |
+        . .venv/bin/activate
+        make desktop
 
     - name: Check if electron-builder works
       working-directory: packages/desktop

.nvmrc

@@ -1 +1 @@
-lts/gallium
+lts/*

CHANGELOG.md

@@ -7,6 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### `@stlite/kernel`
+
+#### Changed
+
+- Update Streamlit to 1.27.0, [#671](https://github.com/whitphx/stlite/pull/671)
+
+### `@stlite/mountable`
+
+#### Added
+
+- `disableProgressToasts` and `disableErrorToasts` options, [#671](https://github.com/whitphx/stlite/pull/671)
+
+#### Changed
+
+- Remove `ctx.disableToast()` and `ctx.enableToast()`, [#671](https://github.com/whitphx/stlite/pull/671)
+
 ## [0.42.5] - 2024-01-19
 
 ### `@stlite/kernel`

DEVELOPMENT.md

@@ -7,7 +7,7 @@ Use the Python version same as the one which the [currently used Pyodide](./pack
 ### Initialize the development environment
 
 ```
-USE_CONSTRAINT_FILE=false make init
+USE_CONSTRAINTS_FILE=false make init
 ```
 
 ### Development of each package

Makefile

@@ -7,8 +7,9 @@ sharing-editor := packages/sharing-editor/build/*
 desktop := packages/desktop/build/*
 kernel := packages/kernel/dist/*
 stlite-server-wheel := packages/kernel/py/stlite-server/dist/stlite_server-0.1.0-py3-none-any.whl
-streamlit_proto := streamlit/frontend/src/lib/proto.d.ts
-streamlit_wheel := packages/kernel/py/streamlit/lib/dist/streamlit-1.24.0-cp311-none-any.whl
+streamlit_proto := streamlit/frontend/lib/src/proto.d.ts
+streamlit_wheel := packages/kernel/py/streamlit/lib/dist/streamlit-1.27.0-cp311-none-any.whl
+streamlit_frontend_lib_prod := streamlit/frontend/lib/dist/*
 
 .PHONY: all
 all: init mountable sharing sharing-editor
@@ -21,8 +22,7 @@ VENV := ./.venv
 NODE_MODULES := ./node_modules
 
 .PHONY: venv
-venv: $(VENV)
-$(VENV): requirements.dev.txt
+venv: requirements.dev.txt
 	[ -d $(VENV) ] || python -m venv $(VENV)
 	. $(VENV)/bin/activate && python -m pip install -U pip && python -m pip install -r requirements.dev.txt
 	@echo "\nPython virtualenv has been set up. Run the command below to activate.\n\n. $(VENV)/bin/activate"
@@ -59,14 +59,14 @@ $(common-react): packages/common-react/src/*.ts yarn_install $(kernel)
 
 .PHONY: mountable
 mountable: $(mountable)
-$(mountable): packages/mountable/src/*.ts packages/mountable/src/*.tsx yarn_install $(kernel) $(common-react)
+$(mountable): packages/mountable/src/*.ts packages/mountable/src/*.tsx yarn_install $(kernel) $(common-react) $(streamlit_frontend_lib_prod)
 	cd packages/mountable; \
 	yarn build
 	@touch $@
 
 .PHONY: sharing
 sharing: $(sharing)
-$(sharing): packages/sharing/src/*.ts packages/sharing/src/*.tsx yarn_install $(kernel) $(sharing-common) $(common-react)
+$(sharing): packages/sharing/src/*.ts packages/sharing/src/*.tsx yarn_install $(kernel) $(sharing-common) $(common-react) $(streamlit_frontend_lib_prod)
 	cd packages/sharing; \
 	yarn build
 	@touch $@
@@ -87,7 +87,7 @@ $(sharing-editor): packages/sharing-editor/src/*.ts packages/sharing-editor/src/
 
 .PHONY: desktop
 desktop: $(desktop)
-$(desktop): packages/desktop/src/*.ts packages/desktop/src/*.tsx packages/desktop/electron/*.ts yarn_install $(kernel) $(common) $(common-react)
+$(desktop): packages/desktop/src/*.ts packages/desktop/src/*.tsx packages/desktop/electron/*.ts yarn_install $(kernel) $(common) $(common-react) $(streamlit_frontend_lib_prod)
 	cd packages/desktop; \
 	yarn build
 	@touch $@
@@ -100,21 +100,24 @@ $(kernel): packages/kernel/src/*.ts $(common) $(stlite-server-wheel) $(streamlit
 	@touch $@
 
 .PHONY: stlite-server-wheel
-$(stlite-server-wheel): $(VENV) packages/kernel/py/stlite-server/stlite_server/*.py
+stlite-server-wheel: $(stlite-server-wheel)
+$(stlite-server-wheel): venv packages/kernel/py/stlite-server/stlite_server/*.py
 	. $(VENV)/bin/activate && \
 	cd packages/kernel/py/stlite-server && \
 	poetry build
 	@touch $@
 
-$(streamlit_proto): $(VENV) streamlit/proto/streamlit/proto/*.proto
+.PHONY: streamlit-proto
+streamlit-proto: $(streamlit_proto)
+$(streamlit_proto): venv streamlit/proto/streamlit/proto/*.proto
 	. $(VENV)/bin/activate && \
-	cd streamlit; \
-	$(MAKE) mini-init
+	$(MAKE) -C streamlit python-init-dev-only && \
+	$(MAKE) -C streamlit protobuf
 	@touch $@
 
 .PHONY: streamlit-wheel
 streamlit-wheel: $(streamlit_wheel)
-$(streamlit_wheel): $(VENV) $(streamlit_proto) streamlit/lib/streamlit/**/*.py streamlit/lib/Pipfile streamlit/lib/setup.py streamlit/lib/bin/* streamlit/lib/MANIFEST.in
+$(streamlit_wheel): venv $(streamlit_proto) streamlit/lib/streamlit/**/*.py streamlit/lib/Pipfile streamlit/lib/setup.py streamlit/lib/bin/* streamlit/lib/MANIFEST.in
 	. $(VENV)/bin/activate && \
 	PYODIDE_VERSION=`python -c "import pyodide_build; print(pyodide_build.__version__)"` && \
 	PYTHON_VERSION=`python -c "import sys; print('.'.join(map(str, sys.version_info[:3])))"` && \
@@ -124,6 +127,11 @@ $(streamlit_wheel): $(VENV) $(streamlit_proto) streamlit/lib/streamlit/**/*.py s
 		exit 1; \
 	fi && \
 	cd streamlit && SNOWPARK_CONDA_BUILD=true $(MAKE) distribution && cd .. && \
-	pyodide py-compile --keep streamlit/lib/dist/streamlit-1.24.0-py2.py3-none-any.whl && \
+	pyodide py-compile --keep streamlit/lib/dist/streamlit-1.27.0-py2.py3-none-any.whl && \
 	mkdir -p $(dir $(streamlit_wheel)) && \
 	cp streamlit/lib/dist/$(notdir $(streamlit_wheel)) $(streamlit_wheel)
+
+.PHONY: streamlit-frontend-lib
+streamlit-frontend-lib: $(streamlit_frontend_lib_prod)
+$(streamlit_frontend_lib_prod): yarn_install $(kernel) $(streamlit_proto) streamlit/frontend/lib/src/**/*.ts streamlit/frontend/lib/src/**/*.tsx streamlit/frontend/lib/package.json streamlit/frontend/lib/tsconfig.json
+	$(MAKE) -C streamlit frontend-lib-prod

package.json

@@ -3,7 +3,9 @@
   "name": "stlite",
   "workspaces": [
     "packages/*",
-    "streamlit/frontend"
+    "streamlit/frontend",
+    "streamlit/frontend/app",
+    "streamlit/frontend/lib"
   ],
   "scripts": {
     "new-version": "lerna version --force-publish",
@@ -22,5 +24,10 @@
     "rimraf": "^3.0.2",
     "wait-on": "^6.0.1"
   },
+  "resolutions": {
+    "immutable": "4.2.3",
+    "webpack": "5.76.0",
+    "@types/react": "17.*"
+  },
   "version": "0.2.0"
 }

packages/common-react/package.json

@@ -14,7 +14,7 @@
     "check:prettier": "prettier --check ."
   },
   "dependencies": {
-    "streamlit-browser": "^1.18.1"
+    "@streamlit/lib": "1.27.0"
   },
   "devDependencies": {
     "@typescript-eslint/eslint-plugin": "^5.49.0",