Skip to content

Conat -- replacing NATS with socketio + sqlite3 + valkey + nodejs #8346

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 634 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
634 commits
Select commit Hold shift + click to select a range
de0a4c2
adjust changefeed keepalive params
williamstein Jun 6, 2025
569cca3
listings failures to query db should return undefined not crash
williamstein Jun 6, 2025
4f20fcf
fix flyout terminal breaking on upgrade of @xterm (unrelated to conat)
williamstein Jun 6, 2025
be128ad
fix issue with keepalive and reduce some excessive logging
williamstein Jun 6, 2025
55988cc
ts fix
williamstein Jun 6, 2025
51b1078
conat: implement significantly enhanced subscription sync; also make …
williamstein Jun 6, 2025
9d8bc9f
ts
williamstein Jun 6, 2025
726e452
just call it force resize instead of kick
williamstein Jun 6, 2025
14d6f7f
fix bugs, tests and fallout from conat client sub sync
williamstein Jun 6, 2025
6db57f0
make terminal history be local to each frame and saved immediately
williamstein Jun 6, 2025
c3424d8
HISTFILE is relative path
williamstein Jun 6, 2025
60dc054
using a subdir for bash history is silly
williamstein Jun 6, 2025
22aa1cf
ts
williamstein Jun 6, 2025
df54854
special terminal env should overwrite parent env
williamstein Jun 6, 2025
2288b9d
conat: fix subscription counter
williamstein Jun 6, 2025
dab36a5
conat sync: fix some leaks in persist clients
williamstein Jun 6, 2025
5786fcc
dkv and dstream could try to save after they close -- fix
williamstein Jun 6, 2025
8354a2c
reducing leaks when closing projects in frontend
williamstein Jun 6, 2025
c122da0
project was initing compute server too much
williamstein Jun 7, 2025
e540868
close project actions if the tab is closed
williamstein Jun 7, 2025
33a6cba
break up some socket tests
williamstein Jun 7, 2025
dafe59c
ts
williamstein Jun 7, 2025
642453a
change frontend to update its /customize store periodically (instead …
williamstein Jun 7, 2025
75082ce
re-implement frontend version check logic
williamstein Jun 7, 2025
60b495c
include new version warning component
williamstein Jun 7, 2025
30e6101
compute server manager -- could hit uncaught error that has meaningfu…
williamstein Jun 7, 2025
516fae5
bunch of testing with lots of packet loss conditions; DELETE all the …
williamstein Jun 7, 2025
29f825b
eliminate "non-ephemeral subscriptions".
williamstein Jun 7, 2025
bdda35d
make it so version is part of customize api
williamstein Jun 7, 2025
660f388
work in progress on project version check
williamstein Jun 7, 2025
769abc9
implement version check for project/compute server
williamstein Jun 7, 2025
904886b
sockets and subscriptions: use 'closed' for the event systematically;…
williamstein Jun 7, 2025
e1a236d
better visibility into what sockets are used for
williamstein Jun 7, 2025
8356d3e
better tracking of sockets (fixes some leaks, maybe); make terminals …
williamstein Jun 7, 2025
aef8ffe
fix terminal restart prompt behavior
williamstein Jun 8, 2025
4580396
mainly -- persist parallelism testing; in particular, lots of servers…
williamstein Jun 8, 2025
ab3b231
temporarily disable merge conflict resolution for open-files (it may …
williamstein Jun 8, 2025
1d913ff
NATS --> Conat; modify open-files to use dko to get proper merge with…
williamstein Jun 8, 2025
08e1134
fix open-files unit test
williamstein Jun 8, 2025
02e19c2
conat: implement batched version of saving kv store data
williamstein Jun 8, 2025
a544b48
conat subscription sync -- behave better when offline
williamstein Jun 8, 2025
e9ecbc2
increase the socket keepalive interval
williamstein Jun 8, 2025
8ccd22e
conat: refactor db changefeed code using a new usage monitor class; s…
williamstein Jun 8, 2025
fb6c244
conat server: per user and global connection limits and monitoring
williamstein Jun 8, 2025
bf9b745
per-user and global limits on persist server
williamstein Jun 8, 2025
8b69362
add requestMany unit test to show how to call many services at once a…
williamstein Jun 8, 2025
3c60225
conat server: add sys service that admins can call, which gives stats…
williamstein Jun 9, 2025
58d76b0
add "pnpm conat-connections"
williamstein Jun 9, 2025
aad79b9
show table of connections
williamstein Jun 9, 2025
11e5776
ssl option for conat server
williamstein Jun 9, 2025
e95a843
Merge branch 'conat' of github.com:sagemathinc/cocalc into conat
williamstein Jun 9, 2025
21d879a
server -- fix ssl handling
williamstein Jun 9, 2025
e2f6a24
hide something that really isn't an error
williamstein Jun 9, 2025
250230d
do not try to write password if don't have perms; better logging of p…
williamstein Jun 9, 2025
6a07bbf
terminal: implement hsy's suggestion for the scope of $HISTFILE
williamstein Jun 9, 2025
eb516ac
missing test for closed
williamstein Jun 9, 2025
4bb01f9
conat: valkey configuration
williamstein Jun 9, 2025
45045d5
remove SECRETS env var before starting projects
williamstein Jun 9, 2025
7a8d6f4
terminal -- fix control code issue
williamstein Jun 9, 2025
4598622
configuration number of socketio servers
williamstein Jun 9, 2025
b8b2891
typescript
williamstein Jun 9, 2025
075f01f
Merge branch 'master' into conat
williamstein Jun 9, 2025
e7efffe
terminal: fix control characters corruption problem properly; also ad…
williamstein Jun 9, 2025
cf5f0ab
remove some paranoid code from terminal that shouldn't be needed (or …
williamstein Jun 9, 2025
7c82e71
fix flyout panel terminal sync properly by using the correct type of …
williamstein Jun 9, 2025
3f3bbb5
conat: rewrite terminal startup to be much faster
williamstein Jun 10, 2025
4c6220c
conat: rewrite server admin to work with multiple servers
williamstein Jun 10, 2025
e88297f
fix sync unit tests
williamstein Jun 10, 2025
d639e14
add ip address to connection stats
williamstein Jun 10, 2025
4e88ee3
refactoring code related to synctables
williamstein Jun 10, 2025
bb82778
better forward address detection
williamstein Jun 10, 2025
9680af0
slight code reorg: move socket.io install to conat package
williamstein Jun 10, 2025
b84bb27
use awaiting's delay instead of explicitly making Promise; make skipp…
williamstein Jun 10, 2025
a415ecf
node 18 is deprecated
williamstein Jun 10, 2025
5a832c0
conat -- fix socket retry DOS bug
williamstein Jun 10, 2025
f1a2626
rewriting new conat code to use a single "until" function for exponen…
williamstein Jun 10, 2025
0d4e65c
make until not catch exceptions; convert more code; refactor some uni…
williamstein Jun 10, 2025
edacb30
Merge branch 'master' into conat
williamstein Jun 10, 2025
ceb76d1
make terminal unit test more robust to slow host
williamstein Jun 10, 2025
83567e9
finish rewriting the async until loops in conat
williamstein Jun 10, 2025
a44c3f7
get rid of the clock skew message, since cocalc sync no longer requir…
williamstein Jun 10, 2025
d903111
add example of conat server
williamstein Jun 10, 2025
bcdf848
Merge branch 'conat' of github.com:sagemathinc/cocalc into conat
williamstein Jun 10, 2025
b22a780
try increasing a test timeout for github ci
williamstein Jun 10, 2025
421b254
add a sync client that would just use conat directly (not used yet); …
williamstein Jun 10, 2025
5e9cd91
delete a bunch of confusing obsolete references to nats from our docs
williamstein Jun 10, 2025
ba32b87
Merge branch 'conat' of github.com:sagemathinc/cocalc into conat
williamstein Jun 10, 2025
e58f188
re-enable logging of file open times, and start/end the timer properly
williamstein Jun 10, 2025
51f0122
fix project copy timeout unit issue (effects courses)
williamstein Jun 10, 2025
79c86fb
eliminate non-implemented notion of lifetime for ephemeral for core-s…
williamstein Jun 10, 2025
85758d1
improve socket reconnect logic
williamstein Jun 11, 2025
3e36208
terminal scrolling - seems not needed and possibly very annoying with…
williamstein Jun 11, 2025
06f6ae8
working on tcp implementation...
williamstein Jun 11, 2025
be0b0b7
misc
williamstein Jun 11, 2025
7d00cdb
socket: servers can close clients, of course
williamstein Jun 11, 2025
3b94e7f
refactor project terminal code - in prep for supporting compute servers
williamstein Jun 11, 2025
f2d815b
use msgpack option to ignore undefined during encode -- much better s…
williamstein Jun 11, 2025
3f6cb55
terminal: supporting compute servers properly
williamstein Jun 11, 2025
669120a
compute server terminal integration
williamstein Jun 11, 2025
753815b
compute servers: support basePath and set CONAT_SERVER in compute ser…
williamstein Jun 11, 2025
92a68c5
a terminal exit message is no longer needed
williamstein Jun 11, 2025
1be3f6f
delete some deprecated code related to counting number of clients
williamstein Jun 11, 2025
b9f6b84
mainly organizing hub to serve various components of conat; also make…
williamstein Jun 11, 2025
c6e29f3
add to README for hub how to start individual conat-related services
williamstein Jun 12, 2025
9093b63
finish removing clients from hub
williamstein Jun 12, 2025
8aef79c
conat_password --> conat-password; add valkey password path (not yet …
williamstein Jun 12, 2025
79fab9d
Merge branch 'master' into conat
williamstein Jun 12, 2025
9c619b0
Merge branch 'master' into conat
williamstein Jun 12, 2025
327c602
HISTFILE - must be an exact path
williamstein Jun 12, 2025
49f028c
fix #8272 -- upgrade zeromq
williamstein Jun 12, 2025
e39e70e
terminal confusion
williamstein Jun 12, 2025
9e75851
even better terminal reset...
williamstein Jun 12, 2025
80740d9
fix auth issue
williamstein Jun 13, 2025
73bb265
path --> termPath for terminal and set HISTFILE better
williamstein Jun 13, 2025
9c5a1f6
...
williamstein Jun 13, 2025
2f1c577
...
williamstein Jun 13, 2025
7c3d9f5
gees
williamstein Jun 13, 2025
064eaca
partial inventory computation via remote persist server
williamstein Jun 13, 2025
4522267
inventory -- implement again in dkv and dstream
williamstein Jun 13, 2025
b4a8911
use ascii-table3 to render inventory table
williamstein Jun 13, 2025
a06e562
make it possible to see the active file tab
williamstein Jun 13, 2025
0045c6c
ts
williamstein Jun 13, 2025
0ea0d84
Merge branch 'master' into conat
williamstein Jun 13, 2025
8c66604
remove unused deps from @cocalc/util (using depcheck)
williamstein Jun 13, 2025
b0f1448
remove unused deps from @cocalc/backend
williamstein Jun 13, 2025
59bb55e
remove all missing deps from @cocalc/hub
williamstein Jun 13, 2025
e240c02
depcheck server and project
williamstein Jun 13, 2025
c2baa72
depcheck on nextjs
williamstein Jun 13, 2025
208d670
depcheck @cocalc/jupyter
williamstein Jun 13, 2025
a9e9358
pg is required for next
williamstein Jun 13, 2025
cf5066f
depcheck database, sync, comm
williamstein Jun 13, 2025
0bdf288
frontend: remove all unused deps; enable all the tests
williamstein Jun 13, 2025
655a499
finish depchecks of everything
williamstein Jun 13, 2025
0743655
fix some version incompat (upgrading editor)
williamstein Jun 13, 2025
885b8ee
inventory: do not complain during unit tests
williamstein Jun 13, 2025
2a32f0a
raise a test timeout
williamstein Jun 13, 2025
6049fb3
remove the @cocalc/terminal package -- I guess I totally rewrote it e…
williamstein Jun 13, 2025
0ab05eb
Merge branch 'conat' of github.com:sagemathinc/cocalc into conat
williamstein Jun 13, 2025
f4ac099
Merge branch 'master' into conat
williamstein Jun 13, 2025
80a4a79
Merge branch 'master' into conat
williamstein Jun 13, 2025
78c520c
Merge branch 'master' into conat
williamstein Jun 13, 2025
6e9f299
disable some obviously very unsafe debugging code (never in prod, of …
williamstein Jun 14, 2025
514c212
add next dist-js only build target (needed for next-free build in k8s)
williamstein Jun 14, 2025
d12a312
do not run inventory when dstream closed
williamstein Jun 14, 2025
c00593b
connection stats -- add both in and out data
williamstein Jun 14, 2025
fcd910b
completely revamp how secret token is set in projects
williamstein Jun 14, 2025
0109271
multi-user secret token permissions
williamstein Jun 14, 2025
76fa45c
disable frontend webgl again -- might be fault of problems and I can'…
Jun 14, 2025
f78426e
rewrite core sage socket connection code using modern style and async…
Jun 14, 2025
11f02c0
de-deprecate sage worksheets
williamstein Jun 14, 2025
60a1749
fix sage worksheet paths
williamstein Jun 14, 2025
ba891e5
ts
williamstein Jun 14, 2025
4f97ce7
fix terminal compute server selector
williamstein Jun 14, 2025
d758bfd
work in progress -- new valkey test
williamstein Jun 15, 2025
4e20aa4
terminal -- better clear
williamstein Jun 15, 2025
ca0836c
jupyter: more unit testing; flag to disable pool; close zmq sockets; …
williamstein Jun 15, 2025
61ceffe
rewriting more of the nteract code (remove bugs/pain/hackiness); addi…
williamstein Jun 15, 2025
1b40d52
rename enchannel-zmq-backend.ts to something meaningful
williamstein Jun 15, 2025
4bf0e18
completely purge all use of execa from cocalc
williamstein Jun 15, 2025
c2bb981
make an attempt to support jupyter in github actions
williamstein Jun 15, 2025
b4eff7f
slow test
williamstein Jun 15, 2025
5b1eb4a
try running test more than one if fail in github actions
williamstein Jun 15, 2025
187f249
jupyter kernel: add pid method, use it, add more tests
williamstein Jun 15, 2025
2d98e43
more jupyter unit tests and rewriting a little bit of how code execut…
williamstein Jun 15, 2025
3d42170
add unit tests of kernel for stateless jupyter api
williamstein Jun 15, 2025
42603fc
add bash command to backend; writing more unit tests for jupyter (mai…
williamstein Jun 15, 2025
0252c34
properly implement ulimit for stateless jupyter api
williamstein Jun 16, 2025
3014a11
make jupyter ulimit test more robust
williamstein Jun 16, 2025
8fcf6a8
fix stateless jupyter api
williamstein Jun 16, 2025
df94b7f
api-client now depends on @cocalc/util
williamstein Jun 16, 2025
59c0e2c
Merge branch 'master' into conat
williamstein Jun 16, 2025
92d9c1f
switch back to upstream ascii table; fix bug with valkey subs
williamstein Jun 16, 2025
5e3bd65
valkey: add unit tests involving using valkey to scale conat
williamstein Jun 16, 2025
b5f2b6f
typo in github ci to install valkey
williamstein Jun 16, 2025
468fbea
unit socketio + valkey + sticky subscriptions
williamstein Jun 17, 2025
3fb0398
add stress tests, including with sticky subs and valkey
williamstein Jun 17, 2025
cec0b37
unit tests: add a delay before considering valkey working
williamstein Jun 17, 2025
f0aabf0
trying out maxRetriesPerRequest:null, since in CI things are breaking
williamstein Jun 17, 2025
94949ca
add a password to the valkey unit tests
williamstein Jun 17, 2025
85fbc14
conat server: suppot creating valkey connection using object with pas…
williamstein Jun 17, 2025
9c867a7
longer timeout for stress test
williamstein Jun 17, 2025
34b728f
more agressive socketio reconnect that doesn't just give up (since th…
williamstein Jun 17, 2025
c9eb492
working on valkey stream trimming test
williamstein Jun 17, 2025
051fcde
support a json string for valkey config
williamstein Jun 17, 2025
4213971
support new format -- sentinel:// -- for configuring valkey
williamstein Jun 17, 2025
022e26f
env variable to disable conat-specific use of valkey
williamstein Jun 17, 2025
82debcb
valkey: make xread timeout configurable
williamstein Jun 17, 2025
face43d
valkey usage: grasping at straws and trying to set a different db for…
williamstein Jun 17, 2025
a100fb3
revert desperate attempt to use separate valkey db
williamstein Jun 17, 2025
78d6f18
touch path multiple times
williamstein Jun 17, 2025
2b7fcf6
run valkey so that it doesn't get left running
williamstein Jun 17, 2025
563876d
update version
williamstein Jun 17, 2025
ec8b93e
fixed subtle race condition with opening documents
williamstein Jun 17, 2025
bcffdf2
minimal proof of concept doing interest sync directly not using its o…
williamstein Jun 17, 2025
9f5ff93
conat core server -- re-implement sticky state sync
williamstein Jun 17, 2025
06b8d3d
conat server: fix unit tests to work with newer model
williamstein Jun 17, 2025
a5a4d74
fix so builds on macos
Jun 17, 2025
b39aca3
Merge branch 'conat' of github.com:sagemathinc/cocalc into conat
Jun 17, 2025
68245f7
valkey: switch to a fork of the adapter with lower latency
williamstein Jun 18, 2025
9fc9ee3
Merge branch 'conat' of github.com:sagemathinc/cocalc into conat
williamstein Jun 18, 2025
b2e683c
test: fix valkey process leak
williamstein Jun 18, 2025
36aec87
switch to using the socketio valkey pub/sub adapter, which makes more…
williamstein Jun 18, 2025
9c4a488
make some more valkey tests more robust
williamstein Jun 18, 2025
0391763
add progress when loading file
williamstein Jun 18, 2025
d16f013
conat server: fix issue with add new nodes and state
williamstein Jun 18, 2025
7ac8200
fix bug involving syncRefCache that I found writing unit tests for pe…
williamstein Jun 18, 2025
adbfabd
platform independent use of sed to patch openapi
williamstein Jun 18, 2025
ebab64b
fix conat unit tests after changing definition of sticky
williamstein Jun 18, 2025
ec9168e
tweak some params to make unit tests in backend more robust
williamstein Jun 18, 2025
8b21f89
a pixel
williamstein Jun 18, 2025
c4ef41f
fix bug in changefeeds/sockets -- it looks like they were sometimes f…
williamstein Jun 18, 2025
0d6c2b1
re-deprecate sagemath worksheets
williamstein Jun 18, 2025
748c9af
fix chat button to be consistent with change I just made to file tab
williamstein Jun 18, 2025
ff4d705
Merge branch 'master' into conat
williamstein Jun 18, 2025
baf74eb
working proof of concept of using the cluster adapter
williamstein Jun 18, 2025
c26068c
conat socketio cluster adapter -- got it working and integrated in an…
williamstein Jun 18, 2025
bd029e5
Merge branch 'master' into conat
williamstein Jun 18, 2025
fed7912
include all env vars when launching socketio cluster adapter
williamstein Jun 18, 2025
0e20c7b
fix another merge conflict
williamstein Jun 18, 2025
c455c77
(1) fix conat core server logging; (2) support sys account in cluster…
williamstein Jun 18, 2025
f4d14f6
test -- don't need to explicitly set logger anymore
williamstein Jun 18, 2025
a9f2bae
conat connections: fix bug in creating table with rejected user
williamstein Jun 18, 2025
7d47cc6
fix depcheck issue found by CI (yeah!)
williamstein Jun 19, 2025
74b419c
new version; having an error user state means not signed in
williamstein Jun 19, 2025
f4d375a
reduce number of clients in a test, which is failing due to too many …
williamstein Jun 19, 2025
1ce8dc0
add pnpm conat-inventory to easily see all streams associated with a …
williamstein Jun 19, 2025
dde0a1e
inventory
williamstein Jun 19, 2025
39dcc8e
name column for the role of a process "role of process" instead of "p…
williamstein Jun 19, 2025
7c3980e
raise changefeed limit per user to 1000
williamstein Jun 19, 2025
b3802b7
make it easier to test running multiple api and persist servers
williamstein Jun 19, 2025
b768bb3
sanitize away CONAT_ vars when starting projects
williamstein Jun 19, 2025
fbf9934
change RAM options in the store (min of 4GB)
williamstein Jun 19, 2025
296aec5
more precise decision about proxying conat socketio server
williamstein Jun 19, 2025
6d604be
set changefeed limit to 500 per user
williamstein Jun 19, 2025
08f2ef1
only configure conat sockets and cluster port via env variables -- ha…
williamstein Jun 19, 2025
e1aecea
disable jupyterlab/vscode launcher buttons
williamstein Jun 19, 2025
5c82234
removing the data about conat set via database
williamstein Jun 19, 2025
58a931a
project actions: make them much more lightweight when tab isn't opened
williamstein Jun 19, 2025
f27e165
make project tour less in your face
williamstein Jun 19, 2025
62e4e33
terminals: fix some issues with project actions being more lightweight
williamstein Jun 19, 2025
caf9fa5
ensure all actions are cleaned up for all open files if you close the…
williamstein Jun 19, 2025
18de50f
any file editing from the browser of any kind wil now renew the idle …
williamstein Jun 19, 2025
1c18472
do not use HISTFILE for flyout terminal (indirectly, by only using it…
williamstein Jun 19, 2025
6d0982e
hub api -- use a queue group
williamstein Jun 19, 2025
86bd517
add support for mesgpackr and make it the default
williamstein Jun 20, 2025
ffb894f
decided to fully revert mesgpackr because (1) it is NOT that much bet…
williamstein Jun 20, 2025
3910376
fix a terminal edge case where it would hang on large paste
williamstein Jun 20, 2025
24028cd
increase listener counts in conat for now
williamstein Jun 20, 2025
eba5899
conat monitor: add "connections per server" table
williamstein Jun 20, 2025
44f50bf
add desc to what is persisted in sqlite storage
williamstein Jun 20, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
20 changes: 17 additions & 3 deletions .github/workflows/make-and-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,6 @@ jobs:
matrix:
# See supported Node.js release schedule at https://nodejs.org/en/about/releases/
node-version:
- "18"
- "20"
pg-version:
- "13.12"
Expand All @@ -92,7 +91,22 @@ jobs:
# cache: "pnpm"
# cache-dependency-path: "src/packages/pnpm-lock.yaml"

- name: Download and install Valkey
run: |
VALKEY_VERSION=8.1.2
curl -LO https://download.valkey.io/releases/valkey-${VALKEY_VERSION}-jammy-x86_64.tar.gz
tar -xzf valkey-${VALKEY_VERSION}-jammy-x86_64.tar.gz
sudo cp valkey-${VALKEY_VERSION}-jammy-x86_64/bin/valkey-server /usr/local/bin/

- name: Set up Python venv and Jupyter kernel
run: |
python3 -m pip install --upgrade pip virtualenv
python3 -m virtualenv venv
source venv/bin/activate
pip install ipykernel
python -m ipykernel install --prefix=./jupyter-local --name python3-local --display-name "Python 3 (Local)"

- run: cd src && npm install -g pnpm
- run: cd src && pnpm run make
- run: cd src && pnpm nats-server-daemon
- run: cd src && pnpm run test
- run: source venv/bin/activate && cd src && pnpm run test || pnpm run test || pnpm run test
- run: cd src && pnpm run depcheck
4 changes: 1 addition & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,5 @@ src/packages/frontend/i18n/extracted.json
src/packages/frontend/i18n/trans/*.compiled.json

**/*.db

src/packages/project/nats/project-env.sh

**/project-env.sh
**/*.bash_history
299 changes: 0 additions & 299 deletions docs/nats/devlog.md

This file was deleted.

Loading