|
| 1 | +Developing |
| 2 | +========== |
| 3 | + |
| 4 | +This repository uses a polylith structure in order to allow the several |
| 5 | +projects under the Runestone umbrella to share code, provide common ways |
| 6 | +of accomplishing similar tasks, and hopefully make it easier for a |
| 7 | +newcomer to contribute to the project. |
| 8 | + |
| 9 | +To get started it will be very helpful to `install |
| 10 | +poetry <https://python-poetry.org/docs/>`__. with poetry installed you |
| 11 | +will need to add two very important plugins. |
| 12 | + |
| 13 | +1. ``poetry self add poetry-polylith-plugin`` |
| 14 | +2. ``poetry self add poetry-multiproject-plugin`` |
| 15 | + |
| 16 | +With those installed go ahead and run ``poetry poly info`` |
| 17 | + |
| 18 | +.. code:: shell |
| 19 | +
|
| 20 | + projects: 7 |
| 21 | + components: 11 |
| 22 | + bases: 5 |
| 23 | + development: 1 |
| 24 | +
|
| 25 | + brick author_server book_server dash_server jobe nginx rsmanage w2p_login_assign_grade development |
| 26 | + ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── |
| 27 | +
|
| 28 | + auth ✔ ✔ - - - - - ✔ |
| 29 | + configuration ✔ ✔ - - - ✔ - - |
| 30 | + data_extract ✔ - - - - - - ✔ |
| 31 | + db ✔ ✔ - - - ✔ - ✔ |
| 32 | + forms ✔ - - - - - - ✔ |
| 33 | + logging ✔ ✔ - - - ✔ - ✔ |
| 34 | + lp_sim_builder - ✔ - - - - - - |
| 35 | + response_helpers ✔ ✔ - - - ✔ - ✔ |
| 36 | + validation ✔ ✔ - - - ✔ - ✔ |
| 37 | + visualization ✔ - - - - - - ✔ |
| 38 | + author_server_api ✔ - - - - - - ✔ |
| 39 | + book_server_api - ✔ - - - - - ✔ |
| 40 | + dash_server_api - - ✔ - - - - - |
| 41 | + rsmanage - - - - - ✔ - ✔ |
| 42 | + web2py_server - - - - - - ✔ ✔ |
| 43 | +
|
| 44 | +This tells you we have 7 projects. There may be more if we haven't kept |
| 45 | +this up to date. |
| 46 | + |
| 47 | +The **projects** listed across the top of the table define the artifacts |
| 48 | +- Docker images or applications, they could be a web application or a |
| 49 | +command line application or whatever. The **bases** - contains the |
| 50 | +public facing API for a project. The **components** contain code that |
| 51 | +supports one or more projects/bases. You can see which projects use |
| 52 | +which base and which components by the check marks in the table. |
| 53 | + |
| 54 | +The goal is to put as much code as possible into the components in a way |
| 55 | +that is very reusable. For example our database code for doing Create, |
| 56 | +Retrieve, Update, and Delete operations in the database is ALL contained |
| 57 | +in the db component. You can probably do a lot of very useful |
| 58 | +development without having to know anything about database programming |
| 59 | +by simply using the functions defined there. |
| 60 | + |
| 61 | +With the structure we have in place, you can run and do development on |
| 62 | +any of the server projects with or without knowing anything about docker |
| 63 | +or containers. |
| 64 | + |
| 65 | +Let's discuss discuss each of the projects at a high level to start |
| 66 | +with. Then you can find detailed documentation for each project in their |
| 67 | +project folder. |
| 68 | + |
| 69 | +**w2p_login_assign_grader** This is a legacy project that uses the |
| 70 | +web2py framework and currently supports - login, assignments, grading, |
| 71 | +and basic student analytics. We are actively working to migrate each of |
| 72 | +those pieces into its own project. |
| 73 | + |
| 74 | +**book_server** The book server is as FastAPI web application that |
| 75 | +serves the pages of each textbook to students and handles the API calls |
| 76 | +from the interactive components of the textbook. It also serves as the |
| 77 | +websocket server for peer instruction. |
| 78 | + |
| 79 | +**author_server** The author server allows the authors of the textbooks |
| 80 | +to build and deploy new versions of their books across the runestone |
| 81 | +system. It also contains functionality for authors and researchers to |
| 82 | +visualize how their textbooks are used, and to create anonymized data |
| 83 | +files for detailed analysis. |
| 84 | + |
| 85 | +**dash_server** This is a new, modern take on the original student |
| 86 | +dashboard, but it will scale up to support very large classes. You can |
| 87 | +work on this 100% in python without needing to know css or javascript as |
| 88 | +it uses the Dash / Plotly framework. |
| 89 | + |
| 90 | +**rsmanage** This is a command line program for managing courses, users, |
| 91 | +and many other aspects of the Runestone system. It is mostly useful for |
| 92 | +people running large scale servers. |
| 93 | + |
| 94 | +**jobe** The jobe server is a custom job runner for compiling and |
| 95 | +running C, C++, and Java programs. |
| 96 | + |
| 97 | +**nginx** The nginx project uses nginx as the traffic director to route |
| 98 | +requests across the various servers that comprise the Runestone system. |
| 99 | + |
| 100 | +Project File Structure |
| 101 | +---------------------- |
| 102 | + |
| 103 | +.. code-block:: text |
| 104 | +
|
| 105 | + 📁. |
| 106 | + ├── 📁bases |
| 107 | + │ └── 📁rsptx |
| 108 | + │ ├── 📁author_server_api |
| 109 | + │ ├── 📁book_server_api |
| 110 | + │ ├── 📁dash_server_api |
| 111 | + │ ├── 📁rsmanage |
| 112 | + │ └── 📁web2py_server |
| 113 | + ├── 📁components |
| 114 | + │ └── 📁rsptx |
| 115 | + │ ├── 📁auth |
| 116 | + │ ├── 📁configuration |
| 117 | + │ ├── 📁data_extract |
| 118 | + │ ├── 📁db |
| 119 | + │ ├── 📁forms |
| 120 | + │ ├── 📁logging |
| 121 | + │ ├── 📁lp_sim_builder |
| 122 | + │ ├── 📁response_helpers |
| 123 | + │ ├── 📁validation |
| 124 | + │ └── 📁visualization |
| 125 | + ├── 📁development |
| 126 | + │ └── core.py |
| 127 | + ├── docker-compose.yml |
| 128 | + ├── 📁docs |
| 129 | + │ ├── 📁build |
| 130 | + │ │ ├── 📁doctrees |
| 131 | + │ │ └── 📁html |
| 132 | + │ ├── 📁graffle |
| 133 | + │ │ └── Phased Approach to Polylith.graffle |
| 134 | + │ ├── 📁images |
| 135 | + │ │ └── RunestoneArch.svg |
| 136 | + │ ├── Makefile |
| 137 | + │ └── 📁source |
| 138 | + ├── 📁projects |
| 139 | + │ ├── 📁author_server |
| 140 | + │ │ ├── 📁dist |
| 141 | + │ │ ├── Dockerfile |
| 142 | + │ │ ├── gitconfig |
| 143 | + │ │ ├── pyproject.toml |
| 144 | + │ │ └── README.md |
| 145 | + │ ├── 📁book_server |
| 146 | + │ │ ├── 📁dist |
| 147 | + │ │ ├── Dockerfile |
| 148 | + │ │ ├── pyproject.toml |
| 149 | + │ │ └── README.md |
| 150 | + │ ├── 📁dash_server |
| 151 | + │ │ ├── 📁cache |
| 152 | + │ │ ├── 📁dist |
| 153 | + │ │ ├── Dockerfile |
| 154 | + │ │ ├── pyproject.toml |
| 155 | + │ │ └── README.md |
| 156 | + │ ├── 📁jobe |
| 157 | + │ ├── 📁nginx |
| 158 | + │ │ ├── Dockerfile |
| 159 | + │ ├── 📁rsmanage |
| 160 | + │ │ ├── 📁dist |
| 161 | + │ │ ├── poetry.lock |
| 162 | + │ │ └── pyproject.toml |
| 163 | + │ └── 📁w2p_login_assign_grade |
| 164 | + │ ├── 📁dist |
| 165 | + │ ├── Dockerfile |
| 166 | + │ └── pyproject.toml |
| 167 | + ├── pyproject.toml |
| 168 | + ├── README.rst |
| 169 | + ├── 📁test |
| 170 | + └── workspace.toml |
| 171 | +
|
| 172 | +
|
| 173 | +Database Setup |
| 174 | +-------------- |
| 175 | + |
| 176 | +The database is a critical component as it is the glue that ties together the various servers. You have a few different options for database setup. |
| 177 | + |
| 178 | +1. Use SQLLite -- this may be ok for very casual use or even light development work, but really is not ideal for any kind of production environment. |
| 179 | +2. Install Postgresql as part of the docker-compose setup |
| 180 | +3. Install Postgresql on your local host (either natively or in a container) |
| 181 | + |
| 182 | +My currently recommended option is number 3. It is what you are probably going to want for production anyway, and I think it gives you the most flexibility for development. I simply installed it on my mac using `` homebrew.`` Linux users can use `` apt`` or whatever. You could even install it in its own ` docker container < https://www.baeldung.com/ops/postgresql-docker-setup>`_ and access it as if it was installed natively. It is easy for services running in docker to access the database service running on the host. Using a URL like `` postgresql://user:[email protected]/runestone_dev`` The key there is the `` host.docker.internal`` tells the process running in the container to connect to the host. Running it on the host also makes it far less surprising when you do a rebuild and suddenly your test data is gone because you dumped the image. |
| 183 | + |
| 184 | +You can connect to the database with one of 3 URLs depending on your server configuration (``SERVER_CONFIG``) environment variable - production, development, or test. Test is really just for unit testing. So you will most often want to use development. The environment variables to set are ``DBURL``, ``DEV_DBURL`` or ``TEST_DBURL``. |
| 185 | + |
| 186 | + |
| 187 | +Authentication |
| 188 | +-------------- |
| 189 | + |
| 190 | +At the time of this writing (April 2023) authentication is a bit over complicated. That is part of what this monorepo project is trying to straighten out. |
| 191 | + |
| 192 | +To run a project, for example the author server main web app: |
| 193 | + |
| 194 | +.. code:: bash |
| 195 | +
|
| 196 | + poetry shell |
| 197 | + uvicorn rsptx.author_server_api.main:app --reload |
| 198 | +
|
| 199 | +The top level docker-compose.yml file combines all of the projects |
| 200 | + |
| 201 | +Each project has a Dockerfile for building an image. These images should |
| 202 | +be push-able to our docker container registry and or the public docker |
| 203 | +container registry |
| 204 | + |
| 205 | +When developing and you need multiple servers running |
| 206 | +===================================================== |
| 207 | + |
| 208 | +Install nginx and configure projects/nginx/runestone.dev for your |
| 209 | +system. You can run nginx in "non daemon mode" using |
| 210 | +``nginx -g 'daemon off;'`` |
| 211 | + |
| 212 | +Set RUNESTONE_PATH -- not sure what for?? set SERVER_CONFIG development |
| 213 | +set WEB2PY_CONFIG development set DEV_DBURL |
| 214 | +postgresql://bmiller:@localhost/runestone_dev set BOOK_PATH |
| 215 | +/path/to/books set WEB2PY_PRIVATE_KEY ?? |
| 216 | + |
| 217 | +.. code:: bash |
| 218 | +
|
| 219 | + poetry shell |
| 220 | +
|
| 221 | + uvicorn rsptx.book_server_api.main:app --reload --host 0.0.0.0 --port 8111 |
| 222 | + cd ~/rs/bases/rsptx/web2py_server |
| 223 | + python web2py.py --no-gui --password whatever --ip 0.0.0.0 --port 8112 |
| 224 | +
|
| 225 | +If startup fails you may be missing a dependency... poetry seems to miss |
| 226 | +greenlet sometimes. But a quick check is to run python and then |
| 227 | + |
| 228 | +.. code:: python |
| 229 | +
|
| 230 | + >>> import rsptx.book_server_api.main |
| 231 | +
|
| 232 | +You will see a more detailed error message about what is missing. |
| 233 | + |
| 234 | +At a minimum you will need to start web2py long enough for you to login |
| 235 | +once. |
0 commit comments