Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor project tree to handle multiple OpenEdx releases #112

Merged
merged 14 commits into from
Sep 9, 2019

Conversation

jmaupetit
Copy link
Contributor

@jmaupetit jmaupetit commented Aug 28, 2019

Purpose

Managing multiple OpenEdx releases by using branches is a complicated workflow on a day to day basis. Once a new release (i.e. a branch for this project) has been created, it will no longer get rebased to master and backporting things is highly error-prone (one should not forget to do so for each release). Hence, in this context, we've decided to move to a simpler file-based tree to deal with releases.

⚠️ This branch should not be merged until all supported releases have been merged/integrated ⚠️

Proposal

We've decided to add all supported releases to the same releases directory:

releases
└── master
    └── bare
        ├── config
        │   ├── cms
        │   └── lms
        ├── data
        │   ├── media
        │   ├── static
        │   └── store
        ├── Dockerfile
        ├── entrypoint.sh
        └── src
            ├── edx-demo-course
            └── edx-platform

In this case, the master release in its "bare" flavor (i.e. as a fresh edxapp installation would be) has its own assets (Dockerfile configuration, data and sources). Another example for the hawthorn.1 release would look like ("bare" and "oee" flavors):

releases
└── hawthorn
    └── 1
        ├── bare
        │   ├── config
        │   │   ├── cms
        │   │   └── lms
        │   ├── data
        │   │   ├── media
        │   │   ├── static
        │   │   └── store
        │   ├── Dockerfile
        │   ├── entrypoint.sh
        │   └── src
        │       ├── edx-demo-course
        │       └── edx-platform
        └── oee
            ├── config
            │   ├── cms
            │   └── lms
            ├── data
            │   ├── media
            │   ├── static
            │   └── store
            ├── Dockerfile
            ├── entrypoint.sh
            └── src
                ├── edx-demo-course
                └── edx-platform

To work on a particular release, you will have to define the following environment variables:

  • EDX_RELEASE (default: master/bare): targeted OpenEdx release (e.g. hawthorn/1/oee) that should match a path relative to the releases directory
  • EDX_RELEASE_REF (default: release-2018-08-29-14.14): this is the Git reference you are targeting from edx-platform repository
  • EDX_DEMO_RELEASE_REF: (default: master) this is the Git reference you are targeting from edx-demo-course repository

Roadmap

  • add support for multiple OpenEdx releases as a file tree
  • move the master branch to the new architecture
  • add support for multiple OpenEdx image builds in the CI
  • move the hawthorn.1 branch to the new architecture
  • move the oee/hawthorn.1 branch to the new architecture
  • add an activate script that defines them for each release (as we do with a python virtualenv), e.g. source releases/hawthorn/1/ooe/activate
  • move the dogwood-fun branch to the new architecture (edit: postponed to a new PR as we need more work on it than checking out files from the related branch)
  • add CI quality jobs (git lint and changelog quality)
  • add a basic HTTP query test on builds to ensure that the production image runs

Fixes #66, #67, #94 , #110

@jmaupetit jmaupetit added feature FUN WIP Work In Progress labels Aug 28, 2019
@jmaupetit jmaupetit self-assigned this Aug 28, 2019
@jmaupetit jmaupetit force-pushed the refactor-project-tree branch 12 times, most recently from 589d51d to acd296d Compare September 3, 2019 12:26
@jmaupetit jmaupetit removed the WIP Work In Progress label Sep 3, 2019
@jmaupetit
Copy link
Contributor Author

Oops, I forgot to update the docs (README). Will fix this soon.

@jmaupetit
Copy link
Contributor Author

Another remark: I think we may factorize our configuration utility.

@lunika
Copy link
Member

lunika commented Sep 3, 2019

As you are the CHANGELOG master, you should probably add one ?

@jmaupetit
Copy link
Contributor Author

Last remark: what about adding a test in the CI that ensure that the build starts and we can query it? This is the least we can do IMO.

.circleci/config.yml Show resolved Hide resolved
.circleci/config.yml Show resolved Hide resolved
.circleci/config.yml Show resolved Hide resolved
Makefile Show resolved Hide resolved
README.md Show resolved Hide resolved
@sampaccoud
Copy link
Contributor

You might want to update your PR description as some explanations are obsolete.

@jmaupetit
Copy link
Contributor Author

You might want to update your PR description as some explanations are obsolete.

@sampaccoud Can you point me where please?

@sampaccoud
Copy link
Contributor

@jmaupetit it seems you have already added what is suggested in the note?

README.md Show resolved Hide resolved
We've checked out hawthorn.1 branch configuration and Dockerfile. We've
also added a default fake SENTRY_DSN for development purpose.

Fix #94
To ease working on a particular release, we've defined an activation
script for each release (e.g. an activate script at the root of each
release folder); the aforementioned script updates the working
environment to target this release.

We've also added an utility to assist / ease quick start activation: use
bin/activate to list select release you want to work on.
We need our CI to build every release to prevent possible side-effects.
When publishing a new image, we rebuild only a single release
corresponding to the release tag.
We've checked out oee/hawthorn.1 branch configuration, Dockerfile,
requirements and generated a patch from our edx-platform fork [1] to
enable our customizable LTI Xblock. This allows us to explicitly store
differences between a bare hawthorn.1 installation and our oee flavor.

[1] https://github.com/openfun/edx-platform/commits/oee/hawthorn.1
lynx is required for courses indexation and bulk email generation.
A little revamp of the basic docs in the project.
When using raw docker-compose commands, the UID and GID environment
variables are supposed to be defined. Hence, to avoid having a warning
message (and potential errors), one should use the following command:

UID=$(id -u) GID=$(id -g) docker-compose

To ease our life, we've added a bin/compose script that can be used as a
shortcut for a configured docker-compose command.
@jmaupetit jmaupetit force-pushed the refactor-project-tree branch from eaebdc3 to a47a05e Compare September 5, 2019 10:19
@jmaupetit
Copy link
Contributor Author

Like other projects, we need to ensure that our CHANGELOGs and git
commits fit with our expectations.
We've added a wait loop at the end of the "make run" and "make dev"
commands to ensure that the database and core services are up before
giving back the command prompt.
Development settings expect the sentry logger to be defined (meaning
that Sentry is configured for this flavor/release). It is not by
default.
@jmaupetit
Copy link
Contributor Author

Finally ready for another (last?) review @sampaccoud & @lunika 🙏

@jmaupetit jmaupetit force-pushed the refactor-project-tree branch 2 times, most recently from 247b862 to ac01898 Compare September 6, 2019 10:32
Makefile Outdated Show resolved Hide resolved
When dealing with Django management commands, most of the time our
servers are already running, meaning that we don't need to spawn a new
stack and use the one that already runs.
To ensure that our production builds are able to start and respond with
a HTTP requests we now start the CMS & LMS production services using
docker-compose and use curl to test them.
@jmaupetit jmaupetit force-pushed the refactor-project-tree branch from ac01898 to 5c1fba0 Compare September 9, 2019 09:28
Makefile Show resolved Hide resolved
@jmaupetit jmaupetit merged commit ccec536 into master Sep 9, 2019
@jmaupetit jmaupetit deleted the refactor-project-tree branch September 9, 2019 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[CI] remove unused edx-platform sources clone
3 participants