Skip to content

Latest commit

 

History

History
273 lines (178 loc) · 10.6 KB

DEVELOPMENT-GUIDE.md

File metadata and controls

273 lines (178 loc) · 10.6 KB

Development Guide

Technology Stack

The following stack is used when developing Codekvast (in alphabetical order):

  1. Angular 8+
  2. AspectJ (in Load-Time Weaving mode)
  3. Chrome headless
  4. Docker Engine - Community 19.03+ and Docker Compose 1.24+ (For running MariaDB and RabbitMQ in development environment)
  5. git-crypt
  6. Github
  7. Gradle (via Gradle wrapper)
  8. Inkscape (SVG graphics)
  9. Java 8, 9, 10, 11, 12, 13 and 14 (managed by SDKMAN).
  10. jq (for parsing JSON in scripts)
  11. Kotlin
  12. Lombok
  13. MariaDB 10+ (Codekvast Dashboard)
  14. Node Package Manager (npm)
  15. NodeJS
  16. Spring Boot
  17. TypeScript
  18. Webpack
  19. Yarn
  20. yq (for parsing YAML in scripts)

Directory structure

The product itself lives under product/.

Server-side stuff lives under product/server, while agent stuff lives under product/agent.

Sample projects to use when testing Codekvast lives under sample/.

Docker-related stuff live under docker/.

Development tools live under tools/.

Provisioning scripts live under deploy/.

NOTE: Some files are encrypted with git-crypt since it stores sensible data like cloud provider credentials!

Authorized developers are enabled to unlock the repo by adding their public GPG keys. See man git-crypt, git-crypt help add-gpg-user and git-crypt help unlock.

Web site

Web pages (i.e., https://www.codekvast.io) lives in the Git repo https://github.com/crispab/codekvast-site.

Development environment

There are a couple of Bash scripts that prepares the development environment.

They work for Ubuntu, and are called tools/install-compilers.sh and tools/prepare-workstation/run.sh. They use Bash and Ansible for setting up the workstation so that it works for Codekvast.

If you run some other OS or prefer to do it by hand, here are the requirements:

git-crypt

Some files in the deploy/ directory is encrypted with git-crypt and GPG public keys. Which files that are encrypted can be seen in deploy/.gitattributes.

You must be added as trusted developer by git-crypt add-gpg-user to access the secrets.

JDK and Node.js

SDKMAN is required.

In SDKMAN, the Java versions that are required are defined by gradle.properties.

Node.js 12+, NPM 3.10+ and Yarn 1.13+ are required. Chrome and Firefox are downloaded on demand for executing system tests.

git-crypt is required for deploying to the cloud.

Use the following command to install OpenJDK 11, git-crypt, Node.js, npm, Chrome and Yarn (Ubuntu, Debian):

curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
sudo apt install openjdk-11-jdk openjdk-11-doc openjdk-8-source git-crypt nodejs
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
echo "deb http://dl.google.com/linux/chrome/deb/ stable main" | sudo tee /etc/apt/sources.list.d/google-chrome.list
sudo apt-get update && sudo apt-get install yarn google-chrome-stable

TypeScript

The Codekvast Dashboard web UI is developed with TypeScript and Angular 7. Twitter Bootstrap is used as CSS framework.

npm and yarn are used for managing the frontend development environment. Webpack is used as frontend bundler.

Docker Engine - Community and Docker Composae

Docker Engine - Community 19.03 or later and Docker Compose 1.24 or later is required for Codekvast Dashboard development.

Install Docker Engine - Community v19.03+ using the official instructions.

Install Docker Compose 1.24+ using the official instructions.

Inkscape

Graphics including the Codekvast logo is crafted in SVG format, and exported to PNG in various variants and sizes. Inkscape is an excellent, free and cross-platform SVG editor.

Build tool

Codekvast uses Gradle as build tool. It uses the Gradle Wrapper, gradlew, which is checked in at the root of the workspace. There is the convenience script tools/src/script/gradlew which simplifies invocation of gradlew. Install that script in your PATH (e.g., /usr/local/bin), chmod +x /usr/local/bin/gradlew and simply use gradlew instead of path/to/gradlew

Continuous Integration

Codekvast is built by Jenkins at http://jenkins.crisp.se on every push, to all branches.

The pipeline is defined by Jenkinsfile.

To access http://jenkins.crisp.se you need to be either a Member or an Outside collaborator of https://github.com/orgs/crispab/people.

Software publishing

Codekvast Agent is published to https://downloads.codekvast.io as well as to jcenter.

You execute the publishing by executing tools/ship-agent.sh in the root of the project.

Preconditions:

  1. curl installed.

  2. Clean workspace (no work in progress).

  3. On the master branch.

  4. Synced with origin (pushed and pulled).

  5. Bintray credentials either in environment variables CODEKVAST_BINTRAY_USER and CODEKVAST_BINTRAY_KEY or as values in in ~/.gradle/gradle.properties:

    codekvast.bintray.user=my-bintray-user

    codekvast.bintray.key=my-bintray-key

  6. my-bintray-user must be member of the Crisp organisation at Bintray.

IDE

Intellij Ultimate Edition 2019+ is the recommended IDE with the following plugins:

  1. Lombok Support (required)
  2. Google Java Format (required)
  3. Angular 2 TypeScript Live Templates (optional)
  4. JavaScript Support (optional)
  5. Karma (optional)
  6. Git (optional)
  7. Github (optional)
  8. Docker (optional)

Do like this to open Codekvast in Intellij the first time:

  1. File -> New -> Project from Existing Sources...
  2. Navigate to the project root
  3. Import project from external model...
  4. Select Gradle
  5. Click Next
  6. Accept the defaults (use the project's Gradle wrapper)
  7. Click Finish

After the import, some settings must be changed:

  1. File > Settings...
  2. Build, Execution, Deployment > Compiler > Annotation Processing
  3. Check Enable annotation processing
  4. Click OK

Code Style

Java

The general editor config for IDEA is stored in .editorconfig. For Java and IDEA, The google-java-format plugin is required!

If you use some other IDE, please make sure to format the code in format as close to this as possible. The google-java-format is available for the major IDEs.

Most important rules:

  1. INDENT WITH SPACES!
  2. Indentation: 2 spaces
  3. Line length: 140
  4. Charset: UTF-8

TypeScript

The formatting of TypeScript is described and enforced by tslint.js files in the projects that use TypeScript. IDEA will automatically pick up and apply these settings when found.

How to do semi-manual end-to-end tests

All of the non-trivial code is covered with unit tests.

Some tricky integrations are covered by proper integration tests where the external part is executing in Docker containers managed by the tests.

There is also a smoke test that launches MariaDB and Codekvast Dashboard, and executes some Web Driver tests. This is just a smoke test though.

To assist manual e2e tests, there is a number of sample apps that are managed by Gradle. They are configured to start with the latest Codekvast collector attached.

How to demo

The following procedure can be used for demo purposes and also when doing development with live data flowing.

  1. Launch 4 terminal windows
  2. In terminal #1 do ./gradlew :product:server:dashboard:bootRun. This will start Codekvast Dashboard that will consume the data files uploaded by the instrumented apps.
  3. In terminal #2 do ./gradlew :sample:jenkins1:run. This will download and start one version of Jenkins with Codekvast attached.
  4. In terminal #3 do ./gradlew :sample:jenkins2:run. This will download and start another version of Jenkins with Codekvast attached.
  5. In terminal #4 do ./gradlew :sample:sample-gradle-application:run. This will launch the short-lived sample.app.SampleApp with Codekvast attached. The SampleApp is handy when you want to correlate source code to the data that is collected by Codekvast.
  6. Open a web browser at http://localhost:8081. It will show the dashboard web interface wher e you can inspect the collected data.

How to do rapid development of the dashboard web app

In addition to the above do this:

  1. Launch a terminal window
  2. cd product/server/dashboard/src/webapp
  3. npm start. It will start an embedded web server on port 8088. It reloads changes to the webapp automatically. It will also refresh the browser automatically.
  4. Open the web browser at http://localhost:8088

Canned REST responses for off-line dashboard webapp development

When running the dashboard webapp from npm start there is a number of canned REST responses available. This makes it possible to develop the webapp with nothing else than npm start running.

The canned responses are really handy when doing frontend development, where live data is strictly not necessary.

End-point /webapp/v1/methods

In the Methods page, the canned response is delivered from disk by searching for the signature ----- (five dashes).

Updating the canned responses

Canned responses has to be re-captured every time the dashboard REST API has been changed.

The canned response for /webapp/v1/methods is captured by executing

curl -X GET --header 'Accept: application/json' 'http://localhost:8081/webapp/v1/methods?signature=%25&maxResults=100'|jq . > product/server/dashboard/src/webapp/src/app/test/canned/v1/MethodData.json
git add product/server/dashboard/src/webapp/src/app/test/canned/v1/MethodData.json

from the root directory while ./gradlew :product:server:dashboard:bootRun is running. When doing the capture, make sure that data from the three above mentioned sample apps is stored in the dashboard.

(The JSON response is piped through jq . to make it more pretty for the human eye.)

File watch limits

On some Linux distros, both IntelliJ IDEA and the Node.js uses the system service inotify to watch directories for changed files.

If the limit is to low, npm start will fail.

If you happen to use Ubuntu, here is the remedy:

Create the file /etc/sysctl.d/60-jetbrains.conf with the following content:

# Set inotify watch limit high enough for IntelliJ IDEA (PhpStorm, PyCharm, RubyMine, WebStorm).
# Create this file as /etc/sysctl.d/60-jetbrains.conf (Debian, Ubuntu), and
# run `sudo sysctl --system` or reboot.
# Source: https://confluence.jetbrains.com/display/IDEADEV/Inotify+Watches+Limit
# 
# More information resources:
# man inotify  # manpage
# man sysctl.conf  # manpage
# cat /proc/sys/fs/inotify/max_user_watches  # print current value in use

fs.inotify.max_user_watches = 524288

Then do sudo sysctl --system to activate the changes.