Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Kamal deployment #1192

Draft
wants to merge 20 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
f8b6f48
feat: add Kamal deployment support
yinho999 Jan 19, 2025
bcd0f87
Merge branch 'master' into kamal-deployment
yinho999 Jan 19, 2025
fbb3502
Merge branch 'master' into kamal-deployment
yinho999 Jan 19, 2025
e6bdeb0
Added test for postgres, sqlite without/with background queue with redis
yinho999 Jan 19, 2025
af50d5f
Added binding 0.0.0.0 to development.yaml
yinho999 Jan 19, 2025
328ccae
removed duplicated dockerfile
yinho999 Jan 20, 2025
60fca18
Merge branch 'master' into kamal-deployment
yinho999 Jan 25, 2025
db6cdfd
feat: add .dockerignore and improve deployment file generation
yinho999 Jan 25, 2025
04a1c4c
style: reorganize imports across multiple files
yinho999 Jan 25, 2025
a75bd59
style: reformat doc comments and reorganize imports
yinho999 Jan 25, 2025
2ec6d2a
feat: backup existing deployment files before generation
yinho999 Jan 26, 2025
e04c8b5
feat: add default development.yaml config to deployment tests
yinho999 Jan 26, 2025
efbd6cc
Merge branch 'master' into kamal-deployment
yinho999 Jan 26, 2025
e124196
style: improve code formatting and readability
yinho999 Jan 26, 2025
b974128
refactor: extract scaffold kind determination into separate function
yinho999 Jan 26, 2025
8b3c713
feat: add Kamal deployment support
yinho999 Jan 26, 2025
c35d27f
Merge branch 'master' into kamal-deployment
yinho999 Jan 26, 2025
5852120
refactor: update server binding configuration in deployment templates
yinho999 Jan 26, 2025
2604d33
Merge remote-tracking branch 'origin/kamal-deployment' into kamal-dep…
yinho999 Jan 26, 2025
52240c7
Merge branch 'master' into kamal-deployment
yinho999 Jan 27, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 28 additions & 5 deletions loco-gen/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,15 @@ pub use rrgen::{GenResult, RRgen};
use serde::{Deserialize, Serialize};
use serde_json::{json, Value};
mod controller;
use colored::Colorize;
use std::{
fs,
path::{Path, PathBuf},
str::FromStr,
sync::OnceLock,
};

use colored::Colorize;

#[cfg(feature = "with-db")]
mod infer;
#[cfg(feature = "with-db")]
Expand All @@ -37,6 +38,7 @@ const DEPLOYMENT_OPTIONS: &[(&str, DeploymentKind)] = &[
("Docker", DeploymentKind::Docker),
("Shuttle", DeploymentKind::Shuttle),
("Nginx", DeploymentKind::Nginx),
("Kamal", DeploymentKind::Kamal),
];

#[derive(thiserror::Error, Debug)]
Expand Down Expand Up @@ -137,6 +139,7 @@ pub enum DeploymentKind {
Docker,
Shuttle,
Nginx,
Kamal,
}
impl FromStr for DeploymentKind {
type Err = ();
Expand All @@ -146,6 +149,7 @@ impl FromStr for DeploymentKind {
"docker" => Ok(Self::Docker),
"shuttle" => Ok(Self::Shuttle),
"nginx" => Ok(Self::Nginx),
"kamal" => Ok(Self::Kamal),
_ => Err(()),
}
}
Expand Down Expand Up @@ -210,6 +214,9 @@ pub enum Component {
kind: DeploymentKind,
fallback_file: Option<String>,
asset_folder: Option<String>,
sqlite: bool,
postgres: bool,
background_queue: bool,
Comment on lines +313 to +315
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that none of the parameters (fallback_file, asset_folder, host, port) and also the new params are strictly necessary.
The deployment function should take a reference to &Config (you already have the config in cli.rs), and each deployment type should be able to utilize the configuration that works best for it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaplanelad I was trying to use &loco-rs::config::Config in loco gen project. However, importing loco-rs within loco-gen will cause cyclic dependency

host: String,
port: i32,
},
Expand Down Expand Up @@ -269,6 +276,9 @@ pub fn generate(rrgen: &RRgen, component: Component, appinfo: &AppInfo) -> Resul
kind,
fallback_file,
asset_folder,
sqlite,
postgres,
background_queue,
host,
port,
} => match kind {
Expand Down Expand Up @@ -298,6 +308,17 @@ pub fn generate(rrgen: &RRgen, component: Component, appinfo: &AppInfo) -> Resul
});
render_template(rrgen, Path::new("deployment/nginx"), &vars)?
}
DeploymentKind::Kamal => {
let vars = json!({
"pkg_name": appinfo.app_name,
"copy_asset_folder": asset_folder.unwrap_or_default(),
"fallback_file": fallback_file.unwrap_or_default(),
"sqlite": sqlite,
"postgres": postgres,
"background_queue": background_queue
});
render_template(rrgen, Path::new("deployment/kamal"), &vars)?
}
},
};

Expand Down Expand Up @@ -361,9 +382,10 @@ pub fn collect_messages(results: &GenerateResults) -> String {

/// Copies template files to a specified destination directory.
///
/// This function copies files from the specified template path to the destination directory.
/// If the specified path is `/` or `.`, it copies all files from the templates directory.
/// If the path does not exist in the templates, it returns an error.
/// This function copies files from the specified template path to the
/// destination directory. If the specified path is `/` or `.`, it copies all
/// files from the templates directory. If the path does not exist in the
/// templates, it returns an error.
///
/// # Errors
/// when could not copy the given template path
Expand Down Expand Up @@ -418,8 +440,9 @@ pub fn copy_template(path: &Path, to: &Path) -> Result<Vec<PathBuf>> {

#[cfg(test)]
mod tests {
use std::{io::Cursor, path::Path};

use super::*;
use std::path::Path;

#[test]
fn test_template_not_found() {
Expand Down
4 changes: 4 additions & 0 deletions loco-gen/src/template.rs
Original file line number Diff line number Diff line change
Expand Up @@ -119,9 +119,13 @@ pub mod tests {
use super::*;
use std::path::Path;

/// Returns the first directory in the included templates.
/// # Panics
#[must_use]
pub fn find_first_dir() -> &'static Dir<'static> {
TEMPLATES.dirs().next().expect("first folder")
}
#[must_use]
pub fn find_first_file<'a>(dir: &'a Dir<'a>) -> Option<&'a File<'a>> {
for entry in dir.entries() {
match entry {
Expand Down
154 changes: 154 additions & 0 deletions loco-gen/src/templates/deployment/kamal/deploy.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
to: "config/deploy.yml"
skip_exists: true
message: "Deploy file generated successfully."
---

# Name of your application. Used to uniquely configure containers.
service: {{pkg_name}}

# Name of the container image.
image: docker_username/{{pkg_name}}

# Deploy to these servers.
servers:
web:
- server_ip_address
# job:
# hosts:
# - 192.168.0.1
# cmd: bin/jobs

# Enable SSL auto certification via Let's Encrypt and allow for multiple apps on a single web server.
# Remove this section when using multiple web servers and ensure you terminate SSL at your load balancer.
#
# Note: If using Cloudflare, set encryption mode in SSL/TLS setting to "Full" to enable CF-to-app encryption.
proxy:
ssl: true
host: domain_name
# Proxy connects to your container on port 80 by default.
app_port: 5150
healthcheck:
interval: 3
path: /_health
timeout: 3

# Credentials for your image host.
registry:
# Specify the registry server, if you're not using Docker Hub
# server: registry.digitalocean.com / ghcr.io / ...
username: docker_username

# Always use an access token rather than real password (pulled from .kamal/secrets).
password:
- KAMAL_REGISTRY_PASSWORD

# Configure builder setup.
builder:
arch: amd64
# Pass in additional build args needed for your Dockerfile.
# args:

# Inject ENV variables into containers (secrets come from .kamal/secrets).
#
# env:
# clear:
# DB_HOST: 192.168.0.2
# secret:
# - RAILS_MASTER_KEY
{% if postgres or background_queue %}
env:
clear:
{% endif -%}
{%- if background_queue %}
REDIS_URL: "redis://{{pkg_name}}-redis"
{%- endif %}
{%- if postgres %}
secret:
- DATABASE_URL
- POSTGRES_PASSWORD
{%- endif %}

# Aliases are triggered with "bin/kamal <alias>". You can overwrite arguments on invocation:
# "bin/kamal logs -r job" will tail logs from the first server in the job section.
#
# aliases:
# shell: app exec --interactive --reuse "bash"

# Use a different ssh user than root
#
# ssh:
# user: app

# Use a persistent storage volume.
#
# volumes:
# - "app_storage:/app/storage"
{% if sqlite -%}
# Use a persistent database volume.
volumes:
# /var/lib/docker/volumes/data/_data
- "data:/usr/app"
{% endif -%}


# Bridge fingerprinted assets, like JS and CSS, between versions to avoid
# hitting 404 on in-flight requests. Combines all files from new and old
# version inside the asset_path.
#
# asset_path: /app/public/assets

# Configure rolling deploys by setting a wait time between batches of restarts.
#
# boot:
# limit: 10 # Can also specify as a percentage of total hosts, such as "25%"
# wait: 2

# Use accessory services (secrets come from .kamal/secrets).
#
# accessories:
# db:
# image: mysql:8.0
# host: 192.168.0.2
# port: 3306
# env:
# clear:
# MYSQL_ROOT_HOST: '%'
# secret:
# - MYSQL_ROOT_PASSWORD
# files:
# - config/mysql/production.cnf:/etc/mysql/my.cnf
# - db/production.sql:/docker-entrypoint-initdb.d/setup.sql
# directories:
# - data:/var/lib/mysql
# redis:
# image: valkey/valkey:8
# host: 192.168.0.2
# port: 6379
# directories:
# - data:/data

{% if postgres or background_queue %}
accessories:
{% endif -%}
{%- if postgres %}
db:
image: postgres:16
host: server_ip_address
port: "127.0.0.1:5432:5432"
env:
clear:
POSTGRES_USER: loco
POSTGRES_DB: {{pkg_name}}_production
secret:
- POSTGRES_PASSWORD
directories:
- data:/var/lib/postgresql/data
{% endif %}
{%- if background_queue %}
redis:
image: valkey/valkey:8
host: server_ip_address
port: "127.0.0.1:6379:6379"
directories:
- data:/data
{% endif %}
47 changes: 47 additions & 0 deletions loco-gen/src/templates/deployment/kamal/docker.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
to: "Dockerfile"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we not using the same docker template that we already have?

I don't think we need to hold two docker templates

Copy link
Contributor Author

@yinho999 yinho999 Jan 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current docker template has a different purpose than the one kamal one. As I can see the current dockerfile does not include sea orm and seems to be just compiling the binaries without running it. Not sure if I merge them or leave them separated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The question is, why should SeaORM be included in this docker file.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are using sea-orm-cli for generating entities. Why would you need to generate entities inside the Dockerfile?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My current dockerfile for kamal setup is serving the server within the container instead of just compiling, which requires database connection like sqlite or postgres. Therefore SeaOrm is kinda required?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait sea orm cli is not required for database connection? Okay I will remove it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check if you can use the current docker file and not create a new one

skip_exists: true
message: "Dockerfile generated successfully."

injections:
- into: config/development.yaml
after: " port: 5150"
content: " # Expose Server on all interfaces\n binding: 0.0.0.0"

---

FROM rust:1.84-slim as builder

WORKDIR /usr/src/

COPY . .

RUN cargo build --release

FROM debian:bookworm-slim
# Install required system dependencies
RUN apt-get update && apt-get install -y \
pkg-config \
libpq-dev \
libssl-dev \
curl \
build-essential \
&& rm -rf /var/lib/apt/lists/*

# Install Rust
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"

# Install sea-orm-cli
RUN cargo install sea-orm-cli
WORKDIR /usr/app
{% if copy_asset_folder -%}
COPY --from=builder /usr/src/{{copy_asset_folder}} /usr/app/{{copy_asset_folder}}
{% endif -%}
COPY --from=builder /usr/src/assets/views /usr/app/assets/views
{% if fallback_file -%}
COPY --from=builder /usr/src/{{fallback_file}} /usr/app/{{fallback_file}}
{% endif -%}
COPY --from=builder /usr/src/config /usr/app/config
COPY --from=builder /usr/src/target/release/{{pkg_name}}-cli /usr/app/{{pkg_name}}-cli

ENTRYPOINT ["/usr/app/{{pkg_name}}-cli","start"]
26 changes: 26 additions & 0 deletions loco-gen/src/templates/deployment/kamal/secrets.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
to: ".kamal/secrets"
skip_exists: true
message: "Secrets file generated successfully."
---
# Secrets defined here are available for reference under registry/password, env/secret, builder/secrets,
# and accessories/*/env/secret in config/deploy.yml. All secrets should be pulled from either
# password manager, ENV, or a file. DO NOT ENTER RAW CREDENTIALS HERE! This file needs to be safe for git.

# Option 1: Read secrets from the environment
KAMAL_REGISTRY_PASSWORD=$KAMAL_REGISTRY_PASSWORD
{% if postgres -%}
# example: export POSTGRES_PASSWORD="loco"
POSTGRES_PASSWORD=$POSTGRES_PASSWORD
# example: export DATABASE_URL="postgresql://loco:$POSTGRES_PASSWORD@{{pkg_name}}-db:5432/{{pkg_name}}_production"
DATABASE_URL=$DATABASE_URL
{% endif %}
# Option 2: Read secrets via a command
# RAILS_MASTER_KEY=$(cat config/master.key)

# Option 3: Read secrets via kamal secrets helpers
# These will handle logging in and fetching the secrets in as few calls as possible
# There are adapters for 1Password, LastPass + Bitwarden
#
# SECRETS=$(kamal secrets fetch --adapter 1password --account my-account --from MyVault/MyItem KAMAL_REGISTRY_PASSWORD RAILS_MASTER_KEY)
# KAMAL_REGISTRY_PASSWORD=$(kamal secrets extract KAMAL_REGISTRY_PASSWORD $SECRETS)
# RAILS_MASTER_KEY=$(kamal secrets extract RAILS_MASTER_KEY $SECRETS)
Loading
Loading