Skip to content

ringoldsdev/efemel

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Efemel

Python as a functional markup language. Solves YAML scaling issues.

License: MIT Python 3.12+ Built with UV Code style: Ruff


🎯 Overview

Efemel replaces markup templating with native Python. Solves configuration scaling issues without DSLs.

Why Python:

  • Native syntax instead of templating DSLs
  • Built-in validation and IDE support
  • Full programming capabilities
  • Testable configurations

Use Cases:

  • CI/CD pipelines
  • Infrastructure as Code
  • Workflow definitions
  • Docker Compose configs
  • Application settings

✨ Features

Core

  • Python-native configuration
  • Multi-format export (JSON/YAML/TOML)
  • Directory structure preservation
  • Parallel processing
  • Glob pattern support
  • IDE integration

Advanced

  • Environment-specific processing
  • Extensible hook system
  • Auto-validation
  • Testable configurations
  • Live reload

πŸ“¦ Installation

To be filled out once compiled and released.


πŸš€ Usage

Basic

# Single file
efemel process config.py --out output/

# Batch processing
efemel process "**/*.py" --out exported_configs/

# Pick specific keys
efemel process "**/*.py" --out output/ --pick result

# Unwrap values
efemel process "**/*.py" --out output/ --unwrap result
### Advanced Usage

#### Environment-Specific Processing
```bash
# Production environment
efemel process "config/**/*.py" --out prod_config/ --env prod

# Development environment  
efemel process "config/**/*.py" --out dev_config/ --env dev

# Staging with custom working directory
efemel process "*.py" --cwd /app/configs --out staging/ --env staging

Advanced

Performance Tuning

# Control parallel workers
efemel process "**/*.py" --out output/ --workers 4

# Single-threaded processing
efemel process "**/*.py" --out output/ --workers 1

Hook-Based Transformations

# Use hook directory
efemel process "**/*.py" --out output/ --hooks hooks/

Parameter Injection

# Pass parameters to Python scripts during processing
efemel process "config.py" --out output/ --param app_name=myapp --param version=2.0.0 --param debug_mode=true

# Use JSON for complex parameters
efemel process "config.py" --out output/ --param 'database_config={"host":"prod-db","port":5432}'

# Multiple parameters
efemel process "**/*.py" --out output/ \
  --param app_name=myapp \
  --param version=2.0.0 \
  --param debug_mode=true \
  --param port=8080 \
  --param environment=production

# Use params file for complex configurations
efemel process "config.py" --out output/ --params-file params.py

# Combine params file with individual params (individual params override file values)
efemel process "config.py" --out output/ --params-file params.py --param app_name=override-app

Core Patterns & Examples

Pattern 1: Parameter Injection

Input (config.py):

# Parameters are injected as global variables
app_name = globals().get("app_name", "default-app")
port = globals().get("port", 3000)
debug_mode = globals().get("debug_mode", False)

Command:

efemel process config.py --out output/ --param app_name=myapp --param port=8080 --param debug_mode=true

Output:

{
  "app_name": "myapp",
  "port": 8080,
  "debug_mode": true
}

Pattern 2: Basic Data Extraction

Input (app_config.py):

app_name = "my-app"
version = "1.0.0"
port = 8080
debug_mode = True

# Private variables (underscore prefix) are ignored
_internal_secret = "hidden"

Command:

efemel process app_config.py --out output/

Output:

{
  "app_name": "my-app",
  "version": "1.0.0",
  "port": 8080,
  "debug_mode": true
}

Pattern 3: Environment-Specific File Overrides

Input Files:

config.py (default):

workers = 2
timeout = 30

config.prod.py (production override):

workers = 8
timeout = 60

main.py (imports the config):

from config import workers, timeout

server_config = {
    "workers": workers,
    "timeout": timeout
}

Default Output (efemel process main.py --out output/):

{
  "server_config": {
    "workers": 2,
    "timeout": 30
  }
}

Production Output (efemel process main.py --out output/ --env prod):

{
  "server_config": {
    "workers": 8,
    "timeout": 60
  }
}

Pattern 4: Composable Configuration Parts

Input (docker_config.py):

# Base configuration
base_service = {
    "restart": "unless-stopped",
    "networks": ["app-network"]
}

# Specific services using base
web_service = {
    **base_service,
    "image": "nginx:alpine",
    "ports": ["80:80"]
}

api_service = {
    **base_service,
    "image": "python:3.12",
    "ports": ["8080:8080"]
}

# Final composition
services = {
    "web": web_service,
    "api": api_service
}

Command:

efemel process docker_config.py --out output/ --pick services

Output:

{
  "services": {
    "web": {
      "restart": "unless-stopped",
      "networks": ["app-network"],
      "image": "nginx:alpine",
      "ports": ["80:80"]
    },
    "api": {
      "restart": "unless-stopped",
      "networks": ["app-network"],
      "image": "python:3.12",
      "ports": ["8080:8080"]
    }
  }
}

Pattern 5: Parameter Files

Params File (params.py):

app_name = "my-app"
version = "2.0.0"
debug_mode = False
port = 8080

database_config = {
    "host": "prod-db.example.com",
    "port": 5432
}

Config File (config.py):

# Use parameters from params file
app_config = {
    "name": app_name,
    "version": version,
    "debug": debug_mode,
    "port": port
}

db_config = database_config

Command:

efemel process config.py --out output/ --params-file params.py

Output:

{
  "app_config": {
    "name": "my-app",
    "version": "2.0.0",
    "debug": false,
    "port": 8080
  },
  "db_config": {
    "host": "prod-db.example.com",
    "port": 5432
  }
}

βš™οΈ Configuration

Command-Line Options

Option Short Type Required Default Description
FILE_PATTERN - str Yes - Glob pattern for Python files
--out -o str Yes - Output directory path
--cwd -c str No "." Working directory for file operations
--env -e str No None Environment name for imports
--workers -w int No CPU_COUNT Number of parallel workers
--hooks -h str No None Path to hooks file or directory
--flatten -f flag No False Flatten directory structure
--clean - flag No False Clean (delete) the output directory before processing
--dry-run - flag No False Show what would be processed without writing files
--pick -p str No None Pick specific keys from the extracted data (can be used multiple times)
--unwrap -u str No None Extract specific values from the processed data, merging them (can be used multiple times)
--param -P str No None Pass custom parameters to processed scripts in key=value format (can be used multiple times)
--params-file - str No None Path to a Python file that will be processed to extract parameters for other files

Hook Configuration

Create custom transformation hooks in Python:

# hooks/output_filename.py
def add_timestamp(context):
    """Add timestamp to output filenames"""
    from datetime import datetime
    output_path = context['output_file_path']
    timestamp = datetime.now().strftime('%Y%m%d')
    new_name = f"{output_path.stem}_{timestamp}{output_path.suffix}"
    context['output_file_path'] = output_path.with_name(new_name)

πŸ“„ License

This project is licensed under the MIT License - see the LICENSE file for details.


πŸš€ Built With


⭐ Star this repository if Efemel helps your workflow!

About

Use python as a markup language and render results into JSON and YAML

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published