diff --git a/docs/guides/software-lifecycle/continuous-delivery/pypi-cd-template.yml b/docs/guides/software-lifecycle/continuous-delivery/pypi-cd-template.yml new file mode 100644 index 000000000..cf497007d --- /dev/null +++ b/docs/guides/software-lifecycle/continuous-delivery/pypi-cd-template.yml @@ -0,0 +1,38 @@ +name: Continuous Delivery for PyPI Project + +on: + push: + branches: + - main # Trigger on push to the 'main' branch + tags: + - 'v*.*.*' # Trigger on tags matching semantic versioning (v1.0.0) + +jobs: + # Job to set up the environment, install dependencies, and publish to PyPI + publish-to-pypi: + runs-on: ubuntu-latest + + steps: + - name: Checkout repository + uses: actions/checkout@v3 + + - name: Set up Python + uses: actions/setup-python@v3 + with: + python-version: '3.x' # Use a specific Python version, e.g., '3.8', '3.9', etc. + + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install build twine # Required for building and publishing to PyPI + + - name: Build the package + run: | + python -m build # This creates the distribution files under the 'dist' directory + + - name: Publish package to PyPI + env: + TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }} # Store PyPI credentials as GitHub secrets + TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }} + run: | + python -m twine upload dist/* # Uploads the package to PyPI diff --git a/docs/guides/software-lifecycle/continuous-delivery/readme.md b/docs/guides/software-lifecycle/continuous-delivery/readme.md new file mode 100644 index 000000000..d30689452 --- /dev/null +++ b/docs/guides/software-lifecycle/continuous-delivery/readme.md @@ -0,0 +1,270 @@ +--- +title: Continuous Delivery +description: A streamlined guide to setting up a robust continuous delivery pipeline, automating releases, and maintaining best practices. +--- + +# Continuous Delivery + +
A streamlined guide to setting up a robust continuous delivery pipeline, automating releases, and maintaining best practices.
+ + + +## Introduction + +### Background + +Continuous Delivery (CD) is the practice of automatically preparing code changes for production release, extending Continuous Integration (CI) to ensure that every validated change is always production-ready. This guide presents a simplified, practical approach to implementing CD through standardized repository selections, naming conventions, and automation. + +### Approach + +Adopt a clear, four-step plan to implement Continuous Delivery effectively: +1. **Choose repositories**. +2. **Adopt standardized naming conventions**. +3. **Automate publishing**. +4. **Maintain the delivery pipeline**. + +```mermaid +graph TD + Repository[Choose Repositories] --> Naming[Adopt Naming
Conventions] --> Automate[Automate Publishing] + Automate --> Maintain[Maintain Pipeline] + Maintain --> Repository +``` + +### Key Use Cases + +- Auto-publishing built artifacts to package managers. +- Standardizing naming conventions across repositories. +- Versioning releases using semantic versioning. +- Distributing test data automatically. +- Automating container image publication. +- Enabling infrastructure-as-code deployment. + +## Quick Start + +The most important step in setting up continuous delivery is choosing the right repositories and implementing proper naming conventions. + +**Key Concepts to Get Started:** + +**[⬇️ Choose a Package Repository](#package-repositories)** based on your artifact type: +- PyPI for Python packages +- Maven Central for Java +- NPM Registry for NodeJS +- ECR (Amazon Elastic Container Registry)/DockerHub for Containers + +**📝 Implement [Standardized Naming Conventions](#naming-conventions):** +- `nasa-[project-org]-[module-name]` for Python +- `gov.nasa.[project-org].[module-name]` for Java +- `@nasa-[project-org]/[module-name]` for NodeJS + +**🚀 Set up [Automated Publishing](#automated-publishing)** using GitHub Actions + +## Step-by-Step Guide + +### 1. Select Package Repositories + +Choose appropriate repositories based on your artifact type: + +#### 1.1 Code Packages + +##### Python Packages +- **Repository**: PyPI +- **Size Limit**: 60MB +- **Cost**: Free +- **Best For**: Python libraries and tools +- **Setup Steps**: + 1. Create account on PyPI + 2. Set up project with `setup.py` or `pyproject.toml` + 3. Configure automated publishing + +##### Java Packages +- **Repository**: Maven Central +- **Size Limit**: No specific limit +- **Cost**: Free +- **Best For**: Java libraries and frameworks +- **Setup Steps**: + 1. Create Sonatype account + 2. Configure Maven settings + 3. Set up GPG signing + +##### NodeJS Packages +- **Repository**: NPM Registry +- **Size Limit**: No specific limit +- **Cost**: Free +- **Best For**: JavaScript/TypeScript packages +- **Setup Steps**: + 1. Create NPM account + 2. Configure package.json + 3. Set up automated publishing + +#### 1.2 Container Images + +##### Public Containers +- **Repository**: GitHub Packages/GitLab Registry +- **Best For**: Open source projects +- **Limitations**: Higher latency for runtime + +##### Private Containers +- **Repository**: Amazon ECR +- **Best For**: Production deployments +- **Features**: Low-latency pulls, private repos + +#### 1.3 Test Data + +##### Small Datasets (<2GB) +- **Repository**: GitHub/GitLab Releases +- **Naming**: `[project-org]-[project-module]-test-dataset` +- **Best For**: Unit test data, small samples + +##### Medium Datasets (2GB-100GB) +- **Repository**: Amazon S3 +- **Features**: Pre-signed URLs, bandwidth control +- **Best For**: Integration test data + +##### Large Datasets (>100GB) +- **Repository**: EOSDIS DAAC (Earth data) or PDS (Planetary data) +- **Best For**: Mission data, large-scale testing + +### 2. Implement Naming Conventions + +#### 2.1 Package Naming +Follow standard naming conventions for each repository type: + +```mermaid +graph TD + A[Package Type] --> B{Language?} + B -->|Python| C[nasa-project-module] + B -->|Java| D[gov.nasa.project.module] + B -->|NodeJS| E[@nasa-project/module] +``` + +#### 2.2 Version Naming +Use semantic versioning (MAJOR.MINOR.PATCH): +- MAJOR: Breaking changes +- MINOR: New features, backward compatible +- PATCH: Bug fixes + +### 3. Automate Publishing + +#### 3.1 GitHub Actions Workflow + +```yaml +name: Publish Package + +on: + release: + types: [published] + +jobs: + publish: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - name: Set up environment + uses: actions/setup-python@v3 + with: + python-version: '3.x' + - name: Build and publish + env: + TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }} + TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }} + run: | + python -m pip install build twine + python -m build + python -m twine upload dist/* +``` + +#### 3.2 Automated Testing Integration + +```mermaid +graph TD + A[Code Change] --> B[Run Tests] + B --> C{Tests Pass?} + C -->|Yes| D[Build Artifact] + C -->|No| E[Fix Issues] + D --> F[Publish to Repository] +``` + +### 4. Maintain Delivery Pipeline + +Regular maintenance tasks: +1. Update repository credentials +2. Monitor publishing success rates +3. Verify artifact integrity +4. Review and update workflows +5. Clean up old artifacts + +### 5. GitHub Actions Workflow Example for PyPI Project Continuous Delivery +Create a `.github/workflows/pypi-cd.yml` file in your GitHub repository with the following [content](pypi-cd-template.yml): + +```yaml +name: Continuous Delivery for PyPI Project + +on: + push: + branches: + - main # Trigger on push to the 'main' branch + tags: + - 'v*.*.*' # Trigger on tags matching semantic versioning (v1.0.0) + +jobs: + # Job to set up the environment, install dependencies, and publish to PyPI + publish-to-pypi: + runs-on: ubuntu-latest + + steps: + - name: Checkout repository + uses: actions/checkout@v3 + + - name: Set up Python + uses: actions/setup-python@v3 + with: + python-version: '3.x' # Use a specific Python version, e.g., '3.8', '3.9', etc. + + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install build twine # Required for building and publishing to PyPI + + - name: Build the package + run: | + python -m build # This creates the distribution files under the 'dist' directory + + - name: Publish package to PyPI + env: + TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }} # Store PyPI credentials as GitHub secrets + TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }} + run: | + python -m twine upload dist/* # Uploads the package to PyPI +``` + +## Frequently Asked Questions (FAQ) + +**Q: How do I handle dependencies between packages?** + +A: Use semantic versioning and dependency ranges to manage package relationships. + +**Q: What about handling sensitive data in artifacts?** + +A: Use private repositories and encrypted secrets in CI/CD pipelines. + +**Q: How often should artifacts be published?** + +A: Publish on every tagged release for stable versions, and optionally for development versions. + +**Q: How to manage large binary artifacts?** + +A: Use specialized repositories like Amazon S3 for large artifacts and reference them in package metadata. + +## Credits + +**Authorship**: +- [Kyongsik Yun](https://github.com/yunks128) + +**Acknowledgements**: +- Thanks to the SLIM team for providing guidance + +## Feedback and Contributions + +We welcome feedback and contributions to help improve and grow this page. Please see our [contribution guidelines](https://nasa-ammos.github.io/slim/docs/contribute/contributing/).