Add Kubernetes support

[ferranbt]

[fnerdman]

Ferran, I've worked on this a while ago, this is what I came up in terms of understanding where kubernetes differs from the current docker-compose implementation and what could be done about it:

Kubernetes Support for Builder Playground: Local Testing Focus

Overview

Builder Playground is currently optimized for Docker Compose, with no existing Kubernetes runner implementation. We need to create a new Kubernetes deployment approach focused on local testing environments (Minikube, K3s, Kind) rather than production deployments.

Key Challenges for Local Kubernetes Testing

1. Lifecycle Management & Command Flow

• Interactive Testing Experience: Need to maintain a developer-friendly experience while working with Kubernetes' declarative model
• Command Simplicity: Current unified cook command should have a Kubernetes equivalent that handles the common testing workflow
• Quick Iteration: Developers need fast setup/teardown cycles for testing blockchain components

2. Local Storage & Configuration

• Shared Directory Access: Finding simpler alternatives to the common output directory ({{.Dir}}) approach for local K8s
• Local Testing Files:
  • ConfigMaps have size limits (~1MB) that may affect validator keystores and genesis files
  • Need straightforward handling of file permissions for local testing
• Development-Friendly Storage: Balancing proper Kubernetes resources with ease of debugging/modification

3. Simplified Networking for Testing

• Service Port Requirements: Creating services for components that may not expose ports in the Docker Compose version
• Local Service Discovery: Adapting the templating system ({{Service "name" "port"}}) for test clusters
• Local Access: Ensuring developers can easily access services from their host machine

4. Developer Experience

• Log Access: Straightforward access to logs for debugging test environments
• State Persistence: Simple persistence for test data between restarts
• Cleanup: Easy cleanup of resources after testing

Pragmatic Implementation Approach for Local Testing

1. Maintain Simple Command Structure

• Keep a single entry point similar to cook but with Kubernetes backend
• Add simple flags like --keep-resources to retain deployments between runs
• Focus on developer-friendly output and monitoring

2. Local Storage Simplifications

• Use HostPath volumes where possible for easier debugging
• Bundle related files into single ConfigMaps by component
• Use emptyDir for ephemeral test data
• Skip complex PersistentVolumeClaims for local testing scenarios

3. Simplified Networking

• Auto-expose key ports via NodePort for local access
• Use standard Kubernetes DNS patterns for service discovery
• Provide helper output showing how to access services (URLs, ports)

4. Testing-Focused Implementation

• Focus on minikube/k3s/kind compatibility
• Include proper cleanup of resources
• Prioritize ease of debugging over production best practices
• Use kubectl port-forward when needed for service access

Implementation Path

1. Create New Kubernetes Runner:

   • Design for local testing environments specifically
   • Focus on working with local Kubernetes environments

2. Simplify Storage Strategy:

   • Map common directories to ConfigMaps/Secrets
   • Use emptyDir volumes for ephemeral state
   • Consider HostPath for local debugging (with appropriate security context)

3. Streamline Service Access:

   • Automatically configure NodePort services for key endpoints
   • Provide connection information after deployment
   • Create standard service naming patterns

4. Developer Tooling:

   • Add shortcuts for log access
   • Create simple resource cleanup
   • Provide status view of running components

Questions

1. Which local Kubernetes environment should we prioritize? (minikube, k3s, kind, etc.)
2. Is consistency with Docker Compose experience important, or can the Kubernetes workflow be different?
3. Should we use the Kubernetes Operator pattern to manage the deployment lifecycle, or keep it simpler with standard resources?
4. Should we support resource limitations for accurate testing, or assume development machines have sufficient resources?
5. How important is secure handling of validator keys in a local testing context?

Next Steps

1. Create a new Kubernetes runner implementation focused on local testing
2. Test with minikube to ensure basic functionality
3. Implement simplified volume mounting strategy
4. Add user-friendly service discovery and port access
5. Document local Kubernetes setup instructions
6. Add CI testing for the Kubernetes deployment path

[ferranbt]

This is my vision for this feature

The primary value of Builder Playground is providing a seamless local development experience for block building testing via Docker Compose. This remains our core focus and recommended approach for local development.

However, there is value in extending support to Kubernetes for specific use cases - not as a replacement for the local development workflow, but as an option for more persistent deployments.

Unlike our Docker Compose support where we control the full lifecycle (starting, stopping, monitoring), I don't envision the same tight lifecycle management for Kubernetes deployments. Instead, I would do something like this:

• Leverage our existing artifact generation capabilities (already done)
• Generate the manifest from the recipes (already done)
• Generate Kubernetes YAML files from that manifest that users can deploy independently.

In CLI it would look like this:

$ playground cook opstack --dry-run --output ./output
$ playground k8s-generate --data ./output
$ ... deploy the K8s yaml files with kubectl

[fnerdman]

Ok, I've collected everyones feedback and took a few hours to design a possible solution and then lay the first bricks: #120