19 KiB
| title | linkTitle | weight | description |
|---|---|---|---|
| Stacks | Stacks | 40 | Platform-level component provisioning via Stacks |
Overview
The stacks and stacks-instances repositories form the core of a GitOps-based system for provisioning Edge Developer Platforms (EDP). They implement a template-instance pattern that enables the deployment of reusable platform components across different environments. The concept of "stacks" originates from the CNOE.io project (Cloud Native Operational Excellence), which can be traced through the evolutionary development from edpbuilder (derived from CNOE.io's EDPbuilder) to infra-deploy.
Key Features of the Everything-as-Code Stacks Approach
This declarative Stacks provisioning architecture is characterized by the following central properties:
Complete Code Declaration
Platform as Code: All Kubernetes resources, Helm charts, and application manifests are declaratively versioned as YAML files. The entire platform topology is traceable in Git.
Configuration as Code: Environment-specific configurations are generated through template hydration, not manually edited. Gomplate transforms generic templates into concrete configurations.
GitOps-Native Architecture
Single Source of Truth: Git is the sole source of truth for the desired state of all infrastructure and platform components.
Declarative State Management: ArgoCD continuously synchronizes the actual state with the desired state defined in Git. Deviations are automatically corrected.
Audit Trail: Every change to infrastructure or platform is documented through Git commits, with author, timestamp, and change description.
Pull-based Deployment: ArgoCD pulls changes from Git, rather than external systems requiring push access to the cluster. This significantly increases security.
Template-Instance Separation
DRY Principle (Don't Repeat Yourself): Common platform components are defined once as templates and reused for all environments.
Environment Promotion: New environments can be quickly created through template hydration. Consistency across environments is guaranteed.
Centralized Maintainability: Updates to stack definitions can be made centrally in the stacks repository and then selectively rolled out to instances.
Customization Points: Despite reuse, environment-specific customizations remain possible through values files and manifest overlays.
Modular Composition
Stack-based Architecture: Platform capabilities are organized into independent, reusable stacks (core, otc, forgejo, observability).
Selective Deployment: Through the STACKS environment variable, only required components can be deployed selectively.
Mix-and-Match: Different stack combinations yield different platform profiles (Development, Production, Observability clusters).
Pluggable Components: New stacks can be added without modifying existing ones.
Environment Agnosticism
Cloud Provider Abstraction: Templates are formulated generically. Provider-specific details are introduced through hydration.
Multi-Cloud Ready: The architecture supports various cloud providers (currently OTC, historically KIND, extensible to AWS/Azure/GCP).
Environment Variables as Interface: All environment-specific aspects are controlled through clearly defined environment variables.
Portable Definitions: Stack definitions can be ported between environments and even cloud providers.
Self-Healing and Drift Detection
Automated Reconciliation: ArgoCD detects deviations from the desired state and corrects them automatically.
Continuous Monitoring: Permanent monitoring of cluster state compared to Git definition.
Declarative State Recovery: After failures or manual changes, the declared state is automatically restored.
Sync Policies: Configurable sync strategies (automated, manual, with pruning) per application.
Secrets Management
Secrets Outside Git: Sensitive data is not stored in Git but generated at runtime or injected from secret stores.
Generated Credentials: Passwords, tokens, and secrets are generated during deployment and directly created as Kubernetes Secrets.
Sealed Secrets Ready: The architecture is compatible with Sealed Secrets or External Secrets Operators for encrypted secret storage in Git.
Credential Rotation: Secrets can be regenerated through re-deployment.
Observability and Auditability
Declarative Monitoring: Observability stacks are part of the Platform-as-Code definition.
Deployment History: Complete history of all deployments and changes through Git log.
ArgoCD UI: Graphical representation of sync status and application topology.
Infrastructure Events: Terraform state changes and Terragrunt outputs document infrastructure changes.
Idempotence and Reproducibility
Idempotent Operations: Repeated execution of the same declaration leads to the same result without side effects.
Deterministic Builds: Same input parameters (Git commit + environment variables) produce identical environments.
Disaster Recovery: Complete environments can be rebuilt from code without restoring backups.
Testing in Production-Like Environments: Development and staging environments are code-identical to production, only with different parameter values.
Purpose in EDP
A 'stack' is the declarative description for the platform provisionning in an EDP installation.
Repository
Code:
Documentation: [Link to component-specific documentation]
The stacks Repository
Purpose and Structure
The stacks repository contains reusable template definitions for platform components. It serves as a central library of building blocks from which Edge Developer Platforms can be composed.
stacks/
└── template/
├── edfbuilder.yaml
├── registry/
│ ├── core.yaml
│ ├── otc.yaml
│ ├── forgejo.yaml
│ ├── observability.yaml
│ └── observability-client.yaml
└── stacks/
├── core/
├── otc/
├── forgejo/
├── observability/
└── observability-client/
Components
edfbuilder.yaml: The central bootstrap definition. This is an ArgoCD Application that references the registry directory and serves as the entry point for the entire platform provisioning.
registry/: Contains ArgoCD ApplicationSets that function as a meta-layer. Each file defines a category of stacks (e.g., core, forgejo, observability) and references the corresponding subdirectory in stacks/.
stacks/: The actual platform components, organized into thematic categories:
- core: Fundamental components such as ArgoCD, CloudNative PostgreSQL, Dex (SSO)
- otc: Cloud-provider-specific components for Open Telekom Cloud (cert-manager, ingress-nginx, StorageClasses)
- forgejo: Git server and CI runners
- observability: Central observability components (Grafana, Victoria Metrics Stack)
- observability-client: Client-side metrics collection for non-observability clusters
Each stack consists of:
- YAML definitions (primarily ArgoCD Applications)
values.yamlfiles for Helm chartsmanifests/directories for additional Kubernetes resources
Templating Mechanism
The templates use Gomplate with delimiter syntax {{{ }}} for environment variables:
repoURL: "https://{{{ .Env.CLIENT_REPO_DOMAIN }}}/{{{ .Env.CLIENT_REPO_ORG_NAME }}}"
path: "{{{ .Env.CLIENT_REPO_ID }}}/{{{ .Env.DOMAIN }}}/stacks/core"
These placeholders are replaced with environment-specific values during the deployment phase.
The stacks-instances Repository
Purpose and Structure
The stacks-instances repository contains the materialized, environment-specific configurations. While stacks provides the blueprints, stacks-instances contains the actual deployment definitions for concrete environments.
stacks-instances/
└── otc/
├── osctest.t09.de/
│ ├── edfbuilder.yaml
│ ├── registry/
│ └── stacks/
├── backup-test-manu.t09.de/
│ ├── edfbuilder.yaml
│ ├── registry/
│ └── stacks/
└── ...
Organizational Principle
The structure follows the schema {cloud-provider}/{domain}/:
- cloud-provider: Identifies the cloud environment (e.g.,
otcfor Open Telekom Cloud) - domain: The fully qualified domain name of the environment (e.g.,
osctest.t09.de)
Each environment replicates the structure of stacks/template, but with resolved template variables and environment-specific customizations.
Usage by ArgoCD
ArgoCD synchronizes directly from this repository. Applications reference paths such as:
source:
path: "otc/osctest.t09.de/stacks/core"
repoURL: "https://edp.buildth.ing/DevFW-CICD/stacks-instances"
targetRevision: HEAD
This enables true GitOps: every change to the configurations is traceable through Git commits and automatically synchronized by ArgoCD in the target environment.
The infra-deploy Repository
Role in the Overall Architecture
The infra-deploy repository is the orchestration layer that coordinates both infrastructure and platform provisioning. It represents the evolution of edpbuilder, which was originally derived from the CNOE.io project's EDPbuilder.
Two-Phase Provisioning
Phase 1: Infrastructure Provisioning
Uses Terragrunt Stacks (experimental feature) to provision cloud resources:
infra-deploy/
├── root.hcl
├── non-prod/
│ ├── tenant.hcl
│ ├── dns_zone/
│ │ ├── terragrunt.hcl
│ │ ├── terragrunt.stack.hcl
│ │ └── terragrunt.values.hcl
│ └── testing/
├── prod/
└── templates/
└── forgejo/
├── terragrunt.hcl
└── terragrunt.stack.hcl
Terragrunt Stacks provision:
- VPC and network segments
- Kubernetes clusters (CCE on OTC)
- Managed databases (RDS PostgreSQL)
- Load balancers and DNS entries
- Security groups and other cloud resources
Phase 2: Platform Provisioning
The script scripts/edp-install.sh executes the following steps:
-
Template Hydration:
- Checkout of the
stacksrepository - Execution of Gomplate to resolve template variables
- Generation of environment-specific manifests
- Checkout of the
-
Instance Management:
- Checkout/update of the
stacks-instancesrepository - During CI execution: commit and push of the new instance
- Checkout/update of the
-
Secrets Management:
- Generation of credentials (database passwords, SSO secrets, API tokens)
- Creation of Kubernetes Secrets
-
Bootstrap:
- Helm-based installation of ArgoCD
- Application of
edfbuilder.yamlor selective registry entries
-
GitOps Handover:
- ArgoCD takes over further synchronization from
stacks-instances - Continuous monitoring and self-healing
- ArgoCD takes over further synchronization from
GitHub Actions Workflows
The .github/workflows/ directory contains three central workflows:
deploy.yaml: Complete deployment pipeline with the following inputs:
- Cluster environment and tenant (prod/non-prod)
- Node flavor and availability zone
- Stack selection (core, otc, forgejo, observability, etc.)
- Infra-catalogue version
plan.yaml: Terraform/Terragrunt plan preview without execution
destroy.yaml: Controlled teardown of environments
Deployment Workflow
The complete provisioning process proceeds as follows:
-
Initiation: GitHub Actions workflow is triggered (manually or automatically)
-
Environment Preparation:
export CLUSTER_ENVIRONMENT=qa-stage cd scripts ./new-otc-env.sh # Creates Terragrunt configuration if new -
Infrastructure Provisioning:
./ensure-cluster.sh otc # Internally executes: # - ./ensure-otc-cluster.sh # - terragrunt stack run apply -
Platform Provisioning:
./edp-install.sh # Executes: # - Checkout of stacks # - Gomplate hydration # - Checkout/update of stacks-instances # - Secrets generation # - ArgoCD installation # - Bootstrap of stacks -
ArgoCD Synchronization: ArgoCD continuously reads from
stacks-instancesand synchronizes the desired state
The CNOE.io Stacks Concept
The term "stacks" originates from the Cloud Native Operational Excellence (CNOE.io) project. The core idea is the composition of platform capabilities from modular, reusable building blocks.
Principles
Modularity: Each stack is a self-contained unit with clear dependencies
Composability: Stacks can be freely combined to create different platform profiles
Declarativeness: All configurations are declarative and GitOps-capable
Environment-agnostic: Templates are generic; environment specifics are introduced through hydration
Stack Selection and Combinations
The environment variable STACKS controls which components are deployed:
# Complete EDP with central observability
STACKS="core,otc,forgejo,observability"
# Application cluster with client-side observability
STACKS="core,otc,forgejo,observability-client"
# Minimal development environment
STACKS="core,forgejo"
Data Flow and Dependencies
┌─────────────────┐
│ GitHub Actions │
│ (deploy.yaml) │
└────────┬────────┘
│
├─> Phase 1: Infrastructure
│ ┌──────────────────┐
│ │ infra-deploy │
│ │ (Terragrunt) │
│ └────────┬─────────┘
│ │
│ v
│ ┌──────────────────┐
│ │ Cloud Provider │
│ │ (OTC) │
│ │ - VPC │
│ │ - K8s Cluster │
│ │ - RDS │
│ └──────────────────┘
│
└─> Phase 2: Platform
┌──────────────────┐
│ edp-install.sh │
└────────┬─────────┘
│
├─> Checkout: stacks (Templates)
│ └─> Gomplate Hydration
│
├─> Checkout/Update: stacks-instances
│
├─> Secrets Generation
│
├─> ArgoCD Installation (Helm)
│
└─> Bootstrap (edfbuilder.yaml)
│
v
┌────────────────┐
│ ArgoCD │
└────────┬───────┘
│
└─> Continuous Synchronization
from stacks-instances
│
v
┌──────────────┐
│ Kubernetes │
│ Cluster │
└──────────────┘
Historical Context: edpbuilder to infra-deploy
The evolution from edpbuilder to infra-deploy demonstrates the maturation of the architecture:
edpbuilder (Origin):
- Directly derived from CNOE.io's
EDPbuilder - Focus on local KIND clusters
- Manual configuration
- Monolithic structure
infra-deploy (Current):
- Production-ready for cloud deployments (OTC)
- Terragrunt-based infrastructure orchestration
- CI/CD integration via GitHub Actions
- Clear separation between infrastructure and platform
- Template-instance separation through stacks/stacks-instances
Technical Particularities
Gomplate Templating
Gomplate is used with custom delimiters {{{ }}} to avoid conflicts with Helm templating ({{ }}):
gomplate --input-dir="stacks/template" \
--output-dir="work" \
--left-delim "{{{" \
--right-delim "}}}"
Terragrunt Experimental Stacks
The use of Terragrunt Stacks requires the experimental flag:
export TG_EXPERIMENT_MODE=true
terragrunt stack run apply
This enables hierarchical organization of Terraform modules with dependency management.
ArgoCD ApplicationSets
The registry pattern uses ArgoCD Applications that reference directories:
source:
path: "otc/osctest.t09.de/stacks/core"
ArgoCD automatically detects all YAML files in the path and synchronizes them as Applications.
Best Practices and Patterns
Immutable Infrastructure: Every environment is fully defined in Git
Secrets Outside Git: Sensitive data is generated at runtime or injected from secret stores
Progressive Rollouts: New environments start as template instances, then are individually customized
Version Pinning: Critical components (Helm charts, Terragrunt modules) are pinned to specific versions
Namespace Isolation: Each stack deploys into dedicated namespaces
Self-Healing: ArgoCD's automated sync policy enables automatic drift correction
Usage Examples
Deployment by Pipeline
The platform deployment is the second part of the EDP installtaion. First there is the infrastructure setup, which ends with a created kubernetes cluster. Then the platform provisioning by the defined stacks is done. Both is runnable by the deploypipelien in infra-deploy:
The green pipeline looks liek this:
Local setup with 'kind'
It's also possible to just run the second part, the stcks provisionning. Then you need to have a kubernetes cluster already running, which is e.g. feasable by a local kind-cluster.
So imagine, you want to to the stacks 'core,observability' on your local machine. Then you can run the local entzr
# have kind insatlled
# in /infra-deploy
# provide a kind cluster
kind delete clusters --all
./scripts/ensure-kind-cluster.sh -r
# provide some emnv vars
export TERRAFORM=/bin/bash
export LOADBALANCER_ID=ABC
export DOMAIN=ABC
export DOMAIN_GITEA=ABC
export OS_ACCESS_KEY=ABC
export OS_SECRET_KEY=ABC
export STACKS=core,observability
# deploy
./scripts/edp-install.sh
Status
Maturity: [Production]