---
title: "Stacks"
linkTitle: "Stacks"
weight: 40
description: >
  Platform-level component provisioning via Stacks
---

{{% alert title="Draft" color="" %}}
**Editorial Status**: This page is currently being developed.

* **Jira Ticket**: [TICKET-6729](https://jira.telekom-mms.com/browse/IPCEICIS-6729)
* **Assignee**: Stephan
* **Status**: Waiting for Review
* **Last Updated**: 2025-11-30
* **TODO**:
  * [ ] Review and finalize content
{{% /alert %}}

## Overview

The `stacks` and `stacks-instances` repositories form the core of a GitOps-based system for provisioning Edge Developer Platforms (EDP). They implement a template-instance pattern that enables the deployment of reusable platform components across different environments. The concept of "stacks" originates from the CNOE.io project (Cloud Native Operational Excellence), which can be traced through the evolutionary development from `edpbuilder` (derived from CNOE.io's `EDPbuilder`) to `infra-deploy`.

## Key Features of the Everything-as-Code Stacks Approach

This declarative Stacks provisioning architecture is characterized by the following central properties:

### Complete Code Declaration

**Platform as Code**: All Kubernetes resources, Helm charts, and application manifests are declaratively versioned as YAML files. The entire platform topology is traceable in Git.

**Configuration as Code**: Environment-specific configurations are generated through template hydration, not manually edited. Gomplate transforms generic templates into concrete configurations.

### GitOps-Native Architecture

**Single Source of Truth**: Git is the sole source of truth for the desired state of all infrastructure and platform components.

**Declarative State Management**: ArgoCD continuously synchronizes the actual state with the desired state defined in Git. Deviations are automatically corrected.

**Audit Trail**: Every change to infrastructure or platform is documented through Git commits, with author, timestamp, and change description.

**Pull-based Deployment**: ArgoCD pulls changes from Git, rather than external systems requiring push access to the cluster. This significantly increases security.

### Template-Instance Separation

**DRY Principle (Don't Repeat Yourself)**: Common platform components are defined once as templates and reused for all environments.

**Environment Promotion**: New environments can be quickly created through template hydration. Consistency across environments is guaranteed.

**Centralized Maintainability**: Updates to stack definitions can be made centrally in the `stacks` repository and then selectively rolled out to instances.

**Customization Points**: Despite reuse, environment-specific customizations remain possible through values files and manifest overlays.

### Modular Composition

**Stack-based Architecture**: Platform capabilities are organized into independent, reusable stacks (core, otc, forgejo, observability).

**Selective Deployment**: Through the `STACKS` environment variable, only required components can be deployed selectively.

**Mix-and-Match**: Different stack combinations yield different platform profiles (Development, Production, Observability clusters).

**Pluggable Components**: New stacks can be added without modifying existing ones.

### Environment Agnosticism

**Cloud Provider Abstraction**: Templates are formulated generically. Provider-specific details are introduced through hydration.

**Multi-Cloud Ready**: The architecture supports various cloud providers (currently OTC, historically KIND, extensible to AWS/Azure/GCP).

**Environment Variables as Interface**: All environment-specific aspects are controlled through clearly defined environment variables.

**Portable Definitions**: Stack definitions can be ported between environments and even cloud providers.

### Self-Healing and Drift Detection

**Automated Reconciliation**: ArgoCD detects deviations from the desired state and corrects them automatically.

**Continuous Monitoring**: Permanent monitoring of cluster state compared to Git definition.

**Declarative State Recovery**: After failures or manual changes, the declared state is automatically restored.

**Sync Policies**: Configurable sync strategies (automated, manual, with pruning) per application.

### Secrets Management

**Secrets Outside Git**: Sensitive data is not stored in Git but generated at runtime or injected from secret stores.

**Generated Credentials**: Passwords, tokens, and secrets are generated during deployment and directly created as Kubernetes Secrets.

**Sealed Secrets Ready**: The architecture is compatible with Sealed Secrets or External Secrets Operators for encrypted secret storage in Git.

**Credential Rotation**: Secrets can be regenerated through re-deployment.

### Observability and Auditability

**Declarative Monitoring**: Observability stacks are part of the Platform-as-Code definition.

**Deployment History**: Complete history of all deployments and changes through Git log.

**ArgoCD UI**: Graphical representation of sync status and application topology.

**Infrastructure Events**: Terraform state changes and Terragrunt outputs document infrastructure changes.

### Idempotence and Reproducibility

**Idempotent Operations**: Repeated execution of the same declaration leads to the same result without side effects.

**Deterministic Builds**: Same input parameters (Git commit + environment variables) produce identical environments.

**Disaster Recovery**: Complete environments can be rebuilt from code without restoring backups.

**Testing in Production-Like Environments**: Development and staging environments are code-identical to production, only with different parameter values.

## Purpose in EDP

A 'stack' is the declarative description for the platform provisionning in an EDP installation.

## Repository

**Code**: 
* [Stacks Templates Repo](https://edp.buildth.ing/DevFW-CICD/stacks)
* [Stacks Instances Repo, used for ArgoCD Gitops](https://edp.buildth.ing/DevFW-CICD/stacks-instances)
* [EDP Stacks Deployment mechanism](https://edp.buildth.ing/DevFW/infra-deploy)

**Documentation**: [Link to component-specific documentation]
* [Outdated: The former 'edpbuilder' as script, derived from CNOE's 'idpbuilder](https://edp.buildth.ing/DevFW/edpbuilder)


## The stacks Repository

### Purpose and Structure

The `stacks` repository contains reusable template definitions for platform components. It serves as a central library of building blocks from which Edge Developer Platforms can be composed.

```
stacks/
└── template/
    ├── edfbuilder.yaml
    ├── registry/
    │   ├── core.yaml
    │   ├── otc.yaml
    │   ├── forgejo.yaml
    │   ├── observability.yaml
    │   └── observability-client.yaml
    └── stacks/
        ├── core/
        ├── otc/
        ├── forgejo/
        ├── observability/
        └── observability-client/
```

### Components

**edfbuilder.yaml**: The central bootstrap definition. This is an ArgoCD Application that references the `registry` directory and serves as the entry point for the entire platform provisioning.

**registry/**: Contains ArgoCD ApplicationSets that function as a meta-layer. Each file defines a category of stacks (e.g., core, forgejo, observability) and references the corresponding subdirectory in `stacks/`.

**stacks/**: The actual platform components, organized into thematic categories:

- **core**: Fundamental components such as ArgoCD, CloudNative PostgreSQL, Dex (SSO)
- **otc**: Cloud-provider-specific components for Open Telekom Cloud (cert-manager, ingress-nginx, StorageClasses)
- **forgejo**: Git server and CI runners
- **observability**: Central observability components (Grafana, Victoria Metrics Stack)
- **observability-client**: Client-side metrics collection for non-observability clusters

Each stack consists of:
- YAML definitions (primarily ArgoCD Applications)
- `values.yaml` files for Helm charts
- `manifests/` directories for additional Kubernetes resources

### Templating Mechanism

The templates use Gomplate with delimiter syntax `{{{ }}}` for environment variables:

```yaml
repoURL: "https://{{{ .Env.CLIENT_REPO_DOMAIN }}}/{{{ .Env.CLIENT_REPO_ORG_NAME }}}"
path: "{{{ .Env.CLIENT_REPO_ID }}}/{{{ .Env.DOMAIN }}}/stacks/core"
```

These placeholders are replaced with environment-specific values during the deployment phase.

## The stacks-instances Repository

### Purpose and Structure

The `stacks-instances` repository contains the materialized, environment-specific configurations. While `stacks` provides the blueprints, `stacks-instances` contains the actual deployment definitions for concrete environments.

```
stacks-instances/
└── otc/
    ├── osctest.t09.de/
    │   ├── edfbuilder.yaml
    │   ├── registry/
    │   └── stacks/
    ├── backup-test-manu.t09.de/
    │   ├── edfbuilder.yaml
    │   ├── registry/
    │   └── stacks/
    └── ...
```

### Organizational Principle

The structure follows the schema `{cloud-provider}/{domain}/`:

- **cloud-provider**: Identifies the cloud environment (e.g., `otc` for Open Telekom Cloud)
- **domain**: The fully qualified domain name of the environment (e.g., `osctest.t09.de`)

Each environment replicates the structure of `stacks/template`, but with resolved template variables and environment-specific customizations.

### Usage by ArgoCD

ArgoCD synchronizes directly from this repository. Applications reference paths such as:

```yaml
source:
  path: "otc/osctest.t09.de/stacks/core"
  repoURL: "https://edp.buildth.ing/DevFW-CICD/stacks-instances"
  targetRevision: HEAD
```

This enables true GitOps: every change to the configurations is traceable through Git commits and automatically synchronized by ArgoCD in the target environment.

## The infra-deploy Repository

### Role in the Overall Architecture

The `infra-deploy` repository is the orchestration layer that coordinates both infrastructure and platform provisioning. It represents the evolution of `edpbuilder`, which was originally derived from the CNOE.io project's `EDPbuilder`.

### Two-Phase Provisioning

**Phase 1: Infrastructure Provisioning**

Uses Terragrunt Stacks (experimental feature) to provision cloud resources:

```
infra-deploy/
├── root.hcl
├── non-prod/
│   ├── tenant.hcl
│   ├── dns_zone/
│   │   ├── terragrunt.hcl
│   │   ├── terragrunt.stack.hcl
│   │   └── terragrunt.values.hcl
│   └── testing/
├── prod/
└── templates/
    └── forgejo/
        ├── terragrunt.hcl
        └── terragrunt.stack.hcl
```

Terragrunt Stacks provision:
- VPC and network segments
- Kubernetes clusters (CCE on OTC)
- Managed databases (RDS PostgreSQL)
- Load balancers and DNS entries
- Security groups and other cloud resources

**Phase 2: Platform Provisioning**

The script `scripts/edp-install.sh` executes the following steps:

1. **Template Hydration**: 
   - Checkout of the `stacks` repository
   - Execution of Gomplate to resolve template variables
   - Generation of environment-specific manifests

2. **Instance Management**:
   - Checkout/update of the `stacks-instances` repository
   - During CI execution: commit and push of the new instance

3. **Secrets Management**:
   - Generation of credentials (database passwords, SSO secrets, API tokens)
   - Creation of Kubernetes Secrets

4. **Bootstrap**:
   - Helm-based installation of ArgoCD
   - Application of `edfbuilder.yaml` or selective registry entries

5. **GitOps Handover**:
   - ArgoCD takes over further synchronization from `stacks-instances`
   - Continuous monitoring and self-healing

### GitHub Actions Workflows

The `.github/workflows/` directory contains three central workflows:

**deploy.yaml**: Complete deployment pipeline with the following inputs:
- Cluster environment and tenant (prod/non-prod)
- Node flavor and availability zone
- Stack selection (core, otc, forgejo, observability, etc.)
- Infra-catalogue version

**plan.yaml**: Terraform/Terragrunt plan preview without execution

**destroy.yaml**: Controlled teardown of environments

## Deployment Workflow

The complete provisioning process proceeds as follows:

1. **Initiation**: GitHub Actions workflow is triggered (manually or automatically)

2. **Environment Preparation**:
   ```bash
   export CLUSTER_ENVIRONMENT=qa-stage
   cd scripts
   ./new-otc-env.sh  # Creates Terragrunt configuration if new
   ```

3. **Infrastructure Provisioning**:
   ```bash
   ./ensure-cluster.sh otc
   # Internally executes:
   # - ./ensure-otc-cluster.sh
   # - terragrunt stack run apply
   ```

4. **Platform Provisioning**:
   ```bash
   ./edp-install.sh
   # Executes:
   # - Checkout of stacks
   # - Gomplate hydration
   # - Checkout/update of stacks-instances
   # - Secrets generation
   # - ArgoCD installation
   # - Bootstrap of stacks
   ```

5. **ArgoCD Synchronization**: ArgoCD continuously reads from `stacks-instances` and synchronizes the desired state

## The CNOE.io Stacks Concept

The term "stacks" originates from the Cloud Native Operational Excellence (CNOE.io) project. The core idea is the composition of platform capabilities from modular, reusable building blocks.

### Principles

**Modularity**: Each stack is a self-contained unit with clear dependencies

**Composability**: Stacks can be freely combined to create different platform profiles

**Declarativeness**: All configurations are declarative and GitOps-capable

**Environment-agnostic**: Templates are generic; environment specifics are introduced through hydration

### Stack Selection and Combinations

The environment variable `STACKS` controls which components are deployed:

```bash
# Complete EDP with central observability
STACKS="core,otc,forgejo,observability"

# Application cluster with client-side observability
STACKS="core,otc,forgejo,observability-client"

# Minimal development environment
STACKS="core,forgejo"
```

## Data Flow and Dependencies

```
┌─────────────────┐
│  GitHub Actions │
│  (deploy.yaml)  │
└────────┬────────┘
         │
         ├─> Phase 1: Infrastructure
         │   ┌──────────────────┐
         │   │  infra-deploy    │
         │   │  (Terragrunt)    │
         │   └────────┬─────────┘
         │            │
         │            v
         │   ┌──────────────────┐
         │   │  Cloud Provider  │
         │   │  (OTC)           │
         │   │  - VPC           │
         │   │  - K8s Cluster   │
         │   │  - RDS           │
         │   └──────────────────┘
         │
         └─> Phase 2: Platform
             ┌──────────────────┐
             │  edp-install.sh  │
             └────────┬─────────┘
                      │
                      ├─> Checkout: stacks (Templates)
                      │   └─> Gomplate Hydration
                      │
                      ├─> Checkout/Update: stacks-instances
                      │
                      ├─> Secrets Generation
                      │
                      ├─> ArgoCD Installation (Helm)
                      │
                      └─> Bootstrap (edfbuilder.yaml)
                          │
                          v
                 ┌────────────────┐
                 │    ArgoCD      │
                 └────────┬───────┘
                          │
                          └─> Continuous Synchronization
                              from stacks-instances
                              │
                              v
                       ┌──────────────┐
                       │  Kubernetes  │
                       │  Cluster     │
                       └──────────────┘
```

## Historical Context: edpbuilder to infra-deploy

The evolution from `edpbuilder` to `infra-deploy` demonstrates the maturation of the architecture:

**edpbuilder** (Origin):
- Directly derived from CNOE.io's `EDPbuilder`
- Focus on local KIND clusters
- Manual configuration
- Monolithic structure

**infra-deploy** (Current):
- Production-ready for cloud deployments (OTC)
- Terragrunt-based infrastructure orchestration
- CI/CD integration via GitHub Actions
- Clear separation between infrastructure and platform
- Template-instance separation through stacks/stacks-instances

## Technical Particularities

### Gomplate Templating

Gomplate is used with custom delimiters `{{{ }}}` to avoid conflicts with Helm templating (`{{ }}`):

```bash
gomplate --input-dir="stacks/template" \
         --output-dir="work" \
         --left-delim "{{{" \
         --right-delim "}}}"
```

### Terragrunt Experimental Stacks

The use of Terragrunt Stacks requires the experimental flag:

```bash
export TG_EXPERIMENT_MODE=true
terragrunt stack run apply
```

This enables hierarchical organization of Terraform modules with dependency management.

### ArgoCD ApplicationSets

The registry pattern uses ArgoCD Applications that reference directories:

```yaml
source:
  path: "otc/osctest.t09.de/stacks/core"
```

ArgoCD automatically detects all YAML files in the path and synchronizes them as Applications.

## Best Practices and Patterns

**Immutable Infrastructure**: Every environment is fully defined in Git

**Secrets Outside Git**: Sensitive data is generated at runtime or injected from secret stores

**Progressive Rollouts**: New environments start as template instances, then are individually customized

**Version Pinning**: Critical components (Helm charts, Terragrunt modules) are pinned to specific versions

**Namespace Isolation**: Each stack deploys into dedicated namespaces

**Self-Healing**: ArgoCD's automated sync policy enables automatic drift correction

## Usage Examples

### Deployment by Pipeline

The platform deployment is the second part of the EDP installtaion. First there is the infrastructure setup, which ends with a created kubernetes cluster. Then the platform provisioning by the defined stacks is done. Both is runnable by the `deploy`pipelien in `infra-deploy`:


![alt text](./deploy-action.png)

The green pipeline looks liek this:

![alt text](./green-deploy-pipeline.png)

### Local setup with 'kind'

It's also possible to just run the second part, the stcks provisionning. Then you need to have a kubernetes cluster already running, which is e.g. feasable by a local kind-cluster.

So imagine, you want to to the stacks 'core,observability' on your local machine. Then you can run the local entzr

```bash
# have kind insatlled
# in /infra-deploy

# provide a kind cluster
kind delete clusters --all
./scripts/ensure-kind-cluster.sh -r

# provide some emnv vars 
export TERRAFORM=/bin/bash
export LOADBALANCER_ID=ABC
export DOMAIN=ABC
export DOMAIN_GITEA=ABC
export OS_ACCESS_KEY=ABC
export OS_SECRET_KEY=ABC
export STACKS=core,observability

# deploy
./scripts/edp-install.sh
```

## Status

**Maturity**: [Production]

## Additional Resources

* [CNOE](https://cnoe.io/docs/overview/cnoe)
