website-and-documentation/content/en/docs/components/orchestration/platform.md

title	linkTitle	weight	description
Platform Orchestration	Platform Orchestration	20	Platform-level component provisioning via Stacks - Orchestrating the platform infrastructure itself

Overview

Platform Orchestration refers to the automation and management of the platform infrastructure itself. This includes the provisioning, configuration, and lifecycle management of all components that make up the Internal Developer Platform (IDP).

In the context of IPCEI-CIS, Platform Orchestration means:

• Platform Bootstrap: Initial setup of Kubernetes clusters and core services
• Platform Services Management: Deployment and management of ArgoCD, Forgejo, Keycloak, etc.
• Infrastructure-as-Code: Declarative management using Terraform and GitOps
• Multi-Cluster Orchestration: Coordination across different Kubernetes clusters
• Platform Stacks: Reusable bundles of platform components (CNOE concept)

Target Audience

Platform Orchestration is primarily aimed at:

• Platform Engineering Teams: Teams that build and operate the IDP
• Infrastructure Architects: Those responsible for the platform architecture
• SRE Teams: Teams responsible for reliability and operations

Key Features

Declarative Platform Definition

The entire platform is defined declaratively as code:

• GitOps-First: Everything is versioned in Git and traceable
• Reproducibility: The platform can be rebuilt at any time
• Environment Parity: Consistency between Dev, Test, and Production
• Auditability: Complete history of all changes

Self-Bootstrapping

The platform can bootstrap itself:

1. Initial Bootstrap: Minimal tool (like idpbuilder) starts the platform
2. Self-Management: After bootstrap, ArgoCD takes over management
3. Continuous Reconciliation: Platform is continuously reconciled with Git state
4. Self-Healing: Automatic recovery on deviations

Stack-based Composition

Platform components are organized as reusable stacks (CNOE concept):

• Modularity: Components can be updated individually
• Reusability: Stacks can be used across different environments
• Composability: Compose complex platforms from simple building blocks
• Versioning: Stacks can be versioned and tested

In IPCEI-CIS: The stacks concept from CNOE is the core organizational principle for platform components.

Multi-Cluster Support

Platform Orchestration supports different cluster topologies:

• Control Plane + Worker Clusters: Centralized control, distributed workloads
• Hub-and-Spoke: One management cluster manages multiple target clusters
• Federation: Coordination across multiple independent clusters

Purpose in EDP

Platform Orchestration is the foundation of the IPCEI-CIS Edge Developer Platform. It enables:

Foundation for Developer Self-Service

Platform Orchestration ensures all services are available that developers need for self-service:

• GitOps Engine (ArgoCD) for continuous deployment
• Source Control (Forgejo) for code and configuration management
• Identity Management (Keycloak) for authentication and authorization
• Observability (Grafana, Prometheus) for monitoring and logging
• CI/CD (Forgejo Actions/Pipelines) for automated build and test

Consistency Across Environments

Through declarative definition, consistency is guaranteed:

• Development, test, and production environments are identically configured
• No "configuration drift" between environments
• Predictable behavior across all stages

Platform as Code

The platform itself is treated like software:

• Version Control: All changes are versioned in Git
• Code Review: Platform changes go through review processes
• Testing: Platform configurations can be tested
• Rollback: Easy rollback on problems

Reduced Operational Overhead

Automation reduces manual effort:

• No manual installation steps
• Automatic updates and patching
• Self-healing on failures
• Standardized deployment processes

Repository

CNOE Reference Implementation: cnoe-io/stacks

CNOE idpbuilder: cnoe-io/idpbuilder

Documentation: CNOE.io Documentation

Getting Started

Prerequisites

• Docker: For local Kubernetes clusters (Kind)
• kubectl: Kubernetes CLI tool
• Git: For repository management
• idpbuilder: CNOE bootstrap tool

Quick Start

Platform Orchestration with CNOE Reference Implementation:

# 1. Install idpbuilder
curl -fsSL https://cnoe.io/install.sh | bash

# 2. Bootstrap platform
idpbuilder create \
  --use-path-routing \
  --package-dir https://github.com/cnoe-io/stacks//ref-implementation

# 3. Wait for platform ready (ca. 10 minutes)
kubectl get applications -A

Verification

Verify the platform is running correctly:

# Get platform secrets (credentials)
idpbuilder get secrets

# Check all ArgoCD applications
kubectl get applications -n argocd

# Expected: All applications "Synced" and "Healthy"

Access URLs (with path-routing):

• ArgoCD: https://cnoe.localtest.me:8443/argocd
• Forgejo: https://cnoe.localtest.me:8443/gitea
• Keycloak: https://cnoe.localtest.me:8443/keycloak

Usage Examples

Use Case 1: Platform Bootstrap

Initial bootstrapping of a new platform instance:

idpbuilder create \
  --use-path-routing \
  --package-dir https://github.com/cnoe-io/stacks//ref-implementation \
  --log-level debug

# Workflow:
# 1. Creates Kind cluster
# 2. Installs ingress-nginx
# 3. Clones and installs ArgoCD
# 4. Installs Forgejo
# 5. Waits for core services
# 6. Creates technical users
# 7. Configures Git repositories
# 8. Installs remaining stacks via ArgoCD

After approximately 10 minutes, the platform is fully deployed.

Use Case 2: Adding New Platform Components

Add new platform components via ArgoCD:

# Create ArgoCD Application for new component
cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://charts.external-secrets.io
    targetRevision: 0.9.9
    chart: external-secrets
  destination:
    server: https://kubernetes.default.svc
    namespace: external-secrets-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
EOF

Use Case 3: Platform Updates

Update platform components:

# 1. Update via Git (GitOps)
cd your-platform-config-repo
git pull

# 2. Update stack version
vim argocd/applications/component.yaml
# Change targetRevision to new version

# 3. Commit and push
git add .
git commit -m "Update component to v1.2.3"
git push

# 4. ArgoCD will automatically sync
# 5. Monitor the update
argocd app sync component --watch

Integration Points

ArgoCD Integration

• Bootstrap: ArgoCD is initially installed via idpbuilder
• Self-Management: After bootstrap, ArgoCD manages itself via Application CRD
• Platform Coordination: ArgoCD orchestrates all other platform components
• Health Monitoring: ArgoCD monitors health status of all platform services

Forgejo Integration

• Source of Truth: Git repositories contain all platform definitions
• GitOps Workflow: Changes in Git trigger platform updates
• Backup: Git serves as backup of platform configuration
• Audit Trail: Git history documents all platform changes
• CI/CD: Forgejo Actions can automate platform operations

Terraform Integration

• Infrastructure Provisioning: Terraform provisions cloud resources for platform
• State Management: Terraform state tracks infrastructure
• Integration: Terraform can be triggered via Forgejo pipelines
• Multi-Cloud: Support for multiple cloud providers

Architecture

Platform Orchestration Flow

┌─────────────────┐
│   idpbuilder    │  Bootstrap Tool
│  (Initial Run)  │
└────────┬────────┘
         │
         ▼
┌─────────────────────────────────────────────────────┐
│              Kubernetes Cluster                      │
│                                                      │
│  ┌──────────────┐         ┌──────────────┐         │
│  │   ArgoCD     │────────▶│   Forgejo    │         │
│  │  (GitOps)    │         │  (Git Repo)  │         │
│  └──────┬───────┘         └──────────────┘         │
│         │                                            │
│         │  Monitors & Syncs                         │
│         │                                            │
│         ▼                                            │
│  ┌──────────────────────────────────────┐          │
│  │     Platform Stacks                  │          │
│  │                                      │          │
│  │  ┌──────────┐  ┌──────────┐        │          │
│  │  │Forgejo   │  │Keycloak  │        │          │
│  │  └──────────┘  └──────────┘        │          │
│  │  ┌──────────┐  ┌──────────┐        │          │
│  │  │Observ-   │  │Ingress   │        │          │
│  │  │ability   │  │          │        │          │
│  │  └──────────┘  └──────────┘        │          │
│  └──────────────────────────────────────┘          │
└─────────────────────────────────────────────────────┘

Platform Bootstrap Sequence

The idpbuilder executes the following workflow:

1. Create Kind Kubernetes cluster
2. Install ingress-nginx controller
3. Install ArgoCD
4. Install Forgejo Git server
5. Wait for services to be ready
6. Create technical users in Forgejo
7. Create repository for platform state in Forgejo
8. Push platform stacks to Forgejo
9. Create ArgoCD Applications for all stacks
10. ArgoCD takes over continuous synchronization

Deployment Architecture

The platform is deployed in different namespaces:

• argocd: ArgoCD and its components
• gitea: Forgejo Git server
• keycloak: Identity and access management
• observability: Prometheus, Grafana, etc.
• ingress-nginx: Ingress controller

Configuration

idpbuilder Configuration

Key configuration options for idpbuilder:

# Path-based routing (recommended for local development)
idpbuilder create --use-path-routing

# Custom package directory
idpbuilder create --package-dir /path/to/custom/packages

# Custom Kind cluster config
idpbuilder create --kind-config custom-kind.yaml

# Enable debug logging
idpbuilder create --log-level debug

ArgoCD Configuration

Important ArgoCD configurations for platform orchestration:

# argocd-cm ConfigMap
data:
  # Enable automatic sync
  application.instanceLabelKey: argocd.argoproj.io/instance
  
  # Repository credentials
  repositories: |
    - url: https://github.com/cnoe-io/stacks
      name: cnoe-stacks
      type: git
  
  # Resource exclusions
  resource.exclusions: |
    - apiGroups:
      - cilium.io
      kinds:
      - CiliumIdentity

Platform Stack Configuration

Configuration of platform stacks via Kustomize:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: platform-system

resources:
  - argocd-app.yaml
  - forgejo-app.yaml
  - keycloak-app.yaml

patches:
  - target:
      kind: Application
    patch: |-
      - op: add
        path: /spec/syncPolicy
        value:
          automated:
            prune: true
            selfHeal: true

Troubleshooting

Platform not reachable

Problem: After idpbuilder create, platform services are not reachable

Solution:

# 1. Check if all pods are running
kubectl get pods -A

# 2. Check ArgoCD application status
kubectl get applications -n argocd

# 3. Check ingress
kubectl get ingress -A

# 4. Verify DNS resolution
nslookup cnoe.localtest.me

# 5. Check idpbuilder logs
idpbuilder get logs

ArgoCD Applications not synchronized

Problem: ArgoCD Applications show status "OutOfSync"

Solution:

# 1. Check application details
argocd app get <app-name>

# 2. View sync status
argocd app sync <app-name> --dry-run

# 3. Force sync
argocd app sync <app-name> --force

# 4. Check for errors in ArgoCD logs
kubectl logs -n argocd deployment/argocd-application-controller

Git Repository Connection Issues

Problem: ArgoCD cannot access Git repository

Solution:

# 1. Verify repository configuration
argocd repo list

# 2. Test connection
argocd repo get https://your-git-repo

# 3. Check credentials
kubectl get secret -n argocd

# 4. Re-add repository with correct credentials
argocd repo add https://your-git-repo \
  --username <user> \
  --password <token>

Platform Orchestration Best Practices

Based on experience and CNCF Guidelines:

1. Start Simple: Begin with the CNOE reference stack, extend gradually
2. Automate Everything: Manual platform changes are anti-pattern
3. Monitor Continuously: Use observability tools for platform health
4. Document Well: Platform documentation is essential for adoption
5. Version Everything: All platform components should be versioned
6. Test Changes: Platform updates should be tested in non-prod
7. Plan for Disaster: Backup and disaster recovery strategies are important
8. Use Stacks: Organize platform components as reusable stacks

Status

Maturity: Production (for CNOE Reference Implementation)

Stability: Stable

Support: Community Support via CNOE Community

Additional Resources

CNOE Resources

• CNOE Official Website
• CNOE GitHub Organization
• CNOE Reference Implementation
• CNOE Community Slack

Platform Engineering

• CNCF Platforms White Paper
• Platform Engineering Maturity Model
• Team Topologies - Organizational patterns

GitOps

• GitOps Working Group
• ArgoCD Best Practices
• GitOps Principles

CNOE Stacks

• Understanding CNOE Stacks
• Creating Custom Stacks