From 46a8c9dbb30f5a18811e69beae131d48d97a3f90 Mon Sep 17 00:00:00 2001 From: Manuel Ganter Date: Tue, 16 Dec 2025 11:16:08 +0100 Subject: [PATCH] added otc stack docs --- .../components/orchestration/stacks/otc.md | 545 +++++++++++++++--- 1 file changed, 472 insertions(+), 73 deletions(-) diff --git a/content/en/docs/components/orchestration/stacks/otc.md b/content/en/docs/components/orchestration/stacks/otc.md index 83f0345..45f506c 100644 --- a/content/en/docs/components/orchestration/stacks/otc.md +++ b/content/en/docs/components/orchestration/stacks/otc.md @@ -1,127 +1,526 @@ --- title: "OTC" linkTitle: "OTC" -weight: 70 -description: OTC +weight: 10 +description: > + Open Telekom Cloud infrastructure components for ingress, TLS, and storage --- -{{% alert title="Draft" color="warning" %}} -**Editorial Status**: This page is currently being developed. - -* **Jira Ticket**: [TBD] -* **Assignee**: [Name or Team] -* **Status**: Draft -* **Last Updated**: YYYY-MM-DD -* **TODO**: - * [ ] Add detailed component description - * [ ] Include usage examples and code samples - * [ ] Add architecture diagrams - * [ ] Review and finalize content -{{% /alert %}} - ## Overview -[Detailed description of the component - what it is, what it does, and why it exists] +The OTC (Open Telekom Cloud) stack provides essential infrastructure components for deploying applications on Open Telekom Cloud environments. It configures ingress routing, automated TLS certificate management, and cloud-native storage provisioning tailored specifically for OTC's Kubernetes infrastructure. + +This stack serves as a foundational layer that other platform stacks depend on for external access, secure communication, and persistent storage. ## Key Features -* [Feature 1] -* [Feature 2] -* [Feature 3] - -## Purpose in EDP - -[Explain the role this component plays in the Edge Developer Platform and how it contributes to the overall platform capabilities] +* **Automated TLS Certificate Management**: Let's Encrypt integration via cert-manager for automatic certificate provisioning and renewal +* **Cloud Load Balancer Integration**: Nginx ingress controller configured with OTC-specific Elastic Load Balancer (ELB) annotations +* **Native Storage Provisioning**: Default StorageClass using Huawei FlexVolume provisioner for block storage +* **Prometheus Metrics**: Built-in monitoring capabilities for ingress traffic and performance +* **High Availability**: Rolling update strategy with minimal downtime +* **HTTP-01 Challenge Support**: ACME validation through ingress for certificate issuance ## Repository -**Code**: [Link to source code repository] +**Code**: [OTC Stack Templates](https://edp.buildth.ing/DevFW-CICD/stacks/src/branch/main/template/stacks/otc) -**Documentation**: [Link to component-specific documentation] +**Documentation**: +* [cert-manager Documentation](https://cert-manager.io/docs/) +* [ingress-nginx Documentation](https://kubernetes.github.io/ingress-nginx/) +* [Open Telekom Cloud Documentation](https://docs.otc.t-systems.com/) ## Getting Started ### Prerequisites -* [Prerequisite 1] -* [Prerequisite 2] +* Kubernetes cluster running on Open Telekom Cloud +* ArgoCD installed (provided by `core` stack) +* Environment variables configured: + - `LOADBALANCER_ID`: OTC Elastic Load Balancer ID + - `LOADBALANCER_IP`: OTC Elastic Load Balancer IP address + - `CLIENT_REPO_DOMAIN`: Git repository domain + - `CLIENT_REPO_ORG_NAME`: Git repository organization + - `CLIENT_REPO_ID`: Client repository identifier + - `DOMAIN`: Domain name for the environment ### Quick Start -[Step-by-step guide to get started with this component] +The OTC stack is deployed as part of the EDP installation process: -1. [Step 1] -2. [Step 2] -3. [Step 3] +1. **Trigger Deploy Pipeline** + - Go to [Infra Deploy Pipeline](https://edp.buildth.ing/DevFW/infra-deploy/actions?workflow=deploy.yaml) + - Click on Run workflow + - Enter a name in "Select environment directory to deploy". This must be DNS Compatible. + - Execute workflow + +2. **ArgoCD Synchronization** + ArgoCD automatically deploys: + - cert-manager with ClusterIssuer for Let's Encrypt + - ingress-nginx controller with OTC load balancer integration + - Default StorageClass for OTC block storage ### Verification -[How to verify the component is working correctly] - -## Usage Examples - -### [Use Case 1] - -[Example with code/commands showing common use case] +Verify the OTC stack deployment: ```bash -# Example commands +# Check ArgoCD applications status +kubectl get application otc -n argocd +kubectl get application cert-manager -n argocd +kubectl get application ingress-nginx -n argocd +kubectl get application storageclass -n argocd + +# Verify cert-manager pods +kubectl get pods -n cert-manager + +# Check ingress-nginx controller +kubectl get pods -n ingress-nginx + +# Verify ClusterIssuer status +kubectl get clusterissuer main + +# Check StorageClass +kubectl get storageclass default ``` -### [Use Case 2] - -[Another common scenario] - -## Integration Points - -* **[Component A]**: [How it integrates] -* **[Component B]**: [How it integrates] -* **[Component C]**: [How it integrates] - ## Architecture -[Optional: Add architectural diagrams and descriptions] +### Component Architecture -### Component Architecture (C4) +The OTC stack consists of three primary components: -[Add C4 Container or Component diagrams showing the internal structure] +**cert-manager**: +- Automates TLS certificate lifecycle management +- Integrates with Let's Encrypt ACME server (production endpoint) +- Uses HTTP-01 challenge validation via ingress +- Creates and manages certificates as Kubernetes resources +- Single replica deployment -### Sequence Diagrams +**ingress-nginx**: +- Kubernetes ingress controller based on Nginx +- Routes external traffic to internal services +- Integrated with OTC Elastic Load Balancer (ELB) +- Supports TLS termination with cert-manager issued certificates +- Rolling update strategy with max 1 unavailable pod +- Prometheus metrics exporter with ServiceMonitor -[Add sequence diagrams showing key interaction flows with other components] +**StorageClass**: +- Default storage provisioner for persistent volumes +- Uses Huawei FlexVolume driver (`flexvolume-huawei.com/fuxivol`) +- SATA block storage type +- Immediate volume binding mode +- Supports dynamic volume expansion -### Deployment Architecture +### Integration Flow -[Add infrastructure and deployment diagrams showing how the component is deployed] +``` +External Traffic → OTC ELB → ingress-nginx → Kubernetes Services + ↓ + cert-manager (TLS certificates) + ↓ + Let's Encrypt ACME +``` ## Configuration -[Key configuration options and how to set them] +### cert-manager Configuration + +**Helm Values** (`stacks/otc/cert-manager/values.yaml`): + +```yaml +crds: + enabled: true +replicaCount: 1 +``` + +**ClusterIssuer** (`stacks/otc/cert-manager/manifests/clusterissuer.yaml`): + +```yaml +apiVersion: cert-manager.io/v1 +kind: ClusterIssuer +metadata: + name: main +spec: + acme: + email: admin@think-ahead.tech + server: https://acme-v02.api.letsencrypt.org/directory + privateKeySecretRef: + name: cluster-issuer-account-key + solvers: + - http01: + ingress: + ingressClassName: nginx +``` + +**Key Settings**: +- CRDs installed automatically +- Production Let's Encrypt ACME endpoint +- HTTP-01 validation through nginx ingress +- ClusterIssuer named `main` for cluster-wide certificate issuance + +### ingress-nginx Configuration + +**Helm Values** (`stacks/otc/ingress-nginx/values.yaml`): + +```yaml +controller: + updateStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + service: + annotations: + kubernetes.io/elb.class: union + kubernetes.io/elb.port: '80' + kubernetes.io/elb.id: {{{ .Env.LOADBALANCER_ID }}} + kubernetes.io/elb.ip: {{{ .Env.LOADBALANCER_IP }}} + ingressClassResource: + name: nginx + allowSnippetAnnotations: true + config: + proxy-buffer-size: 32k + use-forwarded-headers: "true" + metrics: + enabled: true + serviceMonitor: + additionalLabels: + release: "ingress-nginx" + enabled: true +``` + +**Key Settings**: +- **OTC Load Balancer Integration**: Annotations configure connection to OTC ELB +- **Rolling Updates**: Minimizes downtime with 1 pod unavailable during updates +- **Snippet Annotations**: Enabled for advanced ingress configuration (idpbuilder compatibility) +- **Proxy Buffer**: 32k buffer size for handling large headers +- **Forwarded Headers**: Preserves original client information through proxies +- **Metrics**: Prometheus ServiceMonitor for observability + +### StorageClass Configuration + +**StorageClass** (`stacks/otc/storageclass/storageclass.yaml`): + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + annotations: + storageclass.beta.kubernetes.io/is-default-class: "true" + name: default +parameters: + kubernetes.io/hw:passthrough: "true" + kubernetes.io/storagetype: BS + kubernetes.io/volumetype: SATA + kubernetes.io/zone: eu-de-02 +provisioner: flexvolume-huawei.com/fuxivol +reclaimPolicy: Delete +volumeBindingMode: Immediate +allowVolumeExpansion: true +``` + +**Key Settings**: +- **Default StorageClass**: Automatically used when no StorageClass specified +- **OTC Zone**: Provisioned in `eu-de-02` availability zone +- **SATA Volumes**: Block storage (BS) with SATA performance tier +- **Volume Expansion**: Supports resizing persistent volumes dynamically +- **Reclaim Policy**: Volumes deleted when PersistentVolumeClaim is removed + +### ArgoCD Application Configuration + +**Registry Application** (`template/registry/otc.yaml`): +- Name: `otc` +- Manages the OTC stack directory +- Automated sync with prune and self-heal enabled +- Creates namespaces automatically + +**Component Applications**: + +**cert-manager** (referenced in stack): +- Deploys cert-manager Helm chart +- Automated self-healing enabled +- Includes ClusterIssuer manifest for Let's Encrypt + +**ingress-nginx** (`template/stacks/otc/ingress-nginx.yaml`): +- Deploys from official Kubernetes ingress-nginx repository +- Chart version: helm-chart-4.12.1 +- References environment-specific values from stacks-instances repository + +**storageclass** (`template/stacks/otc/storageclass.yaml`): +- Deploys StorageClass manifest +- Managed as ArgoCD Application +- Automated sync with unlimited retries + +## Usage Examples + +### Creating an Ingress with Automatic TLS + +Create an ingress resource that automatically provisions a TLS certificate: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: my-app + namespace: my-namespace + annotations: + cert-manager.io/cluster-issuer: main + nginx.ingress.kubernetes.io/ssl-redirect: "true" +spec: + ingressClassName: nginx + tls: + - hosts: + - myapp.example.com + secretName: myapp-tls + rules: + - host: myapp.example.com + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: my-app-service + port: + number: 80 +``` + +cert-manager will automatically: +1. Detect the ingress with `cert-manager.io/cluster-issuer` annotation +2. Create a Certificate resource +3. Request certificate from Let's Encrypt using HTTP-01 challenge +4. Store certificate in `myapp-tls` secret +5. Renew certificate before expiration + +### Creating a PersistentVolumeClaim + +Use the default OTC StorageClass for persistent storage: + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: my-data + namespace: my-namespace +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + storageClassName: csi-disk +``` + +### Expanding an Existing Volume + +Resize a persistent volume by editing the PVC: + +```bash +# Edit the PVC storage request +kubectl patch pvc my-data -n my-namespace -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}' + +# Verify expansion +kubectl get pvc my-data -n my-namespace +``` + +The volume will expand automatically due to `allowVolumeExpansion: true` in the StorageClass. + +### Custom Ingress Configuration + +Use nginx ingress snippets for advanced routing: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: advanced-app + annotations: + cert-manager.io/cluster-issuer: main + nginx.ingress.kubernetes.io/configuration-snippet: | + more_set_headers "X-Custom-Header: value"; + if ($http_user_agent ~* "bot") { + return 403; + } +spec: + ingressClassName: nginx + rules: + - host: app.example.com + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: app-service + port: + number: 8080 +``` + +## Integration Points + +* **Core Stack**: Requires ArgoCD for deployment orchestration +* **All Application Stacks**: Depends on OTC stack for: + - External access via ingress-nginx + - TLS certificates via cert-manager + - Persistent storage via default StorageClass +* **Observability Stack**: ingress-nginx metrics exported to Prometheus +* **Coder Stack**: Uses ingress and cert-manager for workspace access +* **Forgejo Stack**: Requires ingress and TLS for Git repository access ## Troubleshooting -### [Common Issue 1] +### Certificate Issuance Fails -**Problem**: [Description] +**Problem**: Certificate remains in `Pending` state and is not issued -**Solution**: [How to fix] +**Solution**: +1. Check Certificate status: + ```bash + kubectl get certificate -A + kubectl describe certificate -n + ``` -### [Common Issue 2] +2. Verify ClusterIssuer is ready: + ```bash + kubectl get clusterissuer main + kubectl describe clusterissuer main + ``` -**Problem**: [Description] +3. Check cert-manager logs: + ```bash + kubectl logs -n cert-manager -l app=cert-manager + ``` -**Solution**: [How to fix] +4. Verify HTTP-01 challenge can reach ingress: + ```bash + kubectl get challenges -A + kubectl describe challenge -n + ``` -## Status +5. Common issues: + - DNS not pointing to load balancer IP + - Firewall blocking HTTP (port 80) traffic + - Ingress class not set to `nginx` + - Let's Encrypt rate limits exceeded -**Maturity**: [Production / Beta / Experimental] +### Ingress Controller Not Ready + +**Problem**: ingress-nginx pods are not running or LoadBalancer service has no external IP + +**Solution**: +1. Check ingress controller status: + ```bash + kubectl get pods -n ingress-nginx + kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller + ``` + +2. Verify LoadBalancer service: + ```bash + kubectl get svc -n ingress-nginx + kubectl describe svc ingress-nginx-controller -n ingress-nginx + ``` + +3. Check OTC load balancer annotations: + ```bash + kubectl get svc ingress-nginx-controller -n ingress-nginx -o yaml + ``` + +4. Verify environment variables are set correctly: + - `LOADBALANCER_ID` matches OTC ELB ID + - `LOADBALANCER_IP` matches ELB public IP + +5. Check OTC console for ELB configuration and health checks + +### Storage Provisioning Fails + +**Problem**: PersistentVolumeClaim remains in `Pending` state + +**Solution**: +1. Check PVC status: + ```bash + kubectl get pvc -A + kubectl describe pvc -n + ``` + +2. Verify StorageClass exists and is default: + ```bash + kubectl get storageclass + kubectl describe storageclass default + ``` + +3. Check volume provisioner logs: + ```bash + kubectl logs -n kube-system -l app=csi-disk-plugin + ``` + +4. Common issues: + - Insufficient quota in OTC project + - Invalid zone configuration (must be `eu-de-02`) + - Requested storage size exceeds limits + - Missing IAM permissions for volume creation + +### Ingress Returns 503 Service Unavailable + +**Problem**: Ingress configured but returns 503 error + +**Solution**: +1. Verify backend service exists: + ```bash + kubectl get svc -n + kubectl get endpoints -n + ``` + +2. Check if pods are ready: + ```bash + kubectl get pods -n -l + ``` + +3. Verify ingress configuration: + ```bash + kubectl describe ingress -n + ``` + +4. Check nginx ingress logs: + ```bash + kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller --tail=100 + ``` + +5. Test service connectivity from ingress controller: + ```bash + kubectl exec -n ingress-nginx -- curl http://..svc.cluster.local: + ``` + +### TLS Certificate Shows as Invalid + +**Problem**: Browser shows certificate warning or certificate details are incorrect + +**Solution**: +1. Verify certificate is ready: + ```bash + kubectl get certificate -n + ``` + +2. Check certificate contents: + ```bash + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout + ``` + +3. Ensure certificate covers the correct domain: + ```bash + kubectl describe certificate -n + ``` + +4. Force certificate renewal if expired or incorrect: + ```bash + kubectl delete certificate -n + # cert-manager will automatically recreate it + ``` ## Additional Resources -* [Link to external documentation] -* [Link to community resources] -* [Link to related components] - -## Documentation Notes - -[Instructions for team members filling in this documentation - remove this section once complete] +* [cert-manager Documentation](https://cert-manager.io/docs/) +* [ingress-nginx User Guide](https://kubernetes.github.io/ingress-nginx/user-guide/) +* [Open Telekom Cloud Documentation](https://docs.otc.t-systems.com/) +* [Let's Encrypt Documentation](https://letsencrypt.org/docs/) +* [Kubernetes Ingress Concepts](https://kubernetes.io/docs/concepts/services-networking/ingress/) +* [Kubernetes Storage Classes](https://kubernetes.io/docs/concepts/storage/storage-classes/)