---
title: "OTC"
linkTitle: "OTC"
weight: 10
description: >
  Open Telekom Cloud infrastructure components for ingress, TLS, and storage
---

## Overview

The OTC (Open Telekom Cloud) stack provides essential infrastructure components for deploying applications on Open Telekom Cloud environments. It configures ingress routing, automated TLS certificate management, and cloud-native storage provisioning tailored specifically for OTC's Kubernetes infrastructure.

This stack serves as a foundational layer that other platform stacks depend on for external access, secure communication, and persistent storage.

## Key Features

* **Automated TLS Certificate Management**: Let's Encrypt integration via cert-manager for automatic certificate provisioning and renewal
* **Cloud Load Balancer Integration**: Nginx ingress controller configured with OTC-specific Elastic Load Balancer (ELB) annotations
* **Native Storage Provisioning**: Default StorageClass using Huawei FlexVolume provisioner for block storage
* **Prometheus Metrics**: Built-in monitoring capabilities for ingress traffic and performance
* **High Availability**: Rolling update strategy with minimal downtime
* **HTTP-01 Challenge Support**: ACME validation through ingress for certificate issuance

## Repository

**Code**: [OTC Stack Templates](https://edp.buildth.ing/DevFW-CICD/stacks/src/branch/main/template/stacks/otc)

**Documentation**:
* [cert-manager Documentation](https://cert-manager.io/docs/)
* [ingress-nginx Documentation](https://kubernetes.github.io/ingress-nginx/)
* [Open Telekom Cloud Documentation](https://docs.otc.t-systems.com/)

## Getting Started

### Prerequisites

* Kubernetes cluster running on Open Telekom Cloud
* ArgoCD installed (provided by `core` stack)
* Environment variables configured:
  - `LOADBALANCER_ID`: OTC Elastic Load Balancer ID
  - `LOADBALANCER_IP`: OTC Elastic Load Balancer IP address
  - `CLIENT_REPO_DOMAIN`: Git repository domain
  - `CLIENT_REPO_ORG_NAME`: Git repository organization
  - `CLIENT_REPO_ID`: Client repository identifier
  - `DOMAIN`: Domain name for the environment

### Quick Start

The OTC stack is deployed as part of the EDP installation process:

1. **Trigger Deploy Pipeline**
   - Go to [Infra Deploy Pipeline](https://edp.buildth.ing/DevFW/infra-deploy/actions?workflow=deploy.yaml)
   - Click on Run workflow
   - Enter a name in "Select environment directory to deploy". This must be DNS Compatible.
   - Execute workflow

2. **ArgoCD Synchronization**
   ArgoCD automatically deploys:
   - cert-manager with ClusterIssuer for Let's Encrypt
   - ingress-nginx controller with OTC load balancer integration
   - Default StorageClass for OTC block storage

### Verification

Verify the OTC stack deployment:

```bash
# Check ArgoCD applications status
kubectl get application otc -n argocd
kubectl get application cert-manager -n argocd
kubectl get application ingress-nginx -n argocd
kubectl get application storageclass -n argocd

# Verify cert-manager pods
kubectl get pods -n cert-manager

# Check ingress-nginx controller
kubectl get pods -n ingress-nginx

# Verify ClusterIssuer status
kubectl get clusterissuer main

# Check StorageClass
kubectl get storageclass default
```

## Architecture

### Component Architecture

The OTC stack consists of three primary components:

**cert-manager**:
- Automates TLS certificate lifecycle management
- Integrates with Let's Encrypt ACME server (production endpoint)
- Uses HTTP-01 challenge validation via ingress
- Creates and manages certificates as Kubernetes resources
- Single replica deployment

**ingress-nginx**:
- Kubernetes ingress controller based on Nginx
- Routes external traffic to internal services
- Integrated with OTC Elastic Load Balancer (ELB)
- Supports TLS termination with cert-manager issued certificates
- Rolling update strategy with max 1 unavailable pod
- Prometheus metrics exporter with ServiceMonitor

**StorageClass**:
- Default storage provisioner for persistent volumes
- Uses Huawei FlexVolume driver (`flexvolume-huawei.com/fuxivol`)
- SATA block storage type
- Immediate volume binding mode
- Supports dynamic volume expansion

### Integration Flow

```
External Traffic → OTC ELB → ingress-nginx → Kubernetes Services
                                    ↓
                              cert-manager (TLS certificates)
                                    ↓
                              Let's Encrypt ACME
```

## Configuration

### cert-manager Configuration

**Helm Values** (`stacks/otc/cert-manager/values.yaml`):

```yaml
crds:
  enabled: true
replicaCount: 1
```

**ClusterIssuer** (`stacks/otc/cert-manager/manifests/clusterissuer.yaml`):

```yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: main
spec:
  acme:
    email: admin@think-ahead.tech
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: cluster-issuer-account-key
    solvers:
    - http01:
        ingress:
          ingressClassName: nginx
```

**Key Settings**:
- CRDs installed automatically
- Production Let's Encrypt ACME endpoint
- HTTP-01 validation through nginx ingress
- ClusterIssuer named `main` for cluster-wide certificate issuance

### ingress-nginx Configuration

**Helm Values** (`stacks/otc/ingress-nginx/values.yaml`):

```yaml
controller:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  service:
    annotations:
      kubernetes.io/elb.class: union
      kubernetes.io/elb.port: '80'
      kubernetes.io/elb.id: {{{ .Env.LOADBALANCER_ID }}}
      kubernetes.io/elb.ip: {{{ .Env.LOADBALANCER_IP }}}
  ingressClassResource:
    name: nginx
  allowSnippetAnnotations: true
  config:
    proxy-buffer-size: 32k
    use-forwarded-headers: "true"
  metrics:
    enabled: true
    serviceMonitor:
      additionalLabels:
        release: "ingress-nginx"
      enabled: true
```

**Key Settings**:
- **OTC Load Balancer Integration**: Annotations configure connection to OTC ELB
- **Rolling Updates**: Minimizes downtime with 1 pod unavailable during updates
- **Snippet Annotations**: Enabled for advanced ingress configuration (idpbuilder compatibility)
- **Proxy Buffer**: 32k buffer size for handling large headers
- **Forwarded Headers**: Preserves original client information through proxies
- **Metrics**: Prometheus ServiceMonitor for observability

### StorageClass Configuration

**StorageClass** (`stacks/otc/storageclass/storageclass.yaml`):

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
  name: default
parameters:
  kubernetes.io/hw:passthrough: "true"
  kubernetes.io/storagetype: BS
  kubernetes.io/volumetype: SATA
  kubernetes.io/zone: eu-de-02
provisioner: flexvolume-huawei.com/fuxivol
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
```

**Key Settings**:
- **Default StorageClass**: Automatically used when no StorageClass specified
- **OTC Zone**: Provisioned in `eu-de-02` availability zone
- **SATA Volumes**: Block storage (BS) with SATA performance tier
- **Volume Expansion**: Supports resizing persistent volumes dynamically
- **Reclaim Policy**: Volumes deleted when PersistentVolumeClaim is removed

### ArgoCD Application Configuration

**Registry Application** (`template/registry/otc.yaml`):
- Name: `otc`
- Manages the OTC stack directory
- Automated sync with prune and self-heal enabled
- Creates namespaces automatically

**Component Applications**:

**cert-manager** (referenced in stack):
- Deploys cert-manager Helm chart
- Automated self-healing enabled
- Includes ClusterIssuer manifest for Let's Encrypt

**ingress-nginx** (`template/stacks/otc/ingress-nginx.yaml`):
- Deploys from official Kubernetes ingress-nginx repository
- Chart version: helm-chart-4.12.1
- References environment-specific values from stacks-instances repository

**storageclass** (`template/stacks/otc/storageclass.yaml`):
- Deploys StorageClass manifest
- Managed as ArgoCD Application
- Automated sync with unlimited retries

## Usage Examples

### Creating an Ingress with Automatic TLS

Create an ingress resource that automatically provisions a TLS certificate:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  namespace: my-namespace
  annotations:
    cert-manager.io/cluster-issuer: main
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - myapp.example.com
    secretName: myapp-tls
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-app-service
            port:
              number: 80
```

cert-manager will automatically:
1. Detect the ingress with `cert-manager.io/cluster-issuer` annotation
2. Create a Certificate resource
3. Request certificate from Let's Encrypt using HTTP-01 challenge
4. Store certificate in `myapp-tls` secret
5. Renew certificate before expiration

### Creating a PersistentVolumeClaim

Use the default OTC StorageClass for persistent storage:

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-data
  namespace: my-namespace
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: csi-disk
```

### Expanding an Existing Volume

Resize a persistent volume by editing the PVC:

```bash
# Edit the PVC storage request
kubectl patch pvc my-data -n my-namespace -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

# Verify expansion
kubectl get pvc my-data -n my-namespace
```

The volume will expand automatically due to `allowVolumeExpansion: true` in the StorageClass.

### Custom Ingress Configuration

Use nginx ingress snippets for advanced routing:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: advanced-app
  annotations:
    cert-manager.io/cluster-issuer: main
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "X-Custom-Header: value";
      if ($http_user_agent ~* "bot") {
        return 403;
      }
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-service
            port:
              number: 8080
```

## Integration Points

* **Core Stack**: Requires ArgoCD for deployment orchestration
* **All Application Stacks**: Depends on OTC stack for:
  - External access via ingress-nginx
  - TLS certificates via cert-manager
  - Persistent storage via default StorageClass
* **Observability Stack**: ingress-nginx metrics exported to Prometheus
* **Coder Stack**: Uses ingress and cert-manager for workspace access
* **Forgejo Stack**: Requires ingress and TLS for Git repository access

## Troubleshooting

### Certificate Issuance Fails

**Problem**: Certificate remains in `Pending` state and is not issued

**Solution**:
1. Check Certificate status:
   ```bash
   kubectl get certificate -A
   kubectl describe certificate <cert-name> -n <namespace>
   ```

2. Verify ClusterIssuer is ready:
   ```bash
   kubectl get clusterissuer main
   kubectl describe clusterissuer main
   ```

3. Check cert-manager logs:
   ```bash
   kubectl logs -n cert-manager -l app=cert-manager
   ```

4. Verify HTTP-01 challenge can reach ingress:
   ```bash
   kubectl get challenges -A
   kubectl describe challenge <challenge-name> -n <namespace>
   ```

5. Common issues:
   - DNS not pointing to load balancer IP
   - Firewall blocking HTTP (port 80) traffic
   - Ingress class not set to `nginx`
   - Let's Encrypt rate limits exceeded

### Ingress Controller Not Ready

**Problem**: ingress-nginx pods are not running or LoadBalancer service has no external IP

**Solution**:
1. Check ingress controller status:
   ```bash
   kubectl get pods -n ingress-nginx
   kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller
   ```

2. Verify LoadBalancer service:
   ```bash
   kubectl get svc -n ingress-nginx
   kubectl describe svc ingress-nginx-controller -n ingress-nginx
   ```

3. Check OTC load balancer annotations:
   ```bash
   kubectl get svc ingress-nginx-controller -n ingress-nginx -o yaml
   ```

4. Verify environment variables are set correctly:
   - `LOADBALANCER_ID` matches OTC ELB ID
   - `LOADBALANCER_IP` matches ELB public IP

5. Check OTC console for ELB configuration and health checks

### Storage Provisioning Fails

**Problem**: PersistentVolumeClaim remains in `Pending` state

**Solution**:
1. Check PVC status:
   ```bash
   kubectl get pvc -A
   kubectl describe pvc <pvc-name> -n <namespace>
   ```

2. Verify StorageClass exists and is default:
   ```bash
   kubectl get storageclass
   kubectl describe storageclass default
   ```

3. Check volume provisioner logs:
   ```bash
   kubectl logs -n kube-system -l app=csi-disk-plugin
   ```

4. Common issues:
   - Insufficient quota in OTC project
   - Invalid zone configuration (must be `eu-de-02`)
   - Requested storage size exceeds limits
   - Missing IAM permissions for volume creation

### Ingress Returns 503 Service Unavailable

**Problem**: Ingress configured but returns 503 error

**Solution**:
1. Verify backend service exists:
   ```bash
   kubectl get svc <service-name> -n <namespace>
   kubectl get endpoints <service-name> -n <namespace>
   ```

2. Check if pods are ready:
   ```bash
   kubectl get pods -n <namespace> -l <service-selector>
   ```

3. Verify ingress configuration:
   ```bash
   kubectl describe ingress <ingress-name> -n <namespace>
   ```

4. Check nginx ingress logs:
   ```bash
   kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller --tail=100
   ```

5. Test service connectivity from ingress controller:
   ```bash
   kubectl exec -n ingress-nginx <controller-pod> -- curl http://<service-name>.<namespace>.svc.cluster.local:<port>
   ```

### TLS Certificate Shows as Invalid

**Problem**: Browser shows certificate warning or certificate details are incorrect

**Solution**:
1. Verify certificate is ready:
   ```bash
   kubectl get certificate <cert-name> -n <namespace>
   ```

2. Check certificate contents:
   ```bash
   kubectl get secret <tls-secret-name> -n <namespace> -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout
   ```

3. Ensure certificate covers the correct domain:
   ```bash
   kubectl describe certificate <cert-name> -n <namespace>
   ```

4. Force certificate renewal if expired or incorrect:
   ```bash
   kubectl delete certificate <cert-name> -n <namespace>
   # cert-manager will automatically recreate it
   ```

## Additional Resources

* [cert-manager Documentation](https://cert-manager.io/docs/)
* [ingress-nginx User Guide](https://kubernetes.github.io/ingress-nginx/user-guide/)
* [Open Telekom Cloud Documentation](https://docs.otc.t-systems.com/)
* [Let's Encrypt Documentation](https://letsencrypt.org/docs/)
* [Kubernetes Ingress Concepts](https://kubernetes.io/docs/concepts/services-networking/ingress/)
* [Kubernetes Storage Classes](https://kubernetes.io/docs/concepts/storage/storage-classes/)