added otc stack docs
This commit is contained in:
parent
babd8df7b5
commit
46a8c9dbb3
1 changed files with 472 additions and 73 deletions
|
|
@ -1,127 +1,526 @@
|
|||
---
|
||||
title: "OTC"
|
||||
linkTitle: "OTC"
|
||||
weight: 70
|
||||
description: OTC
|
||||
weight: 10
|
||||
description: >
|
||||
Open Telekom Cloud infrastructure components for ingress, TLS, and storage
|
||||
---
|
||||
|
||||
{{% alert title="Draft" color="warning" %}}
|
||||
**Editorial Status**: This page is currently being developed.
|
||||
|
||||
* **Jira Ticket**: [TBD]
|
||||
* **Assignee**: [Name or Team]
|
||||
* **Status**: Draft
|
||||
* **Last Updated**: YYYY-MM-DD
|
||||
* **TODO**:
|
||||
* [ ] Add detailed component description
|
||||
* [ ] Include usage examples and code samples
|
||||
* [ ] Add architecture diagrams
|
||||
* [ ] Review and finalize content
|
||||
{{% /alert %}}
|
||||
|
||||
## Overview
|
||||
|
||||
[Detailed description of the component - what it is, what it does, and why it exists]
|
||||
The OTC (Open Telekom Cloud) stack provides essential infrastructure components for deploying applications on Open Telekom Cloud environments. It configures ingress routing, automated TLS certificate management, and cloud-native storage provisioning tailored specifically for OTC's Kubernetes infrastructure.
|
||||
|
||||
This stack serves as a foundational layer that other platform stacks depend on for external access, secure communication, and persistent storage.
|
||||
|
||||
## Key Features
|
||||
|
||||
* [Feature 1]
|
||||
* [Feature 2]
|
||||
* [Feature 3]
|
||||
|
||||
## Purpose in EDP
|
||||
|
||||
[Explain the role this component plays in the Edge Developer Platform and how it contributes to the overall platform capabilities]
|
||||
* **Automated TLS Certificate Management**: Let's Encrypt integration via cert-manager for automatic certificate provisioning and renewal
|
||||
* **Cloud Load Balancer Integration**: Nginx ingress controller configured with OTC-specific Elastic Load Balancer (ELB) annotations
|
||||
* **Native Storage Provisioning**: Default StorageClass using Huawei FlexVolume provisioner for block storage
|
||||
* **Prometheus Metrics**: Built-in monitoring capabilities for ingress traffic and performance
|
||||
* **High Availability**: Rolling update strategy with minimal downtime
|
||||
* **HTTP-01 Challenge Support**: ACME validation through ingress for certificate issuance
|
||||
|
||||
## Repository
|
||||
|
||||
**Code**: [Link to source code repository]
|
||||
**Code**: [OTC Stack Templates](https://edp.buildth.ing/DevFW-CICD/stacks/src/branch/main/template/stacks/otc)
|
||||
|
||||
**Documentation**: [Link to component-specific documentation]
|
||||
**Documentation**:
|
||||
* [cert-manager Documentation](https://cert-manager.io/docs/)
|
||||
* [ingress-nginx Documentation](https://kubernetes.github.io/ingress-nginx/)
|
||||
* [Open Telekom Cloud Documentation](https://docs.otc.t-systems.com/)
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Prerequisites
|
||||
|
||||
* [Prerequisite 1]
|
||||
* [Prerequisite 2]
|
||||
* Kubernetes cluster running on Open Telekom Cloud
|
||||
* ArgoCD installed (provided by `core` stack)
|
||||
* Environment variables configured:
|
||||
- `LOADBALANCER_ID`: OTC Elastic Load Balancer ID
|
||||
- `LOADBALANCER_IP`: OTC Elastic Load Balancer IP address
|
||||
- `CLIENT_REPO_DOMAIN`: Git repository domain
|
||||
- `CLIENT_REPO_ORG_NAME`: Git repository organization
|
||||
- `CLIENT_REPO_ID`: Client repository identifier
|
||||
- `DOMAIN`: Domain name for the environment
|
||||
|
||||
### Quick Start
|
||||
|
||||
[Step-by-step guide to get started with this component]
|
||||
The OTC stack is deployed as part of the EDP installation process:
|
||||
|
||||
1. [Step 1]
|
||||
2. [Step 2]
|
||||
3. [Step 3]
|
||||
1. **Trigger Deploy Pipeline**
|
||||
- Go to [Infra Deploy Pipeline](https://edp.buildth.ing/DevFW/infra-deploy/actions?workflow=deploy.yaml)
|
||||
- Click on Run workflow
|
||||
- Enter a name in "Select environment directory to deploy". This must be DNS Compatible.
|
||||
- Execute workflow
|
||||
|
||||
2. **ArgoCD Synchronization**
|
||||
ArgoCD automatically deploys:
|
||||
- cert-manager with ClusterIssuer for Let's Encrypt
|
||||
- ingress-nginx controller with OTC load balancer integration
|
||||
- Default StorageClass for OTC block storage
|
||||
|
||||
### Verification
|
||||
|
||||
[How to verify the component is working correctly]
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### [Use Case 1]
|
||||
|
||||
[Example with code/commands showing common use case]
|
||||
Verify the OTC stack deployment:
|
||||
|
||||
```bash
|
||||
# Example commands
|
||||
# Check ArgoCD applications status
|
||||
kubectl get application otc -n argocd
|
||||
kubectl get application cert-manager -n argocd
|
||||
kubectl get application ingress-nginx -n argocd
|
||||
kubectl get application storageclass -n argocd
|
||||
|
||||
# Verify cert-manager pods
|
||||
kubectl get pods -n cert-manager
|
||||
|
||||
# Check ingress-nginx controller
|
||||
kubectl get pods -n ingress-nginx
|
||||
|
||||
# Verify ClusterIssuer status
|
||||
kubectl get clusterissuer main
|
||||
|
||||
# Check StorageClass
|
||||
kubectl get storageclass default
|
||||
```
|
||||
|
||||
### [Use Case 2]
|
||||
|
||||
[Another common scenario]
|
||||
|
||||
## Integration Points
|
||||
|
||||
* **[Component A]**: [How it integrates]
|
||||
* **[Component B]**: [How it integrates]
|
||||
* **[Component C]**: [How it integrates]
|
||||
|
||||
## Architecture
|
||||
|
||||
[Optional: Add architectural diagrams and descriptions]
|
||||
### Component Architecture
|
||||
|
||||
### Component Architecture (C4)
|
||||
The OTC stack consists of three primary components:
|
||||
|
||||
[Add C4 Container or Component diagrams showing the internal structure]
|
||||
**cert-manager**:
|
||||
- Automates TLS certificate lifecycle management
|
||||
- Integrates with Let's Encrypt ACME server (production endpoint)
|
||||
- Uses HTTP-01 challenge validation via ingress
|
||||
- Creates and manages certificates as Kubernetes resources
|
||||
- Single replica deployment
|
||||
|
||||
### Sequence Diagrams
|
||||
**ingress-nginx**:
|
||||
- Kubernetes ingress controller based on Nginx
|
||||
- Routes external traffic to internal services
|
||||
- Integrated with OTC Elastic Load Balancer (ELB)
|
||||
- Supports TLS termination with cert-manager issued certificates
|
||||
- Rolling update strategy with max 1 unavailable pod
|
||||
- Prometheus metrics exporter with ServiceMonitor
|
||||
|
||||
[Add sequence diagrams showing key interaction flows with other components]
|
||||
**StorageClass**:
|
||||
- Default storage provisioner for persistent volumes
|
||||
- Uses Huawei FlexVolume driver (`flexvolume-huawei.com/fuxivol`)
|
||||
- SATA block storage type
|
||||
- Immediate volume binding mode
|
||||
- Supports dynamic volume expansion
|
||||
|
||||
### Deployment Architecture
|
||||
### Integration Flow
|
||||
|
||||
[Add infrastructure and deployment diagrams showing how the component is deployed]
|
||||
```
|
||||
External Traffic → OTC ELB → ingress-nginx → Kubernetes Services
|
||||
↓
|
||||
cert-manager (TLS certificates)
|
||||
↓
|
||||
Let's Encrypt ACME
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
[Key configuration options and how to set them]
|
||||
### cert-manager Configuration
|
||||
|
||||
**Helm Values** (`stacks/otc/cert-manager/values.yaml`):
|
||||
|
||||
```yaml
|
||||
crds:
|
||||
enabled: true
|
||||
replicaCount: 1
|
||||
```
|
||||
|
||||
**ClusterIssuer** (`stacks/otc/cert-manager/manifests/clusterissuer.yaml`):
|
||||
|
||||
```yaml
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: ClusterIssuer
|
||||
metadata:
|
||||
name: main
|
||||
spec:
|
||||
acme:
|
||||
email: admin@think-ahead.tech
|
||||
server: https://acme-v02.api.letsencrypt.org/directory
|
||||
privateKeySecretRef:
|
||||
name: cluster-issuer-account-key
|
||||
solvers:
|
||||
- http01:
|
||||
ingress:
|
||||
ingressClassName: nginx
|
||||
```
|
||||
|
||||
**Key Settings**:
|
||||
- CRDs installed automatically
|
||||
- Production Let's Encrypt ACME endpoint
|
||||
- HTTP-01 validation through nginx ingress
|
||||
- ClusterIssuer named `main` for cluster-wide certificate issuance
|
||||
|
||||
### ingress-nginx Configuration
|
||||
|
||||
**Helm Values** (`stacks/otc/ingress-nginx/values.yaml`):
|
||||
|
||||
```yaml
|
||||
controller:
|
||||
updateStrategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxUnavailable: 1
|
||||
service:
|
||||
annotations:
|
||||
kubernetes.io/elb.class: union
|
||||
kubernetes.io/elb.port: '80'
|
||||
kubernetes.io/elb.id: {{{ .Env.LOADBALANCER_ID }}}
|
||||
kubernetes.io/elb.ip: {{{ .Env.LOADBALANCER_IP }}}
|
||||
ingressClassResource:
|
||||
name: nginx
|
||||
allowSnippetAnnotations: true
|
||||
config:
|
||||
proxy-buffer-size: 32k
|
||||
use-forwarded-headers: "true"
|
||||
metrics:
|
||||
enabled: true
|
||||
serviceMonitor:
|
||||
additionalLabels:
|
||||
release: "ingress-nginx"
|
||||
enabled: true
|
||||
```
|
||||
|
||||
**Key Settings**:
|
||||
- **OTC Load Balancer Integration**: Annotations configure connection to OTC ELB
|
||||
- **Rolling Updates**: Minimizes downtime with 1 pod unavailable during updates
|
||||
- **Snippet Annotations**: Enabled for advanced ingress configuration (idpbuilder compatibility)
|
||||
- **Proxy Buffer**: 32k buffer size for handling large headers
|
||||
- **Forwarded Headers**: Preserves original client information through proxies
|
||||
- **Metrics**: Prometheus ServiceMonitor for observability
|
||||
|
||||
### StorageClass Configuration
|
||||
|
||||
**StorageClass** (`stacks/otc/storageclass/storageclass.yaml`):
|
||||
|
||||
```yaml
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
annotations:
|
||||
storageclass.beta.kubernetes.io/is-default-class: "true"
|
||||
name: default
|
||||
parameters:
|
||||
kubernetes.io/hw:passthrough: "true"
|
||||
kubernetes.io/storagetype: BS
|
||||
kubernetes.io/volumetype: SATA
|
||||
kubernetes.io/zone: eu-de-02
|
||||
provisioner: flexvolume-huawei.com/fuxivol
|
||||
reclaimPolicy: Delete
|
||||
volumeBindingMode: Immediate
|
||||
allowVolumeExpansion: true
|
||||
```
|
||||
|
||||
**Key Settings**:
|
||||
- **Default StorageClass**: Automatically used when no StorageClass specified
|
||||
- **OTC Zone**: Provisioned in `eu-de-02` availability zone
|
||||
- **SATA Volumes**: Block storage (BS) with SATA performance tier
|
||||
- **Volume Expansion**: Supports resizing persistent volumes dynamically
|
||||
- **Reclaim Policy**: Volumes deleted when PersistentVolumeClaim is removed
|
||||
|
||||
### ArgoCD Application Configuration
|
||||
|
||||
**Registry Application** (`template/registry/otc.yaml`):
|
||||
- Name: `otc`
|
||||
- Manages the OTC stack directory
|
||||
- Automated sync with prune and self-heal enabled
|
||||
- Creates namespaces automatically
|
||||
|
||||
**Component Applications**:
|
||||
|
||||
**cert-manager** (referenced in stack):
|
||||
- Deploys cert-manager Helm chart
|
||||
- Automated self-healing enabled
|
||||
- Includes ClusterIssuer manifest for Let's Encrypt
|
||||
|
||||
**ingress-nginx** (`template/stacks/otc/ingress-nginx.yaml`):
|
||||
- Deploys from official Kubernetes ingress-nginx repository
|
||||
- Chart version: helm-chart-4.12.1
|
||||
- References environment-specific values from stacks-instances repository
|
||||
|
||||
**storageclass** (`template/stacks/otc/storageclass.yaml`):
|
||||
- Deploys StorageClass manifest
|
||||
- Managed as ArgoCD Application
|
||||
- Automated sync with unlimited retries
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Creating an Ingress with Automatic TLS
|
||||
|
||||
Create an ingress resource that automatically provisions a TLS certificate:
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: my-app
|
||||
namespace: my-namespace
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: main
|
||||
nginx.ingress.kubernetes.io/ssl-redirect: "true"
|
||||
spec:
|
||||
ingressClassName: nginx
|
||||
tls:
|
||||
- hosts:
|
||||
- myapp.example.com
|
||||
secretName: myapp-tls
|
||||
rules:
|
||||
- host: myapp.example.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: my-app-service
|
||||
port:
|
||||
number: 80
|
||||
```
|
||||
|
||||
cert-manager will automatically:
|
||||
1. Detect the ingress with `cert-manager.io/cluster-issuer` annotation
|
||||
2. Create a Certificate resource
|
||||
3. Request certificate from Let's Encrypt using HTTP-01 challenge
|
||||
4. Store certificate in `myapp-tls` secret
|
||||
5. Renew certificate before expiration
|
||||
|
||||
### Creating a PersistentVolumeClaim
|
||||
|
||||
Use the default OTC StorageClass for persistent storage:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: my-data
|
||||
namespace: my-namespace
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
storageClassName: csi-disk
|
||||
```
|
||||
|
||||
### Expanding an Existing Volume
|
||||
|
||||
Resize a persistent volume by editing the PVC:
|
||||
|
||||
```bash
|
||||
# Edit the PVC storage request
|
||||
kubectl patch pvc my-data -n my-namespace -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'
|
||||
|
||||
# Verify expansion
|
||||
kubectl get pvc my-data -n my-namespace
|
||||
```
|
||||
|
||||
The volume will expand automatically due to `allowVolumeExpansion: true` in the StorageClass.
|
||||
|
||||
### Custom Ingress Configuration
|
||||
|
||||
Use nginx ingress snippets for advanced routing:
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: advanced-app
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: main
|
||||
nginx.ingress.kubernetes.io/configuration-snippet: |
|
||||
more_set_headers "X-Custom-Header: value";
|
||||
if ($http_user_agent ~* "bot") {
|
||||
return 403;
|
||||
}
|
||||
spec:
|
||||
ingressClassName: nginx
|
||||
rules:
|
||||
- host: app.example.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: app-service
|
||||
port:
|
||||
number: 8080
|
||||
```
|
||||
|
||||
## Integration Points
|
||||
|
||||
* **Core Stack**: Requires ArgoCD for deployment orchestration
|
||||
* **All Application Stacks**: Depends on OTC stack for:
|
||||
- External access via ingress-nginx
|
||||
- TLS certificates via cert-manager
|
||||
- Persistent storage via default StorageClass
|
||||
* **Observability Stack**: ingress-nginx metrics exported to Prometheus
|
||||
* **Coder Stack**: Uses ingress and cert-manager for workspace access
|
||||
* **Forgejo Stack**: Requires ingress and TLS for Git repository access
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### [Common Issue 1]
|
||||
### Certificate Issuance Fails
|
||||
|
||||
**Problem**: [Description]
|
||||
**Problem**: Certificate remains in `Pending` state and is not issued
|
||||
|
||||
**Solution**: [How to fix]
|
||||
**Solution**:
|
||||
1. Check Certificate status:
|
||||
```bash
|
||||
kubectl get certificate -A
|
||||
kubectl describe certificate <cert-name> -n <namespace>
|
||||
```
|
||||
|
||||
### [Common Issue 2]
|
||||
2. Verify ClusterIssuer is ready:
|
||||
```bash
|
||||
kubectl get clusterissuer main
|
||||
kubectl describe clusterissuer main
|
||||
```
|
||||
|
||||
**Problem**: [Description]
|
||||
3. Check cert-manager logs:
|
||||
```bash
|
||||
kubectl logs -n cert-manager -l app=cert-manager
|
||||
```
|
||||
|
||||
**Solution**: [How to fix]
|
||||
4. Verify HTTP-01 challenge can reach ingress:
|
||||
```bash
|
||||
kubectl get challenges -A
|
||||
kubectl describe challenge <challenge-name> -n <namespace>
|
||||
```
|
||||
|
||||
## Status
|
||||
5. Common issues:
|
||||
- DNS not pointing to load balancer IP
|
||||
- Firewall blocking HTTP (port 80) traffic
|
||||
- Ingress class not set to `nginx`
|
||||
- Let's Encrypt rate limits exceeded
|
||||
|
||||
**Maturity**: [Production / Beta / Experimental]
|
||||
### Ingress Controller Not Ready
|
||||
|
||||
**Problem**: ingress-nginx pods are not running or LoadBalancer service has no external IP
|
||||
|
||||
**Solution**:
|
||||
1. Check ingress controller status:
|
||||
```bash
|
||||
kubectl get pods -n ingress-nginx
|
||||
kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller
|
||||
```
|
||||
|
||||
2. Verify LoadBalancer service:
|
||||
```bash
|
||||
kubectl get svc -n ingress-nginx
|
||||
kubectl describe svc ingress-nginx-controller -n ingress-nginx
|
||||
```
|
||||
|
||||
3. Check OTC load balancer annotations:
|
||||
```bash
|
||||
kubectl get svc ingress-nginx-controller -n ingress-nginx -o yaml
|
||||
```
|
||||
|
||||
4. Verify environment variables are set correctly:
|
||||
- `LOADBALANCER_ID` matches OTC ELB ID
|
||||
- `LOADBALANCER_IP` matches ELB public IP
|
||||
|
||||
5. Check OTC console for ELB configuration and health checks
|
||||
|
||||
### Storage Provisioning Fails
|
||||
|
||||
**Problem**: PersistentVolumeClaim remains in `Pending` state
|
||||
|
||||
**Solution**:
|
||||
1. Check PVC status:
|
||||
```bash
|
||||
kubectl get pvc -A
|
||||
kubectl describe pvc <pvc-name> -n <namespace>
|
||||
```
|
||||
|
||||
2. Verify StorageClass exists and is default:
|
||||
```bash
|
||||
kubectl get storageclass
|
||||
kubectl describe storageclass default
|
||||
```
|
||||
|
||||
3. Check volume provisioner logs:
|
||||
```bash
|
||||
kubectl logs -n kube-system -l app=csi-disk-plugin
|
||||
```
|
||||
|
||||
4. Common issues:
|
||||
- Insufficient quota in OTC project
|
||||
- Invalid zone configuration (must be `eu-de-02`)
|
||||
- Requested storage size exceeds limits
|
||||
- Missing IAM permissions for volume creation
|
||||
|
||||
### Ingress Returns 503 Service Unavailable
|
||||
|
||||
**Problem**: Ingress configured but returns 503 error
|
||||
|
||||
**Solution**:
|
||||
1. Verify backend service exists:
|
||||
```bash
|
||||
kubectl get svc <service-name> -n <namespace>
|
||||
kubectl get endpoints <service-name> -n <namespace>
|
||||
```
|
||||
|
||||
2. Check if pods are ready:
|
||||
```bash
|
||||
kubectl get pods -n <namespace> -l <service-selector>
|
||||
```
|
||||
|
||||
3. Verify ingress configuration:
|
||||
```bash
|
||||
kubectl describe ingress <ingress-name> -n <namespace>
|
||||
```
|
||||
|
||||
4. Check nginx ingress logs:
|
||||
```bash
|
||||
kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller --tail=100
|
||||
```
|
||||
|
||||
5. Test service connectivity from ingress controller:
|
||||
```bash
|
||||
kubectl exec -n ingress-nginx <controller-pod> -- curl http://<service-name>.<namespace>.svc.cluster.local:<port>
|
||||
```
|
||||
|
||||
### TLS Certificate Shows as Invalid
|
||||
|
||||
**Problem**: Browser shows certificate warning or certificate details are incorrect
|
||||
|
||||
**Solution**:
|
||||
1. Verify certificate is ready:
|
||||
```bash
|
||||
kubectl get certificate <cert-name> -n <namespace>
|
||||
```
|
||||
|
||||
2. Check certificate contents:
|
||||
```bash
|
||||
kubectl get secret <tls-secret-name> -n <namespace> -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout
|
||||
```
|
||||
|
||||
3. Ensure certificate covers the correct domain:
|
||||
```bash
|
||||
kubectl describe certificate <cert-name> -n <namespace>
|
||||
```
|
||||
|
||||
4. Force certificate renewal if expired or incorrect:
|
||||
```bash
|
||||
kubectl delete certificate <cert-name> -n <namespace>
|
||||
# cert-manager will automatically recreate it
|
||||
```
|
||||
|
||||
## Additional Resources
|
||||
|
||||
* [Link to external documentation]
|
||||
* [Link to community resources]
|
||||
* [Link to related components]
|
||||
|
||||
## Documentation Notes
|
||||
|
||||
[Instructions for team members filling in this documentation - remove this section once complete]
|
||||
* [cert-manager Documentation](https://cert-manager.io/docs/)
|
||||
* [ingress-nginx User Guide](https://kubernetes.github.io/ingress-nginx/user-guide/)
|
||||
* [Open Telekom Cloud Documentation](https://docs.otc.t-systems.com/)
|
||||
* [Let's Encrypt Documentation](https://letsencrypt.org/docs/)
|
||||
* [Kubernetes Ingress Concepts](https://kubernetes.io/docs/concepts/services-networking/ingress/)
|
||||
* [Kubernetes Storage Classes](https://kubernetes.io/docs/concepts/storage/storage-classes/)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue