provider-controller

A Kubernetes operator for provisioning and managing managed Kubernetes clusters via infrastructure providers (Equinix Metal, AWS, etc.).

Multi-Tenancy (KCP Workspaces)

The provider-controller is fully multi-tenant and tenant-agnostic. It runs centrally on vanilla Kubernetes and watches KubernetesCluster requests from any workspace bound to our pool of clusters (via KCP APIExport / APIBinding).

When a tenant creates a request inside their logical KCP workspace, the central controller:

1. Receives the request with a logical ClusterName (which identifies the originating workspace).
2. Derives a deterministic workspace token fingerprint (12-char SHA-256 string).
3. Namespaces provider-facing Crossplane and OTC resources into isolating namespaces matching kcp-<fingerprint> (e.g. kcp-d37f1981a6be).
4. Reconciles and deletes underlying OTC resources safely per-workspace, preventing cross-tenant name collisions and maintaining strict workspace boundary isolation.

Overview

The provider-controller watches KubernetesCluster resources and orchestrates the provisioning of infrastructure. It:

1. Validates provider credentials (via ProviderConfig)
2. Matches requested machine capabilities to available provider flavors (via MachineCapabilities and ProviderMachineFlavor)
3. Creates Crossplane Composite Resources (XRs) to provision infrastructure
4. Generates and stores cluster kubeconfigs as Kubernetes Secrets

Custom Resources

KubernetesCluster (Namespace Scoped)

User-facing CRD for requesting a new managed cluster.

apiVersion: platform.edgeconnect.eu/v1alpha1
kind: KubernetesCluster
metadata:
  name: my-cluster
  namespace: default
spec:
  machineCapabilitiesRef:
    name: my-capabilities
  nodeCount: 3
  nodeDiskSize: 100Gi
  providerRef:
    type: OTC
    configRef:
      name: my-provider
  kubernetes:
    version: 1.28.0

ProviderConfig (Namespace Scoped)

Stores provider credentials and configuration.

apiVersion: platform.edgeconnect.eu/v1alpha1
kind: ProviderConfig
metadata:
  name: my-provider
  namespace: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: default
      name: provider-credentials
      key: credentials

MachineCapabilities (Namespace Scoped)

Describes desired compute resources. The controller discovers matching flavors from all providers.

apiVersion: platform.edgeconnect.eu/v1alpha1
kind: MachineCapabilities
metadata:
  name: my-capabilities
  namespace: default
spec:
  vCPUCount: 4
  memorySize: 8Gi
  gpuModelName: RTX3080

ProviderMachineFlavor (Cluster Scoped)

Registered by provider integrations; describes available machine types.

apiVersion: platform.edgeconnect.eu/v1alpha1
kind: ProviderMachineFlavor
metadata:
  name: otc-c3-large
spec:
  available: true
  providerRef:
    type: OTC
  flavor: c3.large.x86
  capabilities:
    vCPUCount: 24
    memorySize: 512Gi
    gpuModelName: null
  availableZones:
    - us-east-1
    - eu-west-1

Running Locally

# Install CRDs
kubectl apply -f config/crd/

# Run the controller
go run ./cmd/provider-controller/main.go \
  --metrics-bind-address=:8080 \
  --health-probe-bind-address=:8081

Development

Crossplane OTC Bootstrap

Install Crossplane + OTC provider on kind:

make crossplane-kind

For a true from-scratch kind setup, also create the OTC credentials secret expected by ClusterProviderConfig/infra:

make CROSSPLANE_KUBECONFIG=/tmp/devfw-kind.kubeconfig \
  OTC_CREDENTIALS_FILE=/path/to/otc-credentials.txt \
  crossplane-create-otc-secret

Credential file format example (password mode):

auth_url=https://iam.eu-de.otc.t-systems.com/v3
user_name=<OTC_USER>
password=<OTC_PASSWORD>
domain_name=<OTC_DOMAIN>
tenant_name=eu-de

Credential file format example (AK/SK mode):

{
  "access_key": "<OTC_ACCESS_KEY>",
  "secret_key": "<OTC_SECRET_KEY>",
  "auth_url": "https://iam.eu-de.otc.t-systems.com/v3",
  "region": "eu-de"
}

Use an already existing Crossplane instead of installing one locally:

# Local run (watch KCP, write to existing Crossplane cluster)
make KCP_WORKSPACE=root:orgs \
  CROSSPLANE_KUBECONFIG=/path/to/existing-crossplane.kubeconfig \
  run-kcp-local

# In-cluster deployment (stores external Crossplane kubeconfig in secret)
make KCP_WORKSPACE=root:orgs \
  CROSSPLANE_KUBECONFIG=/path/to/existing-crossplane.kubeconfig \
  deploy-kcp-controller

This uses the default setup manifest:

• provider-controller/hack/crossplane-otc-setup.yaml
• and then applies provider-controller/hack/crossplane-otc-providerconfig.yaml after provider CRDs are available.

Create OTC credentials Secret expected by infra ClusterProviderConfig:

kubectl --kubeconfig=/tmp/devfw-kind.kubeconfig -n crossplane-system create secret generic opentelekomcloud-credentials \
  --from-literal=credentials='auth_url=https://iam.eu-de.otc.t-systems.com/v3
user_name=<OTC_USER>
password=<OTC_PASSWORD>
domain_name=<OTC_DOMAIN>
tenant_name=eu-de' \
  --dry-run=client -o yaml | kubectl --kubeconfig=/tmp/devfw-kind.kubeconfig apply -f -

Recommended demo flow with plain YAML apply:

1. Setup Crossplane + OTC on kind:

make crossplane-kind

2. Show OTC demo YAML files to edit:

make otc-config

Edit these files:

• hack/examples/demo/02-providerconfig.yaml
• hack/examples/demo/03-kubernetescluster.yaml

hack/examples/demo/01-secret.yaml contains only the demo Namespace — no credentials are committed. The OTC credentials Secret (demo/otc-creds) is generated from your AK/SK in step 3.

KubernetesCluster can omit vpcID and subnetID. In that case provider-controller creates VpcV1 and SubnetV1 resources automatically and wires CCE ClusterV3 using Crossplane references.

3. Generate the demo credentials Secret in the target KCP workspace from your AK/SK:

make KCP_WORKSPACE=root:orgs demo-secret-create \
  OTC_ACCESS_KEY=<ak> OTC_SECRET_KEY=<sk>

This applies the demo Namespace and creates Secret demo/otc-creds (key credentials) with access_key, secret_key, auth_url, region, and tenant_name. Override defaults with DEMO_REGION, OTC_AUTH_URL, and OTC_PROJECT_NAME if needed. No credentials are stored in the repository.

4. Apply demo example resources (ProviderConfig + KubernetesCluster only):

make KCP_WORKSPACE=root:orgs \
  CROSSPLANE_KUBECONFIG=/tmp/devfw-kind.kubeconfig \
  demo-apply-cluster

demo-apply-cluster now performs plain kubectl apply on the example YAML files and does not require .env files.

5. Wait for the OTC cluster and worker node pool, then fetch the workload kubeconfig:

make KCP_WORKSPACE=root:orgs \
  CROSSPLANE_KUBECONFIG=/tmp/devfw-kind.kubeconfig \
  demo-up
make KCP_WORKSPACE=root:orgs demo-kubeconfig

demo-up waits for VpcV1, SubnetV1, ClusterV3 (Synced + Ready) and the NodePoolV3 worker pool (Synced).

Worker node placement and capacity (SoldOut)

OTC node-flavor capacity is region/AZ/time dependent. If the worker NodePoolV3 stays in OTC status SoldOut, the requested flavor has no capacity in the chosen AZ. To avoid this:

• KubernetesCluster.spec.nodeAvailableZone: random lets OTC place workers in any AZ that has capacity (the demo sets this). This is independent of the control-plane availableZone.
• If the whole region is SoldOut for a flavor, pick a different spec.nodeFlavor (e.g. s3.xlarge.4, c4.xlarge.4) and/or lower spec.nodeCount, then re-apply. demo-up prints a SoldOut warning when it detects this condition.

The controller always writes literal clusterId and keyPair values onto the NodePoolV3 (never Crossplane *Ref fields), so the node pool can be both created and cleanly deleted — Crossplane does not resolve references during deletion, which would otherwise deadlock teardown.

Generate CRDs and DeepCopy

make generate
make manifests

Build the controller image

make docker-build

Roadmap

• Crossplane integration for actual infrastructure provisioning
• MachineCapabilities auto-discovery from provider APIs
• Multi-provider failover and load-balancing
• Cluster lifecycle management (upgrades, scaling, deletion)
• Integration with KCP for logical cluster workspaces