kcp-karmada-provider

Platform provider controller that bridges kcp tenant-facing APIs to Karmada compute orchestration. Watches FleetWorkload and KubernetesCluster resources in kcp virtual workspaces and translates them into Karmada resources and cluster-operator Cluster CRs.

Module path: edp.buildth.ing/devfw-cicd/kcp-karmada-provider

Architecture

┌─────────────────────────────────────────────────────────┐
│                     kcp (API layer)                     │
│                                                         │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │
│  │ tenant-alpha │  │ tenant-beta  │  │ tenant-gamma │  │
│  │ FleetWorkload│  │ FleetWorkload│  │ FleetWorkload│  │
│  │ Fleet        │  │ Fleet        │  │ Fleet        │  │
│  │ Placement    │  │ Placement    │  │ Placement    │  │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘  │
│         │                 │                 │           │
│  ┌──────┴─────────────────┴─────────────────┴───────┐  │
│  │  APIExport "edge-portal-fleet"                   │  │
│  │  → APIExportEndpointSlice (virtual workspace)    │  │
│  └──────────────────────┬───────────────────────────┘  │
└─────────────────────────┼──────────────────────────────┘
                          │ watches via VW endpoint
                          ▼
               ┌─────────────────────┐
               │ kcp-karmada-provider│
               │                     │
               │ 1. Discover VW URLs │
               │ 2. Watch FleetWL    │
               │ 3. Read Policy+Fleet│
               │ 4. Translate        │
               │ 5. Apply to Karmada │
               │ 6. Update status    │
               └─────┬─────────┬─────┘
                     │         │
          ┌──────────┘         └──────────┐
          ▼                               ▼
┌──────────────────────┐   ┌──────────────────────────┐
│  Karmada (compute)   │   │  Management Plane        │
│                      │   │                          │
│  PropagationPolicy   │   │  cluster-operator        │
│  Deployment/Service  │   │  Cluster CRs → KIND/OTC  │
│  ResourceBindings    │   │                          │
└──────────────────────┘   └──────────────────────────┘

Design

ADRs live in docs/adr/ (numbered, TEMPLATE.md for new ones):

• ADR-0001: Karmada as dataplane dispatcher
• ADR-0002: Immutable revision for rollback
• ADR-0003: Split APIResourceSchemas

Rolling update and rollback design: docs/rolling-update-design.md.

Reconciliation Flow

FleetWorkload

1. Discover — Read APIExportEndpointSlice from kcp, extract virtual workspace endpoints.
2. Watch — Start dynamic informers for FleetWorkload and KubernetesCluster across all tenant namespaces.
3. Translate — Convert tenant resources into Karmada objects: deterministic namespace (kcp-{hash}), PropagationPolicy, Deployment/Service.
4. Apply — Create or update Karmada resources via merge patch.
5. Rollout — Execute rolling update waves, manage evictions, track revisions.
6. Status — Write conditions, placements, rollout state back to kcp.
7. Cleanup — On deletion, remove Karmada resources and release the finalizer.

KubernetesCluster

1. Create a Cluster CR on the management plane (cluster-operator provisions the cluster).
2. When Ready, join the cluster to Karmada via karmadactl.
3. Update the KubernetesCluster status in kcp.

Translation Rules

Portal concept	Karmada concept
Fleet.spec.clusterRefs	PropagationPolicy.spec.placement.clusterAffinity.clusterNames
Fleet.spec.selector	PropagationPolicy.spec.placement.clusterAffinity.labelSelector
PlacementPolicy.spec.requirements	fieldSelector (well-known) / labelSelector (custom)
PlacementPolicy.strategy=Weighted	replicaSchedulingType=Divided, preference=Weighted
PlacementPolicy.strategy=Spread	replicaSchedulingType=Divided + spreadConstraints
PlacementPolicy.strategy=FixedReplicas	replicaSchedulingType=Duplicated

Tenant Isolation

• Tenants interact only with kcp workspaces — they never see Karmada.
• Workspace identity is derived from the kcp.io/cluster annotation.
• All Karmada objects carry edge-connect.eu/source-workspace, edge-connect.eu/source-name labels for correlation.

Flags

All flags can be set via environment variables (see defaults).

Flag	Env	Default	Description
--kcp-kubeconfig	KCP_KUBECONFIG		Path to kcp kubeconfig (provider workspace)
--karmada-kubeconfig	KARMADA_KUBECONFIG		Path to Karmada control plane kubeconfig
--mgmt-kubeconfig	MGMT_KUBECONFIG		Path to management-plane kubeconfig
--mgmt-namespace	MGMT_NAMESPACE	default	Namespace for Cluster CRs on the management plane
--node-id	NODE_ID		Stable ID for this provider node
--namespace		edge-connect-system	Namespace for operations
--api-export-name		edge-portal-fleet	Name of the kcp APIExport
--kcp-external-url	KCP_EXTERNAL_URL		Rewrites internal VW endpoint URLs
--reconcile-interval		30s	Endpoint discovery interval
--health-addr		:8081	Address for /healthz and /readyz

Local Development

Prerequisites

• Go 1.22+
• Docker (for integration tests)
• karmadactl on PATH (for cluster join/unjoin)

Build

make build        # → bin/kcp-karmada-provider
make test         # unit tests
make vet          # go vet
make lint         # golangci-lint
make fmt          # gofmt

Run locally

make run          # builds + runs with env vars KCP_KUBECONFIG, KARMADA_KUBECONFIG, MGMT_KUBECONFIG

Docker

make docker-build

Image: gcr.io/distroless/static-debian12:nonroot + statically linked binary.

Code generation

make generate     # regenerate deepcopy from //+kubebuilder markers
make manifests    # regenerate CRDs under config/crd
make tidy         # go mod tidy

Integration Testing

The integration harness creates a fully self-contained local stack: 3 kind clusters, Karmada, kcp, and the provider binary.

make integration-up                   # kind + karmada + kcp (no provider)
make integration-up WITH_PROVIDER=1   # + build and start the provider
make test-integration                 # run integration tests
make integration-down                 # tear everything down

The harness installs all tools (kind, karmadactl, kcp) into test/integration/.work/bin/ — no global installs needed. Kubeconfigs land in test/integration/.work/kubeconfigs/.

Running the portal against the harness

After make integration-up WITH_PROVIDER=1, copy the consumer kubeconfig to the portal:

cp test/integration/.work/kubeconfigs/kcp-consumer.kubeconfig \
   ~/Dev/edge-connect-portal-poc/.kcp-karmada.kubeconfig
cd ~/Dev/edge-connect-portal-poc
bun run kcp:seed:karmada
bun run dev:karmada

Project Layout

main.go                         Entrypoint, flags, client setup
internal/
  controller/
    controller.go               Main reconcile loop, informer-based watch, finalizers
    cluster_reconciler.go       KubernetesCluster → cluster-operator + Karmada join
  endpoint/
    discovery.go                APIExportEndpointSlice watcher, VW client factory
  translate/
    translate.go                FleetWorkload → Karmada PropagationPolicy + workloads
  karmada/
    applier.go                  Apply/delete resources on Karmada control plane
    binding.go                  Read Karmada ResourceBindings for per-cluster status
  status/
    updater.go                  Write status back to kcp FleetWorkload
  rollout/
    engine.go                   Rolling update orchestration (waves, suspension)
    operations.go               Rollback/restart operation parsing
    eligibility.go              Cluster eligibility filtering for auto-migration
    eviction.go                 Eviction merging into placements
  revision/
    minter.go                   Create FleetWorkloadRevision snapshots
    gc.go                       Garbage-collect old revisions
    rollback.go                 Restore state from a previous revision
  cluster/
    state.go                    Cluster health, metrics, dwell tracking
    live_state.go               Live cluster state from KubernetesCluster + ClusterMetrics
  conditions/
    conditions.go               Kubernetes-native status condition helpers
  platform/
    platform.go                 GVR constants, finalizer helpers, label utilities
api/
  platform/v1alpha1/            Go types for platform CRDs (deepcopy generated)
config/
  kcp/                          APIExport + 13 APIResourceSchema YAMLs for kcp
  kcp-karmada-rbac.yaml         RBAC for the provider workspace
  kcp-karmada-tenant-rbac.yaml  RBAC for tenant workspaces
  kcp-karmada-apibinding.yaml   APIBinding for tenant workspaces
test/
  integration/harness/          Shell scripts for local kind+karmada+kcp stack
docs/
  adr/                          Architecture Decision Records
  rolling-update-design.md      Rolling update and rollback design

IPCEI-CIS Reference Architecture

This controller is an EdgeConnect platform deliverable mapped to the Service Orchestration and Cloud-Edge Platform layers of the IPCEI-CIS Reference Architecture v3.0.

RA Layer	RA Component	Coverage
Service Orchestration	Service Orchestrator — multi-cluster scheduling	Primary (via Karmada)
Cloud-Edge Platform	Workload Deployment Manager (WDM)	Primary
Cloud-Edge Platform	Multi-Cluster Manager (MCM)	Primary
Cloud-Edge Platform	Cloud-Edge Federation — kcp virtual workspaces	Partial
Virtualization	Container Infrastructure Service Manager	Partial (via cluster-operator)

Origin

Originally extracted from the fed8ra monorepo; now fully independent with its own label domain (edge-connect.eu/*) and lifecycle.