Carbon Data Collector

Kubernetes-native controller that enriches Karmada clusters with real-time carbon intensity data from ENTSO-E (European Network of Transmission System Operators for Electricity).

Version: 0.2.0
Status: Production Ready
Project: IPCEI-CIS Sustainability Initiative
Organization: Deutsche Telekom AG

---

Overview

The Carbon Data Collector is a Kubernetes controller that automatically fetches electricity carbon intensity data from the ENTSO-E Transparency Platform and makes it available to the Karmada Carbon Scheduler via Custom Resource Definitions (CRDs). It enables zero-configuration carbon-aware workload placement across European datacenters.

Key Features

• Automated Data Collection: Hourly updates from ENTSO-E API for 23+ European zones
• Kubernetes-Native Storage: Uses ClusterCarbonMetrics CRD for data persistence
• Zero-Configuration Discovery: Automatically detects clusters via carbon.edge-connect.eu/entso-zone labels
• Comprehensive Metrics: Carbon intensity (gCO2/kWh) + renewable share (%)
• Production Ready: Health checks, Prometheus metrics, graceful error handling
• Lightweight: Minimal resource footprint, runs as single deployment

---

Architecture

┌────────────────────────────────────────────────────────────┐
│           Carbon Data Collector Controller                  │
│  ┌──────────────────────────────────────────────────────┐ │
│  │  Reconciliation Loop (every 1 hour)                  │ │
│  │  1. List Karmada Cluster resources                   │ │
│  │  2. Find clusters with carbon.edge-connect.eu/entso-zone label  │ │
│  │  3. Query ENTSO-E API for each zone                  │ │
│  │  4. Calculate carbon metrics                         │ │
│  │  5. Create/Update ClusterCarbonMetrics CRDs          │ │
│  └──────────────────────────────────────────────────────┘ │
└────────────────┬────────────────────────────────▲──────────┘
                 │                                 │
                 │ Queries generation data         │ Reads labels
                 ↓                                 │
┌─────────────────────────────┐     ┌─────────────┴──────────┐
│      ENTSO-E REST API       │     │  Karmada Clusters      │
│  Document A75 (Actual Gen)  │     │  with zone labels      │
│  23+ European bidding zones │     │                        │
└─────────────────────────────┘     └────────────────────────┘
                 │
                 │ Creates/Updates CRDs
                 ↓
┌─────────────────────────────────────────────────────────────┐
│              ClusterCarbonMetrics CRDs                       │
│  apiVersion: carbon.edge-connect.eu/v1alpha1              │
│  kind: ClusterCarbonMetrics                                 │
│  metadata:                                                  │
│    name: member1                                            │
│  spec:                                                      │
│    clusterName: member1                                     │
│    zone: DE                                                 │
│    zoneProvider: ENTSOE                                     │
│  status:                                                    │
│    carbonIntensity: 199.1  # gCO2/kWh                      │
│    renewableShare: 0.72    # 0.0-1.0                       │
│    lastUpdate: "2026-01-19T14:00:00Z"                      │
└─────────────────────────────────────────────────────────────┘
                 │
                 │ Read by Carbon Scheduler
                 ↓
┌─────────────────────────────────────────────────────────────┐
│         Karmada Carbon Scheduler Plugin                      │
│  - Reads ClusterCarbonMetrics CRDs                          │
│  - Scores clusters based on carbon intensity                │
│  - No API calls, pure CRD reads                             │
└─────────────────────────────────────────────────────────────┘

Data Flow:

1. Collector watches Karmada Clusters (every hour)
2. For each cluster with carbon.edge-connect.eu/entso-zone label:
   • Query ENTSO-E API for zone's generation mix
   • Calculate carbon intensity and renewable share
3. Create/Update ClusterCarbonMetrics CRD
4. Scheduler reads CRD during scheduling decisions

---

Quick Start

Prerequisites

• Karmada v1.12.0+ installed and running
• kubectl configured with Karmada contexts (karmada-apiserver, karmada-host)
• ENTSO-E API Token (free registration at transparency.entsoe.eu)
• Go 1.22+ (for building from source)

Get ENTSO-E API Token

1. Register at ENTSO-E Transparency Platform
2. Navigate to: Account → Web API → Generate Token
3. Save your token (format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)

Note: Token generation may take up to 24 hours for new accounts.

Installation

# 1. Clone repository
git clone https://edp.buildth.ing/DevFW-CICD/carbon-data-collector.git
cd carbon-data-collector

# 2. Build
make build

# 3. Build and load Docker image (for local kind cluster)
cd ../karmada-local
make build-carbon-collector

# 4. Label your clusters
kubectl --context karmada-apiserver label cluster member1 carbon.edge-connect.eu/entso-zone=DE
kubectl --context karmada-apiserver label cluster member2 carbon.edge-connect.eu/entso-zone=FR
kubectl --context karmada-apiserver label cluster member3 carbon.edge-connect.eu/entso-zone=SE

# 5. Deploy collector
make deploy-carbon-collector
# You will be prompted for your ENTSO-E API token

Verify Deployment

# Check pod status
kubectl --kubeconfig ~/.kube/karmada.config --context karmada-host \
  -n karmada-system get pods -l app=carbon-data-collector

# Check logs
kubectl --kubeconfig ~/.kube/karmada.config --context karmada-host \
  -n karmada-system logs -l app=carbon-data-collector --tail=50

# View ClusterCarbonMetrics CRDs
kubectl --context karmada-apiserver get clustercarbonmetrics

# View specific cluster's data
kubectl --context karmada-apiserver get clustercarbonmetrics member1 -o yaml

---

Configuration

Required: Cluster Labels

Each Karmada cluster must have a zone label to enable carbon data collection:

# Label clusters with ENTSO-E zone codes
kubectl --context karmada-apiserver label cluster <cluster-name> carbon.edge-connect.eu/entso-zone=<zone-code>

Example:

kubectl --context karmada-apiserver label cluster member1 carbon.edge-connect.eu/entso-zone=DE
kubectl --context karmada-apiserver label cluster member2 carbon.edge-connect.eu/entso-zone=FR
kubectl --context karmada-apiserver label cluster member3 carbon.edge-connect.eu/entso-zone=SE

Supported Zone Codes:

Zone Code	Region	Description
DE	Germany	German electricity market
FR	France	French electricity market
SE	Sweden	Swedish electricity market
NO-NO1	Norway	Oslo region
NO-NO2	Norway	Kristiansand region
DK-DK1	Denmark	Western Denmark
DK-DK2	Denmark	Eastern Denmark
NL	Netherlands	Dutch electricity market
BE	Belgium	Belgian electricity market
AT	Austria	Austrian electricity market
CH	Switzerland	Swiss electricity market
IT-NORD	Italy	Northern Italy
ES	Spain	Spanish electricity market
PT	Portugal	Portuguese electricity market
PL	Poland	Polish electricity market
CZ	Czech Republic	Czech electricity market

Full zone list: See docs/reference/zones.md

Required: ENTSO-E API Token Secret

The collector requires an ENTSO-E API token stored in a Kubernetes secret:

# Automatically created during deployment
# Or create manually:
kubectl --kubeconfig ~/.kube/karmada.config --context karmada-host \
  -n karmada-system create secret generic carbon-scheduler-secret \
  --from-literal=entsoe-api-token=<your-token-here>

Environment Variables

# ENTSO-E API Configuration
ENTSOE_API_TOKEN          # API token (required, from secret)

# Refresh Interval
REFRESH_INTERVAL=1h       # Data update frequency (default: 1h)
                          # Valid units: h (hours), m (minutes)
                          # Example: 30m, 2h

# Label Keys (customize if needed)
ZONE_LABEL_KEY=carbon.edge-connect.eu/entso-zone
CITY_LABEL_KEY=topology.kubernetes.io/city
LATITUDE_LABEL_KEY=topology.kubernetes.io/latitude
LONGITUDE_LABEL_KEY=topology.kubernetes.io/longitude
COUNTRY_LABEL_KEY=topology.kubernetes.io/country

# Operational
ENABLE_FALLBACK=true      # Enable geographic fallback (default: true)

---

ClusterCarbonMetrics CRD

Schema

apiVersion: carbon.edge-connect.eu/v1alpha1
kind: ClusterCarbonMetrics
metadata:
  name: member1  # Must match Cluster name
spec:
  # Specification (immutable after creation)
  clusterName: member1
  zone: DE  # ENTSO-E zone code
  zoneProvider: ENTSOE
status:
  # Status (updated hourly by collector)
  carbonIntensity: 199.1  # gCO2/kWh
  renewableShare: 0.72    # 0.0-1.0 (72% renewable)
  lastUpdate: "2026-01-19T14:00:00Z"

Accessing Data

# List all ClusterCarbonMetrics
kubectl --context karmada-apiserver get clustercarbonmetrics

# View specific cluster
kubectl --context karmada-apiserver get clustercarbonmetrics member1 -o yaml

# Get carbon intensity for all clusters
kubectl --context karmada-apiserver get clustercarbonmetrics \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.carbonIntensity}{"\n"}{end}'

# Get renewable share for all clusters
kubectl --context karmada-apiserver get clustercarbonmetrics \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.renewableShare}{"\n"}{end}'

---

Troubleshooting

No ClusterCarbonMetrics CRDs Created

Symptom: kubectl get clustercarbonmetrics returns no resources

Solutions:

1. CRD not installed

   kubectl --context karmada-apiserver get crd clustercarbonmetrics.carbon.edge-connect.eu
   # If missing, redeploy collector (includes CRD)
   cd karmada-local && make deploy-carbon-collector
   

2. Clusters not labeled

   kubectl --context karmada-apiserver get clusters \
     -o custom-columns=NAME:.metadata.name,ZONE:.metadata.labels.carbon\.karmada\.io/zone
   kubectl --context karmada-apiserver label cluster member1 carbon.edge-connect.eu/entso-zone=DE
   

3. Collector not running

   kubectl --kubeconfig ~/.kube/karmada.config --context karmada-host \
     -n karmada-system get pods -l app=carbon-data-collector
   kubectl --kubeconfig ~/.kube/karmada.config --context karmada-host \
     -n karmada-system logs -l app=carbon-data-collector --tail=100
   

ENTSO-E API Errors

Symptom: Logs show Failed to query ENTSO-E API

Solutions:

1. Invalid API token

   kubectl --kubeconfig ~/.kube/karmada.config --context karmada-host \
     -n karmada-system delete secret carbon-scheduler-secret
   cd karmada-local && make deploy-carbon-collector
   

2. Rate limiting

   • Increase REFRESH_INTERVAL to reduce request frequency
   • Default 1h interval is safe for 3 zones

---

Development

Local Development Setup

# 1. Setup Karmada local environment
cd karmada-local
make setup

# 2. Build collector image
make build-carbon-collector

# 3. Label clusters
make label-clusters-carbon

# 4. Deploy collector
make deploy-carbon-collector

Running Tests

# Unit tests
cd carbon-data-collector
go test ./pkg/...

# Integration tests (requires ENTSO-E token)
export ENTSOE_API_TOKEN=<your-token>
go test ./test/integration/...

---

Documentation

• Getting Started: docs/getting-started/quickstart.md
• ENTSO-E Setup: docs/getting-started/entsoe-setup.md
• Architecture: docs/concepts/architecture.md
• Zone Reference: docs/reference/zones.md
• Configuration: docs/reference/configuration.md

---

Contributing

See CONTRIBUTING.md

---

Publication note

This repository is published as a result of the EU-funded project DTAG Beitrag zur Entwicklung eines Multi-Provider Edge-Cloud-Kontinuums (Deutsche Telekom IPCEI-CIS), part of the 8ra initiative. It is provided for publication and dissemination purposes and is effectively read-only: external contributions, pull requests, and issue-based collaboration are not accepted unless explicitly requested by the project consortium. See CONTRIBUTING.md and SECURITY.md.

---

License

Unless stated otherwise:

• Source code is licensed under the Apache License 2.0 — see LICENSE-CODE.
• Documentation and other textual materials are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0) — see LICENSE-DOCS.

Attribution and consortium information are recorded in NOTICE. Third-party components are documented in THIRD-PARTY-NOTICES.md.

EU funding notice and disclaimer

Funded by the European Union – NextGenerationEU under Grant Agreement No. 13IPC005.

The views and opinions expressed are solely those of the author(s) and do not necessarily reflect the views of the European Union or the European Commission. Neither the European Union nor the European Commission can be held responsible for them.

See EU-DISCLAIMER.md.

---

Contact

• Project: IPCEI-CIS Sustainability Initiative
• Organization: Telekom Deutschland GmbH
• Repository: https://edp.buildth.ing/DevFW-CICD/carbon-data-collector