CiSizer

Right-size your CI/CD runner resources automatically using real usage data.

CI Sizer collects CPU and memory metrics from CI/CD workloads via /proc, aggregates them over time, and computes Kubernetes resource requests and limits that match actual usage. Deploy the collector as a sidecar in your CI pods and the receiver as a central service — it stores metrics, serves sizing recommendations, and provides a web UI for visibility.

CI Sizer is part of the IPCEI-CIS cloud infrastructure initiative. Within the IPCEI-CIS edge framework, it serves as the resource optimization component — measuring actual CI/CD workload consumption and feeding right-sized resource recommendations back into the runner provisioning pipeline via GARM integration.

Features

• Automatic resource sizing — computes Kubernetes requests/limits from historical run data (p95 CPU, peak memory, configurable buffers)
• Zero-instrumentation collection — reads /proc directly; no agent or code changes in your CI jobs
• Container-aware grouping — maps processes to containers via cgroup paths
• Energy & carbon scoring — estimates per-run energy (Wh) and CO2 (gCO2eq) using the TEADS power model with Energy Charts real-time generation data (primary), FfE projection (DE-only fallback), static table (last resort)
• Web UI — browse metrics, sizing recommendations, and overrides at /ui
• Scoped push tokens — HMAC-SHA256 tokens scoped to org/repo/workflow/job; a compromised token can't read data
• OIDC & gateway auth — supports direct OIDC login (Dex, Keycloak, Entra ID) or API gateway JWT forwarding
• GARM integration — automated runner sizing via WebSocket lifecycle events
• 3-phase confidence-gated sizing — unknown (bootstrap 4Gi), learning (3× headroom), confident (tight staircase buffer)
• Multi-provider support — Forgejo Actions, GitHub Actions, GitLab CI (ci_provider: forgejo, github, gitlab)
• Zero historical metrics returns 200 with meta.confidence_phase=unknown and bootstrap defaults (not 404)
• OOM commit status notifications — Forgejo + GitLab + GitHub; only --notify-token required, base URL auto-detected from run_url

Intelligent Memory Management

CI Sizer prevents OOM (Out of Memory) kills through a confidence-gated sizing system:

• Unknown workloads get generous bootstrap defaults (4 GiB) on their first run
• Learning phase uses 3× headroom until enough clean samples are collected
• Confident phase applies tight, optimized limits based on observed peaks
• OOM detection via cgroup v2 memory.events oom_kill counter — exponential backoff (limit × 2^consecutiveOOMs), capped at node ceiling
• Early warnings notify via CI commit status when memory usage exceeds 90% of limit
• Node ceiling auto-detected from /proc/meminfo (90% of node RAM), or explicit --max-memory / --max-cpu flags

No manual tuning required. Only RECEIVER_NOTIFY_TOKEN needed for CI status notifications.

Quick Start

Prerequisites

• Go 1.26+ and Make (build from source), or Docker (container path)
• curl and jq for token generation examples

Build & Run

# Build both binaries
make all

# Start the receiver
./receiver --addr=:8080 --db=metrics.db \
  --read-token=my-secret-token --hmac-key=my-hmac-key

# Generate a scoped push token
PUSH_TOKEN=$(curl -s -X POST http://localhost:8080/api/v1/token \
  -H "Authorization: Bearer my-secret-token" \
  -H "Content-Type: application/json" \
  -d '{"organization":"my-org","repository":"my-repo","workflow":"ci.yml","job":"build"}' \
  | jq -r .token)

# Run the collector (in a CI pod or locally for testing)
./collector --interval=2s --top=5 \
  --push-endpoint=http://localhost:8080/api/v1/metrics \
  --push-token=$PUSH_TOKEN

# Stop the collector (Ctrl+C) — it pushes a run summary on shutdown
# Then query sizing recommendations:
curl -H "Authorization: Bearer my-secret-token" \
  http://localhost:8080/api/v1/sizing/repo/my-org/my-repo/ci.yml/build

Docker Compose

# Start the receiver
docker compose -f test/docker/docker-compose-stress.yaml up -d --build receiver

# Generate a push token and start the collector
PUSH_TOKEN=$(curl -s -X POST http://localhost:9080/api/v1/token \
  -H "Authorization: Bearer dummyreadtoken" \
  -H "Content-Type: application/json" \
  -d '{"organization":"test-org","repository":"stress-test","workflow":"stress-test-workflow","job":"heavy-workload"}' \
  | jq -r .token)

COLLECTOR_PUSH_TOKEN=$PUSH_TOKEN \
  docker compose -f test/docker/docker-compose-stress.yaml up -d --build collector

# Stop collector to trigger summary push, then query results
docker compose -f test/docker/docker-compose-stress.yaml stop collector
curl -H "Authorization: Bearer dummyreadtoken" \
  http://localhost:9080/api/v1/metrics/repo/test-org/stress-test/stress-test-workflow/heavy-workload

For Kubernetes deployment examples, see docs/configuration.md.

Architecture

┌─────────────────────────────────────────────┐       ┌──────────────────────────┐
│  CI/CD Pod (shared PID namespace)           │       │  Receiver Service        │
│                                             │       │                          │
│  ┌───────────┐  ┌────────┐  ┌───────────┐   │       │  POST /api/v1/metrics    │
│  │ collector │  │ runner │  │ sidecar   │   │       │         │                │
│  │           │  │        │  │           │   │       │         ▼                │
│  │ reads     │  │        │  │           │   │  push │  ┌────────────┐          │
│  │ /proc for │  │        │  │           │   │──────▶│  │  SQLite    │          │
│  │ all PIDs  │  │        │  │           │   │       │  └────────────┘          │
│  └───────────┘  └────────┘  └───────────┘   │       │         │                │
│                                             │       │         ▼                │
└─────────────────────────────────────────────┘       │  GET /api/v1/metrics/... │
                                                      │  GET /api/v1/sizing/...  │
                                                      │       (sizer)            │
                                                      └──────────────────────────┘

The collector runs as a sidecar in CI pods with shared PID namespace. It samples /proc on a configurable interval, groups processes by container via cgroup paths, and pushes a run summary to the receiver on shutdown (SIGINT/SIGTERM).

The receiver stores metric summaries in SQLite, exposes query and sizing APIs, and serves a web UI at /ui. Internally, the receiver is decomposed into focused subpackages: auth/ (OIDC, gateway JWT, middleware), store/ (SQLite persistence), sizing/ (algorithm and overview aggregation), reporting/ (dashboard KPIs and aggregation), garm/ (GARM WebSocket client), and pushtoken/ (HMAC token generation). The sizer aggregates the N most recent runs, selects a CPU percentile, applies buffers, and outputs Kubernetes-ready resource values.

For internal package details and data flow, see docs/sizing.md.

Documentation

Guide	Description
API Reference	All endpoints, auth, request/response examples
Sizing Algorithm	Algorithm steps, buffers, floors, overrides, enforcement modes
Configuration	All collector and receiver flags, environment variables, K8s examples
Dex / OIDC Integration	Direct OIDC auth setup, claim mapping, session cookies
API Gateway Auth	APISIX gateway mode, JWT forwarding, JWKS verification
GARM Integration	WebSocket events, token flow, run number resolution
GitLab CI Integration	MutatingAdmissionWebhook for GitLab Runner sidecar injection
Deployment Modes	Forgejo, GitHub, and GitLab deployment configurations
Energy Calculation	TEADS power model, Energy Charts real-time data, hardware profiling
Energy Impact Design	Energy PoC design document and scoring model
Cgroup Mapping	Design alternatives for container process mapping

Background references (Linux primitives this project builds on):

Document	Description
Process cgroups by PID	Reading /proc/<PID>/cgroup for container identification
/proc/stat in containers	Why /proc/stat shows host data and how to use per-process stats

Configuration

Essential flags to get started. For the full reference (all flags, env vars, CI context variables, K8s examples), see docs/configuration.md.

Collector (most common):

Flag	Env Var	Description	Default
--interval	—	Collection interval (e.g., 5s, 1m)	5s
--push-endpoint	—	Receiver URL to push metrics to	—
--push-token	COLLECTOR_PUSH_TOKEN	Bearer token for push auth	—
--top	—	Number of top processes to include	5
--log-level	—	debug, info, warn, error	info

Receiver (most common):

Flag	Env Var	Description	Default
--addr	—	HTTP listen address	:8080
--db	—	SQLite database path	metrics.db
--read-token	RECEIVER_READ_TOKEN	Token for read/admin endpoints	—
--hmac-key	RECEIVER_HMAC_KEY	Secret for push token generation	—
--auth-mode	RECEIVER_AUTH_MODE	none, oidc, or gateway	auto
--cpu-sizing-mode	RECEIVER_CPU_SIZING_MODE	observe or enforce	observe
--max-memory	RECEIVER_MAX_MEMORY	Node memory ceiling (e.g. 16Gi)	—
--max-cpu	RECEIVER_MAX_CPU	Node CPU ceiling (e.g. 8)	—
--notify-enabled	RECEIVER_NOTIFY_ENABLED	Enable OOM commit status notifications	false
--notify-base-url	RECEIVER_NOTIFY_BASE_URL	CI system URL for notifications	—
--notify-token	RECEIVER_NOTIFY_TOKEN	API token for notifications	—

Authentication

CI Sizer uses a two-tier token model: a pre-shared read token for query/admin endpoints, and scoped push tokens (HMAC-SHA256) for collectors. Three auth modes are supported: none (token-only), oidc (direct OIDC login), and gateway (external API gateway with JWT forwarding).

Setup guides: Dex / OIDC Integration | API Gateway Auth

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines on development workflow, code style, and pull requests.

make all    # fmt + vet + lint + build (run before every commit)
make test   # run all tests

License

EUPL-1.2 — see LICENSE.

Funding and Support

This open source project is part of activities carried out within the Important Project of Common European Interest on Next Generation Cloud Infrastructure and Services (IPCEI-CIS) — FKZ 13IPC005, an EU initiative to build a sovereign, interoperable and energy-efficient cloud-to-edge infrastructure in Europe.

The work in this repository has been supported and co-funded by Deutsche Telekom in the context of IPCEI-CIS, where Deutsche Telekom contributes its expertise in secure, sustainable connectivity and cloud-edge platform orchestration for a European cloud-edge continuum.

Where applicable, further national or European public funding instruments associated with IPCEI-CIS may also have contributed to the development of this software.