K8s-MCP CRD Discoverability Harness

A ready-to-use test rig for measuring how well an LLM agent discovers and drives EdgeConnect CRDs through a generic Kubernetes MCP server. The premise (AI technical-strategy sync, 2026-07-21): post-KRM, CRDs are inherently discoverable — schema, descriptions, CRUD — so a generic server should carry them with no custom MCP or Skills work.

Scope: assemble the harness, not run it. Producing the go/no-go verdict is a separate story. Full scope, including what is explicitly excluded: docs/tickets/.

The 30-second mental model

One question: can an agent, given only a generic Kubernetes tool, discover and correctly drive EdgeConnect CRDs — well enough that a bespoke EdgeXR MCP server is unnecessary?

Everything else is machinery to answer that fairly. Three different things could explain a failure, so each gets its own knob:

Knob	Values	Isolates	Today
Arm	oracle / unhinted / hinted	Whose fault a failure is — the task set, the agent cold, or the agent un-helped	all three
Tier	1 / 2 / 3	How real the cluster is — bare CRDs / fake controllers / kcp	Tier 1
Archetype	generic vs per-resource	The shape of the server — why there are ~4 verbs, not one tool per CRD	settled, not a live knob

And the output that matters is not the go/no-go — that is ~90% predictable. It is the CRD-authoring feedback the failures generate (confusable kinds, thin descriptions, unguessable fields), which lands while the CRDs can still change. Why: docs/background.md.

Layout

Path	What
crds/	Six real, pinned EdgeConnect CRDs + PROVENANCE.md (source OCI chart, digest, refresh). Schema fidelity by construction.
examples/	Real known-valid Tenant CRs from poc-core-deploy used as preconditions, plus reference/ — the manifests the oracle arm applies.
mcp/	SERVER_CHOICE.md (shortlist + rationale), config.toml, mcp.json (client registration).
cluster/	kind-cluster.yaml + TIERS.md (tier 1→3 escalation, and what is out of scope).
tasks/	README.md (why these tasks — the design rationale), task-schema.json and the golden sets — the only home of "correct".
tools/	Python + uv: harness.py (loading + grading), validate.py (CI), run_eval.py (the three arms), confusion.py (which kind the agent reached for).
runs/	Verbatim transcripts and provenance, one directory per run — committed, because a result that only existed on a laptop is not a result.
SYNTHETIC_DESCRIPTIONS.md	Convention for AI-filling missing descriptions on future WIP CRDs (marked, domain-faithful).
docs/	Background & mandate, decision register, external research digest, pre-registered expectations, ticket scope.
AGENTS.md	Contributor/agent conventions — including the rules that keep the experiment unbiased.

Deliverables → where

What IPCEICIS-10004 asked for, and where it landed. The full wording of each deliverable lives in the ticket doc; this is only the index.

Deliverable	Delivered as
Server picked, rationale recorded	mcp/SERVER_CHOICE.md — containers/kubernetes-mcp-server v0.0.65
Representative CRD set, pinned	crds/ — 6 tenancy CRDs, growth path to ~25 in PROVENANCE.md
Synthetic descriptions where missing	SYNTHETIC_DESCRIPTIONS.md — none needed; this set is authentic
Cluster tiers reachable	cluster/TIERS.md
Golden task set with known-correct outcomes	tasks/ — 11 tasks over 3 CRDs and 2 API groups; why these

The three arms

One runner, one grading path, three questions. --arm is mandatory — an unlabelled hinted run answers a different question than it appears to.

Arm	Who authors the manifest	Needs	Answers
oracle	a reference manifest, applied with kubectl	a cluster	Is the task set right — does the substrate really accept and reject what we claim?
unhinted	the model, via MCP, told nothing about CRDs	cluster + model	Is KRM inherently discoverable?
hinted	the model, told where schemas live	cluster + model	Is it usable if we help?

What the arms do and do not license you to say

The bespoke server is never built, so it is never measured. What stands in for it is the gap between unhinted and hinted — the hint is the cheapest curation anyone could ship (four sentences of generic KRM technique, no EdgeConnect specifics). Read the gap as an upper bound on the cheap fix and a lower bound on what curation is worth:

• gap ≈ 0 — helping does not help. A bespoke server is unlikely to rescue it either; the problem is the schemas or the model, not the interface.
• gap large — curation pays. But it says a prompt paid, not that a bespoke server is needed; server instructions are far cheaper than a second server, and that is the real comparison the decision needs.

Two confounds are live and neither is closed by more runs of the current setup. Model capability: an all-fail unhinted arm is equally consistent with "KRM is undiscoverable" and "this model is too weak", which is why the model matrix (docs/decisions.md) is a prerequisite for any headline, not a nice-to-have. Missing verb: this server has no explain tool, so "generic MCP cannot surface CRD schemas" and "this server lacks one verb" currently predict the same result (mcp/SERVER_CHOICE.md).

Quickstart

One toolchain on every platform: uv. Nothing to install first — uv run resolves the environment on the way in.

# 0. sanity-check the harness itself — no cluster, no model, no network
uv run tools/validate.py

# 1. control arm: prove the task set is right before trusting any agent result
#    (creates a disposable kind cluster, applies crds/, tears it down)
uv run tools/run_eval.py --arm oracle

# 2. the measurement (needs an OpenAI-compatible endpoint: OPENAI_BASE_URL, AGENT_MODEL)
uv sync --extra runner
uv run tools/run_eval.py --arm unhinted --list-tools   # MCP smoke test, no model
uv run tools/run_eval.py --arm unhinted

# 3. which kind did it reach for? analysis over runs/ - no cluster, no model
uv run tools/confusion.py

Needs docker and kind. Every run gets its own cluster: the CRDs are all cluster-scoped, so there is no namespace to confine a run to, and an agent that cannot find the CRD reaches for a Namespace of its own that would otherwise be read as pre-existing state by the next run. --context <name> runs against a cluster you already have and makes isolation your problem.

Every arm writes a verbatim transcript per task under runs/, with a provenance record — server version, model, CRD digest, tier and arm. Those files are committed: this repo is the home of the harness, the results, and the analysis of those results.

Reading a result

A run writes two files. Open runs/<timestamp>-<arm>/review.md — one page, attention-first, with the agent's own answer quoted next to what the task asked for. run.json beside it is the verbatim record (provenance, transcripts, tool calls, resulting objects); you should not need to open it.

Verdict	Means
✅ PASS / ❌ FAIL	Objective — checked against resulting cluster state.
👀 REVIEW	A human judges this one. A precondition would satisfy its checks anyway, so passing them earns nothing — the signal is the agent's prose (did it report the owners? did it surface the rejection honestly?). A failing check still fails the row outright.
✂️ TRUNC	The run hit --max-turns and was cut off. Never a pass or a fail — the limit is there to stop a runaway agent, and scoring it would penalise the agent that explored more. Re-run at a higher limit if you need a verdict.
➖ ?	Nothing to check — the task produces no cluster state.

A verdict says whether the agent was right. uv run tools/confusion.py says what it did instead — a matrix of asked for against the kind it actually authored, read back out of the transcripts, so it needs no cluster and no re-run. That is the deliverable prediction P1 asks for: failures are expected to cluster on near-synonym kinds, and this is where that shows up or doesn't.

Why manual rather than LLM-as-judge: an automated judge is its own project, and at this sample size hand-reading three rows is cheaper and more trustworthy. Recorded in docs/decisions.md.

On the agent arms, tasks whose assertions a precondition would satisfy anyway are marked REVIEW, never PASS — otherwise the read-back and rejection tasks report green without the agent having done anything. They can still FAIL: a precondition can make a check green without the agent's help, but it cannot make one red, so a red check is the agent's doing and needs no human. The oracle arm ignores the flag entirely: it authors the manifests, so its checks stay the control.

Grading a run by hand

review.md is written once per timestamped run and never regenerated, so the reviewer edits it in place. For each 👀 REVIEW section:

1. Read the agent's quoted answer against Done means and Answer should mention.
2. Overwrite that section's stub line — **Verdict:** unjudged — replace with… — with a verdict and one line of reasoning, e.g. **Verdict:** fail — named the right kind but invented spec.tier; never read the CRD.
3. Commit the edited file. The judgement is then part of the record (invariant 5), attributable to the run that produced it.

Anything left unjudged is reported as ungraded — never as a pass.

Verified live (2026-08-05, grader version 3): --arm oracle — 10 of 10 auto-graded tasks pass on a disposable kind cluster, Kubernetes v1.30.13 (the discovery task produces no cluster state and is judge-only). Create/read/update/delete accepted; both CEL guards (name==slug, slug immutability) reject as expected, the rejection now read from the write attempt itself; descriptions confirmed reachable by reading the CRD object. The four tasks added under IPCEICIS-10194 — two ambiguity probes, one unstated-reference probe, one free-form-map probe — pass on the control, so the substrate accepts what they claim. Not yet reproduced in CI. Back-to-back agent arms on the same substrate, qwen3.6-35b at temperature 0: unhinted 0 pass · 4 fail · 3 review, hinted 3 pass · 1 fail · 3 review. Those two tallies were produced by grader version 1 on the 7-task set, and are not comparable with anything recorded since: three of the fourteen rows were runs cut off at --max-turns and scored anyway. They are kept unedited and were not re-graded — what changed and why is in docs/decisions.md.

Any cluster works, but a disposable kind one is the default — what each tier does and does not prove is in cluster/TIERS.md. Refreshing the pinned CRDs is documented in crds/PROVENANCE.md.

Key finding (the gating open question)

Does the server surface CRD descriptions to the model, or only OpenAPI types? — Yes, but only indirectly, and that indirection is itself a measurable discoverability signal. Mechanism, caveats and the list_output dependency: mcp/SERVER_CHOICE.md.

Open choices and what would overturn them: docs/decisions.md. Why any of this is being measured: docs/background.md.