DevFW-CICD/k8s-mcp-eval
Watch
24
0
Fork
You've already forked k8s-mcp-eval
 0
Code Issues Pull requests 2 Projects Releases Packages Wiki Activity Actions

IPCEICIS-10005 #5

Open
david.brozovics wants to merge 22 commits from IPCEICIS-10005 into main
pull from: IPCEICIS-10005
merge into: DevFW-CICD:main
Conversation 1 Commits 22 Files changed 119 +203082 -87
david.brozovics commented 2026-08-17 09:10:14 +00:00
Collaborator
Copy link
No description provided. 
Twelve further runs take every model x arm x reasoning cell to N=3, and the
headline changes: the hint's value does not depend on the model, it depends on
whether the model is reasoning.

  Qwen 35B  think off   24% (13/25/33)  ->  85% (88/88/78)   +61
  gpt-oss   no switch   24% (13/22/38)  ->  81% (67/88/88)   +57
  Qwen 35B  think on    41% (50/29/43)  ->  56% (56/50/63)   +15

Two model families land in the same place once reasoning is controlled. What
looked like "qwen and gpt-oss disagree about whether help helps" was the
uncontrolled variable raised in review.

Corrects three claims, all of which looked clean at N=1:
  - the hinted/unhinted token-cost ratio
  - the cold reasoning effect (50 vs 12 was the luckiest and unluckiest runs)
  - model-dependence of the arm gap

Also narrows the CRD-design ask: the Tenant/Namespace collision disappears
entirely in the best configuration, so it is configuration-dependent rather
than an unconditional property of the name. That lowers the urgency of a
rename without removing the reason for one, and crd-design.md is the artifact
meant to go to other teams, so it should not overstate.

The gemma run is kept although it graded nothing: it is the evidence that the
model 504s at the gateway on a realistic tool-calling payload.
Three runs of the best configuration at the minimum catalogue, against the
existing three at six CRDs:

  3 CRDs   75% (71/67/88)   ~128k prompt tokens/task
  6 CRDs   85% (88/88/78)   ~165k prompt tokens/task

No measurable accuracy effect - the ranges overlap and the nominal direction
is the opposite of the concern. Context cost rises sub-linearly: double the
catalogue, ~29% more prompt tokens. Neither answers the make-or-break
question, because 6 CRDs is not 25; it demonstrates the instrument.

Verdict recorded: conditional go for discoverability and CRUD, no-go for
unattended operation, scalability unproven. The deciding factor for accuracy
is two sentences of guidance rather than the server. The blocker is
behavioural - agents delete and recreate objects to work around immutability,
then report success - which a bespoke server would not fix either.
📚 docs: add skilled arm decisions, expectations and ticket entry (IPCEICIS-10318)
Some checks failed
verify / validate (pull_request) Successful in 5s
Details
verify / oracle (pull_request) Failing after 4m26s
Details
b32280c972 
Two problems found while building a task for the false-absence failure mode.

1. tenant-update-owner-derived: the portal label must be set to the owning
   group's name, a value present only in the live object. That turns a read
   into a write, so "the tenant does not exist" (it does) now fails on state
   instead of only in prose. On first contact it caught both a false absence
   (qwen) and a false SUCCESS (gpt-oss invented a portalLabel label and
   reported the portal updated) - the latter had never been elicited before.

2. cluster_provider_strategy = "disabled". With "kubeconfig" the server
   exposed every context in the operator's kubeconfig, not just the run's
   disposable cluster. All 19 agent runs to date reached a docker-desktop
   context: a real cluster the harness neither created nor owns, with write
   tools and cluster-admin available. Disposable-cluster isolation only ever
   held for the current context.

Consequence: the tool surface changed (21 -> 20 tools), so results before and
after are not directly comparable and the matrix needs re-running.
Today's re-run burned two hours producing nothing usable. Two changes so that
does not repeat.

1. ERROR_STREAK: three consecutive endpoint failures end the run. Six retries
   against a 60s gateway timeout is ~7 minutes per task, so a dead endpoint
   was costing hours before anyone noticed. Completed tasks stay recorded.

2. The verdict section carries a warning. Its numbers were produced with the
   server able to see the whole kubeconfig. The confined re-run got one clean
   repetition of all six cells before qwen's endpoint started returning 504s:
   every qwen cell is lower (hinted/no-think 85% -> 50% on the same 11 tasks)
   while gpt-oss hinted is unchanged (81% -> 88%). Either the confinement hurt
   or qwen was already degrading; that is not separable yet, so the section
   says so rather than picking the flattering reading.

Also adds OAM/KubeVela Application and Component to the collision vocabulary -
PC&OF are offering their kcp environment, and those are the kinds on it.

The nine runs from today are kept: the six clean ones are rep 1 under the new
config, and the contaminated three are the record of the endpoint failing.
24 runs: qwen3.6-35b on the shared vLLM (4 cells x 3) and gpt-oss-120b on the
MMS gateway (2 cells x 3), all under the confined server config.

gpt-oss is the clean comparison - same endpoint, same build, same 11 tasks,
only the isolation config differs:

  unhinted   24% -> 4%   (0/0/13)
  hinted     81% -> 72%  (75/67/75)

So confining the server genuinely lowered scores; it was not one model being
ill, which was the open question yesterday. Cold discovery took the larger
hit, so the unhinted arm had been leaning on having a second cluster to
explore rather than on reading schemas. The earlier unhinted numbers were
flattering rather than representative.

Percentages are now withdrawn rather than flagged. The same cell measured 85%
on one build and 52% on another, N=3 both times, so levels are not a property
of KRM usability and must not be quoted. The proposed gates are re-expressed
as gaps, which is what reproduced: hinted beats unhinted every time, by 13 to
68 points, across two model families, two endpoints, two builds and two
isolation configs.

tenant-update-owner-derived failed in all 18 runs that included it. No
configuration measured reliably does read-then-write on a value it must fetch
from the object itself.
Reviewed-on: #4
docs(10318): cite the gap not the level, and reject the response_contains 'fix'
Some checks failed
verify / validate (pull_request) Successful in 11s
Details
verify / oracle (pull_request) Failing after 4m59s
Details
0d7b3e0a0e 
Two corrections to the skilled-arm ticket after 10005's percentages were
withdrawn.

1. The rationale quoted "worth roughly 60 points" and "the 85% figure" as a
   floor. Both are withdrawn: the same cell measured 85% on one build and 52%
   on another, and confining the server moved gpt-oss from 24% -> 4% unhinted
   and 81% -> 72% hinted. Levels move ~30 points with infrastructure. The gap
   is what reproduced - 13 to 68 points, every cell, two model families, two
   endpoints, two builds, two isolation configs - so the ticket now cites that.

   Adds the comparability constraint this cost us: hinted:v1 and v2 must run
   in the same session against the same endpoint, or the comparison reports
   the gateway rather than the prompt.

2. The Jira copy carries a scope item to "fix the response_contains grader"
   because tenant-discover-kind scores 0%. It is not scored 0, it is not
   scored - it produces no cluster state, and response_contains is review-only
   by design and enforced by validate.py. The 0% and the "~91% oracle ceiling"
   are both the all-tasks denominator; over auto-graded tasks the control arm
   is 10/10.

   Grading it would reintroduce the 10196 bug: the needles are Tenant,
   tenancy.edgeconnect.eu and slug, which any agent that lists CRDs emits.
   gpt-oss unhinted described Tenant perfectly and then authored a Namespace
   on every create task - full marks on that check, worst performer measured.

   The underlying gap is real, so the ticket points at the fix that works:
   make discovery produce state, as tenant-update-owner-derived did for false
   absence.
Some checks failed
verify / validate (pull_request) Successful in 11s
Details
verify / oracle (pull_request) Failing after 4m59s
Details
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin IPCEICIS-10005:IPCEICIS-10005
git switch IPCEICIS-10005

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff IPCEICIS-10005
git switch IPCEICIS-10005
git rebase main
git switch main
git merge --ff-only IPCEICIS-10005
git switch IPCEICIS-10005
git rebase main
git switch main
git merge --no-ff IPCEICIS-10005
git switch main
git merge --squash IPCEICIS-10005
git switch main
git merge --ff-only IPCEICIS-10005
git switch main
git merge IPCEICIS-10005
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Robby.Fritzsch
martin.mccaffery
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
DevFW-CICD/k8s-mcp-eval!5
No description provided.