-
Notifications
You must be signed in to change notification settings - Fork 1
Expand file tree
/
Copy pathimplementation-guide.html
More file actions
1040 lines (942 loc) · 79.4 KB
/
Copy pathimplementation-guide.html
File metadata and controls
1040 lines (942 loc) · 79.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 64 64'%3E%3Crect width='64' height='64' fill='%23000000'/%3E%3Ctext x='32' y='39' text-anchor='middle' font-family='monospace' font-size='20' fill='white'%3EV1%3C/text%3E%3C/svg%3E">
<title>Project Benchmark Internals</title>
<style>
:root {
--bg: #050607;
--surface: #0b0e11;
--surface-2: #101419;
--text: #f4f6f8;
--body: #c5ccd2;
--muted: #909aa3;
--faint: #626c75;
--line: #293139;
--line-strong: #46515b;
--cyan: #4bd8ff;
--green: #8de878;
--amber: #ffc75a;
--red: #ff716c;
--violet: #c19cff;
--mono: "SFMono-Regular", Consolas, "Liberation Mono", monospace;
--sans: Inter, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
--content: 960px;
}
* { box-sizing: border-box; }
html { scroll-behavior: smooth; }
body { margin: 0; color: var(--text); background: var(--bg); font-family: var(--sans); line-height: 1.72; }
a { color: inherit; text-underline-offset: 3px; }
code, pre, .mono { font-family: var(--mono); }
code { color: #e4e9ed; font-size: 0.9em; }
p code, li code, td code, figcaption code, .note code { padding: 0.08em 0.3em; border: 1px solid #303a43; color: #ffffff; background: #141a20; border-radius: 3px; }
.file-ref { display: inline-block; padding: 0.05em 0.35em; border-bottom: 1px solid var(--cyan); color: #8ce9ff; background: rgba(75, 216, 255, 0.09); font-family: var(--mono); font-size: 0.88em; line-height: 1.45; text-decoration: none; }
.file-ref::before { margin-right: 0.2em; color: var(--faint); content: "./"; }
.file-ref:hover { color: #ffffff; background: rgba(75, 216, 255, 0.16); }
button { font: inherit; }
table { border-collapse: collapse; }
:focus-visible { outline: 2px solid var(--cyan); outline-offset: 3px; }
.skip-link { position: fixed; z-index: 100; top: 8px; left: 8px; padding: 8px 11px; color: var(--bg); background: var(--cyan); transform: translateY(-180%); }
.skip-link:focus { transform: translateY(0); }
.progress { position: fixed; z-index: 60; top: 0; left: 0; width: 0; height: 2px; background: var(--cyan); }
.topbar { position: sticky; z-index: 40; top: 0; border-bottom: 1px solid var(--line); background: rgba(5, 6, 7, 0.94); backdrop-filter: blur(12px); }
.topbar-inner { display: flex; align-items: center; justify-content: space-between; max-width: 1320px; min-height: 52px; margin: 0 auto; padding: 0 24px; }
.topbar-title, .topbar a { font-family: var(--mono); font-size: 11px; text-decoration: none; }
.topbar-title { color: var(--cyan); }
.topbar a { color: var(--muted); }
.layout { display: grid; grid-template-columns: 230px minmax(0, var(--content)); gap: 58px; justify-content: center; padding: 60px 24px 100px; }
.toc { position: sticky; top: 82px; align-self: start; max-height: calc(100vh - 104px); overflow: auto; }
.toc h2 { margin: 0 0 12px; color: var(--faint); font-family: var(--mono); font-size: 10px; letter-spacing: 0.12em; text-transform: uppercase; }
.toc ol { display: grid; gap: 1px; margin: 0; padding: 0; list-style: none; }
.toc a { display: grid; grid-template-columns: 28px 1fr; padding: 6px 0; color: var(--muted); font-size: 11px; line-height: 1.4; text-decoration: none; }
.toc a:hover { color: var(--text); }
.toc a span { color: var(--faint); font-family: var(--mono); }
.toc a[aria-current="true"] { color: var(--text); }
.toc a[aria-current="true"] span { color: var(--cyan); }
.toc-note { margin-top: 25px; padding-top: 18px; color: var(--faint); border-top: 1px solid var(--line); font-family: var(--mono); font-size: 9px; line-height: 1.65; }
main { min-width: 0; }
.report-header { padding-bottom: 56px; border-bottom: 1px solid var(--line-strong); }
.kicker, .section-index, .figure-label, .label { color: var(--cyan); font-family: var(--mono); font-size: 10px; letter-spacing: 0.1em; text-transform: uppercase; }
h1, h2, h3, h4 { color: #c9e2eb; line-height: 1.15; letter-spacing: -0.025em; }
h1 { max-width: 820px; margin: 12px 0 24px; color: #d8edf4; font-size: clamp(2.7rem, 6vw, 5.1rem); letter-spacing: -0.055em; }
h2 { margin: 0 0 14px; color: var(--cyan); font-size: clamp(1.8rem, 3.4vw, 2.7rem); }
h3 { margin: 40px 0 12px; font-size: 1.35rem; }
h4 { margin: 28px 0 8px; font-size: 1rem; }
p { max-width: 78ch; margin: 0 0 18px; color: var(--body); }
.abstract { max-width: 74ch; color: #d8dde1; font-size: 1.12rem; }
.scope-note { display: grid; grid-template-columns: 140px 1fr; margin-top: 32px; padding: 18px 0; border-top: 1px solid var(--line); border-bottom: 1px solid var(--line); }
.scope-note strong { color: var(--amber); font-family: var(--mono); font-size: 10px; text-transform: uppercase; }
.scope-note p { margin: 0; color: var(--muted); font-size: 12px; }
.section { padding: 72px 0; border-bottom: 1px solid var(--line-strong); scroll-margin-top: 70px; }
.section-head { position: relative; margin-bottom: 34px; }
.section-index { display: inline-flex; align-items: center; justify-content: center; min-width: 34px; height: 22px; margin: 0 0 12px; padding: 0 6px; border: 1px solid var(--line-strong); border-left: 2px solid var(--cyan); color: var(--cyan); line-height: 1; }
.section-deck { max-width: 75ch; margin: 0; color: var(--muted); }
.subsection { margin-top: 44px; }
.lead-invariant { margin: 34px 0; padding: 20px 22px; overflow: auto; border-left: 3px solid var(--cyan); color: var(--text); background: var(--surface); font-size: 14px; }
.invariant-formula { font-family: var(--mono); white-space: nowrap; }
.invariant-explainer { max-width: 78ch; margin: 10px 0 0; color: var(--muted); font-family: var(--sans); font-size: 12px; line-height: 1.55; }
.lead-invariant .fail { color: var(--red); }
.lead-invariant .pass { color: var(--green); }
.note { max-width: 78ch; margin: 25px 0; padding: 16px 18px; border-left: 2px solid var(--amber); color: var(--body); background: rgba(255, 199, 90, 0.06); font-size: 13px; }
.note strong { color: var(--amber); }
.danger { border-left-color: var(--red); background: rgba(255, 113, 108, 0.06); }
.danger strong { color: var(--red); }
figure { margin: 42px 0; }
figcaption { max-width: 80ch; margin-top: 13px; color: var(--muted); font-size: 11px; }
.figure-label { display: block; margin-bottom: 12px; color: var(--faint); }
.diagram { padding: 24px 0; border-top: 1px solid var(--line-strong); border-bottom: 1px solid var(--line-strong); }
.system-flow { display: grid; grid-template-columns: 1fr 34px 1.15fr 34px 1.15fr 34px 1fr; align-items: stretch; }
.system-node { padding: 16px; border-left: 2px solid var(--line-strong); background: linear-gradient(90deg, var(--surface), transparent); }
.system-node.skill { border-color: var(--cyan); }
.system-node.runner { border-color: var(--violet); }
.system-node h4 { margin: 20px 0 7px; }
.system-node p { margin: 0; color: var(--muted); font-size: 11px; }
.system-type { color: var(--faint); font-family: var(--mono); font-size: 9px; text-transform: uppercase; }
.arrow { display: grid; place-items: center; color: var(--faint); font-family: var(--mono); }
.authoring-flow { display: grid; grid-template-columns: 120px 1fr; border-top: 1px solid var(--line); }
.lane-name, .lane-events { min-height: 74px; border-bottom: 1px solid var(--line); }
.lane-name { padding: 15px 12px 15px 0; color: var(--muted); font-family: var(--mono); font-size: 10px; }
.lane-events { display: grid; grid-template-columns: repeat(6, 1fr); }
.event { padding: 13px 10px; color: var(--faint); border-left: 1px solid var(--line); font-size: 10px; line-height: 1.45; }
.lane-name.header, .lane-events.header { min-height: 44px; color: var(--faint); background: var(--surface); font-family: var(--mono); font-size: 9px; letter-spacing: 0.06em; text-transform: uppercase; }
.lane-name.header { padding-top: 13px; }
.lane-events.header .event { padding-top: 13px; color: var(--faint); background: var(--surface); }
.event.user { color: var(--amber); background: rgba(255, 199, 90, 0.06); }
.event.agent { color: var(--cyan); background: rgba(75, 216, 255, 0.06); }
.event.helper { color: var(--violet); background: rgba(193, 156, 255, 0.06); }
.event.worker { color: var(--green); background: rgba(141, 232, 120, 0.06); }
.candidate-proof { display: grid; grid-template-columns: minmax(0, 1fr) 210px; gap: 20px; align-items: stretch; }
.proof-matrix { display: grid; grid-template-columns: 1.35fr repeat(3, 1fr); border-top: 1px solid var(--line); border-left: 1px solid var(--line); }
.proof-cell { min-height: 54px; padding: 11px 12px; border-right: 1px solid var(--line); border-bottom: 1px solid var(--line); color: var(--muted); font-size: 10px; line-height: 1.4; }
.proof-cell.header { min-height: 40px; color: var(--faint); background: var(--surface); font-family: var(--mono); font-size: 9px; letter-spacing: 0.06em; text-transform: uppercase; }
.proof-cell.probe { color: var(--body); font-family: var(--mono); }
.proof-cell.pass, .proof-cell.fail { display: grid; place-items: center; font-family: var(--mono); font-size: 11px; text-transform: uppercase; }
.proof-cell.pass { color: var(--green); background: rgba(141, 232, 120, 0.05); }
.proof-cell.fail { color: var(--red); background: rgba(255, 113, 108, 0.07); }
.proof-verdict { display: flex; flex-direction: column; justify-content: space-between; padding: 17px; border-top: 2px solid var(--cyan); background: var(--surface); }
.proof-verdict .label { color: var(--cyan); }
.proof-verdict strong { display: block; margin: 20px 0 7px; color: var(--text); font-size: 14px; line-height: 1.3; }
.proof-verdict p { margin: 0; color: var(--muted); font-size: 10px; line-height: 1.5; }
.proof-verdict code { display: block; margin-top: 18px; color: var(--green); font-size: 10px; }
.guard-flow { display: grid; grid-template-columns: 1fr 42px 1fr 42px 1fr; align-items: center; }
.guard-step { min-height: 152px; padding: 16px; border-top: 2px solid var(--line-strong); background: var(--surface); }
.guard-step h4 { margin-top: 24px; }
.guard-step p { margin: 0; color: var(--muted); font-size: 11px; }
.guard-step.prepare { border-color: var(--violet); }
.guard-step.review { border-color: var(--cyan); }
.guard-step.accept { border-color: var(--green); }
.tree, .code-block { position: relative; max-width: 100%; margin: 22px 0; padding: 18px 20px; overflow: auto; border: 1px solid var(--line); color: #dce2e6; background: #020304; font-size: 11px; line-height: 1.7; white-space: pre; }
.file-panel-heading { margin: 28px 0 0; padding: 11px 20px; border: 1px solid var(--line); border-bottom: 0; color: var(--cyan); background: var(--surface); font-family: var(--mono); font-size: 0.9rem; letter-spacing: 0; }
.file-panel-heading + .tree, .file-panel-heading + .code-block { margin-top: 0; }
.copy-wrap { position: relative; }
.copy-button { position: absolute; z-index: 2; top: 10px; right: 10px; min-width: 48px; min-height: 32px; border: 1px solid var(--line-strong); color: var(--muted); background: var(--surface-2); cursor: pointer; font-family: var(--mono); font-size: 9px; text-transform: uppercase; }
.code-key { color: var(--cyan); }
.code-value { color: var(--green); }
.code-comment { color: var(--faint); }
.table-scroll { max-width: 100%; margin: 28px 0; overflow-x: auto; }
.technical-table { width: 100%; min-width: 710px; border-top: 1px solid var(--line-strong); font-size: 12px; line-height: 1.55; }
.technical-table th, .technical-table td { padding: 12px 13px; border-bottom: 1px solid var(--line); vertical-align: top; text-align: left; }
.technical-table thead th { color: var(--faint); background: var(--surface); font-family: var(--mono); font-size: 9px; letter-spacing: 0.06em; text-transform: uppercase; }
.technical-table tbody th { width: 19%; color: var(--text); font-weight: 600; }
.technical-table tbody td { color: var(--body); }
.yes { color: var(--green); }
.no { color: var(--red); }
.conditional { color: var(--amber); }
.workspace-layers { display: grid; grid-template-columns: 1fr 42px 1fr; gap: 0; align-items: stretch; }
.layer-inputs { display: grid; gap: 1px; background: var(--line); }
.layer { padding: 15px; background: var(--surface); }
.layer b { display: block; margin-bottom: 5px; font-size: 12px; }
.layer p { margin: 0; color: var(--muted); font-size: 10px; }
.layer:nth-child(1) { border-left: 2px solid var(--cyan); }
.layer:nth-child(2) { border-left: 2px solid var(--violet); }
.layer:nth-child(3) { border-left: 2px solid var(--green); }
.workspace-output { padding: 20px; border: 1px solid var(--line-strong); background: var(--surface); }
.workspace-output h4 { margin-top: 22px; }
.workspace-output ul { margin: 12px 0 0; padding-left: 18px; color: var(--muted); font-size: 11px; }
.phase-boundaries { display: grid; grid-template-columns: repeat(4, 1fr); border-top: 1px solid var(--line-strong); border-left: 1px solid var(--line); }
.phase { min-width: 0; border-right: 1px solid var(--line); border-bottom: 1px solid var(--line); }
.phase h4 { margin: 0; padding: 14px; border-bottom: 1px solid var(--line); }
.phase ul { display: grid; gap: 0; margin: 0; padding: 0; list-style: none; }
.phase li { padding: 9px 13px; color: var(--muted); border-bottom: 1px solid rgba(41, 49, 57, 0.6); font-size: 10px; }
.phase li:last-child { border-bottom: 0; }
.phase.preflight { border-top: 2px solid var(--amber); }
.phase.setup { border-top: 2px solid var(--green); }
.phase.solver { border-top: 2px solid var(--cyan); }
.phase.evaluator { border-top: 2px solid var(--violet); }
.validation-figure { display: grid; grid-template-columns: 1fr 50px 1fr; align-items: stretch; }
.validation-state { padding: 18px; background: var(--surface); }
.validation-state.base { border-top: 2px solid var(--red); }
.validation-state.reference { border-top: 2px solid var(--green); }
.validation-state h4 { display: flex; justify-content: space-between; margin: 0 0 18px; }
.exit { font-family: var(--mono); font-size: 11px; }
.validation-state.base .exit { color: var(--red); }
.validation-state.reference .exit { color: var(--green); }
.validation-state p { margin: 0; color: var(--muted); font-size: 11px; }
.component-list { display: grid; grid-template-columns: 190px 1fr; border-top: 1px solid var(--line-strong); }
.component-list dt, .component-list dd { margin: 0; padding: 12px 0; border-bottom: 1px solid var(--line); }
.component-list dt { padding-right: 20px; color: var(--cyan); font-family: var(--mono); font-size: 10px; }
.component-list dd { color: var(--body); font-size: 12px; }
.references { display: grid; grid-template-columns: 1fr 1fr; gap: 0 24px; margin-top: 28px; }
.reference { padding: 12px 0; color: var(--body); border-bottom: 1px solid var(--line); font-size: 12px; text-decoration: none; }
.reference code { display: block; margin-bottom: 3px; color: var(--cyan); }
.footer { padding-top: 28px; color: var(--faint); font-family: var(--mono); font-size: 9px; }
@media (max-width: 1080px) {
.layout { grid-template-columns: minmax(0, var(--content)); }
.toc { display: none; }
.system-flow { grid-template-columns: 1fr; gap: 9px; }
.system-flow .arrow { transform: rotate(90deg); }
}
@media (max-width: 720px) {
.topbar-inner { padding: 0 16px; }
.layout { padding: 38px 16px 70px; }
.section { padding: 54px 0; }
.scope-note { grid-template-columns: 1fr; gap: 8px; }
.authoring-flow { grid-template-columns: 90px 1fr; overflow-x: auto; }
.lane-events { min-width: 720px; }
.candidate-proof, .guard-flow, .workspace-layers, .validation-figure { grid-template-columns: 1fr; gap: 9px; }
.guard-flow .arrow, .workspace-layers .arrow, .validation-figure .arrow { transform: rotate(90deg); }
.phase-boundaries { grid-template-columns: 1fr 1fr; }
.component-list { grid-template-columns: 1fr; }
.component-list dt { padding-bottom: 3px; border-bottom: 0; }
.component-list dd { padding-top: 3px; }
.references { grid-template-columns: 1fr; }
}
@media (prefers-reduced-motion: reduce) { html { scroll-behavior: auto; } }
</style>
</head>
<body>
<a class="skip-link" href="#main">Skip to guide</a>
<div class="progress" id="progress" aria-hidden="true"></div>
<header class="topbar">
<div class="topbar-inner"><span class="topbar-title">PROJECT BENCHMARK INTERNALS / V1</span><a href="../README.md">README ↗</a></div>
</header>
<div class="layout">
<aside class="toc" aria-label="Guide contents">
<h2>Contents</h2>
<ol>
<li><a href="#model"><span>00</span>System model</a></li>
<li><a href="#authoring"><span>01</span>Task creation</a></li>
<li><a href="#task-contract"><span>02</span>Task contract</a></li>
<li><a href="#configuration"><span>03</span>Configuration loading</a></li>
<li><a href="#workspace"><span>04</span>Workspace construction</a></li>
<li><a href="#container"><span>05</span>Container environment</a></li>
<li><a href="#auth"><span>06</span>Authentication and agents</a></li>
<li><a href="#validation"><span>07</span>Task validation</a></li>
<li><a href="#execution"><span>08</span>Running the experiment</a></li>
<li><a href="#results"><span>09</span>Identity and results</a></li>
<li><a href="#components"><span>10</span>Component index</a></li>
</ol>
</aside>
<main id="main">
<header class="report-header">
<span class="kicker">Implementation guide · skill and runner</span>
<h1>Benchmark Skill</h1>
<p class="abstract">This document follows a benchmark from repository history to its final,
standardized result. It explains how the skill finds and creates tasks and how helper scripts
protect the user's working copy. It also covers configuration checks and the boundaries
between Docker phases.</p>
<div class="scope-note"><strong>Reading scope</strong><p>The README is an operational introduction.
This page explains the implementation in depth. Rules that define required behavior remain
in their source documents, which are linked here instead of copied into a second specification.</p></div>
</header>
<section class="section" id="model">
<div class="section-head"><span class="section-index">00</span><div><h2>System model</h2><p class="section-deck">The repository has two parts: one creates benchmark tasks, and the other runs them. They use the same file formats but have different jobs and access to different information.</p></div></div>
<p>The <code>bench-this</code> skill manages task creation. It studies project
history, proposes candidate changes, coordinates task authors, and reviews each prompt and
evaluator. It also adapts the project environment and pauses when the user must make a
decision. Its output is a <code>benchmarks/</code> directory inside the target repository.</p>
<p><code>agent-bench</code> runs completed tasks. It does not decide whether a historical change
makes a good task. Instead, it loads the task files, creates clean workspaces, runs each
Docker phase with fixed limits, checks the selected model, verifies evaluator output, and
saves the results. Generated benchmarks use the bundled runner at
<code>benchmarks/_vendor/agent_bench/</code>. The maintained source in this repository is
<code>src/agent_bench/</code>.</p>
<div class="lead-invariant"><div class="invariant-formula">valid(task) = setup(base) ∧ <span class="fail">evaluator(base) = 1</span> ∧ setup(reference) ∧ <span class="pass">evaluator(reference) = 0</span></div><p class="invariant-explainer">Setup must succeed for both commits. The evaluator must fail on the base commit, then pass on the reference commit. Here, <code>0</code> means success and <code>1</code> means an ordinary evaluator failure.</p></div>
<p>The base and reference commits calibrate the task. The base commit does not yet have the
requested behavior. The reference commit proves that the public requirements can pass, but
the solver never sees its code. A measured run asks whether one agent configuration, called
a treatment, can reproduce the behavior from the base commit using only public task information.</p>
<figure>
<span class="figure-label">Figure 1 · Major data flow</span>
<div class="diagram system-flow">
<div class="system-node"><span class="system-type">Evidence</span><h4>Git history</h4><p>Exact base and reference commits, code changes, public behavior, and available dependencies.</p></div>
<div class="arrow" aria-hidden="true">→</div>
<div class="system-node skill"><span class="system-type">Task creation</span><h4>Skill + helpers</h4><p>Find candidates, request approval, create tasks in isolation, check them, and prepare shared setup.</p></div>
<div class="arrow" aria-hidden="true">→</div>
<div class="system-node"><span class="system-type">Portable package</span><h4>benchmarks/</h4><p>Task requirements, hidden evaluators, Docker image, setup, agent configurations, and bundled runner.</p></div>
<div class="arrow" aria-hidden="true">→</div>
<div class="system-node runner"><span class="system-type">Task execution</span><h4>Runner + Docker</h4><p>Validation records, isolated runs, usage data, and standardized results.</p></div>
</div>
<figcaption>The skill turns repository history into a portable benchmark package. The runner
uses that package without loading the skill or judging the task against repository history.</figcaption>
</figure>
</section>
<section class="section" id="authoring">
<div class="section-head"><span class="section-index">01</span><div><h2>How tasks are created</h2><p class="section-deck">The agent does most of the task-creation work, but the user controls key decisions. Searching history cannot change files. Task files appear only after candidate approval, and treatment files appear only after the user chooses an agent configuration.</p></div></div>
<h3>Discovery covers the repository's lifespan</h3>
<p>The skill searches the repository's main line of development across older, middle, and
recent changes before it narrows the candidate list. Commit titles are not enough. The skill
reads relevant code changes and tests, checks surrounding code and related commits, and
compares several plausible choices. If the local Git history is incomplete, it reports what
it could and could not inspect.</p>
<p>For each promising change, the skill confirms the exact relationship between the base and
reference commits. It examines the public behavior before and after the change and checks
that an evaluator can observe every proposed requirement.</p>
<h3>Candidate quality is a behavioral argument</h3>
<p>A candidate is split into meaningful groups of public requirements. Each group must be able
to fail on its own while the others pass. This test prevents small implementation details,
such as constants or helper functions, from being counted as separate behaviors. A useful
task has several independent outcomes and cannot be solved by copying one small change.</p>
<figure>
<span class="figure-label">Figure 2 · Behavioral independence test</span>
<div class="diagram candidate-proof">
<div class="proof-matrix" role="img" aria-label="Three probes. In each probe, one requirement group fails while the other two pass.">
<div class="proof-cell header">Counterfactual probe</div><div class="proof-cell header">Group A</div><div class="proof-cell header">Group B</div><div class="proof-cell header">Group C</div>
<div class="proof-cell probe">Remove behavior A</div><div class="proof-cell fail">Fails</div><div class="proof-cell pass">Passes</div><div class="proof-cell pass">Passes</div>
<div class="proof-cell probe">Remove behavior B</div><div class="proof-cell pass">Passes</div><div class="proof-cell fail">Fails</div><div class="proof-cell pass">Passes</div>
<div class="proof-cell probe">Remove behavior C</div><div class="proof-cell pass">Passes</div><div class="proof-cell pass">Passes</div><div class="proof-cell fail">Fails</div>
</div>
<div class="proof-verdict"><div><span class="label">Candidate verdict</span><strong>Independent, observable outcomes</strong><p>Each public requirement detects a distinct missing behavior, not merely a private implementation fragment.</p></div><code>quality argument: supported</code></div>
</div>
<figcaption>Counterfactual probes establish that the requirement groups are behaviorally
distinct. If removing one behavior makes several groups fail together, the proposed split
needs revision.</figcaption>
</figure>
<p>The candidate summary identifies the public entry point, the dependencies available at both
commits, and any small test substitutes the evaluator needs. The skill rejects a candidate
if fair evaluation would require reading the submitted source or checking private calls. It
also rejects tasks that need large parts of a framework rebuilt, rely on unreliable production
services, or test output that the prompt does not disclose. After presenting the requested
number of candidates, the skill stops for the user's first decision.</p>
<figure>
<span class="figure-label">Figure 3 · Authoring ownership and decision gates</span>
<div class="diagram authoring-flow">
<div class="lane-name header">Actor</div><div class="lane-events header"><div class="event">Inventory</div><div class="event">Candidate review</div><div class="event">Approval gate</div><div class="event">Task creation</div><div class="event">Treatment gate</div><div class="event">Measured run</div></div>
<div class="lane-name">User</div><div class="lane-events"><div class="event user">places repository in scope</div><div class="event">—</div><div class="event user">approves candidate</div><div class="event">—</div><div class="event user">chooses treatment</div><div class="event user">authorizes run</div></div>
<div class="lane-name">Coordinator</div><div class="lane-events"><div class="event agent">explores history</div><div class="event agent">probes + presents</div><div class="event">—</div><div class="event agent">reviews + validates</div><div class="event agent">generates config</div><div class="event agent">runs matrix</div></div>
<div class="lane-name">Task worker</div><div class="lane-events"><div class="event">—</div><div class="event">—</div><div class="event worker">authors prompt + evaluator</div><div class="event worker">reports evidence</div><div class="event">—</div><div class="event">—</div></div>
<div class="lane-name">Helper scripts</div><div class="lane-events"><div class="event">—</div><div class="event">—</div><div class="event helper">scaffold + prepare</div><div class="event helper">accept guard</div><div class="event helper">configure</div><div class="event">—</div></div>
</div>
<figcaption>The user chooses the scope and experiment. After approval, the coordinating agent
manages task creation and validation. Task-writing workers operate in isolation and cannot
create more workers.</figcaption>
</figure>
<h3>The scaffold makes the benchmark self-contained</h3>
<p>After approval, <a class="file-ref" href="../skills/bench-this/scripts/scaffold.py">scaffold.py</a> copies the skill's bundled
<code>assets/benchmarks/</code> directory into the target. It will not overwrite a non-empty
destination. It creates a Docker image name from the repository name and a short hash of the
checkout path, then makes the runner, setup script, and example evaluator executable. The
path hash keeps two local clones with the same name from sharing a Docker image tag by accident.</p>
<p>The scaffold includes the runner and the pure-Python part of PyYAML under
<code>_vendor/</code>. Before it starts the command-line interface,
<code>benchmarks/run.py</code> tells Python to load packages from that directory. The target
can therefore run the benchmark with its normal Python interpreter. It does not need to
install this development package or use the target project's dependency manager.</p>
<h3>Task authoring is isolated from the user's checkout</h3>
<p>The task worker can inspect the repository, but it never edits the user's working copy. That
working copy may contain unfinished changes, untracked files, or part of an existing benchmark.
Instead, a helper creates a temporary area with three separate views of the repository:</p>
<pre class="tree">user's target checkout/ protected; unchanged during authoring
temporary scratch/
├── repository/ authoring clone at the target's current HEAD
│ └── benchmarks/tasks/id/ the only task bundle the worker writes
├── base/ fixed checkout of the base commit
└── reference/ fixed checkout of the reference commit</pre>
<p><a class="file-ref" href="../skills/bench-this/scripts/task_workspace.py">task_workspace.py</a> <code>prepare</code> constructs this layout in four steps:</p>
<ol>
<li><p><strong>Check the commits.</strong> The helper expands both commit names to full hashes.
It rejects identical commits, reversed order, or a reference commit that does not come after
the base commit.</p></li>
<li><p><strong>Record the working copy.</strong> It hashes the current commit, staged and
unstaged changes, and untracked files outside the task directory. It does not read ignored
files, which helps keep local secrets out of temporary metadata.</p></li>
<li><p><strong>Create the writing copy.</strong> The helper makes a separate clone at the
target's current commit. It replaces any copied benchmark directory with the shared
scaffold and creates the same empty task structure every time. The worker writes
<code>prompt.md</code>, public files, and hidden tests here.</p></li>
<li><p><strong>Create fixed historical copies.</strong> Separate working directories expose
the base and reference source trees. The worker uses them to observe old and new behavior
and test the evaluator without changing the writing copy.</p></li>
</ol>
<p>The helper returns the exact paths for the writing copy, both historical copies, the output,
and the temporary directory. It also stores hashes of the protected working copy and shared
benchmark files. After the worker finishes, the coordinator reviews the task there. Only the
later <code>accept</code> command copies the reviewed task into the user's working copy.</p>
<figure>
<span class="figure-label">Figure 4 · Guarded task integration</span>
<div class="diagram guard-flow">
<div class="guard-step prepare"><span class="label">Prepare</span><h4>Isolate state</h4><p>Resolve ancestry, hash target state, create authoring clone, base/reference worktrees, and task skeleton.</p></div>
<div class="arrow" aria-hidden="true">→</div>
<div class="guard-step review"><span class="label">Review</span><h4>Inspect the bundle</h4><p>Coordinator checks approved scope, assertion ledger, public entry point, evaluator techniques, setup evidence, and group coverage.</p></div>
<div class="arrow" aria-hidden="true">→</div>
<div class="guard-step accept"><span class="label">Accept once</span><h4>Recheck and install safely</h4><p>Verify hashes of the target and shared files, reject symbolic links and forbidden patterns, then replace the destination in one operation.</p></div>
</div>
<figcaption>A task can be accepted only once. If acceptance fails, the temporary workspace is
closed. It remains available for diagnosis, but another attempt requires a newly prepared
workspace instead of a manual copy or retry.</figcaption>
</figure>
<p>Before it copies a task, the workspace guard verifies the required paths, prompt, evaluator,
requirement headings, result marker, task identity, and symbolic links. It also looks for
forbidden evaluator patterns and changes to the protected working copy or shared benchmark
files. The coordinator first performs a broader review. Every prompt and evaluator section
must map to an approved requirement group, and every hidden assertion must be fair. The task
must also stay within the approved scope. Passing on the reference commit is necessary, but
it is not enough by itself.</p>
<h3>The repeatable workspace check</h3>
<p><a class="file-ref" href="../skills/bench-this/scripts/task_workspace.py">task_workspace.py</a>
exposes a shared, read-only <code>check</code> command for both task workers and the
orchestrator:</p>
<div class="copy-wrap">
<button class="copy-button" type="button" data-copy="workspace-check-command">Copy</button>
<pre class="code-block" id="workspace-check-command">python <skill-directory>/scripts/task_workspace.py check <scratch-root></pre>
</div>
<p>The command is used with the scratch path returned by <code>prepare</code>, not the target
repository. A failed check exits nonzero and leaves the scratch workspace available for diagnosis.</p>
<div class="table-scroll">
<table class="technical-table">
<thead><tr><th>Actor</th><th>When it runs</th><th>What the check protects</th></tr></thead>
<tbody>
<tr><th>Task worker</th><td>After authoring and direct behavior checks, then again after each correction.</td><td>Required manifest, prompt, public suite, hidden evaluator, group headings and result protocol; forbidden evaluator techniques; assigned task ID; unchanged shared benchmark infrastructure.</td></tr>
<tr><th>Orchestrator</th><td>After worker and verifier reports and its own review, then again after every correction.</td><td>All worker checks plus the protected target state and an empty destination, so acceptance cannot overwrite an existing task or integrate a stale review.</td></tr>
</tbody>
</table>
</div>
<p>A worker must report a successful check with its handoff. The orchestrator owns the
final check: after it passes, the bundle must not be edited before the single
<code>accept</code> command. <code>check</code> never copies files into the target; unlike
<code>accept</code>, it can be repeated safely while the bundle is being corrected.</p>
</section>
<section class="section" id="task-contract">
<div class="section-head"><span class="section-index">02</span><div><h2>The task contract</h2><p class="section-deck">Each task combines public requirements with a private evaluator. A manifest connects both to exact commits in the repository's history.</p></div></div>
<pre class="tree">tasks/<task-id>/
├── task.yaml commit binding, paths, command, groups, timeout
├── prompt.md solver-visible public contract
├── public/ optional files overlaid into the solver workspace
├── public-tests/ required solver-visible scored checks
└── hidden-tests/
├── run.sh private evaluator entrypoint
└── evaluate.py additional private evaluator files are allowed</pre>
<p><code>task.yaml</code> records the full base and reference commit hashes. It also names the
prompt, public files, required public and hidden suites, evaluator commands, nonempty result
groups, and optional solver time limit. A task ID must use lowercase letters and hyphens and must match its directory
name. All paths stay inside the task directory; absolute paths and <code>..</code> are rejected.</p>
<p>The prompt describes the public inputs, outputs, errors, side effects, constraints, existing
behavior, and any new interface. Every scored requirement group uses an exact Markdown heading,
making it easy to connect the prompt to the reported result. Commit hashes, pull-request
context, the historical patch, reference code, and hidden test cases remain private.</p>
<h3>A complete illustrative task</h3>
<p>This small fictional task shows how the files fit together. The historical project already
provides <code>mailstore.MailStore.from_file()</code>, <code>add()</code>, and
<code>emails()</code>. The reference change made duplicate checks ignore letter case. Real
benchmark tasks should cover more independent behaviors, as required by the quality policy.</p>
<h4 class="file-panel-heading"><code>task.yaml</code>: bind history to the contract</h4>
<pre class="code-block"><span class="code-key">version:</span> <span class="code-value">1</span>
<span class="code-key">id:</span> duplicate-email
<span class="code-key">base_commit:</span> 0123456789abcdef0123456789abcdef01234567
<span class="code-key">reference_commit:</span> fedcba9876543210fedcba9876543210fedcba98
<span class="code-key">prompt:</span> prompt.md
<span class="code-key">public_directory:</span> public
<span class="code-key">public_tests_directory:</span> public-tests
<span class="code-key">public_test_command:</span> /bin/sh /public-tests/run.sh /workspace
<span class="code-key">public_test_groups:</span>
- duplicate-basic
<span class="code-key">hidden_tests_directory:</span> hidden-tests
<span class="code-key">test_command:</span> /bin/sh /evaluator/run.sh /workspace
<span class="code-key">requirement_groups:</span>
- duplicate-detection
- persisted-state
- compatibility
- error-contract
<span class="code-key">solver_timeout_seconds:</span> null</pre>
<p>The solver never sees this file because it contains commit and evaluator details.
<code>solver_timeout_seconds: null</code> means the task adds no time limit for the solver.
Project-level limits still apply to setup and evaluation.</p>
<h4 class="file-panel-heading"><code>prompt.md</code>: disclose the public behavior</h4>
<pre class="code-block"># Reject duplicate email addresses
Update the existing `mailstore.MailStore` behavior. The public seed file is
`benchmark-fixtures/seed-emails.json`.
## duplicate-detection
`MailStore.add(email)` must reject an address already in the store after
case-insensitive normalization.
## persisted-state
A rejected duplicate must not change the stored email sequence.
## compatibility
Previously stored addresses must remain available, and adding a distinct address
must continue to succeed.
## error-contract
Duplicate rejection must raise an error whose public `code` is `duplicate_email`
and whose `email` value is the normalized duplicate address.</pre>
<p>The headings exactly match the group IDs in <code>task.yaml</code>. The prompt states every
scored behavior, including the error fields and case-normalization rule. It does not tell the
solver which files or algorithm to change.</p>
<h4 class="file-panel-heading"><code>public/</code>: files intentionally added to the workspace</h4>
<pre class="tree">public/
└── benchmark-fixtures/
└── seed-emails.json</pre>
<pre class="code-block">{"emails":["owner@example.test"]}</pre>
<p>During a measured run, the runner copies <code>public/</code> into the workspace root. The file
above therefore becomes <code>/workspace/benchmark-fixtures/seed-emails.json</code>. This
directory is for disclosed fixtures, starter assets, and other public task context. It must
not contain hidden checks, reference code, commit details, or evaluator logic. Task files are
copied after treatment files, so a task file wins if both use the same path. The files remain
available during setup, solving, and evaluation.</p>
<h4 class="file-panel-heading"><code>public-tests/</code>: visible checks, canonical scoring</h4>
<p>The runner copies this required suite to
<code>/workspace/.agent-bench-public-tests/</code>. The solver may inspect, run, or edit that
copy. Evaluation instead mounts the unchanged task-owned directory read-only at
<code>/public-tests</code> and executes <code>public_test_command</code>. Public checks use the
same structured result protocol as hidden checks, but their result IDs and completion are
reported separately. The runner also records whether the final workspace modified or deleted
any supplied public test file; that integrity signal does not alter the score.</p>
<p>The public entrypoint resolves its helper files relative to its own location rather than
hard-coding <code>/public-tests</code>. The same source can therefore run from both the
solver-visible copy and the canonical evaluation mount.</p>
<p>Public checks should expose a representative success or integration case. Hidden checks may
exercise the same disclosed behavior, but must use different concrete inputs, combinations,
boundaries, or compatibility conditions. Copying one check into both suites would make the
public/hidden split uninformative.</p>
<h4 class="file-panel-heading"><code>hidden-tests/run.sh</code>: stable container entrypoint</h4>
<pre class="code-block">#!/bin/sh
set -eu
workspace=${1:?workspace path is required}
exec python3 /evaluator/evaluate.py \
"$workspace" \
"$workspace/benchmark-fixtures/seed-emails.json"</pre>
<p>The runner starts this script with the manifest's <code>test_command</code>. Hidden files appear
read-only at <code>/evaluator</code>, and the solver's edited project appears at
<code>/workspace</code>. This script installs nothing, so <code>setup.sh</code> must prepare every
dependency the evaluator needs.</p>
<h4 class="file-panel-heading"><code>hidden-tests/evaluate.py</code>: exercise the public API</h4>
<pre class="code-block">import json
import sys
from pathlib import Path
workspace = Path(sys.argv[1])
fixture = Path(sys.argv[2])
sys.path.insert(0, str(workspace))
groups = {
"duplicate-detection": False,
"persisted-state": False,
"compatibility": False,
"error-contract": False,
}
candidate_error = False
try:
import mailstore
except (ImportError, SyntaxError) as error:
# This module is public and already imports at the base commit. Breaking that
# stable import is direct evidence of a submitted-product failure.
candidate_error = True
print(f"candidate import failed: {error}")
else:
store = mailstore.MailStore.from_file(fixture)
before = list(store.emails())
duplicate_error = None
try:
store.add("Owner@Example.Test")
except (AttributeError, TypeError) as error:
# The requested API may be absent at the historical base. Record an
# ordinary behavior failure while allowing independent checks to run.
print(f"duplicate behavior unavailable: {error}")
except Exception as error:
duplicate_error = error
groups["duplicate-detection"] = duplicate_error is not None
groups["persisted-state"] = list(store.emails()) == before
groups["error-contract"] = (
getattr(duplicate_error, "code", None) == "duplicate_email"
and getattr(duplicate_error, "email", None) == "owner@example.test"
)
try:
store.add("new@example.test")
except (AttributeError, TypeError) as error:
print(f"compatible add unavailable: {error}")
else:
after = list(store.emails())
groups["compatibility"] = (
"owner@example.test" in after and "new@example.test" in after
)
payload = {"version": 1, "groups": groups, "candidate_error": candidate_error}
print("AGENT_BENCH_RESULT: " + json.dumps(payload, separators=(",", ":")))
raise SystemExit(0 if all(groups.values()) and not candidate_error else 1)</pre>
<p>This evaluator imports a stable public module, calls public methods, and checks both the error
and the resulting state. It does not read the submitted source, inspect private calls, or
compare the solver's patch with the reference implementation. A real evaluator usually tests
more cases in each group. More assertions do not create more groups, and every group must still pass.</p>
<h3>Evaluator protocol</h3>
<p>The evaluator tests the same public interface that a user or calling program would use. Its
exact inputs and edge cases may remain private, but they cannot introduce requirements that
are missing from the prompt or existing compatibility rules. The evaluator may print normal
diagnostic messages, but it must print exactly one structured result line:</p>
<pre class="code-block">AGENT_BENCH_RESULT: {"version":1,"groups":{"duplicate-detection":true,"persisted-state":false},"candidate_error":false}</pre>
<p><a class="file-ref" href="../src/agent_bench/evaluator.py">evaluator.py</a> accepts protocol
version <code>1</code>, known fields only, exactly the groups listed in the manifest, and real
JSON boolean values. Exit <code>0</code> means every group passed and no candidate error occurred.
Exit <code>1</code> means at least one group failed or a candidate error occurred. Any other exit
code, missing or repeated marker, invalid JSON, or mismatch between the report and exit code
counts as a benchmark infrastructure failure.</p>
<div class="note"><strong><code>candidate_error</code> has one specific meaning.</strong> It shows
that the solver's edited project cannot import or compile. A requested API may be missing at
the historical base; that is a normal failed requirement if the evaluator can continue. Broken
evaluator code, fixtures, or dependencies are benchmark infrastructure failures.</div>
</section>
<section class="section" id="configuration">
<div class="section-head"><span class="section-index">03</span><div><h2>Loading configuration safely</h2><p class="section-deck">Before a run starts, the runner checks three YAML files: the project environment, the task, and the agent configuration. It converts them into records that cannot change during the run.</p></div></div>
<p><a class="file-ref" href="../src/agent_bench/config.py">config.py</a> loads YAML with
<code>safe_load</code>, checks its structure, and immediately resolves paths on the host.
Project configuration selects one Docker image and Dockerfile. It also defines the setup
command, number of repetitions, optional model prices, and default time limits for the solver,
setup, and evaluator. Version 1 does not support Docker Compose. Setup and evaluator must have
positive time limits; the solver limit may be <code>null</code>, as it is in the scaffold.</p>
<p>A treatment is the complete agent configuration being tested. Its file selects the command-line
command-line agent tool, called a harness, and a fixed model. When OpenCode needs them, it also
selects the provider and agent. A treatment may add files to the tool's home directory or the
solver workspace, choose an authentication profile, and supply extra command arguments.
Supported harnesses are Copilot CLI and OpenCode. OpenCode providers
are limited to <code>amazon-bedrock</code>, <code>github-copilot</code>,
<code>openai</code>, and <code>opencode-go</code>. Amazon Bedrock configurations also pin
an AWS region in their OpenCode settings.
The value <code>auto</code> is rejected because it would let the harness choose a model. That
would make results harder to reproduce and prevent the runner from checking one expected model.</p>
<div class="table-scroll">
<table class="technical-table">
<thead><tr><th>Loader</th><th>Input</th><th>Important guarantees</th></tr></thead>
<tbody>
<tr><th><code>load_project</code></th><td><code>benchmark.yaml</code></td><td>Version 1, one existing Dockerfile, a setup command, valid time limits and repetitions, and prices that are zero or greater.</td></tr>
<tr><th><code>load_task</code></th><td><code>tasks/*/task.yaml</code></td><td>Directory/ID match, commit-like hashes, contained paths, existing prompt/evaluator, unique group IDs, inherited timeout.</td></tr>
<tr><th><code>load_harness</code></th><td><code>configurations/*/configuration.yaml</code></td><td>Matching directory and ID, supported harness and provider, fixed model, existing public-file directories, safe profile ID, and text arguments.</td></tr>
</tbody>
</table>
</div>
<h3>Creating the same treatment every time</h3>
<p><a class="file-ref" href="../skills/bench-this/scripts/configure.py">configure.py</a>
turns approved values into a complete configuration directory. It checks IDs and refuses to
replace an existing directory or copy a skill tree through a symbolic link. It first writes
everything to a temporary directory, then renames that directory into place in one operation.
For OpenCode, it writes an <code>opencode.json</code> that enables only the chosen provider,
fixes the small model, and disables sharing.</p>
<p>Optional skills are copied into that treatment's
<code>workspace/.agents/skills/<name>/</code> after validating their frontmatter and
directory name. The generator enables OpenCode skill permission when required, but upstream
activation instructions must still be added explicitly. Skills, external tool servers,
workspace instructions, and reasoning settings belong to the treatment rather than the shared
baseline. This separation makes their effect measurable.</p>
</section>
<section class="section" id="workspace">
<div class="section-head"><span class="section-index">04</span><div><h2>Building the solver workspace</h2><p class="section-deck">The runner creates a fresh source tree for every measured run. It never checks out commits in the user's repository or cleans the user's files.</p></div></div>
<p><a class="file-ref" href="../src/agent_bench/workspace.py">workspace.py</a> uses
<code>git archive</code> to export the task's base commit. Before extracting the archive, it
checks every path for safety. It then removes any historical <code>benchmarks/</code> directory
and checks that no Git or benchmark metadata remains. The result contains project source at
one commit, not the repository's history.</p>
<p><code>prepare_workspace</code> next copies in public treatment files, followed by public task
files. If both provide the same path, the task file replaces the treatment file. Neither set
may contain symbolic links, which could otherwise redirect a copy or mounted directory to an
unrelated host path. Placeholder <code>.gitkeep</code> files are ignored.</p>
<figure>
<span class="figure-label">Figure 5 · Repeatable workspace assembly</span>
<div class="diagram workspace-layers">
<div class="layer-inputs">
<div class="layer"><b>1 · Archived base commit</b><p>Project files only; no .git, benchmarks, reference commit, or PR metadata.</p></div>
<div class="layer"><b>2 · Treatment files</b><p>Public settings, instructions, skills, or integration files for one experimental treatment.</p></div>
<div class="layer"><b>3 · Task files</b><p>Fixtures or other files intentionally visible for this task; copied last.</p></div>
</div>
<div class="arrow" aria-hidden="true">→</div>
<div class="workspace-output"><span class="label">Temporary host directory</span><h4>Solver-visible /workspace</h4><ul><li>writable by the solver</li><li>prepared before the prompt runs</li><li>reused by the offline evaluator</li><li>deleted with the cell's temporary root</li></ul></div>
</div>
<figcaption>The runner passes the prompt directly to the agent command instead of writing it
into the workspace. Hidden tests are also absent while the solver works; the runner mounts
them only during evaluation.</figcaption>
</figure>
</section>
<section class="section" id="container">
<div class="section-head"><span class="section-index">05</span><div><h2>Docker image and phase boundaries</h2><p class="section-deck">Every task uses the same Docker image. Each phase, however, receives only the files, settings, network access, and time limit it needs.</p></div></div>
<p>The image has two jobs. First, it reproduces the historical project's build and test
environment. Second, it provides the selected coding-agent command-line tool. The scaffold
starts with Node 22 on Debian and installs Git, Python, certificates, Copilot CLI, and OpenCode.
Benchmark authors add any project runtimes and native libraries. During image creation,
<a class="file-ref" href="../src/agent_bench/runner.py">runner.py</a> uses only the small
<code>benchmarks/</code> directory as Docker's build input. Project source is mounted later,
when a phase runs.</p>
<p><code>setup.sh</code> receives the <code>/workspace</code> path and may use the network. It must
prepare the project for both the solver's normal checks and the hidden evaluator. During setup,
the runner exposes only this script at the read-only <code>/benchmark</code> path. The rest of
the benchmark directory stays hidden. Setup must work on both the base and reference commits
of every task.</p>
<figure>
<span class="figure-label">Figure 6 · Container access during each phase</span>
<div class="diagram phase-boundaries">
<div class="phase preflight"><h4>Model check</h4><ul><li>empty <code>/workspace</code></li><li>temporary credential home</li><li>network: available</li><li>fixed test prompt</li><li>timeout ≤ 180 seconds</li></ul></div>
<div class="phase setup"><h4>Project setup</h4><ul><li>workspace: read/write</li><li><code>/benchmark</code>: read-only</li><li>no credentials</li><li>network: bridge</li><li>positive setup timeout</li></ul></div>
<div class="phase solver"><h4>Solver</h4><ul><li>workspace: read/write</li><li>temporary auth home</li><li>network: bridge</li><li>hidden tests absent</li><li>optional solver timeout</li></ul></div>
<div class="phase evaluator"><h4>Evaluator</h4><ul><li>modified workspace</li><li><code>/evaluator</code>: read-only</li><li>no credentials</li><li>network: none</li><li>positive evaluator timeout</li></ul></div>
</div>
<figcaption>Every automated phase uses the same image and runs as the host user's numeric ID.
The container receives only approved environment variables, cannot gain extra privileges,
has a process limit, and cannot access the Docker control socket.</figcaption>
</figure>
<p><a class="file-ref" href="../src/agent_bench/docker.py">docker.py</a> builds the Docker command
directly. It resolves every mounted path, gives each container a unique name, and captures
standard output and error separately while also saving both to phase logs. If a phase times
out, the runner removes the container itself because Docker's <code>--rm</code> option acts only
after a normal exit. It then finishes collecting output and reports a <code>CommandTimeout</code>
with a safe explanation and paths to the logs.</p>
<div class="note"><strong>Why installation paths matter.</strong> Containers run as the host user,
not as <code>root</code>, and receive a temporary home directory. A tool installed under
<code>/root</code> will therefore be unavailable. The same is true for a symbolic link that
eventually points there.</div>
</section>
<section class="section" id="auth">
<div class="section-head"><span class="section-index">06</span><div><h2>Authentication and agent commands</h2><p class="section-deck">Credentials stay outside the repository. For each run, the runner checks the selected profile, copies it into a temporary home directory, and verifies the model before exposing project source.</p></div></div>
<p>Profiles live at <code>~/.agent-bench/auth/<profile>/<harness>/</code>. Interactive
<code>auth login</code> normally runs in the benchmark image with no project-source mount.
Copilot asks the user to complete its device login, while OpenCode receives one explicit
provider. After an OpenCode login, the runner removes generated caches, logs, databases,
package data, and plugin links. It keeps only
<code>.local/share/opencode/auth.json</code>, containing credentials for that provider, and
restricts the file so only its owner can read or write it.</p>
<p>Amazon Bedrock is the provider-specific exception. The runner itself prompts for the
Bedrock bearer API key only when a user invokes the manual login command. Agent-assisted
setup uses the skill's non-interactive <code>provision_auth.py</code> helper instead, so
the orchestrator never launches a command that waits for input. Both paths store the same
narrow external profile. The runner forwards the secret to provider containers as
<code>AWS_BEARER_TOKEN_BEDROCK</code> without placing the value in Docker process
arguments. The credential file is not copied into the temporary OpenCode home.</p>
<p>Before a run, <code>validate_auth_profile</code> rejects a missing profile or any symbolic link.
For OpenCode, it also rejects unexpected files, invalid JSON, credentials for extra providers,
or a provider mismatch. Bedrock profiles must contain exactly one valid bearer token.
<code>stage_home</code> copies file-based profiles into a new temporary directory, then adds
the treatment's agent settings. This order prevents saved credentials from silently replacing
settings that belong to the experiment.</p>
<h3>How the runner starts each agent</h3>
<div class="table-scroll">
<table class="technical-table">
<thead><tr><th>Concern</th><th>Copilot adapter</th><th>OpenCode adapter</th></tr></thead>
<tbody>
<tr><th>Command mode</th><td>Prompt mode, JSON output, no user questions, autonomous local tools.</td><td><code>opencode run</code>, JSON events, provider-qualified model, configured agent and workspace.</td></tr>
<tr><th>Restrictions</th><td>GitHub tool denied, remote export and auto-update disabled.</td><td>Enabled provider constrained by treatment <code>opencode.json</code>; sharing disabled by generator.</td></tr>
<tr><th>Environment</th><td><code>HOME</code>, <code>COPILOT_HOME</code>, model, allow-all flag, no color.</td><td><code>HOME</code> plus XDG config, data, and cache roots inside disposable home; Bedrock additionally receives its runner-managed bearer token.</td></tr>
<tr><th>Identity evidence</th><td>Last resolved model found in structured output.</td><td>INFO stream lines; every observed provider/model identity must match the configured pair.</td></tr>
</tbody>
</table>
</div>
<p>Before exposing source code, the runner sends a fixed test prompt in an empty workspace. This
preflight check confirms that authentication works and that the harness selected the expected
model. It can run for at most 180 seconds, or less when the project gives the solver a shorter
default limit. Missing or unexpected model information counts as an infrastructure failure,
and the logs are kept. The <code>doctor</code> command combines this check with validation of
Docker, the image, task files, and the authentication profile.</p>
</section>
<section class="section" id="validation">
<div class="section-head"><span class="section-index">07</span><div><h2>Proving that a task is valid</h2><p class="section-deck">Validation checks that one evaluator reliably distinguishes the two historical commits. It does not run a coding agent and needs no treatment credentials.</p></div></div>
<p><a class="file-ref" href="../src/agent_bench/runner.py">runner.py</a> begins validation by
resolving the Docker image to an exact, unchanging ID. It hashes the task and validation
environment, creates a unique directory for the attempt, and writes a receipt with status
<code>running</code>. It checks the base commit first and the reference commit second. For each,
it exports a clean workspace, runs setup, runs the required public suite, runs
the same hidden evaluator, and records both outcomes. The receipt is safely replaced after
every step, so it never contains a partial write.</p>
<figure>
<span class="figure-label">Figure 7 · Historical control pair</span>
<div class="diagram validation-figure">
<div class="validation-state base"><h4>Base commit <span class="exit">exit 1</span></h4><p>Setup passes. Evaluator reaches public behavior and reports ordinary false groups. <code>candidate_error</code> is forbidden here.</p></div>
<div class="arrow" aria-hidden="true">+</div>
<div class="validation-state reference"><h4>Reference commit <span class="exit">exit 0</span></h4><p>Setup passes. The exact same evaluator reports every declared requirement group true.</p></div>
</div>
<figcaption>Only this exact pair—base exits <code>1</code>, reference exits <code>0</code>—produces
the final status <code>validated</code>. Another completed pair is <code>invalid</code>. Setup,
evaluator, timeout, or interruption failures produce <code>infrastructure_error</code>. A
required public suite must independently produce the same base <code>1</code>, reference
<code>0</code> pair.</figcaption>
</figure>
<p>An exit code of <code>1</code> does not by itself prove that the base failed for the right reason.
Validation also looks for signs that the test never started correctly, such as missing modules
or commands, import and syntax errors, startup crashes, or no discovered tests. These signs make
the attempt an infrastructure failure, even if the test tool returned <code>1</code>. The base
also cannot report <code>candidate_error: true</code>: it must reach a real behavior check rather
than fail during import or compilation.</p>
<p><code>validate-tasks</code> can validate several tasks at once, up to a fixed worker limit.
Within one task, it still checks base before reference. Each worker has separate attempt and
log directories. To keep terminal output readable, parallel workers save their detailed logs
and print progress messages one at a time. A verbose retry of one task streams its phase output.
Changing the shared image or setup requires every task to be checked again. A task-only change
requires revalidation of that task alone.</p>
<p>Each attempt has its own receipt, and the newest receipt is copied to the task's
<code>latest.json</code> through a safe temporary-file replacement. The receipt records the image,
task and environment hashes, commits, setup and evaluator states, log paths, final status, and
error category. This file is the official validation record; terminal output and worker reports
are not.</p>
</section>
<section class="section" id="execution">
<div class="section-head"><span class="section-index">08</span><div><h2>Running the experiment</h2><p class="section-deck">The runner creates one isolated run for every requested combination of task, treatment, and repetition. A normal failure becomes a standardized result row, so it does not stop the remaining runs.</p></div></div>
<p><a class="file-ref" href="../src/agent_bench/runner.py">runner.py</a> first finds all tasks and
treatments. Repeated <code>--task</code> and <code>--configuration</code> options select exact IDs.
The runner checks that repetitions and worker count are positive, then creates every requested
task–treatment–repetition combination. The full command gets one sortable experiment ID, and
each individual run gets its own ID. An unknown requested ID causes an error instead of being ignored.</p>
<p>By default, the runner handles up to three runs at once, but never starts more workers than
there are runs. <code>--verbose</code> uses one worker so detailed phase output stays readable.
In normal parallel mode, every run saves complete logs while the terminal shows orderly progress.
Only the coordinator writes result files. As each run finishes, it appends one JSON line and
refreshes the experiment summary.</p>
<h3>One cell, step by step</h3>
<ol>
<li><p>Create the log directory, hashes of the task and treatment, a temporary home, and a temporary workspace.</p></li>
<li><p>Copy the selected authentication profile and agent settings into the temporary home.</p></li>
<li><p>Confirm authentication and model identity in an empty workspace. Prepare source only after this check succeeds.</p></li>
<li><p>Export the base commit, copy in public treatment and task files, then run project setup.</p></li>
<li><p>Read <code>prompt.md</code> and build the command for the fixed agent and model. Run it with writable source, the temporary home, and network access.</p></li>
<li><p>Read token use and provider-reported cost from both output streams. If the solver failed, record that fact but still evaluate because it may have made useful edits.</p></li>
<li><p>Run canonical public tests and the hidden evaluator in new containers without credentials or network access. Check both structured reports for consistency.</p></li>
<li><p>Pass the run only if the solver and every configured evaluator suite exit <code>0</code>. Otherwise, assign a specific failure category.</p></li>
</ol>
<div class="note"><strong>The evaluator still runs after a solver failure.</strong> This preserves
useful evidence about the edited workspace. The run still counts as a solver failure, even if
the evaluator passes. A successful run requires both processes to exit successfully.</div>
<div class="copy-wrap">
<button class="copy-button" type="button" data-copy="matrix-command">Copy</button>
<pre class="code-block" id="matrix-command"><span class="code-comment"># User selects the scope; the coordinator normally invokes the runner.</span>
./benchmarks/run.py run \
--task <task-a> --task <task-b> \
--configuration <configuration-a> --configuration <configuration-b> \
--repetitions 3 --jobs 3</pre>
</div>
</section>
<section class="section" id="results">
<div class="section-head"><span class="section-index">09</span><div><h2>Input identity, usage, and results</h2><p class="section-deck">Readable IDs tell people what was selected. Content hashes prove exactly which inputs were used. Results keep both.</p></div></div>
<h3>Hashes of the real inputs</h3>
<p><a class="file-ref" href="../src/agent_bench/identity.py">identity.py</a> hashes both labels and
content lengths, preventing two different input layouts from producing the same sequence of
bytes. The task hash covers commits, evaluator command, time limit, group order, prompt, public
files, public tests, and hidden tests. The treatment hash covers the harness, provider, model, agent, profile
name, arguments, harness files, and workspace files. It deliberately excludes credential contents.</p>
<p>The validation-environment hash covers the exact Docker image ID, setup command, setup and
evaluator time limits, and <code>setup.sh</code>. Summaries group results by both treatment ID
and treatment hash. Two different versions of a treatment therefore cannot be combined silently.</p>
<h3>Standardizing usage data</h3>
<p><a class="file-ref" href="../src/agent_bench/telemetry.py">telemetry.py</a> reads one JSON event
per line, even when a diagnostic prefix appears first. Because CLI versions wrap events in
different ways, it searches nested objects for known token fields. It adds input, output,
reasoning, cache-read, and cache-write tokens separately. A reported cost of zero is treated
as unknown rather than free. The same events provide model identity; OpenCode's stable
<code>providerID</code> and <code>modelID</code> log fields provide another check.</p>
<p>A cost reported by the provider takes priority. If the provider reports no cost, the runner
estimates one only for OpenCode models that have explicit per-million-token prices in
<code>benchmark.yaml</code>. Reported and estimated costs remain separate fields. The summary
combines them only when it calculates cost per successful run.</p>
<div class="table-scroll">
<table class="technical-table">
<thead><tr><th>Failure kind</th><th>Trigger</th><th>Interpretation</th></tr></thead>
<tbody>
<tr><th class="conditional">incorrect</th><td>Solver exited normally; evaluator returned a valid report with false groups.</td><td>A measured correctness failure.</td></tr>
<tr><th class="no">candidate</th><td>Evaluator supplied direct <code>candidate_error</code> evidence for broken product import/compile.</td><td>A failed submitted workspace, distinguished from ordinary behavior.</td></tr>
<tr><th class="conditional">solver</th><td>The agent command exited with a failure.</td><td>A solver failure, even if later evaluation happens to pass.</td></tr>
<tr><th class="no">phase / infrastructure</th><td>Model check, setup, evaluator protocol, Docker, timeout, quota, or startup failure.</td><td>Not counted as an incorrect solution.</td></tr>
</tbody>
</table>
</div>
<p><a class="file-ref" href="../src/agent_bench/report.py">report.py</a> writes one sorted JSON
object per line to <code>results/runs.jsonl</code>. After each completed experiment, it
appends a labeled section to <code>results/summary.md</code> without discarding earlier
experiment summaries. Each section contains a configuration-level comparison and a
per-task breakdown. Both report public, hidden, and combined completion; the task table
also exposes token usage, cost, solver time, runtime, and failure reason. Configuration
rows additionally summarize pass rate, public-test mutation telemetry, native and
estimated cost, and cost per successful run. Each run keeps its complete or partial phase
logs, including output captured before a timeout.</p>
</section>
<section class="section" id="components">
<div class="section-head"><span class="section-index">10</span><div><h2>Component and script index</h2><p class="section-deck">Use this index to find the file that implements each concept in the guide.</p></div></div>
<h3>Skill-facing helpers</h3>
<dl class="component-list">
<dt>scaffold.py</dt><dd>Copies the self-contained benchmark package into a target and gives its local Docker image a name that will not clash with another checkout.</dd>
<dt>task_workspace.py</dt><dd>Prepares isolated task-writing workspaces, runs the repeatable read-only <code>check</code> guard, accepts a reviewed bundle once, and discards the workspace.</dd>
<dt>configure.py</dt><dd>Safely creates a fixed Copilot or OpenCode treatment and can copy portable skill directories into it.</dd>
</dl>
<h3>Runner modules</h3>
<dl class="component-list">
<dt>cli.py</dt><dd>Defines commands, interactive login, OpenCode profile cleanup, progress output, and readable error messages.</dd>
<dt>config.py</dt><dd>Checks project, task, and treatment YAML, then converts it into resolved configuration records.</dd>
<dt>models.py</dt><dd>Defines configuration records, validation receipts, and standardized run results.</dd>
<dt>scaffold.py</dt><dd>Creates the scaffold used by the installed <code>agent-bench init</code> command.</dd>
<dt>workspace.py</dt><dd>Exports safe historical copies, adds public files, checks credential profiles, and prepares temporary home directories.</dd>
<dt>harnesses.py</dt><dd>Builds Copilot and OpenCode commands, prepares their environments, reads usage, and checks model identity.</dd>
<dt>telemetry.py</dt><dd>Standardizes token and cost data from JSON event logs and extracts provider and model identity.</dd>
<dt>docker.py</dt><dd>Builds images and runs restricted containers with explicit files, settings, network access, logs, and cleanup.</dd>
<dt>evaluator.py</dt><dd>Reads the structured requirement-group report and checks that it agrees with the evaluator's exit code.</dd>
<dt>identity.py</dt><dd>Computes repeatable SHA-256 hashes for tasks, treatments, and validation environments.</dd>
<dt>runner.py</dt><dd>Coordinates diagnostics, authentication checks, setup, solver and evaluator runs, validation, parallel work, and result recording.</dd>
<dt>report.py</dt><dd>Appends JSON-line results and writes a Markdown comparison that separates different input hashes.</dd>
<dt>errors.py</dt><dd>Separates configuration, infrastructure, model-identity, and timeout failures into clear categories.</dd>
</dl>
<h3>Repository maintenance scripts</h3>
<dl class="component-list">
<dt>sync_vendored_runner.py</dt><dd>Copies the maintained <code>src/agent_bench/</code> source into the skill scaffold and checks that both copies match.</dd>
<dt>create_synthetic_fixture.py</dt><dd>Creates a disposable two-commit project for a quick test of the real runner lifecycle.</dd>
<dt>format_markdown.py</dt><dd>Wraps Markdown prose at 100 columns while leaving layout-sensitive structures unchanged.</dd>
</dl>
<h3>Command ownership in normal use</h3>
<div class="table-scroll">
<table class="technical-table">
<thead><tr><th>Command or helper</th><th>Normal owner</th><th>Why users can run it directly</th></tr></thead>
<tbody>
<tr><th>Discovery / scaffold / prepare / accept</th><td>Coordinator agent after the relevant user gate.</td><td>Auditable helpers and developer diagnosis.</td></tr>
<tr><th><code>validate</code>, <code>build</code>, <code>validate-tasks</code></th><td>Coordinator agent during task creation.</td><td>CI, maintenance, and focused manual diagnosis.</td></tr>
<tr><th><code>auth login</code></th><td>User, because the provider may require an interactive browser or device flow.</td><td>Explicit credential creation without source exposure.</td></tr>
<tr><th><code>doctor</code>, <code>run</code></th><td>Coordinator after explicit run authorization.</td><td>Automation and direct hands-on experiments remain supported.</td></tr>
</tbody>
</table>
</div>
<h3>Normative references</h3>
<div class="references">
<a class="reference" href="../skills/bench-this/SKILL.md"><code>SKILL.md</code>Complete coordinator workflow and policy</a>
<a class="reference" href="../skills/bench-this/references/task-quality.md"><code>task-quality.md</code>Candidate, prompt, and evaluator quality gates</a>
<a class="reference" href="../skills/bench-this/references/setup.md"><code>setup.md</code>Image, setup, validation, and concurrency contract</a>
<a class="reference" href="../skills/bench-this/references/formats.md"><code>formats.md</code>Manifest and evaluator protocol reference</a>
<a class="reference" href="../skills/bench-this/references/configuration.md"><code>configuration.md</code>Treatment, auth, overlay, and run policy</a>
<a class="reference" href="../README.md"><code>README.md</code>Concise operational introduction</a>
</div>
</section>
<footer class="footer">END · project benchmark implementation guide · v1</footer>
</main>
</div>
<script>
// Clipboard access may be unavailable on file:// pages, so retain a selection fallback.
async function copyBlock(button) {
const source = document.getElementById(button.dataset.copy);
try {