Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
consensus_pack_id: FOWOCO-INTENT-BOUNDARY-CONSENSUS
version: 0.1.0
status: provisional_assuming_reviewer_b_agreement
rule_version: '1.1'
depends_on_pr: 31
source:
path: data/intent/hr_intent_dataset.jsonl
record_count: 1340
sha256: 4f4ebfdd4170a78def33e31edbed8315921c0b67934f5ff8612595dcd479bed2
inputs:
- reviewer_code: A
path: data/review/intent_boundary_review_a.csv
sha256: 09c6abd939f9c32e7425e27b3451e32b57f1b8f228f042afbf2d952460a283af
completed: true
- reviewer_code: B
path: data/review/intent_boundary_review_b.csv
sha256: 182cb27b4f8a1c18c05753b7336a49b2c37108bba117aa1e24fbfeb0c29a1f58
completed: false
generation:
mode: provisional_assume_b_agrees
candidate_count: 361
agreement_status_counts:
ASSUMED_PENDING_B_CONFIRMATION: 335
NEEDS_ADJUDICATION: 26
output:
path: data/review/intent_boundary_review_consensus.csv
row_count: 361
sha256: d727ea51aedfb8e460d8dddde9d5635194f876181f86266193e6c6160f06a6be
source_application:
allowed: false
block_reasons:
- Reviewer B 판정이 모든 행에 기록되지 않음
- AGREED가 아닌 consensus 행이 존재함
limitations:
- ASSUMED_PENDING_B_CONFIRMATION은 Reviewer B의 실제 판정이 아님
- NEEDS_ADJUDICATION과 DISAGREED는 합의 검수 전 원본에 반영하지 않음
- source_application.allowed가 true인 최종 합의본만 원본 반영에 사용함
362 changes: 362 additions & 0 deletions fowoco-knowledge/data/review/intent_boundary_review_consensus.csv

Large diffs are not rendered by default.

36 changes: 34 additions & 2 deletions fowoco-knowledge/docs/INTENT_AB_REVIEW.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

- 규칙 기준: Intent 라벨 규칙 v1.1
- 원본 데이터: `data/intent/hr_intent_dataset.jsonl`
- 검수 상태: 시작 전
- 검수 상태: Reviewer A 완료, Reviewer B 대기
- 선행 작업: PR #29 승인 및 `main` 반영

이 검수팩은 1,340건 전체를 다시 펼치지 않고, 규칙 v1.1에서 합의가 필요한 경계
Expand Down Expand Up @@ -57,7 +57,39 @@ Reviewer A와 B의 독립 검수 결과가 모두 제출된 뒤 합의본을 만
5. 불일치 항목만 합의 검수하고 최종 consensus 데이터를 만든다.
6. 합의 전에는 원본 1,340건과 원본 checksum을 변경하지 않는다.

## 5. 한계
## 5. Provisional consensus

Reviewer B 결과가 아직 없는 동안 다음 명령으로 Reviewer A 판정을 기준으로 한
임시 합의안을 만들 수 있다.

```bash
python -m fowoco_knowledge build-intent-consensus --assume-b-agrees
```

이 결과에서 `ASSUMED_PENDING_B_CONFIRMATION`은 Reviewer B의 실제 판정이 아니다.
Reviewer A의 `NEEDS_DISCUSSION`은 `NEEDS_ADJUDICATION`으로 유지하며 자동 합의하지
않는다. 이 상태에서는 `source_application.allowed`가 `false`이고 원본 반영 명령도
실패한다.

Reviewer B CSV가 제출되면 같은 명령을 `--assume-b-agrees` 없이 실행해 실제
일치·불일치 목록을 만든다. `DISAGREED`와 `NEEDS_ADJUDICATION`은 합의 검수 후
양쪽 CSV의 판정 또는 별도 합의 기록을 확정해야 한다.

## 6. 원본 반영

모든 행에 Reviewer B 판정이 있고 모든 `agreement_status`가 `AGREED`일 때만 별도
출력 파일을 만들 수 있다.

```bash
python -m fowoco_knowledge apply-intent-consensus \
--output data/intent/hr_intent_dataset.consensus-preview.jsonl
```

출력 파일의 Schema, ID·레코드 수, evidence exact substring, 중복, 개인정보 패턴을
검증한 뒤 원본 교체와 `data/intent/manifest.yaml`의 version·SHA-256을 갱신한다.
그 다음 PR #33의 고정 seed로 Train/Validation split과 checksum을 다시 생성한다.

## 7. 한계

- 키워드 기반 후보 추출이므로 모든 의미 오류를 보장하지 않는다.
- 후보로 선정됐다는 사실은 현재 라벨이 잘못됐다는 뜻이 아니다.
Expand Down
2 changes: 2 additions & 0 deletions fowoco-knowledge/knowledge/manifest.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ datasets:
intent_boundary_review_manifest: data/review/intent_boundary_review_manifest.yaml
intent_boundary_review_a: data/review/intent_boundary_review_a.csv
intent_boundary_review_b: data/review/intent_boundary_review_b.csv
intent_boundary_consensus_manifest: data/review/intent_boundary_consensus_manifest.yaml
intent_boundary_review_consensus: data/review/intent_boundary_review_consensus.csv
review_policy:
minimum_reviewers_for_official_knowledge: 2
all_outbound_messages_require_hr_approval: true
Expand Down
85 changes: 85 additions & 0 deletions fowoco-knowledge/src/fowoco_knowledge/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,14 @@

from .engine import RequestEvaluator
from .ingestion import OfficialDataPipeline
from .intent_consensus import (
ConsensusNotFinalError,
apply_consensus,
build_consensus_manifest,
build_consensus_rows,
write_consensus_csv,
write_manifest,
)
from .repository import KnowledgeRepository
from .validation import KnowledgeValidator

Expand Down Expand Up @@ -62,6 +70,26 @@ def build_parser() -> argparse.ArgumentParser:
)
industry_parser.add_argument("query")
industry_parser.add_argument("--limit", type=int, default=20)

consensus_parser = subparsers.add_parser(
"build-intent-consensus",
help="Compare Reviewer A/B results and build the Intent consensus pack",
)
consensus_parser.add_argument(
"--assume-b-agrees",
action="store_true",
help="Build a clearly marked provisional proposal from Reviewer A results",
)

apply_parser = subparsers.add_parser(
"apply-intent-consensus",
help="Apply only a fully agreed Intent consensus to a separate JSONL output",
)
apply_parser.add_argument("--consensus", type=Path)
apply_parser.add_argument("--manifest", type=Path)
apply_parser.add_argument("--source", type=Path)
apply_parser.add_argument("--output", type=Path, required=True)
apply_parser.add_argument("--allow-exclude", action="store_true")
return parser


Expand Down Expand Up @@ -130,4 +158,61 @@ def main(argv: Sequence[str] | None = None) -> int:
print(f"{row['industry_id']}\t{row['middle_category']}\t{row['business_content_ko']}")
return 0

if args.command == "build-intent-consensus":
reviewer_a_path = repository.root / "data/review/intent_boundary_review_a.csv"
reviewer_b_path = repository.root / "data/review/intent_boundary_review_b.csv"
output_path = repository.root / "data/review/intent_boundary_review_consensus.csv"
manifest_path = repository.root / "data/review/intent_boundary_consensus_manifest.yaml"
rows = build_consensus_rows(
reviewer_a_path,
reviewer_b_path,
assume_b_agrees=args.assume_b_agrees,
)
write_consensus_csv(output_path, rows)
manifest = build_consensus_manifest(
root=repository.root,
reviewer_a_path=reviewer_a_path,
reviewer_b_path=reviewer_b_path,
output_path=output_path,
rows=rows,
assume_b_agrees=args.assume_b_agrees,
)
write_manifest(manifest_path, manifest)
print(
f"BUILT\t{manifest['status']}\t{len(rows)}\t"
f"source_apply_allowed={manifest['source_application']['allowed']}"
)
return 0

if args.command == "apply-intent-consensus":
consensus_path = args.consensus or (
repository.root / "data/review/intent_boundary_review_consensus.csv"
)
if not consensus_path.is_absolute():
consensus_path = repository.root / consensus_path
manifest_path = args.manifest or (
repository.root / "data/review/intent_boundary_consensus_manifest.yaml"
)
if not manifest_path.is_absolute():
manifest_path = repository.root / manifest_path
source_path = args.source or (repository.root / "data/intent/hr_intent_dataset.jsonl")
if not source_path.is_absolute():
source_path = repository.root / source_path
output_path = args.output
if not output_path.is_absolute():
output_path = repository.root / output_path
try:
changed_count, record_count = apply_consensus(
consensus_path=consensus_path,
source_path=source_path,
output_path=output_path,
manifest_path=manifest_path,
allow_exclude=args.allow_exclude,
)
except ConsensusNotFinalError as error:
print(f"BLOCKED: {error}")
return 1
print(f"APPLIED\tchanged={changed_count}\trecords={record_count}\t{output_path}")
return 0

return 2
Loading