feat: account MinIO in knowledge-base quotas

[MoeexT]

原有知识库

新建知识库

[Copilot]

Pull request overview

This PR extends knowledge-base quota accounting to include MinIO-retained source objects (in addition to Elasticsearch index storage) by introducing a durable storage-object ledger, strict storage metadata reads, and reconciliation/backfill tooling to keep the ledger accurate.

Changes:

• Add a knowledge_storage_object_t ledger table + DAL/service layer helpers to record authoritative MinIO object sizes per KB/tenant.
• Update quota computation and deletion/upload flows to commit/release MinIO storage charges and to include MinIO bytes in quota usage.
• Add strict, paginated ES source-reference enumeration and maintenance CLI/service tests for backfill + reconciliation.

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/sdk/vector_database/test_elasticsearch_core.py	Adds coverage for strict ES composite-aggregation pagination and error behavior.
test/sdk/storage/test_minio.py	Adds coverage for strict MinIO size semantics (missing vs operational error).
test/backend/test_knowledge_storage_reconciliation_cli.py	Tests CLI wiring/exit codes for reconciliation/backfill operations.
test/backend/services/test_vectordatabase_service.py	Tests that ledger releases happen only after successful MinIO deletion and new cleanup semantics.
test/backend/services/test_quota_service.py	Tests combining ES + MinIO bytes and tenant totals including orphaned ledger rows.
test/backend/services/test_knowledge_storage_service.py	Unit tests for KB upload accounting helpers (commit/compensate/byte aggregation).
test/backend/services/test_knowledge_storage_reconciliation_service.py	Unit tests for conservative backfill and reconcile flows.
test/backend/services/test_file_management_service.py	Extends upload/delete tests for KB-only accounting, batch-size enforcement, and tenant-scoped delete semantics.
test/backend/database/test_knowledge_storage_object_db.py	Unit tests for new ledger DAL idempotency, constraints, and aggregations.
test/backend/database/test_client.py	Tests MinIO client strict size passthrough.
test/backend/database/test_attachment_db.py	Tests strict MinIO size helper normalization and missing semantics.
test/backend/app/test_quota_app.py	Ensures usage responses expose only composite totals (no component fields) and preserve null limits.
test/backend/app/test_file_management_app.py	Updates delete impl signature plumbing and adds forbidden mapping for tenant-ownership failures.
sdk/nexent/vector_database/elasticsearch_core.py	Adds get_documents_detail_strict using paginated composite aggregation (raises on ES failures).
sdk/nexent/storage/storage_client_base.py	Adds abstract strict size API to storage client base.
sdk/nexent/storage/minio.py	Implements strict MinIO size lookup with “None if missing, raise otherwise” semantics.
frontend/app/[locale]/resource-manage/components/resources/KnowledgeList.tsx	Displays composite per-KB usage from quota data and always refreshes quota usage.
deploy/sql/migrations/v2.5.0_0811_add_kb_storage_object_ledger.sql	Adds new durable KB storage ledger table + indexes via migration.
deploy/sql/init.sql	Adds the ledger table + indexes to fresh-init schema.
backend/services/vectordatabase_service.py	Adds KB source cleanup using ledger + strict ES enumeration; releases charges after confirmed deletion.
backend/services/quota_service.py	Includes committed MinIO bytes in KB + tenant totals and adds tenant-scoped cache invalidation.
backend/services/knowledge_storage_service.py	New KB upload accounting service helpers (context resolution, commit, compensation, byte totals).
backend/services/knowledge_storage_reconciliation_service.py	New reconciliation/backfill service and helpers for canonical reference parsing + cache invalidation.
backend/services/file_management_service.py	KB-only quota enforcement uses complete-batch sizing and commits ledger rows post-upload; tenant-aware delete reconciles ledger.
backend/knowledge_storage_reconciliation.py	Adds CLI entry point for backfill/reconcile operations.
backend/database/knowledge_storage_object_db.py	New DAL for committing, listing, aggregating, and soft-deleting ledger rows (idempotent + conflict-safe).
backend/database/db_models.py	Adds ORM model + constraints/indexes for knowledge_storage_object_t.
backend/database/client.py	Exposes get_file_size_strict on MinioClient wrapper.
backend/database/attachment_db.py	Adds get_file_size_from_minio_strict helper.
backend/apps/file_management_app.py	Passes tenant/user context into delete flow and maps tenant-ownership failure to 403.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

❌ Patch coverage is 90.06772% with 44 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
backend/services/vectordatabase_service.py	83.33%	17 Missing and 2 partials ⚠️
backend/services/knowledge_storage_service.py	88.29%	9 Missing and 2 partials ⚠️
backend/services/file_management_service.py	87.14%	5 Missing and 4 partials ⚠️
backend/database/knowledge_storage_object_db.py	97.02%	0 Missing and 3 partials ⚠️
sdk/nexent/storage/storage_client_base.py	50.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!