feat: add modusgraph-gen code generator with private field support and accessor name overrides#13
Closed
mlwelles wants to merge 185 commits into
Closed
feat: add modusgraph-gen code generator with private field support and accessor name overrides#13mlwelles wants to merge 185 commits into
mlwelles wants to merge 185 commits into
Conversation
This PR removes the WithDataDir() function, which implies that the datadirectory is optional and enforces it on NewDefaultConfig, since on its own, the NewDefaultConfig will not spin up a new modusDB instance. Addressed in https://linear.app/hypermode/issue/DGR-822/modusdb-newdefaultconfig-fails
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.29.0 to 0.31.0. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/golang/crypto/commit/b4f1988a35dee11ec3e05d6bf3e90b695fbd8909"><code>b4f1988</code></a> ssh: make the public key cache a 1-entry FIFO cache</li> <li><a href="https://github.com/golang/crypto/commit/7042ebcbe097f305ba3a93f9a22b4befa4b83d29"><code>7042ebc</code></a> openpgp/clearsign: just use rand.Reader in tests</li> <li><a href="https://github.com/golang/crypto/commit/3e90321ac7bcee3d924ed63ed3ad97be2079cb56"><code>3e90321</code></a> go.mod: update golang.org/x dependencies</li> <li><a href="https://github.com/golang/crypto/commit/8c4e668694ccbaa1be4785da7e7a40f2ef93152b"><code>8c4e668</code></a> x509roots/fallback: update bundle</li> <li>See full diff in <a href="https://github.com/golang/crypto/compare/v0.29.0...v0.31.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/hypermodeinc/modusDB/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This PR adds the initial issue and PR templates, a changelog, code of conduct, contributing guide, and security reporting instructions. We can enhance these over time.
**Description** Initialize Trunk for broader linting and security monitoring. Fix addressable issues. **Checklist** - [x] Code compiles correctly and linting passes locally
**Description** add readme to repo
Add []int, []float64, []bool, []time.Time private slice fields to Studio test entity. Verify generated Get/Set/Append/Remove/RemoveFunc signatures for each primitive slice type.
…truct Private fields can't be validated by go-playground/validator directly (it panics on reflect.Value.Interface() for unexported fields). Solve this by generating a ValidateWith(ctx, validator) method that builds a temporary exported mirror struct with the same values and validate tags, then delegates to validator.StructCtx(). Changes: - model: add ValidateTag string field to capture raw validate tags - parser: store full validate tag in ValidateTag - generator: add hasValidateTags/fieldsWithValidation helpers - marshal.go.tmpl: generate ValidateWith method when validate tags exist - client.go: add SelfValidator interface, validateOne helper that checks for SelfValidator before falling back to StructCtx Custom validators registered on the *validator.Validate instance work automatically since the mirror struct preserves the raw validate tags. Add Studio test fields with validate tags (name: required/min/max, yearFounded: gte/lte, revenue: gte) and generator tests verifying ValidateWith method generation, mirror struct tags, and field assignments.
…agged ValidateWith mirror struct now includes ALL fields (not just those with validate tags) so custom validators registered by field name or type work correctly. The mirror is always generated for entities with private fields, even if no validate tags are present. Add comprehensive validation integration tests: - TestSelfValidatorDispatch: 8 subtests covering valid/invalid values for string (required, min), int (gte/lte boundaries), float64 (range), and boundary value testing - TestSelfValidatorWithCustomValidator: 4 subtests verifying custom validator registration (studio_code format) works through ValidateWith - TestValidateOneDispatchesSelfValidator: 3 subtests verifying engine dispatches to SelfValidator for private-field entities and falls back to StructCtx for regular entities
Add accessor:"." struct tag for per-field name overrides and initialism-aware toCamelCase using the canonical golang.org/x/lint list (38 entries). Fields like 'id' now generate ID()/SetID() automatically; explicit overrides take precedence.
…d packages The marshal template hardcoded only context, encoding/json, and modusgraph imports. When entity fields use external types (scalars.UUID, enums.ArchiveStatus) or aliased packages (dg "github.com/dolan-in/dgman/v2"), the generated _marshal_gen.go files would fail to compile. Changes: - Add marshalFields() helper returning all non-UID/DType fields for import computation - Refactor externalImports() to return ImportSpec with alias support, stripping pointer/slice prefixes before extracting the package qualifier - Update marshal, accessors, and options templates to emit aliased imports - Add tests for aliased imports, pointer-prefixed types, and marshal import coverage - Regenerate golden files
|
You're iterating quickly on this pull request. To help protect your rate limits, cubic has paused automatic reviews on new pushes for now—when you're ready for another review, comment |
The CLI template generates string flags for external types (enums, scalars) but the entity setters expect the typed values. This adds explicit type conversions (e.g., enums.ArchiveStatus(c.ArchiveStatus)) and imports the external packages into the generated CLI main.go. Changes: - Update cliConvert() to wrap external types with a type conversion - Add packageExternalImports() to collect imports across all entities - Update CLI template to emit external package imports
toDgraphMap dereferences struct pointers with v.Elem() before checking the DgraphMapper interface. Since DgraphMap() uses a pointer receiver, the dereferenced struct value's method set excludes it, so the type assertion always failed. This caused all entities with private fields to fall through to dgman's reflection-based MutateBasic path, which crashes on unexported fields during the post-mutation setUIDs walk. Fix: introduce asDgraphMapper() helper that checks both v.Interface() (value receivers) and v.Addr().Interface() (pointer receivers). Replace all three DgraphMapper assertion sites in toDgraphMap with this helper.
The marshal template's default case unconditionally emitted fields with external types (e.g., scalars.UUID). Since scalars.UUID is a string type alias, this sent empty strings to Dgraph, triggering @upsert unique constraint violations on the id predicate. Fix: use a typed zero-value comparison (var zero T; if e.field != zero) for all external/unknown scalar types in the DgraphMap template.
The DgraphMapper path bypasses dgman's reflectwalk pipeline, which means dgraph.type and blank UIDs are not automatically set. Without dgraph.type, inserted nodes have no type and type-based queries return nothing. Fix: mutateWithMap now sets dgraph.type from the Go struct name and assigns a blank UID when missing, then sends the map directly to Dgraph via the gRPC Mutate API and writes the allocated UID back to the struct.
Instead of producing a map and bypassing dgman's mutation pipeline, entities with private fields now generate an all-exported 'reflectable' shadow struct. Before mutation, ToReflectable() copies field values to the shadow; dgman mutates it normally via reflectwalk; FromReflectable() copies UID/DType back. This eliminates toDgraphMap, asDgraphMapper, mutateWithMap, and the direct Dgraph API calls — dgman handles dgraph.type, blank UIDs, and post-mutation UID writeback as designed. The reflectable struct carries the same json and dgraph tags as the original, so schema generation, predicate resolution, and indexing all work correctly through dgman's existing code paths.
dgman's query builder reflects on the struct to determine which predicates to project. Private fields are invisible to reflection, so queries returned only UID/DType. The generated entity client now queries using the reflectable struct (all-exported, same predicates) for Get, List, and Search, then converts back to the entity via JSON round-trip through UnmarshalJSON.
Replace the modusgraph-level Reflectable interface and mutateFunc with dgman's HasReflectable interface. Entity structs now implement ToReflectable()/FromReflectable() which dgman calls directly. Template changes: - marshal.go.tmpl: ToReflectable/FromReflectable replace QueryModel/MutateModel/MutateBack. UnmarshalJSON uses map[string]json.RawMessage (no reflectable allocation on query path). MarshalJSON added for symmetry. - entity.go.tmpl: Get/List/Search pass the entity directly to dgman instead of creating a reflectable proxy and doing a JSON round-trip. client.go: - Remove Reflectable interface and mutateFunc. - Insert/Update pass the entity directly to tx.MutateBasic. go.mod: replace dgman with mlwelles/dgman feature/private-field-support.
Dgraph uses optimistic concurrency control; concurrent mutations on overlapping predicates cause transaction aborts (dgo.ErrAborted). Add a retry loop in process() that creates a fresh TxnContext and retries up to maxRetries times (default 3). Configurable via WithMaxRetries(n) client option. This gives Insert(), Upsert(), and Update() retry for free.
Add RetryPolicy type and WithRetry method on Client, modeled after dgraph4j's client.withRetry(). Retry is opt-in per call site, not default behavior. Uses exponential backoff with jitter (defaults: 5 retries, 100ms base, 5s max, 10% jitter). Context cancellation is respected during backoff sleeps. Default Insert/Upsert/Update behavior is unchanged.
Prepare for gRPC support by abstracting mutation submission behind a mutator interface. The embedded path (namespaceMutator) preserves existing behavior. The shared loadData() pipeline handles both modes.
LoadData() loads RDF/JSON data files via the embedded engine or gRPC. The grpcMutator sends batched mutations with CommitNow=true over gRPC, receiving blank node UID mappings from the server for cross-batch resolution. WithSchema(path) option applies schema before loading.
Add grpcUIDAllocator that leases UIDs in bulk (10K at a time) via dgo.AllocateUIDs, enabling local blank node resolution for the gRPC path — same as the embedded engine. With UIDs pre-resolved, mutations are independent, so the consumer is now 8 concurrent workers instead of a single goroutine. Progress ticker moved outside the errgroup to avoid blocking completion.
…nWorkers Export LoadOptions struct and fields. Add WithLoadOptions for passing a full struct. Defaults match upstream: BatchSize=1000, MutationWorkers=1. Callers tune via WithBatchSize/WithMutationWorkers or WithLoadOptions.
…d LoadData Add 19 new tests covering: - LoadOptions defaults, zero/negative values, explicit values - All LoadOpt functions: WithBatchSize, WithMutationWorkers, WithSchema, WithLoadOptions (struct-based), composition/override behavior - RetryPolicy.delay() exponential growth, max cap, jitter bounds - isAbortedError with direct, wrapped, and non-abort errors - WithRetry: non-abort errors (no retry), success on first try, MaxRetries=0 (single call) - LoadData: without schema, bad schema path, empty data dir, with individual options, with struct-based WithLoadOptions
…ator - Move LoadOptions, WithSchema, WithBatchSize, WithMutationWorkers to load sub-package: load.Options, load.WithSchema, etc. - Add retry with exponential backoff to grpcMutator.mutate() for transaction aborts from concurrent mutations - Add load.FileMatcher interface, load.FileSort, load.FilterFiles() method for configurable file discovery pipeline - FilterFiles() applies FileMatcher per-item; nil matcher matches all - Remove old live_opts.go from root package
Code conventions: - DefaultExtensions: mutable var → function returning copy - Remove pkg/errors, use fmt.Errorf with %%w consistently - grpcMutator uses RetryPolicy.delay() instead of duplicate backoff - grpcUIDAllocator uses uint64 throughout (no int overflow) - CommitNow set before retry loop, not inside - WithRetry: remove dead lastErr, use range loop - Remove isAbortedError wrapper, inline errors.Is - Magic 10000 → defaultNqchBufferSize constant - Add godoc on Namespace.Load - Remove stale CSV TODO - Embedded LoadData now routes through loadData() respecting all options File renames: - live.go → loaddata.go - live_grpc.go → loaddata_grpc.go - live_grpc_test.go → loaddata_test.go New integration tests: - FileMatch filtering (verifies only matched files are loaded) - Multiple RDF files from one directory - Gzipped RDF loading - Blank node resolution across files
Author
|
Closing in favor of landing the work directly. This branch was substantially reworked since the PR opened: the private-field serialization approach (and its dgman fork) described in the original PR title was replaced by the public-fielded schema + opt-in generated wrapper layer (see docs/specs/2026-05-18-public-wrapper-types-design.md). The finished work is being merged directly rather than through this stale PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
modusgraph-gen, a code generation tool that reads Go structs withjson/dgraphtags and produces typed CRUD clients, query builders, iterators, and a CLI — with full private field support and customizable accessor names.Core Generator
client_gen.goClientwith per-entity sub-clientspage_options_gen.goFirst(n)/Offset(n)pagination helpersiter_gen.goSearchIter/ListIter(Go 1.23+iter.Seq2)<entity>_gen.goGet,Add,Update,Delete,Search,List<entity>_options_gen.go<entity>_query_gen.go<entity>_accessors_gen.go<entity>_marshal_gen.goDgraphMap(),UnmarshalJSON(),ValidateWith()(private fields only)cmd/<pkg>/main.goPrivate Field Support
Entities with unexported fields get:
name→Name()/SetName())*Entity, bareEntity, orvalidate:"max=1"fields →*TypesignaturesAppend(variadic),Remove(by UID),RemoveFunc(predicate)Append,Remove(by value),RemoveFuncDgraphMap()for write-path serialization (bypasses dgman's reflect limitation on unexported fields)UnmarshalJSON()for read-path deserializationValidateWith(ctx, validator)for struct validation via a mirror struct with exported fieldsjsontag → skip;dgraph:"-"→ explicit skipAccessor Name Overrides (New)
Initialism-aware naming:
toCamelCasenow recognizes the canonical Go lint initialisms (38 entries fromgolang.org/x/lint). Fields likeidproduceID()/SetID()automatically.Per-field overrides via
accessorstruct tag:The override applies everywhere: getters, setters,
Append*,Remove*,With*Option,UnmarshalJSONalias,ValidateWithmirror, and CLI struct fields.Engine Changes
DgraphMapperinterface:Insert/Updatedetect entities with private fields and route through map-based mutationInsertRawfor raw struct insertionWithValidator(...)andSelfValidatorinterfaceEdge Type Coverage
All edge variants are supported and tested:
[]Entity,[]*Entity— multi-edge (getter, setter, append, remove, removeFunc)*Entity— singular edge (getter, setter)Entity(bare value) — singular edge (getter returns*Entity, setter accepts*Entity)[]Entity/[]*Entity+validate:"max=1"orvalidate:"len=1"— singular edgeZero New Dependencies
The generator uses only the Go standard library (
go/ast,go/parser,text/template,embed).Usage
//go:generate go run github.com/matthewmcneely/modusgraph/cmd/modusgraph-genTesting
toCamelCasewith 33 initialism/compound/edge cases,accessorNamewith override and fallback,accessortag parsingparseValidateTag,dgraph:"-", private field detection, singular edge detection,accessortaggo vet/go buildclean