Skip to content

feat: add modusgraph-gen code generator with private field support and accessor name overrides#13

Closed
mlwelles wants to merge 185 commits into
matthewmcneely:mainfrom
mlwelles:feature/add-modusgraphgen
Closed

feat: add modusgraph-gen code generator with private field support and accessor name overrides#13
mlwelles wants to merge 185 commits into
matthewmcneely:mainfrom
mlwelles:feature/add-modusgraphgen

Conversation

@mlwelles
Copy link
Copy Markdown

@mlwelles mlwelles commented Mar 25, 2026

Summary

Adds modusgraph-gen, a code generation tool that reads Go structs with json/dgraph tags and produces typed CRUD clients, query builders, iterators, and a CLI — with full private field support and customizable accessor names.

Core Generator

Output Purpose
client_gen.go Typed Client with per-entity sub-clients
page_options_gen.go First(n) / Offset(n) pagination helpers
iter_gen.go Auto-paging SearchIter / ListIter (Go 1.23+ iter.Seq2)
<entity>_gen.go Per-entity CRUD: Get, Add, Update, Delete, Search, List
<entity>_options_gen.go Functional options per scalar field
<entity>_query_gen.go Fluent query builder with filters, ordering, pagination
<entity>_accessors_gen.go Getters, setters, slice/edge helpers (private fields only)
<entity>_marshal_gen.go DgraphMap(), UnmarshalJSON(), ValidateWith() (private fields only)
cmd/<pkg>/main.go Kong CLI with per-entity subcommands and raw DQL query

Private Field Support

Entities with unexported fields get:

  • Getters/setters with Go-idiomatic names (nameName() / SetName())
  • Singular edge accessors for *Entity, bare Entity, or validate:"max=1" fields → *Type signatures
  • Multi-edge helpers: Append (variadic), Remove (by UID), RemoveFunc (predicate)
  • Primitive slice helpers: Append, Remove (by value), RemoveFunc
  • DgraphMap() for write-path serialization (bypasses dgman's reflect limitation on unexported fields)
  • UnmarshalJSON() for read-path deserialization
  • ValidateWith(ctx, validator) for struct validation via a mirror struct with exported fields
  • Field opt-out: no json tag → skip; dgraph:"-" → explicit skip

Accessor Name Overrides (New)

Initialism-aware naming: toCamelCase now recognizes the canonical Go lint initialisms (38 entries from golang.org/x/lint). Fields like id produce ID() / SetID() automatically.

Per-field overrides via accessor struct tag:

type Widget struct {
    id   string `json:"id,omitempty" accessor:"ID"`
    url  string `json:"url,omitempty"`           // auto: URL() / SetURL()
    name string `json:"name,omitempty"`           // auto: Name() / SetName()
}

The override applies everywhere: getters, setters, Append*, Remove*, With*Option, UnmarshalJSON alias, ValidateWith mirror, and CLI struct fields.

Engine Changes

  • DgraphMapper interface: Insert/Update detect entities with private fields and route through map-based mutation
  • InsertRaw for raw struct insertion
  • Validator support via WithValidator(...) and SelfValidator interface

Edge Type Coverage

All edge variants are supported and tested:

  • []Entity, []*Entity — multi-edge (getter, setter, append, remove, removeFunc)
  • *Entity — singular edge (getter, setter)
  • Entity (bare value) — singular edge (getter returns *Entity, setter accepts *Entity)
  • []Entity / []*Entity + validate:"max=1" or validate:"len=1" — singular edge

Zero New Dependencies

The generator uses only the Go standard library (go/ast, go/parser, text/template, embed).

Usage

//go:generate go run github.com/matthewmcneely/modusgraph/cmd/modusgraph-gen

Testing

  • Unit tests: toCamelCase with 33 initialism/compound/edge cases, accessorName with override and fallback, accessor tag parsing
  • Golden file tests: all entity types including Studio (private field test fixture with all edge variants)
  • Parser tests: parseValidateTag, dgraph:"-", private field detection, singular edge detection, accessor tag
  • Generator tests: accessor output, marshal output, CLI output, validator output, edge variants, external imports
  • All existing modusGraph tests pass unchanged
  • go vet / go build clean

ryanfoxtyler and others added 30 commits October 2, 2024 15:12
This PR removes the WithDataDir() function, which implies that the
datadirectory is optional and enforces it on NewDefaultConfig, since on
its own, the NewDefaultConfig will not spin up a new modusDB instance.
Addressed in

https://linear.app/hypermode/issue/DGR-822/modusdb-newdefaultconfig-fails
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from
0.29.0 to 0.31.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/golang/crypto/commit/b4f1988a35dee11ec3e05d6bf3e90b695fbd8909"><code>b4f1988</code></a>
ssh: make the public key cache a 1-entry FIFO cache</li>
<li><a
href="https://github.com/golang/crypto/commit/7042ebcbe097f305ba3a93f9a22b4befa4b83d29"><code>7042ebc</code></a>
openpgp/clearsign: just use rand.Reader in tests</li>
<li><a
href="https://github.com/golang/crypto/commit/3e90321ac7bcee3d924ed63ed3ad97be2079cb56"><code>3e90321</code></a>
go.mod: update golang.org/x dependencies</li>
<li><a
href="https://github.com/golang/crypto/commit/8c4e668694ccbaa1be4785da7e7a40f2ef93152b"><code>8c4e668</code></a>
x509roots/fallback: update bundle</li>
<li>See full diff in <a
href="https://github.com/golang/crypto/compare/v0.29.0...v0.31.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=golang.org/x/crypto&package-manager=go_modules&previous-version=0.29.0&new-version=0.31.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/hypermodeinc/modusDB/network/alerts).

</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This PR adds the initial issue and PR templates, a changelog, code of
conduct, contributing guide, and security reporting instructions. We can
enhance these over time.
**Description**

Initialize Trunk for broader linting and security monitoring. Fix
addressable issues.

**Checklist**

- [x] Code compiles correctly and linting passes locally
**Description**

add readme to repo
Add []int, []float64, []bool, []time.Time private slice fields to
Studio test entity. Verify generated Get/Set/Append/Remove/RemoveFunc
signatures for each primitive slice type.
…truct

Private fields can't be validated by go-playground/validator directly
(it panics on reflect.Value.Interface() for unexported fields). Solve
this by generating a ValidateWith(ctx, validator) method that builds a
temporary exported mirror struct with the same values and validate tags,
then delegates to validator.StructCtx().

Changes:
- model: add ValidateTag string field to capture raw validate tags
- parser: store full validate tag in ValidateTag
- generator: add hasValidateTags/fieldsWithValidation helpers
- marshal.go.tmpl: generate ValidateWith method when validate tags exist
- client.go: add SelfValidator interface, validateOne helper that checks
  for SelfValidator before falling back to StructCtx

Custom validators registered on the *validator.Validate instance work
automatically since the mirror struct preserves the raw validate tags.

Add Studio test fields with validate tags (name: required/min/max,
yearFounded: gte/lte, revenue: gte) and generator tests verifying
ValidateWith method generation, mirror struct tags, and field assignments.
…agged

ValidateWith mirror struct now includes ALL fields (not just those with
validate tags) so custom validators registered by field name or type
work correctly. The mirror is always generated for entities with private
fields, even if no validate tags are present.

Add comprehensive validation integration tests:
- TestSelfValidatorDispatch: 8 subtests covering valid/invalid values
  for string (required, min), int (gte/lte boundaries), float64 (range),
  and boundary value testing
- TestSelfValidatorWithCustomValidator: 4 subtests verifying custom
  validator registration (studio_code format) works through ValidateWith
- TestValidateOneDispatchesSelfValidator: 3 subtests verifying engine
  dispatches to SelfValidator for private-field entities and falls back
  to StructCtx for regular entities
Add accessor:"." struct tag for per-field name overrides and
initialism-aware toCamelCase using the canonical golang.org/x/lint
list (38 entries). Fields like 'id' now generate ID()/SetID()
automatically; explicit overrides take precedence.
@mlwelles mlwelles changed the title feat: add modusgraph-gen code generator with private field support feat: add modusgraph-gen code generator with private field support and accessor name overrides Apr 2, 2026
…d packages

The marshal template hardcoded only context, encoding/json, and modusgraph
imports. When entity fields use external types (scalars.UUID, enums.ArchiveStatus)
or aliased packages (dg "github.com/dolan-in/dgman/v2"), the generated
_marshal_gen.go files would fail to compile.

Changes:
- Add marshalFields() helper returning all non-UID/DType fields for import computation
- Refactor externalImports() to return ImportSpec with alias support, stripping
  pointer/slice prefixes before extracting the package qualifier
- Update marshal, accessors, and options templates to emit aliased imports
- Add tests for aliased imports, pointer-prefixed types, and marshal import coverage
- Regenerate golden files
@cubic-dev-ai
Copy link
Copy Markdown

cubic-dev-ai Bot commented Apr 14, 2026

You're iterating quickly on this pull request. To help protect your rate limits, cubic has paused automatic reviews on new pushes for now—when you're ready for another review, comment @cubic-dev-ai review.

mlwelles added 19 commits April 14, 2026 14:37
The CLI template generates string flags for external types (enums, scalars)
but the entity setters expect the typed values. This adds explicit type
conversions (e.g., enums.ArchiveStatus(c.ArchiveStatus)) and imports the
external packages into the generated CLI main.go.

Changes:
- Update cliConvert() to wrap external types with a type conversion
- Add packageExternalImports() to collect imports across all entities
- Update CLI template to emit external package imports
toDgraphMap dereferences struct pointers with v.Elem() before checking
the DgraphMapper interface. Since DgraphMap() uses a pointer receiver,
the dereferenced struct value's method set excludes it, so the type
assertion always failed. This caused all entities with private fields to
fall through to dgman's reflection-based MutateBasic path, which crashes
on unexported fields during the post-mutation setUIDs walk.

Fix: introduce asDgraphMapper() helper that checks both v.Interface()
(value receivers) and v.Addr().Interface() (pointer receivers). Replace
all three DgraphMapper assertion sites in toDgraphMap with this helper.
The marshal template's default case unconditionally emitted fields with
external types (e.g., scalars.UUID). Since scalars.UUID is a string type
alias, this sent empty strings to Dgraph, triggering @upsert unique
constraint violations on the id predicate.

Fix: use a typed zero-value comparison (var zero T; if e.field != zero)
for all external/unknown scalar types in the DgraphMap template.
The DgraphMapper path bypasses dgman's reflectwalk pipeline, which means
dgraph.type and blank UIDs are not automatically set. Without dgraph.type,
inserted nodes have no type and type-based queries return nothing.

Fix: mutateWithMap now sets dgraph.type from the Go struct name and assigns
a blank UID when missing, then sends the map directly to Dgraph via the
gRPC Mutate API and writes the allocated UID back to the struct.
Instead of producing a map and bypassing dgman's mutation pipeline,
entities with private fields now generate an all-exported 'reflectable'
shadow struct. Before mutation, ToReflectable() copies field values to
the shadow; dgman mutates it normally via reflectwalk; FromReflectable()
copies UID/DType back. This eliminates toDgraphMap, asDgraphMapper,
mutateWithMap, and the direct Dgraph API calls — dgman handles
dgraph.type, blank UIDs, and post-mutation UID writeback as designed.

The reflectable struct carries the same json and dgraph tags as the
original, so schema generation, predicate resolution, and indexing all
work correctly through dgman's existing code paths.
dgman's query builder reflects on the struct to determine which predicates
to project. Private fields are invisible to reflection, so queries returned
only UID/DType. The generated entity client now queries using the
reflectable struct (all-exported, same predicates) for Get, List, and
Search, then converts back to the entity via JSON round-trip through
UnmarshalJSON.
Replace the modusgraph-level Reflectable interface and mutateFunc with
dgman's HasReflectable interface. Entity structs now implement
ToReflectable()/FromReflectable() which dgman calls directly.

Template changes:
- marshal.go.tmpl: ToReflectable/FromReflectable replace
  QueryModel/MutateModel/MutateBack. UnmarshalJSON uses
  map[string]json.RawMessage (no reflectable allocation on query path).
  MarshalJSON added for symmetry.
- entity.go.tmpl: Get/List/Search pass the entity directly to dgman
  instead of creating a reflectable proxy and doing a JSON round-trip.

client.go:
- Remove Reflectable interface and mutateFunc.
- Insert/Update pass the entity directly to tx.MutateBasic.

go.mod: replace dgman with mlwelles/dgman feature/private-field-support.
Dgraph uses optimistic concurrency control; concurrent mutations on
overlapping predicates cause transaction aborts (dgo.ErrAborted).
Add a retry loop in process() that creates a fresh TxnContext and
retries up to maxRetries times (default 3). Configurable via
WithMaxRetries(n) client option.

This gives Insert(), Upsert(), and Update() retry for free.
Add RetryPolicy type and WithRetry method on Client, modeled after
dgraph4j's client.withRetry(). Retry is opt-in per call site, not
default behavior. Uses exponential backoff with jitter (defaults:
5 retries, 100ms base, 5s max, 10% jitter). Context cancellation
is respected during backoff sleeps.

Default Insert/Upsert/Update behavior is unchanged.
Prepare for gRPC support by abstracting mutation submission behind
a mutator interface. The embedded path (namespaceMutator) preserves
existing behavior. The shared loadData() pipeline handles both modes.
LoadData() loads RDF/JSON data files via the embedded engine or gRPC.
The grpcMutator sends batched mutations with CommitNow=true over gRPC,
receiving blank node UID mappings from the server for cross-batch
resolution. WithSchema(path) option applies schema before loading.
Add grpcUIDAllocator that leases UIDs in bulk (10K at a time) via
dgo.AllocateUIDs, enabling local blank node resolution for the gRPC
path — same as the embedded engine. With UIDs pre-resolved, mutations
are independent, so the consumer is now 8 concurrent workers instead
of a single goroutine. Progress ticker moved outside the errgroup
to avoid blocking completion.
…nWorkers

Export LoadOptions struct and fields. Add WithLoadOptions for passing
a full struct. Defaults match upstream: BatchSize=1000, MutationWorkers=1.
Callers tune via WithBatchSize/WithMutationWorkers or WithLoadOptions.
…d LoadData

Add 19 new tests covering:
- LoadOptions defaults, zero/negative values, explicit values
- All LoadOpt functions: WithBatchSize, WithMutationWorkers, WithSchema,
  WithLoadOptions (struct-based), composition/override behavior
- RetryPolicy.delay() exponential growth, max cap, jitter bounds
- isAbortedError with direct, wrapped, and non-abort errors
- WithRetry: non-abort errors (no retry), success on first try,
  MaxRetries=0 (single call)
- LoadData: without schema, bad schema path, empty data dir,
  with individual options, with struct-based WithLoadOptions
…ator

- Move LoadOptions, WithSchema, WithBatchSize, WithMutationWorkers to
  load sub-package: load.Options, load.WithSchema, etc.
- Add retry with exponential backoff to grpcMutator.mutate() for
  transaction aborts from concurrent mutations
- Add load.FileMatcher interface, load.FileSort, load.FilterFiles()
  method for configurable file discovery pipeline
- FilterFiles() applies FileMatcher per-item; nil matcher matches all
- Remove old live_opts.go from root package
Code conventions:
- DefaultExtensions: mutable var → function returning copy
- Remove pkg/errors, use fmt.Errorf with %%w consistently
- grpcMutator uses RetryPolicy.delay() instead of duplicate backoff
- grpcUIDAllocator uses uint64 throughout (no int overflow)
- CommitNow set before retry loop, not inside
- WithRetry: remove dead lastErr, use range loop
- Remove isAbortedError wrapper, inline errors.Is
- Magic 10000 → defaultNqchBufferSize constant
- Add godoc on Namespace.Load
- Remove stale CSV TODO
- Embedded LoadData now routes through loadData() respecting all options

File renames:
- live.go → loaddata.go
- live_grpc.go → loaddata_grpc.go
- live_grpc_test.go → loaddata_test.go

New integration tests:
- FileMatch filtering (verifies only matched files are loaded)
- Multiple RDF files from one directory
- Gzipped RDF loading
- Blank node resolution across files
@mlwelles
Copy link
Copy Markdown
Author

Closing in favor of landing the work directly. This branch was substantially reworked since the PR opened: the private-field serialization approach (and its dgman fork) described in the original PR title was replaced by the public-fielded schema + opt-in generated wrapper layer (see docs/specs/2026-05-18-public-wrapper-types-design.md). The finished work is being merged directly rather than through this stale PR.

@mlwelles mlwelles closed this May 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants