fix: release temporary cast arrays after ownership transfer#898
fix: release temporary cast arrays after ownership transfer#898fallintoplace wants to merge 4 commits into
Conversation
zeroshade
left a comment
There was a problem hiding this comment.
The production fix is correct and worth having — ArraySpan.TakeOwnership retains the buffers (arrow/compute/exec/span.go), so the ba.bldr.NewArray() temporary was leaking without the defer result.Release().
But both new tests fail:
cast_test.go:1682: invalid memory size exp=192, got=0
cast_test.go:1694: invalid memory size exp=32896, got=0
--- FAIL: TestCasts/TestBinaryToBinaryViewReleasesTemporaryResult
--- FAIL: TestCasts/TestBinaryViewToBinaryReleasesTemporaryResult
The scope is created after in is allocated, but the test releases in inside the scope before CheckSize, so outstanding drops below the baseline (exp = size of in, got = 0). Either call scope.CheckSize(c.T()) before in.Release(), or allocate in inside the scope. Once the harness is fixed these will lock the fix in nicely. Verified against the PR head with go test.
|
@fallintoplace nudge — the fix itself is correct ( |
zeroshade
left a comment
There was a problem hiding this comment.
The production fix here (defer result.Release() after TakeOwnership) is correct — TakeOwnership retains, so dropping the builder's ref is right. The two new tests need rework though: they fail as written, and even after fixing the ordering they don't bind the cast to the checked allocator, so they'd pass regardless of the fix. Details plus a verified formulation inline. Requesting changes on the tests only.
| } | ||
|
|
||
| func (c *CastSuite) TestBinaryToBinaryViewReleasesTemporaryResult() { | ||
| scope := memory.NewCheckedAllocatorScope(c.mem) |
There was a problem hiding this comment.
Two problems with this test:
- It fails as written — the scope is opened here, before
inis allocated, butCheckSizeruns whileinis still alive, so it reportsexp=0, got=192. - More importantly, the cast runs on
context.Background()→DefaultAllocator, notc.mem, so the checked scope never sees the temporary. It passes with or without the production fix (I confirmed both), so it doesn't actually guard the leak.
To exercise the fix, allocate in first, open the scope after it, and bind the cast to the checked allocator:
in, _, err := array.FromJSON(c.mem, arrow.BinaryTypes.Binary, strings.NewReader(`["aGk=", "dGhpcyBpcyB0aGUgZmlyc3QgdGVzdCE=", null]`))
c.Require().NoError(err)
scope := memory.NewCheckedAllocatorScope(c.mem)
ctx := exec.WithAllocator(context.Background(), c.mem) // arrow/compute/exec
out, err := compute.CastArray(ctx, in, compute.SafeCastOptions(arrow.BinaryTypes.BinaryView))
c.Require().NoError(err)
out.Release()
scope.CheckSize(c.T())
in.Release()I verified this fails before the fix (exp=192, got=33088) and passes after.
| } | ||
|
|
||
| func (c *CastSuite) TestBinaryViewToBinaryReleasesTemporaryResult() { | ||
| scope := memory.NewCheckedAllocatorScope(c.mem) |
There was a problem hiding this comment.
Same two issues as the binary→view test above: it fails as written, and once the ordering is fixed it still doesn't bind the cast to c.mem, so it's vacuous. Apply the same pattern — build in, then open the scope, run the cast through exec.WithAllocator(context.Background(), c.mem), release out, CheckSize, then release in.
Summary
Problem
CastBinaryToBinaryViewandCastBinaryViewToBinaryeach create a temporary result array withba.bldr.NewArray(), transfer ownership of its buffers without.TakeOwnership(result.Data()), and then return without releasing the temporary array.TakeOwnershiponly transfers buffer ownership, so the temporary array retains its own reference and can leak memory per cast.Change
defer result.Release()after creating the temporary result array in:CastBinaryToBinaryViewCastBinaryViewToBinaryTakeOwnershipbehavior intact.Impact
Testing
arrow/compute/cast_test.go:TestBinaryToBinaryViewReleasesTemporaryResultTestBinaryViewToBinaryReleasesTemporaryResultmemory.NewCheckedAllocatorScopeafter releasing cast outputs and inputs.