Question on backend refcounting

[gquintard]

Hi,

I'm working on backend/director abstraction in rust, and I can't wrap my head around one question, which is usually a sign I'm looking at the issue the wrong way.

So, I can call VRT_AddDirector to create a director, VCL_DelDirector to delete it, and the VCL code will use VRT_AssignBackend to increase the refcount.

But I don't see what prevents the vmod from creating a director, returning a VCL_BACKEND and then destroying the director before the VCL has a chance to call VRT_AssignBackend.
Conversely, if VRT_AssignBackend has the time to be called, then VCL_DelDirector should break because the refcount won't end up being 0?

I'm not doubting the code works, but I'm struggling to understand how it does. Is there a guarantee I missed that the vmod writer needs to maintain?

[bsdphk]

There are a lot of variable lifetimes which are not as well defined as we'd like.
Strictly speaking it is the vmod's problem to keep the BACKEND variable it returned valid "long enough".

Anybody who wants that BACKEND to stick around, will grab a ref to it and they will do so while the current "job" (ie: request/backend/special-stuff) runs, so the vmod is not allowed to remove the backend again, until that task finished.

Enforcing it would cost a LOT of locking.

[gquintard]

Hum, that is problematic. With the current API, the only approach that I can see working is:

• vmod wants to return VCL_BACKEND
• vmod, takes a ref and stashes the pointer in a PRIV_TASK
• it returns the pointer
• by the time it's time to free the priv, you can count on whoever was interested in the backend having grabbed the their own ref

but that's 2 refs pretty much every time you return a backend/director. That's woefully inefficient, and conceptually, having to take a reference so that another guy has the time to take a reference makes my brain hurt.

Could we have the vmod be in charge of increasing the refcount, and whoever uses the backend be in charge of releasing it?

[nigoroll]

Related: #3949 which also links to nigoroll/libvmod-dynamic@d8eea02 - this is basically the "only approach" @gquintard has described

#3599 (comment) is basically still the summary of what I have to say here. I am still willing to work on the implementation.