[Rd] Finalizer execution order question

[Jeroen Ooms]

Given an externalptr object 'pool' which protects an R object 'prot':

  # SEXP prot = (a dynamically updated list with handles)
  SEXP pool =  R_MakeExternalPtr(p, R_NilValue, prot);
  R_RegisterCFinalizerEx(pool, fin_pool, TRUE);

WRE explains that 'prot' remains in existence as long as 'pool' is
around. Does this also mean 'prot' still exists when the finalizer of
'pool' gets executed?

Long story: I am running into an issue with the next generation of the
curl package which uses dynamic pools of (many) http request handles.
Both the pool and the handles are externalptr objects with finalizers
that clean up after themselves.

When the pool goes out of scope, its finalizer has to loop over
pending handles to release them. However, to my surprise, the
finalizers of the handles get executed *before* the finalizer of
'pool'. Therefore the handles have destroyed themselves before I can
properly take hem out of the pool. This is exactly what I was trying
to prevent by putting the handles in a 'prot' list.

I wrote a simple package [1] that illustrates this issue. To run the example:

  devtools::install_github("jeroenooms/test")
  library(test)
  pool <- make_pool()
  show_handles(pool)
  rm(pool)
  gc()

What this example is supposed to illustrate is that even though 'prot'
gets protected from GC while the externalptr is around, the finalizers
in 'prot' have already executed when the externalptr gets finalized.

What is even stranger (to me) is that the SEXPs in 'prot' still seem
in tact during the finalizer of externalptr (ASAN is not giving
use-after-free warnings either). It's just that their finalizers have
already executed. So that leaves the pool finalizer in the odd
position of seeing only the post-mortem SEXP contents of the handles.
This does provide a workaround in my case, but can we rely on this?

So my question: is this expected behavior? What would be an
alternative approach to ensure that handles stay protected until the
pool has been cleaned and finalized (rather than running all
finalizers in the order they were registered)?


[1] https://github.com/jeroenooms/test