[Rd] S3 lookup rules changed in R 3.6.1

[Konrad Rudolph]

Oh, I had missed that that code path is now enabled by default. It’s worth
noting that the commented-out test in that commit also still succeeds if
invoked via `getS3method`. So at the very least there’s now an
inconsistency in the lookup performed by R internally (via `UseMethod`) and
`getS3method`, which is probably unintentional.

I see how the change is beneficial by preventing surprising behaviour in a
corner case. Unfortunately it also breaks at least one published package
[1], and if I understand correctly it no longer conforms to the documented
behaviour (quoted in my initial message), which even explicitly mentions
non-namespace environments.

[1] https://github.com/klmr/modules/issues/147

On Wed, Oct 9, 2019 at 11:23 PM Duncan Murdoch <murdoch.duncan using gmail.com>
wrote:

> On 09/10/2019 3:22 p.m., Konrad Rudolph wrote:
> > tl;dr: S3 lookup no longer works in custom non-namespace environments as
> of
> > R 3.6.1. Is this a bug?
>
> I don't know whether this was intentional or not, but a binary search
> through the svn commits finds that the errors started in this one:
>
> ------------------------------------------------------------------------
> r75127 | hornik | 2018-08-13 09:58:47 -0400 (Mon, 13 Aug 2018) | 2 lines
> Changed paths:
>     M /trunk/src/main/objects.c
>     M /trunk/tests/reg-tests-1a.R
>
> Have S3 methods lookup by default look for the S3 registry in the topenv
> of the generic.
> ------------------------------------------------------------------------
>
> Duncan Murdoch
>
> >
> > I am implementing S3 dispatch for generic methods in environments that
> are
> > not
> > packages. I am trying to emulate the R package namespace mechanism by
> > having a
> > “namespace” environment that defines generics and methods, but only
> exposes
> > the
> > generics themselves, not the methods.
> >
> > To make S3 lookup work when using the generics, I am using
> > `registerS3method`.
> > While this method itself has no extensive documentation, the
> documentation
> > of
> > `UseMethod` contains this relevant passage:
> >
> >> Namespaces can register methods for generic functions. To support this,
> >> ‘UseMethod’ and ‘NextMethod’ search for methods in two places: in the
> >> environment in which the generic function is called, and in the
> > registration
> >> data base for the environment in which the generic is defined
> (typically a
> >> namespace). So methods for a generic function need to be available in
> the
> >> environment of the call to the generic, or they must be registered. (It
> > does
> >> not matter whether they are visible in the environment in which the
> > generic is
> >> defined.) As from R 3.5.0, the registration data base is searched after
> > the
> >> top level environment (see ‘topenv’) of the calling environment (but
> > before
> >> the parents of the top level environment).
> >
> > This used to work but it stopped working in R 3.6.1 and I cannot figure
> out
> > (a)
> > why, and (b) how to fix it. Unfortunately I am unable to find the
> relevant
> > information by reading the R source code, even when “diff”ing what seem
> to
> > be
> > the only even remotely relevant changes [1].
> >
> > The R NEWS merely list the following change for R 3.6.0:
> >
> >> * S3method() directives in ‘NAMESPACE’ can now also be used to perform
> > delayed
> >>    S3 method registration.
> >> […]
> >> * Method dispatch uses more relevant environments when looking up class
> >>    definitions.
> >
> > Unfortunately it is not clear to me what exactly this means.
> >
> > Here’s a minimal example code that works under R 3.5.3 but breaks under
> > R 3.6.1
> > (I don’t know about 3.6.0).
> >
> > ```
> > # Define “package namespace”:
> > ns = new.env(parent = .BaseNamespaceEnv)
> > local(envir = ns, {
> >      test = function (x) UseMethod('test')
> >      test.default = function (x) message('test.default')
> >      test.foo = function (x) message('test.foo')
> >
> >      .__S3MethodsTable__. = new.env(parent = .BaseNamespaceEnv)
> >      .__S3MethodsTable__.$test.default = test.default
> >      .__S3MethodsTable__.$test.foo = test.foo
> >
> >      # Or, equivalently:
> >      # registerS3method('test', 'default', test.default)
> >      # registerS3method('test', 'foo', test.foo)
> > })
> >
> > # Expose generic publicly:
> > test = ns$test
> >
> > # Usage:
> > test(1)
> > test(structure(1, class = 'foo'))
> > ```
> >
> > Output in R up to 3.5.3:
> >
> > ```
> > test.default
> > test.foo
> > ```
> >
> > Output in R 3.6.1:
> >
> > ```
> > Error in UseMethod("test") :
> >    no applicable method for 'test' applied to an object of class
> > "c('double', 'numeric')"
> > ```
> >
> > It’s worth noting that the output of `.S3methods` is the same for all R
> > versions, and from my understanding of its output, this *should* indicate
> > that
> > S3 lookup should behave identically, too. Furthermore, lookup via
> > `getS3method`
> > succeeds in all R versions, and (again, in my understanding) the logic of
> > this
> > function should be identical to the logic of R’s internal S3 dispatch:
> >
> > ```
> > getS3method('test', 'default')(1)
> > getS3method('test', 'foo')(1)
> > ```
> >
> > Conversely, specialising an existing generic from a loaded package works.
> > E.g.:
> >
> > ```
> > local(envir = ns, {
> >      print.foo = function (x) message('print.foo')
> >      registerS3method('print', 'foo', print.foo)
> > })
> >
> > print(structure(1, class = 'foo'))
> > ```
> >
> > This prints “print.foo” in all R versions as expected.
> >
> > So my question is: Why do the `test(…)` calls in R 3.6.1 no longer
> trigger
> > S3
> > method lookup in the generic function’s environment? Is this behaviour by
> > design
> > or is it a bug? If it’s by design, why does `getS3method` still use the
> old
> > behaviour? And, most importantly, how can I fix my definition of `ns` to
> > make
> > S3 dispatch for non-exposed methods work again?
> >
> > … actually I just found a workaround:
> >
> > ```
> > ns$.packageName = 'not important'
> > ```
> >
> > This marks `ns` as a package namespace. To me, the documentation seems to
> > imply
> > that this shouldn’t be necessary (and it previously wasn’t). Furthermore,
> > the
> > code for `registerS3method` explicitly supports non-package namespace
> > environments. Unfortunately this workaround is not satisfactory because
> > pretending that the environment is a package namespace, when it really
> > isn’t,
> > might break other things.
> >
> > [1] See r75273; there’s also r74625, which changes the actual lookup
> > mechanism
> >      used by `UseMethod`, but that seems even less relevant, because it
> is
> >      disabled unless a specific environment variable is set.
> >
>
>

-- 
Konrad Rudolph

	[[alternative HTML version deleted]]