[R] optim

[Sundar Dorai-Raj]

nwew wrote:
> Dear R-helpers,
> 
> The function optim implements algorithms that I would like to use.
> 
> 
> I have function implemented in R, which given the parameters of which 
> minimization is to take place returns a scalar as well as the gradient.
> 
> Unfortunately optim requires two function _fn_ and _gr_ where fn returns the 
> function value and gr the gradient. Splitting my function in two functions 
> would be easy, however I am wondering if evaluating both is not doubling the 
> the very high computational costs. Most of the computational intensive 
> operations are identical if computing the function value and gradient.
> 
> Question: is there a way to tweek optim that only one function evaluation is 
> necessary? Are ther other implementations of the algorithm, which do assume 
> that the function to be minimized returns the function value and the gradient 
> as well?
> 
> Thanks
> Eryk.
> 

Hi, Eryk,

?optim does not *require* the "gr" argument. If one is not supplied, 
numerical gradients are used for which optim evaluates "fn" twice. 
However, in many cases analytical gradients can both improve numerical 
accuracy and computational speed.

Trivial example:

fn <- function(beta) {
   sum((y - x %*% beta)^2)
}

gr <- function(beta) {
   colSums(-2 * x * drop((y - x %*% beta)))
}

set.seed(1)
n <- 10000
p <- 5
g <- factor(rep(LETTERS[1:p], each = n/p))
x <- model.matrix(~g)
beta <- rnorm(p)
y <- drop(x %*% beta + rnorm(n))
start <- rep(0, p)

system.time(f1 <- optim(start, fn, gr, hessian = TRUE))
# [1] 0.47 0.00 0.48   NA   NA

system.time(f2 <- optim(start, fn, hessian = TRUE))
# [1] 0.54 0.00 0.53   NA   NA

f1$par
#[1] -0.6408643  0.2128109 -0.8378791  1.5983054  0.3366216

f2$par
#[1] -0.6408643  0.2128109 -0.8378791  1.5983054  0.3366216

f1$hessian
#      [,1] [,2] [,3] [,4] [,5]
#[1,] 20000 4000 4000 4000 4000
#[2,]  4000 4000    0    0    0
#[3,]  4000    0 4000    0    0
#[4,]  4000    0    0 4000    0
#[5,]  4000    0    0    0 4000

f2$hessian
#      [,1]         [,2] [,3]         [,4] [,5]
#[1,] 20000 4.000000e+03 4000 4.000000e+03 4000
#[2,]  4000 4.000000e+03    0 2.273737e-07    0
#[3,]  4000 0.000000e+00 4000 0.000000e+00    0
#[4,]  4000 2.273737e-07    0 4.000000e+03    0
#[5,]  4000 0.000000e+00    0 0.000000e+00 4000
#

Does this answer your question? Furthermore, have you read the chapter 
in MASS that discusses optim?

HTH,

--sundar