Statistical Quasi-Newton: A New Look at Least Change

• Activity Number: 80
• Type: Contributed
• Date/Time: Monday, August 9, 2004 : 8:30 AM to 10:20 AM
• Sponsor: Section on Statistical Computing
• Author(s): Scott Vander Wiel*+ and Chuanhai Liu
• Companies: Bell Laboratories and Texas A&M University
• Keywords: BFGS ; DFP ; negative Broyden family ; Wishart model

Abstract

A new quasi-Newton method is derived by modeling curvature uncertainty with a Wishart distribution and by following a certain least change principle to determine a Hessian estimate that preserves accuracy across iterations. The new update is in the Broyden class but uses a negative parameter, outside the convex range usually regarded as the "safe-zone'' for Broyden updates. Although Newton steps based on this update tend to be too long, optimal step sizes can be estimated from the Wishart model. In numerical comparisons to BFGS, the new algorithm converges with about 20% fewer iterations and gradient evaluations and about 10% fewer function evaluations on a suite of standard test functions. Our statistical framework provides a simple way to understand differences among various Broyden updates such as BFGS and DFP and shows that these methods do not preserve Hessian accuracy. In fact, BFGS, DFP and all other updates with non-negative Broyden parameters tend to inflate Hessian estimates. Numerical results on three new test functions validate these conclusions.