Emmanuel Nwaeze (IJAEST) INTERNATIONAL JOURNAL OF ADVANCED ENGINEERING SCIENCES AND TECHNOLOGIES

Vol No. 2, Issue No. 2, 181 - 186

A MODIFIED NONLINEAR CONJUGATE

GRADIENT METHOD FOR SOLVING
NONLINEAR REGRESSION PROBLEMS

 k  min f ( x k  d k ), (3)
Emmanuel Nwaeze 
Department of Mathematics,
University of Ilorin, Ilorin,
Nigeria. d k is a vector in the form
e-mail: nwaezeema@yahoo.com
  g k 1 , k 0
d k 1   (4)
 g k 1   k d k , k  1

T
Abstract—In regression analysis, scientists face the
problem of constructing a nonlinear mathematical function that in which g k  g ( x k )  f ( x k ) and  k is a parameter of
has the best fit to a series of experimental data points. Thus in the CGM, i.e. different s determine different CGMs. For
this paper, we present a Modified Nonlinear Conjugate Gradient
instance, the following are formulae for some popular CGMs:
ES
Method (MNCGM) for solving the nonlinear regression
Problems without linearizing the form of the function. Testing
the MNCGM on some problems (including polynomial regression
1 
g kT ( g k  g k 1 )
PkT1 ( g k  g k 1 )
[5]

problems with many parameters) confirms the possibility of 2

constructing a function that has the best fit to given experimental
gk
 
2
2
[3]
data points. The results obtained also show that the method is g k 1
amenable to nonlinear regression analysis and computer
g kT ( g k  g k 1 )
automation.   3
2
[7]
g k 1
A
2
I. INTRODUCTION g k 1
 
4
[1]
The Modified Nonlinear Conjugate Gradient Method T
d k yk
(MNCGM) seeks to minimize an objective function f
Where y k  g k 1  g k and || . || denotes the Euclidean
min f (x), (1)
IJ

norm.
where x  R,
N
R N is an N-dimensional Euclidean space
The global convergence properties of the above
and f is n – times differentiable. g (x) is the gradient vector. variants of CGM have already been established by many
A conjugate Gradient Method (CGM) for solving (1) uses an authors including Powell[10], Dai and Yuan[1] and
iterative scheme: Zoutendijk[8].The cubic interpolation technique is among the
x k 1  x k   k d k , k  0, 1, 2,... (2) most widely applicable and useful procedure for establishing
convergence results of CGM. This technique (cubic
where x0  R N is an initial point,  k , the step size at
interpolation) minimizes the one-dimensional function
iteration k, is defined by

ISSN: 2230-7818 @ 2011 http://www.ijaest.iserp.org. All rights Reserved. Page 181

 Emmanuel Nwaeze (IJAEST) INTERNATIONAL JOURNAL OF ADVANCED ENGINEERING SCIENCES AND TECHNOLOGIES
Vol No. 2, Issue No. 2, 181 - 186

 ( )  f ( x k  d k ) (5)  and k are step lengths of the procedure.

through an iterative procedure: Hence the value of  such that

  ( k )  p 2  p1 
 k 1   k  ( k   k 1 ) , (6) f ( x k   k d k )  optimize f ( x k  d k ), (7)
 ( k )   ( k 1 )  2 p 2   0

 ( k 1 )   ( k ) becomes available. Bunday and Garside[2] and others used

p1   ( k 1 )   ( k )  3 , cubic interpolation technique extensively in solving nonlinear
 k 1   k
optimization problems. Dai and Yuan used Wolfe conditions
 
1
2
p 2  sign( k   k 1 ) p1   ( k 1 ) ( k ) , 2 to establish the global convergence of a nonlinear conjugate
gradient method as stated in the algorithm below.

2 1. The Least-Squares Polynomial of degree m:

g k 1
ALGORITHM I: ( 
4
 T
, y k  g k 1  g k ) Pn ( x)  a0  a1 x  a 2 x 2  . . .  a n x n , (9)
d k yk
where a 0 , a1 , ..., a n are constants to be determined

T
Step 1 Given x1   v , d 1   g 1 , k  1, if g1  0 then stop.
Step 2 Compute an  k  0 that satisfies Wolfe conditions . and n  m 1.
Step 3 Let x k 1  x k   k d k . If g k 1  0 then stop.
2. The Least-Squares Exponential function
Step 4 Compute  k by  4 and generate d k 1 by (1.4),
k  k  1, go to step 2. Pn ( x)  be ax (10)

function (model) of the form:

ES
Nonlinear regression involves a general mathematical
Pn ( x)  bx a
where a and b are constants to be determined.
Pn ( x)  f ( x1 , x2 , ...,xm , a0 , a1 , a 2 , ..., a n ) (8)
3. The Least-Squares Rational function of degree n + r:
where ( a 0 , a1 , a 2 , ..., a n ) are regression parameters to a
a 0  a1 x  a 2 x 2  . . .  a n x n
set of m tabulated values of x1 , x 2 , ...,x m (independent Pn ( x)  , (11)
b0  b1 x  b2 x 2  .. .  br x r
A
variables) versus P (dependent variable) and f is nonlinear
where a 0 , a1 , ..., a n ; b0 , b1 , ..., br are constants
with respect to the regression parameters. In regression
to be determined.
analysis, we find the best-fitting function to a given set of
points by minimizing the sum of the squares of the offsets (the
4. The Multivariable Nonlinear function
residuals) of the points from the function. The sum of the
IJ

a 0  a1 x1  a 2 x 2  . . .  a n x n
squares of the offsets is used instead of the offset absolute Pn ( x)  (12)
b0  b1 x1  b2 x 2  .. .  br x r
values because this allows the residuals to be treated as a
where a 0 , a1 , ..., a n ; b0 , b1 , ..., br are constants
continuous differentiable quantity. NCGM is then used to
minimize the sum of the squares of the offsets. The best-fit to be determined.
function that approximates a set of data points: Using any of the model functions above, the Modified
Nonlinear Conjugate Gradient Method seeks to minimize the
{ ( xi , yi ) | i  1, 2, ..., m } may take one of the following offset (error function)

forms, depending on the relationship between the x i and y i

values.

ISSN: 2230-7818 @ 2011 http://www.ijaest.iserp.org. All rights Reserved. Page 182

 Emmanuel Nwaeze (IJAEST) INTERNATIONAL JOURNAL OF ADVANCED ENGINEERING SCIENCES AND TECHNOLOGIES
Vol No. 2, Issue No. 2, 181 - 186

m
F (a)    y i  Pn ( x i )  or
2
PROBLEM I: Construct the least squares polynomial of
i 1
m degree 3 for the data below.
F (a, b)    y i  Pn ( xi ) 
2
(13)
i 1 xi 0 1 2 3 4
over a, b where a   and b   .
n r
yi 0 12 156 174 408

II. MODIFIED NONLINEAR CONJUGATE GRADIENT f ( x)  10 x  5x 2  7 x 3 , n = 3, m = 5)

METHOD

The Modified Nonlinear Conjugate Gradient Method PROBLEM II: Construct the least squares polynomial of
is Algorithm (I) driven by cubic interpolation line search degree 10 for the data below.
procedure (10). This method inherits Dai and Yuan[1]
convergence results of Algorithm (I) since cubic interpolation xi 0 0.1 0.2 0.3 0.4 0.5
technique satisfies Wolfe conditions implicitly. The
Algorithm is as follows. yi 5 5.266 5.484 5.676 5.872 6.138

T
ALGORITHM II
i. Input initial values x0 and d 0   g 0 . 0.6 0.7 0.8 0.9 1 1.1
ii. Repeat:
6.635 7.745 10.292 15.962 28 52.337
a. Find step length  k such that

b.
by (10)
Compute new point:
ES
f ( x k   k d k )  min f ( x k  d k ) ,
 0 f ( x)  5  3 x  4 x 2  7 x 3  9 x 4  6 x 5
 8 x 6  3x 7  4 x 8  10 x 9  x 10
x k 1  x k   k d k (n = 10, m = 12)
c. Update search direction:
Dk 1   g k 1   k d k , PROBLEM III: Construct the least squares exponential
function of the form (10) for the data below.
2 xi 1 1.25 1.5 1.75 2
A
g k 1
k  5.1 5.79 6.53 7.45 8.46
T
d k yk yi
y k  g k 1  g k n = 2, m = 5
d. Check for optimality of g:
Terminate iteration at step m when g m is so small that x m PROBLEM IV: Construct the least squares nonlinear function
IJ

of the form (1.12) for the data below (n = 10, m = 12).

is an acceptable estimate of the optimal point x * of f . If not
0 1 2 3 4
optimal set k  k  1. x1
9.5 8.5 7.5 6.5 5.5
x2
III. NUMERICAL EXPERIMENTS yi -4.5 0.234 0.079 0.023 -0.013

5 6 7 8 9
The following constitute the test problems for the
CGM on which Algorithm II was implemented in Visual Basic 4.5 3.5 2.5 1.5 0.5

6.0: -0.046 -0.08 -0.143 -0.275 -0.982

ISSN: 2230-7818 @ 2011 http://www.ijaest.iserp.org. All rights Reserved. Page 183

 Emmanuel Nwaeze (IJAEST) INTERNATIONAL JOURNAL OF ADVANCED ENGINEERING SCIENCES AND TECHNOLOGIES
Vol No. 2, Issue No. 2, 181 - 186

SOLUTION TO PROBLEM I

T
Time taken = 0.82 seconds.
ES
P3 ( x)  0.000096  9.998656 x  4.999029 x 2  6.999837 x3
P3 (3)  174.0004
Time taken = 0.45 seconds.

P3 (5)  799.997298 P10 ( x)  5.000068  2.959044 x  3.371729 x 2

 3.597307 x 3  1.12348 x 4  0.105908 x 5 
3.356655 x 6  5.336969 x 7  5.086751x 8 
SOLUTION TO PROBLEM II 3.829213 x 9  3.434991x 10
P10 (2)  7683.00162
A
P10 (0.35)  5.77066
P10 (1)  27.99988

SOLUTION TO PROBLEM III

IJ

ISSN: 2230-7818 @ 2011 http://www.ijaest.iserp.org. All rights Reserved. Page 184

 Emmanuel Nwaeze (IJAEST) INTERNATIONAL JOURNAL OF ADVANCED ENGINEERING SCIENCES AND TECHNOLOGIES
Vol No. 2, Issue No. 2, 181 - 186

 0.119524  0.090346 x1  0.075362 x 2

P( x) 
 0.132537  0.231999 x1 x 2
P(6, 3.6)  0.08000128
P(5.5, 4)  0.06335777

IV. REMARK ON NUMERICAL RESULTS

A cursory look at the tables reveals that results
Time taken = 1.06 seconds.
obtained with the new conjugate gradient method is in close
P( x)  3.06697e 0.506879x
agreement with the given data. Attainment of global optimum
P(1.5)  6.560118
is possible with the new method.
P(3)  14.031814

V. CONCLUSION
SOLUTION TO PROBLEM IV

T
Herein, we present a Modified Nonlinear Conjugate
Gradient Method (MNCGM) for solving nonlinear regression
problems to scientists and engineers. Testing the MNCGM on
some problems (including polynomial regression problems
ES with many parameters) confirms the possibility of obtaining
the global optimum of the objective function with high
accuracy. The results obtained also show that the method is
amenable to nonlinear regression analysis and computer
automation.

REFERENCES
A
[1] Y.H. Dai and Y. Yuan : “A nonlinear
conjugate gradient method with a strong
global convergence property”. SIAM J.
Optim. 10, 177-182 (1999)
[2] B. D. Bunday and G. R. Garside :
IJ

“Optimization methods in Pascal”. Printed

and bound in Britain by Edward Arnold ltd,
London (1987)
[3] R. Fletcher and G. M. Reeves : “Function
minimization by conjugate gradients”.
Computer J. 7, 149-154 (1964)
[4] E. Polak, “Optimization: Algorithms and
Consistent Approximations”, vol. 124 of
Applied Mathematical Sciences, Springer,
New York, NY, USA, 1997.
Time taken = 0.41 seconds.

ISSN: 2230-7818 @ 2011 http://www.ijaest.iserp.org. All rights Reserved. Page 185

 Emmanuel Nwaeze (IJAEST) INTERNATIONAL JOURNAL OF ADVANCED ENGINEERING SCIENCES AND TECHNOLOGIES
Vol No. 2, Issue No. 2, 181 - 186

[5] M. Hestenes, and E. Stiefel: “Method of

conjugate gradients for solving linear
systems”. J. Res. Nat. Bur. Standards 49,
409-436 (1952)
[6] M. Ledvij “Curve Fitting Made Easy”.
Industrial Physicist 9, 24-27, Apr./May
2003.

[7] E. Polak and G. Ribiére: Note sur la

convergence de directions conjugées. Rev.
Francaise Informat Recherche Opertionelle,
3e Année, 16, 35-43 (1969)
[8] G. Zoutendijk “Nonlinear programming,
computational methods, in Integer and
Nonlinear Programming”, J. Abadie, Ed.,

T
pp. 37–86, North-Holland, Amsterdam,
The Netherlands, 1970.
[9] C. Lawson and R. Hanson “Solving Least
Squares Problems”. Englewood Cliffs, NJ:
Prentice-Hall, 1974.
[10]
ES
M. J. D. Powell “Restart procedures for the
conjugate gradient method”, Mathematical
Programming, vol. 12, no. 2, pp. 241–
254, 1977.

[11] P. Lancaster and K. Šalkauskas “Curve and

Surface Fitting: An Introduction”.

London: Academic Press, 1986.
A
IJ

ISSN: 2230-7818 @ 2011 http://www.ijaest.iserp.org. All rights Reserved. Page 186