Ateneo de Naga University

College of Engineering
Department of ECE and CpE
Navigational Aids [ECTE002]
GREAT CIRCLE CALCULATIONS
Great Circle Distance
1. Haversine Formula
lat
+ cos lat1 cos lat 2
2
c = 2atan2 a, 1 a
a = sin2
d = Rc

sin2

long
2

Where : R is the earths radius R = 6371 km);

Note: angles need to be in radians to pass to trigonometric functions

2. Spherical Law of Cosine

d = R arccos[ sin lat1 sin lat 2 + (cos lat1 )(cos lat 2 )(cos long)]
3. Equirectangular Approximation
x = long 2 long1
y = lat 2 lat1
d = R x2 + y2

cos

lat1 + lat 2
2

Bearing
= atan2 sin long cos lat 2 , cos lat1 sin lat 2 sin lat1 cos lat 2 cos long
Midpoint
Bx = cos lat 2 (cos long)
By = cos lat 2 (sin long)
lat m = atan2 sin lat1 + sin lat 2 ,

cos lat1 + Bx

+ By2

long m = long1 + atan2(By , Bx + cos lat1 )

Destination point given distance and bearing from start point
Given a start point, initial bearing, and distance, this will calculate the destination point and final
bearing travelling along a (shortest distance) great circle arc.
d
d
lat 2 = arcsin sin lat1 cos
+ cos lat1 sin
cos
R
R
d
d
long 2 = long1 + atan2 sin sin
cos lat1 , cos sin lat1 sin lat 2
R
R

John Joshua F. Montaez || Compilation of Formulae || BS ECE- V ||

GE51 || Navigational Aids || 1 | P a g e

Rhumb lines Distance

tan + lat 2
4
= ln

tan 4 + lat1
lat
q=
(cos lat for EW line)

d = R lat 2 + q2 long 2

RHUMB LINES CALCULATIONS

Where: long is less than 180o taking the shortest route

Bearing
= atan2(long, )
Midpoint
lat1 + lat 2
lat m =
2
lat1
f1 = tan +
4
2
lat 2
f2 = tan +
4
2
lat m
fm = tan +
4
2
long 2 long1 ln fm + long1 ln f2 long 2 ln f1
long m =
f
ln 2
f1

Destination point given distance and bearing from start point

Given a start point and a distance d along constant bearing , this will calculate the destination
point. If you maintain a constant bearing along a rhumb line, you will gradually spiral in towards one of
the poles.
d
lat 2 = lat1 + cos
R
d sin
long 2 = long1 +
R q
Intersection of two paths given start points and bearings
This is a rather more complex calculation than most others on this page, but I've been asked for it
a number of times. This comes from Ed Williams aviation formulary.
lat
long
+ cos lat1 cos lat 2 sin2
2
2
sin lat 2 sin lat1 cos 12
a = arccos
sin 12 cos lat1
sin lat1 sin lat 2 cos 12
b = arccos
sin 12 cos lat 2
12 = 2 arcsin sin2

John Joshua F. Montaez || Compilation of Formulae || BS ECE- V ||

GE51 || Navigational Aids || 2 | P a g e

If sin long 2 long1 > 0

12 = a
21 = 2 b
Else
12 = 2 a
21 = b
1 = 13 12 + %2
2 = 21 23 + %2
3 = arccos( cos 1 cos 2 + sin 1 sin 2 cos 12 )
13 = atan2[ sin 12 sin 1 sin 2 , (cos 2 + cos 1 cos 3 )]
lat 3 = arcsin(sin lat1 cos 13 + cos lat1 sin 13 cos 13 )
long13 = atan2[ sin 13 sin 13 cos lat1 , (cos 13 sin lat1 sin lat 3 )]
long 3 = long1 + long13 + %2
Note:
if sin 1 = 0 and sin 2 = 0: infinite solutions
if sin 1 sin 2 < 0: ambiguous solution
this formulation is not always well-conditioned for meridional or equatorial lines

John Joshua F. Montaez || Compilation of Formulae || BS ECE- V ||

GE51 || Navigational Aids || 3 | P a g e