Professional Documents
Culture Documents
Automation in Construction
journal homepage: www.elsevier.com/locate/autcon
a r t i c l e i n f o a b s t r a c t
Article history: Large mining excavators impose numerous challenges to the operator during a dig. A semi-automated or fully
Accepted 6 November 2012 automated machine that could assist the operator during excavation would positively affect efciency, ore se-
Available online 30 December 2012 lectivity, and reduce machine wear at the same time. Besides, the dependence on operator skills and atten-
tiveness will be reduced. In order to realize any control on an excavator, it is prudent to know the dig
Keywords:
patterns followed by the operator during a real time dig. This paper aims to classify the dig by analyzing
Mining
Excavator
the actual task performed by the operator. The trajectories are classied based on the location and angle of
Trajectory the bucket. Both 2-dimensional and 3-dimensional analyses of the bucket trajectory were performed on a
Bucket EX 3500 mining excavator at the Newmont open pit mine using Arial Performance Analysis System (APAS).
2012 Elsevier B.V. All rights reserved.
0926-5805/$ see front matter 2012 Elsevier B.V. All rights reserved.
http://dx.doi.org/10.1016/j.autcon.2012.11.006
R. Tiwari et al. / Automation in Construction 31 (2013) 128139 129
The automation of an excavator can be divided into different cat- Cartesian coordinate system. A time delay control with switching action
egories. The rst category consists of tele-operation, in which the op- with integral sliding surface was discussed by Lee and Chang [15]. The
erator is physically removed from the machine, but is still in charge of controller was designed for a straight-line motion of a 21-ton heavy-
controlling the machine to follow the required trajectory. Thus the duty excavator. Swing velocity tracking control was achieved through
operator controls the machine from a remote place via hand control- adaptive robust control by Yao et al. [16]. Non-linearity in the system
lers (joysticks, or steering wheel). However, if certain phases of the was considered in the design and analysis of the controller through
operation are automated then it is also possible that the operator simulation.
may control more than one machine [10]. In order to facilitate the development of a set of useful software
Supervisory control [11] is the second category of machine automa- kinematics for mining equipment, it was necessary to evaluate and
tion. Here the operator performs main decision making tasks while the classify actual machine trajectories used during mine excavation.
robot is used for repetitive operations. Plonecki et al. [5] discussed auto- The following questions are raised in order to derive suitable param-
mation of heavy machines to assist the machine operator. They separat- eters for the software-dened kinematics for machine automation:
ed the tasks between robot and operator with the operator having to
play the main decision and control role. Pre-programmed, predictable 1. What are the motion characteristics of the bucket during digging
operations were utilized under standard working conditions for realiza- and loading under different conditions?
tion of tool motions that otherwise were difcult to control manually or 2. What is the similarity between consecutive digging and loading
to correct the trajectory dened by the operator for optimal control. On cycles?
similar lines, Alami et al. [12] proposed control architecture for mobile 3. What types of adjustments, to the machine motion, are needed
robots, which reportedly integrated human supervisory direction from one cycle to another?
into an automated robotic machine. According to their architecture, 4. What internal relationships may exist between motion parameters
a human supervisor observed the motion and position of an end ef- such as bucket position, bucket angle, and movement velocities?
fector, and provided mission guidance to a trajectory planner. The
trajectory planner selected a software control protocol correspond- For answers to these questions, it was necessary to evaluate and
ing with the mission guidance instruction and provided power and classify the semi-repetitive machine motion patterns observed during
motion control to the machine actuators (e.g. hydraulic pistons and typical mining operations. Of particular interest is the trajectory of the
cylinders), to execute the mission guidance instruction. The software bucket and its relationship to the coordination of multiple machine
control protocol iterated every step needed, including every actuator links. Trajectory evaluation was accomplished using a multi-camera
adjustment, for the end effector to execute the desired trajectory. computer vision method aligned with a model based image analysis
Complete automation is the third category of automation. Here system [17].
the operator is eliminated from the control loop altogether. An adap-
tive control strategy with real-time self-tuning ability was studied by 3. Mine data collection
Chiang et al. [13]. A new hydraulic valve system was mounted on a
large excavator, and a PID controller was utilized for valve control. Mine data were collected in cooperation with the Lone Tree Mine,
State space control along with MRAS (model reference adaptive sys- Nevada. In order to develop effective video data collection methods, a
tem) was employed for the desired path tracking. Lee et al. [14] doc- number of video samples were collected and transferred to University
umented the control of a heavy-duty excavator under working speed of Nevada Reno (UNR) for processing. As illustrated in Fig. 2, three
conditions. A time delay controller was designed based on a dynamic video cameras were used to record the motion of several pieces of
model of the machine to control the movement of the bucket tip in a mining equipment, during normal operation at an open pit mine.
a) b)
d) c)
Fig. 1. Loading cycle consisting of (a) bucket placement at the digging front, (b) lling the bucket, (c) hoisting the load over the loading truck and (d) dumping the load.
130 R. Tiwari et al. / Automation in Construction 31 (2013) 128139
Fig. 3. Mine data video from three cameras around shovel EX 3500.
R. Tiwari et al. / Automation in Construction 31 (2013) 128139 131
Fig. 4. (a) Video clip frame of hand-held marker during laboratory experiment and (b) 2D trajectory of hand-moved marker-tracking point (tracked using APAS in image
coordinate).
Fig. 5. APAS marker tracking test on Bobcat 225 machine showing initial and nal machine positions with marker trajectory overlay in image coordinate (inset: for close-up of
boxed region).
was subsequently increased to about 4 in. to improve tracking. Fig. 5 marker trajectories (red) are shown in the gure. These trajectories
shows the preliminary results of one marker-tracking test using 4-inch closely followed the motion of the two markers on the excavator arm,
markers on the Bobcat 225 excavator. The last frame of the video is as the arm was lifted up.
shown superimposed over the rst frame, so that the starting and end- Fig. 5 inset shows a close-up view of marker B along with the prelim-
ing positions of the machine are visible on the same plot. Two selected inary tracking data for this marker point. The error between the plotted
Fig. 6. Control points (left) and tracking point (right) along with the xed point (inside square).
132 R. Tiwari et al. / Automation in Construction 31 (2013) 128139
Fig. 7. Orthographic view of the tracking point (left) and nal tracked trajectory of the markers (right).
a) b)
0 0
100 100
200 200
300 300
400 400
0 100 200 300 400 500 600 700 0 100 200 300 400 500 600 700
Fig. 8. Marker point trajectories plotted in camera coordinates on initial video frame; (a) Type 2-excavation example, (b) Type 4-excavation example. The axes are in image coor-
dinate and can be converted to actual trajectory measurement using the measurements in Appendix A.
marker trajectory and the actual center of the marker is attributed to points were selected so that they cover a large part of the frame in
the fact that pixels near the edge of the black marker appear in the order to maximize the accuracy of the APAS results. A variety of
video to be darker than those in its center. This preliminary analysis video clips were selected for initial processing, spanning various dig-
did not account for errors due to slight variations in apparent marker ging conditions and digging patterns. This representative sample
color. However, additional image processing techniques are expected was evaluated using the APAS software. The tracked markers were
to eliminate this problem. viewed in one of the APAS module (see Fig. 7).
Trajectories were processed by using APAS for marker tracking and
then feeding the output 2D marker trajectories (in image coordinates),
3.1.2. 2D marker tracking into a newly developed multi-frame version of the model-based tting
The APAS software was used to track the markers on the shovels. Ini- algorithm [17]. The output of the multi-frame tting algorithm was a 3D
tial analysis was carried out using one camera view for each video clip bucket trajectory, with its associated bucket angle curve. The trajectory
with ve control points and four motion tracking marker points. Fig. 6 was plotted over the initial frame of the video for visual conrmation of
shows the 4 points used for tracking and one xed point in the back- the t results (Fig. 8).
ground. MATLAB was used to provide the coordinates of the initial
control points (from the rst frame of the video) for APAS in image
coordinate. Since the y-axis of MATLAB and APAS are opposite to 3.1.2.1. Trajectory classication. Analysis of the bucket digging trajec-
each other, it was required to change the sign of the y coordinates be- tories observed in the videos for the Hitachi EX3500 front shovels
fore feeding it to APAS. showed four major classes of bucket trajectories. These classications
The initial set of marker points was selected to facilitate a focused were based on the initial bucket angle, and on whether the bucket
evaluation of the bucket trajectories, since bucket trajectory data are was initially placed at the base of the pile, or midway up the pile. Ex-
likely the most important for development of software-based ma- amples of each type of trajectory, along with the associated bucket
chine kinematics which improves digging efciency. The control angle (see denition in Fig. 9) plotted as a function of horizontal
bucket displacement, are shown in Figs. 1013. The initial position
of the bucket corresponds to zero displacement of the bucket in the
plots. Table 1 summarizes the four main trajectory types observed
in the videos.
3.1.2.1.1. Type 1 trajectory. This trajectory starts from the base of
the pile with a negative bucket angle, as shown in Fig. 10. The initial
position of the bucket is close to the machine body. The operator
rst rolls the bucket to a larger (but still negative) bucket angle, ei-
ther immediately, or with a slight forward bucket motion. After
x
rolling the bucket, the operator follows a trajectory similar to the
Type 2 classication below. The distinctive feature of the Type 1
class is a sharp increase in bucket angle near the beginning of the tra-
Fig. 9. Denition of bucket angle with respect to the horizontal. jectory, whereas the Type 2 bucket angle increases smoothly.
R. Tiwari et al. / Automation in Construction 31 (2013) 128139 133
Fig. 10. Examples of Type 1 trajectory shapes (top) and corresponding bucket angles Fig. 12. Examples of Type 3 trajectory shapes (top) and corresponding bucket angles
(bottom). (bottom).
Fig. 11. Examples of Type 2 trajectory shapes (top) and corresponding bucket angles Fig. 13. Examples of Type 4 trajectory shapes (top) and corresponding bucket angles
(bottom). (bottom).
134 R. Tiwari et al. / Automation in Construction 31 (2013) 128139
Fig. 14. Three-camera view showing control points for 3D analysis of a loading cycle using APAS.
3.1.2.1.3. Type 3 trajectory. The bucket starts at the base of the pile
and is initially moved nearly horizontally as in Type 2. However, the
Type 3 trajectory rapidly curves upward as the bucket is quickly lled,
as shown in Fig. 12. The main difference from Type 2 is that the initial
bucket angle is nearly zero, and increases dramatically as the maxi-
mum bucket capacity is approached.
3.1.2.1.2. Type 2 trajectory. The bucket starts at the base of the pile
and is rst pushed forward nearly horizontally (effectively smoothing
the previously excavated area just in front of the pile). The bucket
edge trajectory then gradually curves upward, and becomes approxi-
mately parallel to the slope face. The bucket is then lifted out of the
pile and (almost simultaneously) begins to swing toward a nearby
haul truck where it is to be emptied. Type 2 trajectories are character-
ized by an initial bucket angle, which is signicantly negative. The
bucket angle increases approximately linearly, but remains negative
for up to 50% of the arc length of the trajectory (Fig. 11).
Table 1
Summary of observed excavation trajectory types.
Fig. 17. Sample bucket trajectories calculated using APAS viewed from the side of the Fig. 19. Block diagram for Machine Simulation.
shovel.
3.1.2.1.4. Type 4 trajectory. The bucket starts midway up the pile, the presence of insurmountable obstacles, such as large boulders
and moves in an approximately linear trajectory, which is nearly par- hidden beneath the pile, or due to misjudgment of bucket ll
allel to the slope face, as shown in Fig. 13. The bucket is then lifted rate. Operator redirection can take a variety of forms, including
and moved toward the haul truck as described in Type 2. The bucket sudden lifting or lowering of the bucket, large, sudden changes
angle begins near zero, and increases semi-linearly for most of the in bucket angle, or even temporary reversal of bucket motion.
trajectory with a somewhat sharper increase typically near the end. 3) Trajectory scaling proportional or non-proportional changes in
Four major classes of front shovel trajectories were observed in this overall trajectory dimensions. Proportional scaling does not affect
research. While it is possible that additional types may be found in the the aspect ratio of the trajectory, but changes only its overall size
future, it is believed that the majority of digging trajectories can be to match the size of the pile. Non-proportional scaling allows for
placed into a relatively small number of such classes. Each class can be variations in the slope angle of the pile, by allowing the trajectory
described in terms of a parameterized general trajectory shape, and a to be lengthened in only one direction.
set of possible deviations from the general shape. Distinguishing be-
tween the general shape of a trajectory, and the deviations from that It is assumed that the last two categories of deviation are best
general shape, facilitates the integration of human and robotic control. controlled by the operator, while the general trajectory can be easily
Computerized robotic control is best suited for performance of the gen- performed robotically through the use of pre-programmed, parame-
eral task, while the human operator is much more capable of evaluating terized trajectory shapes, realized by a software-controlled machine
the need for some types of deviations, due to rapidly changing environ- kinematics.
mental factors such as variations in diggability. In addition, it is assumed that there must be a well-dened rela-
Several kinds of deviations that may be commonly observed: tionship between the direction of general trajectory motion and the
bucket angle. Clearly, such a relationship must exist to facilitate ef-
1) Reactive path variations small, transient changes in bucket direc- cient loading of the bucket, and indeed, the data suggest that there
tion due to resistance produced by non-uniform density of mate- is such a relationship. Both the existence of this relationship and its
rial. These are produced by reaction forces exerted on the bucket importance for efciency suggest that the bucket angle might best
by the material, and are facilitated by elastic deformation of machine
links, shaking or bouncing of the machine's undercarriage, and other
similar phenomena. Reactive variations generally appear as noise
on the trajectory plot and do not signicantly affect the overall tra-
jectory shape. Joystick Sensor
Voltage
2) Operator redirection moderate to large changes in bucket direc- Voltage Sensor Machine
Joysticks Signals Calibration (or Machine
tion made by the operator. This type of deviation may occur due to
(or Joystick Simulation)
Simulation)
Measured
Link Angles Valve
(1, 2, 3) voltage
Current desired
Valve
machine angles Kinematics
characteristics
1h = 1hold + d1h Transformation
2h = 2hold + d2h (Real and virtual
3h = 3hold + d3h machine models)
Differential angles
(d1h, d2h, d3h) Desired Actuator
Velocities (u1, u2, u3)
Fig. 18. Sample bucket trajectories calculated using APAS viewed from in front of the
shovel. Fig. 20. Block diagram for Kinematics Transformation.
136 R. Tiwari et al. / Automation in Construction 31 (2013) 128139
Fig. 21. (a) Bucket pin motion control. Motion in Cartesian coordinates at 45 and (b) 0 slope angles.
Fig. 22. Open-loop bucket pin control with improved machine calibration. Cartesian coordinates at (a) 45 and (b) 0 slope angles.
Fig. 23. Motion in Cartesian coordinates at 45 slope angle. Bucket pin motion control without angle stabilization (a) with feedback control and (b) with open-loop control. Three
consecutive cycles are shown.
R. Tiwari et al. / Automation in Construction 31 (2013) 128139 137
be controlled by the robotic portion of the system, allowing the oper- control algorithms in an attempt to further improve the machine per-
ator even more freedom to concentrate on controlling the deviations formance. Recent test results are shown in Fig. 23.
from the general trajectory. Optimization of the relationship between
bucket motion direction and bucket angle is seen to be important for
5. Conclusion
reducing the digging force and related power consumption.
In order to test the feasibility of controlling the machine bucket to Appendix A. EX 3500 shovel marker coordinates
follow the above trajectory, a control algorithm described briey in
Appendix A and described in detail in Reference [8] was used. Paramet- (All dimensions in meters unless otherwise specied)
ric trajectory testing was performed in both simulation platform devel- Track
oped in MATLAB as well as on an experimental Bobcat 435 excavator.
Simulation provided a safe environment in which to test, debug, and
demonstrate the feasibility of new algorithms, to control the bucket
trajectory.
Block diagrams of the machine simulation and the kinematics-
transforming differential control algorithm are shown in Figs. 19 and
20, respectively.
Linear trajectories can be generated at any angle by rotating the x
yz Cartesian system around a horizontal axis. Examples of the simu-
lation output are shown in Fig. 21.
Markers A and B are on the right side of the machine while Marker Serial number #103 Serial number #112
markers C and D are on the left side of the machine.
x y z x y z
Side of the cab
A 5.893 2.4896 0 5.918 2.413 2.375
B 5.893 2.4896 0 5.906 2.4892 2.324
C 6.464 0 0 6.464 0 1.791
D 6.636 0.0254 0 6.464 0 0.2413
Table A.1
Notations used in Fig. A.4: differential trajectory generation with coordinate
transformation.
h i
f i
s
s : The pre-dened trajectory feature that is always available to the supervisor
to copy into the motion. This term is a function of the movement along
the trajectory through s. The super-operator can re-wind s and start at the
beginning for a new motion.
v(t): The real-time execution velocity along the pre-dened trajectory, con-
trolled by the operator. If v = 0, the machine becomes fully manual.
J: Machine kinematics Jacobian matrix.
Jc1 : Coordinate transformation feature for manual correction or fully manual
control. Several Jc1 choices can be provided for selection. If Jc1 is the unit
matrix, the machine is controlled directly in the joint space, acting like a
manual machine.
c
x_ i : Correction or manual motion control velocities guided by the operator.
Fig. A.3. Marker positions on the back of the cab the machine.
R. Tiwari et al. / Automation in Construction 31 (2013) 128139 139
[2] H. Hemami, Modeling, analysis and preliminary studies for automatic scooping,
Journal of Advanced Robotics, Japan 8 (5) (1994) 511529.
[3] X.D. Huang, L.E. Bernold, Control model for robot backhoe excavation and obsta-
cle handling, in: Proc. Robotics for Challenging Environments, 1994, pp. 97105.
[4] J. Maciejewski, A. Jarzebowski, W. Trampczynski, On the efciency of soil digging
process, Engineering Transactions 49 (4) (2001) 599621.
[5] L. Ponecki, W. Trampczynski, J. Cendrowicz, A conception of the assisting system
for the hydraulic excavator operator, Automation in Construction 7 (5) (1998)
401411.
[6] A.S. Hall, P.R. McAree, A study of the interaction between operator style and ma-
chine capability for a hydraulic mining excavator, Proceedings of the Institution
of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science 219
(2005) 477.
[7] E. Bisse, A. Hemami, E.K. Boukas, A comparison of the required energy in loading for
four scooping strategies, in: Proceedings the Third International Symposium on Mine
Mechanization and Automation, Golden, Colorado, June 1995, pp. 2-172-28.
[8] G. Danko, Operator control architecture, in: Proceedings of the 4th Regional Sym-
posium on Computer Applications in the Mineral Industries, Tampere, Finland,
2001, pp. 253266.
[9] A. Himami, Fundamental analysis of automatic excavation, Journal of Aerospace
Engineering 8 (4) (1995) 175179.
[10] L.J. Mirabelli, Mining automation program, in: NAANSA/USGS, Automation Con-
ference, 2000.
[11] T.B. Sheridan, Telerobotics, Automation, and Human Supervisory Control, MIT
Press, 1992.
[12] R. Alami, R. Chatila, S. Fleury, M. Ghallab, F. Ingrand, An architecture for autono-
my, International Journal of Robotics Research 17 (4) (1998) 315337.
[13] M.H. Chiang, C.C. Huang, Experimental implementation of complex path tracking
control for large robotic hydraulic excavator, International Journal of Advanced
Manufacturing Technology 23 (2004) 126132.
[14] S.-J. Lee, P.-H. Chang, Y.-M. Kwon, An experimental study on Cartesian tracking
control of automated excavator system using TDC-based robust control design,
in: Proc. of the American Control Conference, 1999, pp. 31803185.
[15] S.-U. Lee, P.-H. Chang, Control of a heavy-duty robotics excavator using time delay
control with switching action with integral sliding surface, in: Proc. of IEEE Inter-
national conference on Robotics and Automation, 2001, pp. 39553960.
[16] B. Yao, J. Zhang, D. Koehler, J. Litherland, High performance swing velocity-tracking
control of hydraulic excavator, in: Proc. of the American Control Conference, 1998,
pp. 818822.
[17] G. Danko, J. Knowles, Rashi Tiwari, Model based camera vision evaluation of ma-
chine motion, in: Proceedings Application of Computer and Operational Research
in Mining Industry, 2005, pp. 513520.
[18] http://www.hitachi-c-m.com/au/products/excavator/face/index.html.
[19] Ariel Dynamics Inc., Ariel performance analysis system, http://www.arielnet.com/
start/apas/default.html 2004.
Fig. A.4. Continuous differential control architecture with workspace sensor input.
References
[1] S. Singh, R. Simmons, Task planning for robotic excavation, in: IEEE/RSJ Interna-
tional Conference on Intelligent Robots and Systems, 1992, pp. 12841291.