C o v e r S t o r y S AFE T Y SYSTEMS

STEERING WHEEL FOR ACTIVE

DRIVER STATE DETECTION
The steady increase in equipment in driver assistance and communications systems has made it necessary to
look at driver workload and risk of accidents. The flood of information may even overload the driver. Although a
lot is known via sensors such as cameras, radar and lidar about the environment around the vehicle, the condi-
tion in the vehicle is still relatively unknown. Takata presents systems in the steering wheel, such as hand detec-
tion and integrated camera that allow biometric condition monitoring of the driver. The safety risk by additional
occupation is minimised.

36 www.autotechreview.com
 Authors
STEFANIE ESSERS
is Director for EMEA Core Engineering
& Research at Takata AG
in Berlin (Germany).
JASON LISSEMAN
is Director North America Core
Engineering for Steering Wheel
Systems at Takata Holdings, Inc. in
Auburn Hills, Michigan (USA).
HEIKO RUCK
is Director for EMEA Application
Engineering for Steering Wheels and
Airbags at Takata AG in
Aschaffenburg (Germany).
autotechreview Jul y 2 016 Vo lum e 5 | I s su e 7
DRIVER WORKLOAD AND meantime, one is faced with increasing
ACCIDENT RISK numbers of accidents due to distraction
[3]. People feel safer in today’s cars than
Car buyers are being increasingly confron- ever before. They use their smart phones
ted with the field of autonomous driving for SMS texting, selecting music, tele-
by the media. One such showcase is at phone and continue with activities such
the CES conference in Las Vegas, where as eating or drinking.
for the last few years many vehicle manu- Interestingly, Landau [1] had already
facturers and suppliers have presented included driver monitoring as a future
advanced driver assistance systems device and this is what will be discussed
(ADAS) and vehicles with varying degrees more extensively in the following.
of autonomous driving functionality.
In 2002, Landau [1] did intensive
research on automation. He specifically DATA COLLECTION
evaluated the influence of automation in
cars. The relationship between driver If we look at vehicles in the D segment
workload and accident risk and the and above, we are now able to “see” a
degree of available assistance and com- 360° surround view of the vehicle,
munication systems was already con- exceeding the human senses. Lidar
sidered. It was already anticipated that sensor systems, for example, are able to
after facilitating the driving task and detect and classify obstacles 300 m
thereby reducing the accident risk and ahead of the vehicle. Exterior sensors
driver workload, we would come to a typically supply one to multiple smart
turning point. The increasing number of assistance systems today, which belong
ADAS and sensor data fusion would lead to one of three categories:
to a situation of increased driver workload :: Safety systems (collision warning, ped-
and accident risk because of an increased estrian protection, emergency
demand on driver attention, while driving braking);
from vehicle assistance systems and :: Vehicle control (ACC stop-and-go, lane
nomadic devices. keeping); and
Both forecasts have come true. In :: Driver support (traffic sign recognition,
highly equipped cars, people sometimes blind spot detection).
are overloaded in managing the available With automatic parking and traffic jam
assistance and infotainment systems par- assistance it is already experienced
allel to their main task of driving [2]. In the today what will evolve further – a com-
2030
Explanation 2025
Autonomous
Safety Vehicle Driver Autonomous
systems support
2020 city driving
control highway
Highway driving
assist Emergency
avoidance
steering
Traffic jam Emergency
assist stop assist
2015
Intelligent
light
Traffic sign
Pedestrian Automatic
recog
detection parking
nition
Lane
Emergency
keeping
braking
Collision
warning ACC
stop and go Blind spot
detection
Parking
assist
1 ADAS characteristics – chronological development towards combined assistance systems
37
C o v e r S t o r y S AFE T Y SYSTEMS

Driver controls Driver controls Driver must Driver does not No driver needed System can
continuously continuously permanently monitor permanently monitor in the specific automatically handle
longitudinal and longitudinal or the system the system; has to be application all situations during
lateral steering lateral steering potentially in a the entire trip;
Automation ←→ Driver

position to monitor no driver required

System can
automatically handle
all situations in
System takes over specific application
longitudinal and
System takes over lateral control in a
longitudinal and specific application;
lateral control in a detects system
specific application boundaries and calls
System takes the
for driver to take over
respective other
with a sufficient time
function
reserve
No engaging
vehicle system active

Level 0 Level 1 Level 2 Level 3 Level 4 Level 5

Driver only Assistant Partially Highly Fully Driverless
automated automated automated

System
overrules
2 Five levels of autonomous driving (Source: VDA)

bination of vehicle control and driver
support based on sensor data fusion.
Within the next decade we will see
combined ADAS that will include safety
features, vehicle control and driver sup-
port in one system. This item allows the
car to drive automatically, 1.
Future cars will not only offer
increased assistance based on sensor data
fusion but they will also be fully connec-
ted – offering additional new functionalit-
ies to the occupants, including the driver.
As automation increases, the feedback
loop between vehicle and driver will be
influenced. The German Association of
the Automotive Industry (VDA) [4] as well
as the German Federal Highway Research
Institute (BASt) [5] provided definitions in
2014 that picked up on earlier automation
classifications and added a fifth level: the
autonomous driverless car, 2.
Today’s modern cars are just at the
threshold of moving from partial to highly
automated driving – level 2 to level 3, ②.
The driver will be allowed to step out of
the loop for a limited amount of time.
Nevertheless, the safety of the occupants
must at all times be ensured. This
requires knowledge of the driver’s status
and activities. So, what does that mean
for the role of the driver?
THE FUTURE ROLE OF THE DRIVER
The driver will become partly passive in
the loop, and will no longer be provided
with information solely about the system
state and the ADAS. In fact, the intelli-
gence of the vehicle (electronic control
unit, ECU) will monitor and assess
information about the driver status. How-
ever, from today’s perspective we
approach a technical gap, 3: There is an
“intelligent”, connected vehicle, which is
very aware of everything in its environ-
ment. And there is a driver who can be
related to any of the following statuses –
focused and stable; distracted by second-
ary or tertiary tasks; drowsy; aggressive;
bored and overloaded.

POSSIBILITIES FOR
DRIVER STATE DETECTION

Unfortunately, the “intelligent” vehicle

does not see the driver’s behaviour via its
external sensors so far. It has very good
“eyes” for the exterior – but it is missing
all the senses to be aware of its inner side.
3 Technical gap between well-known environment recognition via “intelligent” The MIT Agelab [6] made a model of all
vehicle (left) and unknown driver behaviour (right) necessary data that is needed to detect the

38 www.autotechreview.com

driver status. Ultimately this requires a
lot of data, some of which can be
provided by existing vehicle perform-
ance, or environmental status devices,
some of which will drive the develop-
ment of new sensors and monitoring sys-
tems that do not exist today.
However, the real key will be interrog-
ating this data to build a holistic picture
of the drivers’ mental workload, level and
sources of distraction, fitness level and
intentions. In order to accomplish this,
there are three main sensing focus areas:
:: Biometrics (heart rate, respiration, skin
conductance);
:: Visual attention and fatigue (gaze
measurements or percental eye clos-
ure); and
;; Emotions (face recognition or voice
analysis).
Many of these systems exist in the med-
ical industry, and in the automotive
industry these measurements are already
taken to determine the driver-fitness
safety. As mentioned previously, people
eat, drink, talk on the phone and just
recently in the last years’ text, tweet and
browse the internet while driving.
Two aspects of these additional driver
activities are dangerous. On the one hand,
it is the visible behaviour of the driver; he
takes his hand off the wheel and his eyes
off the road. On the other hand, the GUIs
of mobile devices are not designed for
in-vehicle usage. For example, the user
autotechreview Jul y 2 016 Vo lum e 5 | I s su e 7
4 Steering wheel with integrated lightbar as warning function (left) and a sec-
tioned view of the steering wheel with a hands-on detection sensor mat (right)
interface of a smart phone app does not
have to reflect the standards for automot-
ive usage [7]. The safety risk posed by this
distracted behaviour is proven by the mul-
titude of statistics that are commonly
available [8, 9].
DETECTION OF HAND CONTACT AT
STEERING WHEEL
In order to reduce the safety risk of addi-
tional driver activities, a first step for the
near future could be the addition of a
steering wheel hand detection system.
This Hands-On Detection (HOD) system
enables the ability to identify when the
driver is in control of the car and to detect
situations when this is not the case.
By utilising hidden capacitive sensors
underneath the steering wheel rim surface
material, it is possible to detect not only
hand contact but the location of contact
on the steering wheel for all vehicle
speeds, including stationary vehicles. The
first generation of HOD steering wheels is
in production today. HOD can also be
combined with a high visibility wheel
mounted light warning function that can
remind the driver to put their hands back
on the steering wheel, 4. This is done to
quickly communicate vehicle state and
can attract visual attention back to the
forward looking zone.
CAMERA-BASED DRIVER
STATE DETECTION
Combining this functionality with an
interior-focused vision system allows the
“intelligent” vehicle to monitor the visual
attention of the driver. There are different
regions, where a driver is likely to focus:
at the road, infotainment system, smart-
phone or off-road.
With the “inner eyes”, the vehicle is
able to track whether the driver is pay-
ing full attention and therefore “in the
loop” or whether the driver is distracted
and needs increased assistance or sup-
port. With a camera and illumination
via infrared light emitting diode
(IR-LED) included in the steering wheel,
5 (right), we have one of the best per-
spectives on the driver’s face and an
optimal viewing angle for further meas-
urements such as gaze direction, occu-
pant size and position, heart rate extrac-
ted out of skin colour variances, identi-
fication and age estimation.
All these measurements support both
passive and active safety functions. For
almost full automation, additional
measurements like eye closure for
drowsiness detection and pupillometry
for workload measurement become
increasingly important.
With smart vision algorithms we will
soon be able to more accurately detect
39
C o v e r S t o r y S AFE T Y SYSTEMS
5 Steering wheel with biometric sensors in the rim (left) as well as integrated camera and infrared light emitting diode (right)
age, gender, posture, stature and identity
of the occupants, which will in turn
enable the car to personalise the passive
safety restraints for optimum performance
and optimise the active safety system by
adapting the automation.
VITAL STEERING WHEEL
One area of interest would be expanding
these measurements to include an Electro
Cardiogram (ECG), skin conductance and
respiratory rate with further sensors
within the steering wheel or the seatbelt.
Such data could help derive the fitness
level, workload and emotional state of the
driver. More measurements such as brain
waves would be beneficial; but combining
these mentioned three core measurements
with vision data it is possible to get a clas-
sification of stress level or workload, fit-
ness level or wellness, emotional state
and driver intention.
Continued sensing research is expected
to deliver alternative non-intrusive ways
to gain this data, but as of today the
majority of these measurements can only
be derived by direct contact. As the steer-
ing wheel is the only location in the
vehicle for direct skin contact, it repres-
ents the ideal location to capture such
data. The steering wheel prototype in ⑤
(left) is one such wheel, which is
equipped with independent biometric
sensors around the steering wheel rim (as
sensor mat), which allows skin conduct-
ance and heart rate measurements in a
reliable and accurate manner.
However, this is just one potential
40
solution, because hand position is not
stable and in autonomous mode, the
hands may not always be on the wheel.
In general, no matter whether visual or
surface sensors are used, redundant meas-
urements are key for stable classification
results and for a valid prognosis of driver
behaviour. Last but not the least, a smart
vehicle will also need a “brain” – a
powerful processing platform with com-
plex algorithm – that combines all neces-
sary data to classify the driver state and to
adapt the automation and the interior to
the needs.
CONCLUSION FOR
DRIVER STATE SENSING
Takata continues to strive to bring all
these pieces into a complete system, such
as the virtual co-pilot. This would be the
“personification” of the vehicle and its
assistance systems.
While the assistance systems and
automation become more and more
powerful, the interface has not evolved
significantly. For a complex automation
task more than several on/ off buttons of
various assistance systems are needed. In
fact, seamless, natural interaction
between two “intelligent” parties –
between vehicle and driver – will be
expected from the customer.
From a safety or misuse perspective,
some level of driver and/ or occupant
sensing is indispensable to do automa-
tion safely. The steering wheel will play
a significant role in detecting driver state.
After all, the vehicle on an autonomous
drive from Munich to Hamburg should
know if the driver is healthy and ulti-
mately still there.
REFERENCES
[1] Landau, K.: The development of driver assist-
ance systems following usability criteria. In: Beha-
viour & Information Technology 21 (2002), No. 5,
pp. 341-344
[2] Jahn, G.; Oehme, A.; Rösler, D.; Krems, J. F.:
Kompetenzerwerb im Umgang mit Fahrerinforma-
tionssystemen. Berichte der Bundesanstalt für
Straßenwesen, Abt. Fahrzeugtechnik, Heft F 47,
Bremerhaven: Wirtschaftsverlag NW, 2011
[3] Klauer, S. G.; Dingus, T. A.; Neale, V. L.; Sud-
weeks, J. D.; Ramsey, D. J.: The impact of driver
inattention on near-crash/crash risk – an analysis
using the 100-car naturalistic driving study.
NHTSA, April 2006, National Technical Informa-
tion Service, Springfield, Virginia, 22161
[4] VDA: Automatisiertes Fahren. In: https://www.
vda.de/de/themen/innovation-und-technik/automat-
isiertes-fahren.html, last access on 23 January 2015
[5] Gasser, T.: Herausforderungen automatischen
Fahrens und Forschungsschwerpunkte. 6. Tagung
Fahrerassistenz, Tüv Süd, München, 2013
[6] Coughlin, J. F.; Mehler, B.; Reimer, B.: Monit-
oring, managing, and motivating driver safety and
well-being. In: Pervasive Computing, 10 (2011),
No. 3, pp. 14-21
[7] Alliance of Automobile Manufacturers (AAM),
Driver Focus-Telematics Working Group: State-
ment of principles, criteria and verification proced-
ures on driver interactions with advanced in-ve-
hicle information and communication systems.
Washington, DC, 2002
[8] U. S. Department of Transportation: Facts and
statistics. In: http://www.distraction.gov/content/
get-the-facts/facts-and-statistics.html, last access
on 23 January 2015
[9] Jolley, D.: Ford survey shows many European
drivers ignore distracted driving risks. In: http://
europe.autonews.com/article/20120414/
ANENE/120419968, last access on 23 January 2015
Read this article on
www.autotechreview.com
www.autotechreview.com