Second Delaware Immersion Suit

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 1 of 25 PageID #: 1
IN THE UNITED STATES DISTRICT COURT

FOR THE DISTRICT OF DELAWARE
IMMERSION CORPORATION
Civil Action No. ____________
vs.
APPLE INC.,
JURY TRIAL DEMANDED

AT&T INC., and
AT&T MOBILITY LLC
COMPLAINT FOR PATENT INFRINGEMENT

Plaintiff Immersion Corporation ("Immersion") brings this action for patent infringement
against Apple Inc., AT&T Inc., and AT&T Mobility LLC (collectively, "Defendants"), and
alleges as follows:
NATURE OF THE ACTION
1.
Immersion is a leading innovator and developer of haptic technology. Since its
founding in 1993, Immersion has developed and licensed haptic products and intellectual
property across diverse industries and applications, including, for example, medical devices,
medical training simulations, game systems and controllers, automotive devices, touchscreen
controls for appliances and office equipment, and mobile electronic devices. In recent years, a
primary focus of Immersion's business has been developing products for the mobile electronic
device industry, including software products for implementing advanced haptic effects, including
pressure-enabled haptic effects, on mobile electronic devices.
2.
"Haptics" refers to the science of touch. Haptics in consumer electronic devices
provides touch or tactile sensations to the users of electronic devices. Because of the importance
of the sense of touch to the way people perceive their surroundings and the things with which
they interact, haptics can greatly enhance the usability and functionality of consumer electronic
6251724
devices. Haptic sensations in such devices are typically created by actuators, or motors, which
create a vibration.
3.
When haptics is implemented on a mobile phone (for example, through
Immersion's TouchSense software), users can experience a vibrating force as they push a
virtual button, select a graphical object, scroll through a list, or receive a message. As another
example, when haptics is implemented in a laptop computer (for example, in the laptop's
touchpad), users can experience a vibrating force when they press down on the touchpad.
Haptics may, for example, confirm or provide cues that an action has been performed.
Predefined haptic effects may be incorporated in an electronic device to allow certain haptic
effects to be associated with certain actions and allow applications to access and use the
predefined haptic effects library. As another example, when implemented in a mobile game,
users can feel an engine reverberation, the crack of the bat meeting the ball, or the release of a
slingshot. The presence of haptics in electronic devices provides a new level of user interaction
through haptic cues, which provide users with a more rich and immersive user experience.
4.
In electronic devices, haptic effects are typically managed and controlled by
embedded software, and integrated into device user interfaces and applications via embedded
control software APIs. Applications running on a mobile electronic device call these APIs to
implement the haptic effects. These APIs in turn cause specific haptic effect commands to be
sent to an actuator in the electronic device, resulting in the associated haptic feedback effect.
More sophisticated applications may provide a variety of tactile sensations. For example, user
actions may trigger different haptic effects and thus communicate different types of information.
This information may be conveyed by varying the type, duration, intensity, or frequency of the
6251724
-2-
tactile sensations. This enables the creation of different haptic effects so that users can easily
distinguish, for example, a calendar alert from a text message alert or an incoming call.
5.
Pressure-enabled haptics is a form of haptics. For example, pressure-enabled
sensors may measure the pressure levels of contacts on the touchscreen. Separate haptic effects
may be configured for different interactions with graphical objects, for example, touching
graphical objects on the touchscreen and applying pressure at varying pressure levels. Different
haptic effects may be provided depending on the pressure level of the contact on the touchscreen
in relation to predefined pressure thresholds.
6.
Haptics, including pressure-enabled haptics, can be incorporated into many
features of a smartphone or laptop. For example, it is especially useful in electronic devices

containing touchscreens, which tend to have very few physical buttons.
For example, a
particular haptic effect may be used when the pressure level of the contact on the touchscreen is
less than a first pressure threshold, or a particular haptic effect may be used when the pressure
level of the contact on the touchscreen is between the first pressure threshold and a second
pressure threshold. As another example, haptic effects in a pressure-enabled mobile device may
be provided depending on whether certain conditions are met by the user's interaction with the
pressure-enabled touch sensor. For example, a user may interact with a displayed graphical
object on the touchscreen by applying pressure to the touchscreen. The pressure-enabled touch
sensor may receive contact data from the interaction and the mobile device may determine a
gesture based on the contact data. As an example, if the pressure is greater than a pressure
threshold and the change in pressure is greater than a change in pressure threshold and a time
interval has elapsed, a haptic effect may be provided. In this way, the user's intended interaction
6251724
-3-
may be accurately determined based on contact data from the pressure-enabled touch sensor, and
the user experience enhanced.
7.
Haptic effects, including pressure-enabled haptics, may also be used to confirm
receipt of a command and indicate that a command is unrecognized. For example, a user may
issue a command by interacting with a graphical object on a touchscreen, e.g., by applying a
certain level of pressure to the touchscreen. If the user input is recognized and the command is
determined, the mobile electronic device may provide a haptic effect confirming receipt of the
command. Otherwise, the mobile electronic device may provide a second haptic effect. The
second haptic effect may indicate, for example, that the user input is unrecognized or that a
corresponding command was not found. Haptic feedback confirming receipt of a command
provides a more realistic and responsive user experience, informing the user via tactile sensation
whether the user's input was recognized and the command was determined.
8.
Immersion's hard work and ingenuity in the field of haptics has resulted in
extensive intellectual property protection for Immersion's innovations. This protection includes
more than 2,100 world-wide granted and pending patents, including the patents-in-suit.
9.
Haptics play a central role in Defendants' products. For example, important
features of the current generation of Apple iPhones are promoted under the names "3D Touch"
and "Taptic Engine."
These features of the Apple iPhone 6s and 6s Plus provide haptic
feedback, including feedback based on pressure-sensitive interactions with the touch screen.
Apple and AT&T tout this functionality on their websites.
See, e.g., Apple, iPhone 6s
3DTouch, http://www.apple.com/iphone-6s/3d-touch/ (3D Touch):
6251724
-4-
See also, e.g., http://www.apple.com/iphone-6s/technology/ (Taptic Engine):
See also, e.g., https://www.att.com/cellphones/iphone/iphone-6s.html (3D Touch advertised by

AT&T):
6251724
-5-
10.
Similarly, important haptic features of the Apple MacBook, Apple MacBook Pro
13-inch, and Apple MacBook Pro 15-inch are promoted under the names "Force Touch" and
"Taptic
Engine."
Apple
touts
this
http://www.apple.com/macbook/design/:
6251724
-6-
functionality
on
its
website.
See
11.
Defendants are capitalizing on Immersion's innovation and success by selling
mobile devices that infringe Immersion's patents. Defendants are utilizing Immersion's patented
inventions without license or authority from Immersion. Immersion has brought this action to
remedy Defendants' infringement.
PARTIES
12.
Immersion is a Delaware corporation with its principal place of business located
at 50 Rio Robles, San Jose, CA 95134. Immersion is the owner of the intellectual property rights
at issue in this action. As noted above, Immersion is a leading innovator and developer of haptic
feedback technologies. Since its founding in 1993, Immersion has developed and licensed haptic
feedback products and intellectual property across diverse industries and applications, including
mobile electronic devices.
13.
Defendant Apple Inc. ("Apple") is a California corporation with its principal place
of business at 1 Infinite Loop, Cupertino, CA 95014.

14.
Apple makes, uses, offers to sell, sells, and imports into the United States mobile
electronic devices, including the Apple MacBook, Apple MacBook Pro 13-inch, Apple
MacBook Pro 15-inch, Apple iPhone 6s, and Apple iPhone 6s Plus. Apple maintains one or
more retail stores and numerous employees in this judicial district. Apple's retail stores in this
6251724
-7-
judicial district offer to sell and do sell infringing devices, including the Apple MacBook, Apple
MacBook Pro 13-inch, Apple MacBook Pro 15-inch, Apple iPhone 6s, and Apple iPhone 6s
Plus.
15.
Defendant AT&T Inc. is a Delaware corporation with its principal place of
business at 208 S. Akard Street, Dallas, TX 75202. Defendant AT&T Mobility LLC is a
Delaware limited liability company with its principal place of business at 1025 Lenox Park
Boulevard NE, Atlanta, GA 30319. AT&T Mobility LLC is a wholly-owned subsidiary of
AT&T Inc. Together, AT&T Inc. and AT&T Mobility LLC will be referred to as "AT&T."
16.
AT&T uses, offers to sell, sells, and imports into the United States mobile
electronic devices, including the Apple iPhone 6s and Apple iPhone 6s Plus. AT&T maintains
one or more retail stores and numerous employees in this judicial district. AT&T's retail stores
in this judicial district offer to sell and do sell infringing devices, including the Apple iPhone 6s
and Apple iPhone 6s Plus.
JURISDICTION AND VENUE
17.
This is an action for patent infringement arising under the patent laws of the
United States of America, 35 U.S.C. 1, et seq., including 35 U.S.C. 271. This Court has
subject matter jurisdiction over the matters pleaded herein under 28 U.S.C. 1331 and 1338(a).
18.
This Court has personal jurisdiction over Defendants because, among other
reasons, Defendants have done business in this District, have committed and continue to commit
acts of patent infringement in this District, and have harmed and continue to harm Immersion in
this District, by, among other things, using, selling, and offering for sale infringing products in
this District. Moreover, the Defendants have placed infringing products into the stream of
6251724
-8-
commerce by shipping those products into this District or knowing that the products would be
shipped into this District.
19.
Venue is proper in this District pursuant to 28 U.S.C. 1400(b) and 1391(b)-(c)
because, among other reasons, Defendants are subject to personal jurisdiction in this District and
have committed acts of infringement in this District, including selling and distributing infringing
products in this District.
20.
Joinder of Defendants in this action is proper under 35 U.S.C. 299 because,
among other reasons, Immersion asserts a right to relief against Defendants with respect to or
arising out of the same transaction, occurrence, or series of transactions or occurrences relating
to the making, using, importing into the United States, offering for sale, or selling of common
accused products, including the Apple iPhone 6s and Apple iPhone 6s Plus, and questions of fact
common to all Defendants will arise in this action.
FIRST CLAIM FOR RELIEF FOR PATENT INFRINGEMENT
(U.S. Patent No. 8,749,507)
21.
Immersion incorporates by reference paragraphs 1 through 20 as if set forth here
22.
Immersion is the owner of the entire right, title, and interest in and to U.S. Patent
in full.
No. 8,749,507 (the "'507 patent"), entitled "Systems and Methods for Adaptive Interpretation of
Input from a Touch-Sensitive Input Device," which was duly issued on June 10, 2014. A copy
of the '507 patent is attached as Exhibit A.
23.
The '507 patent, in general, describes advances in providing haptic feedback in
certain mobile electronic devices such as mobile phones that receive contact data from
interactions with a touchscreen. The '507 patent teaches, among other things, systems and
6251724
-9-
methods in which the mobile electronic device determines a pressure and a change in pressure
based on contact data. The mobile electronic device outputs a haptic effect if the pressure is
greater than a pressure threshold, the change in pressure is greater than a change in pressure
threshold, and an interval has elapsed. Further details of the invention are described in the
specification and the claims.
24.
Defendants have infringed and are currently infringing the '507 patent, in
violation of 35 U.S.C. 271, by, among other things, making, using, offering for sale, selling,
and/or importing within this judicial district and elsewhere in the United States, without license
or authority, products and/or processes falling within the scope of one or more claims of the '507
patent, including without limitation the Apple iPhone 6s and Apple iPhone 6s Plus.
25.
Based on the information presently available to it, Immersion alleges that at least
the following apparatuses, products, devices, processes, methods, acts, or other instrumentalities
infringe or are covered by claims of the '507 patent: the Apple iPhone 6s and Apple iPhone 6s
Plus.
Immersion makes this preliminary identification of infringing apparatuses, products,
devices, processes, methods, acts, or other instrumentalities without the benefit of discovery or
claim construction in this action, and expressly reserves the right to augment, supplement, and
revise its identifications based on additional information obtained through discovery or
otherwise.
26.
Apple, AT&T, and their customers directly infringe the '507 patent using the
Apple iPhone 6s and Apple iPhone 6s Plus. The accused devices receive contact data from a
user's finger contacting the touch screen of the devices.
The accused devices determine
interactions with displayed graphical objects. Then, the accused devices determine a "Peek" or
"Pop" press using pressure, change in pressure, and timing interval information. Additional
6251724
- 10 -
details relating to the accused devices and their infringement are in the possession of
Defendants.
27.
Defendants also indirectly infringe the '507 patent. Defendants are aware of the
'507 patent at least from the date of this Complaint. Additionally, on information and belief,
Defendants were aware of the '507 patent before the date of this Complaint, including without
limitation through Defendants' knowledge of Immersion and Immersion's disclosure of its
patents
on
its
website
(formerly
located
at
http://www.immersion.com/haptics-
technology/patents/index.html and currently located at http://www.immersion.com/legal/

#patents). For example, Immersion's website specifically lists the '507 patent. Additionally, on
information and belief, the relevance of Immersion's leading portfolio of haptic patents is wellknown
to
Defendants.
See,
e.g.,
http://www.benzinga.com/analyst-ratings/analyst-
color/13/08/3816746/immersion-others-could-get-a-boost-when-haptics-market-r
("When
it
comes to haptics, Immersion is usually the first company that comes to mind."). Defendants
knew that the Apple iPhone 6s and Apple iPhone 6s Plus infringed the '507 patent, or at a
minimum believed there was a high probability that the Apple iPhone 6s and Apple iPhone 6s
Plus were covered by the '507 patent, but willfully blinded themselves to Immersion's patent and
the infringing nature of the Apple iPhone 6s and Apple iPhone 6s Plus.
28.
Defendants induced and are actively inducing infringement of the '507 patent, in
violation of 35 U.S.C. 271(b), by, among other things, actively and knowingly aiding and
abetting others to directly make, use, offer for sale, sell, and/or import within this judicial district
and elsewhere in the United States, without license or authority, products and/or processes
falling within the scope of one or more claims of the '507 patent, including without limitation the
Apple iPhone 6s and Apple iPhone 6s Plus. For example, Defendant Apple provides directions,
6251724
- 11 -
instruction manuals, guides, and/or other materials that encourage and facilitate infringing use by
others.
See, e.g., http://www.apple.com/iphone-6s/3d-touch/; http://www.apple.com/iphone-
6s/technology/; http://help.apple.com/iphone/9/. Defendant AT&T similarly provides directions,

others.
See,
e.g.,
http://www.att.com/wireless/iphone/;
https://www.att.com/cellphones/
iphone/iphone-6s.html; http://www.att.com/esupport/iphonefaqs.jsp. The Defendants have sold

and are selling these products with the knowledge and intent that customers who buy the
products will use the products for their infringing use and therefore that customers have been and
are directly infringing the '507 patent.
29.
Defendants have contributorily infringed and are currently contributorily
infringing the '507 patent, in violation of 35 U.S.C. 271(c), by, among other things, selling,
offering for sale, and/or importing within this judicial district and elsewhere in the United States,
without license or authority, products or components of products which constitute a material part
of the '507 patent, knowing that such products and/or components are especially made or
especially adapted for use in the infringement of the '507 patent, and not staple articles or
commodities of commerce suitable for substantial noninfringing use.
30.
Defendants' infringement of the '507 patent has been and continues to be willful
and deliberate. Despite knowledge of the '507 patent, Defendants have acted and are acting
despite an objectively high likelihood that their actions constitute patent infringement. This
objective risk was and is known to the Defendants, and is also so obvious that it should have
been known to the Defendants.
31.
As a direct and proximate result of Defendants' infringement of the '507 patent,
Immersion has suffered and will continue to suffer irreparable injury for which there is no
6251724
- 12 -
adequate remedy at law. Immersion has also been damaged and, until an injunction issues, will
continue to be damaged in an amount yet to be determined.
SECOND CLAIM FOR RELIEF FOR PATENT INFRINGEMENT
(U.S. Patent No. 7,808,488)
32.
33.
in full.
No. 7,808,488 (the "'488 patent"), entitled "Method and Apparatus for Providing Tactile
Sensations," which was duly issued on Oct. 5, 2010. A copy of the '488 patent is attached as
Exhibit B.
34.
certain mobile electronic devices such as mobile phones that have, among other things, a
touchscreen and an actuator. The '488 patent teaches, among other things, systems and methods
in which the mobile electronic device displays on the touchscreen a graphical object. When the
user touches the graphical object on the touchscreen using his or her finger, a signal is generated
indicating an object contacting the touchscreen and a pressure of the contact. The mobile
electronic device generates an actuator signal to the actuator to output a haptic effect based on
the user's interaction with the graphical object on the touchscreen. Haptic effects are output
based on, among other things, the pressure of the contact in relation to pressure thresholds.
Further details of the invention are described in the specification and the claims.
35.
6251724
- 13 -
36.
Plus.
otherwise.
37.
Apple iPhone 6s and Apple iPhone 6s Plus. When a user presses at a pressure level to trigger a
"Peek" interaction on a software-generated object on the touch screen of the accused devices, the
processor receives a signal indicating the application of pressure from a user's finger contacting
the touch screen, and transmits an actuator signal to the actuator in the accused devices (e.g., the
"Taptic Engine"), causing the actuator to output a first tactile sensation. Similarly, when the user
then presses at a greater pressure level to trigger a "Pop" interaction, a different, second tactile
sensation is produced. Additional details relating to the accused devices and their infringement
are in the possession of Defendants.
38.
6251724
- 14 -
patents
on
its
website
(formerly
located
at

to
Defendants.
See,
e.g.,
("When
it
39.
others.

others.
See,
e.g.,
6251724
- 15 -
40.
41.
42.
THIRD CLAIM FOR RELIEF FOR PATENT INFRINGEMENT
(U.S. Patent No. 8,581,710)
43.
in full.
6251724
- 16 -
44.
No. 8,581,710 (the "'710 patent"), entitled "Systems and Methods for Haptic Confirmation of
Commands," which was duly issued on Nov. 12, 2013. A copy of the '710 patent is attached as
Exhibit C.
45.
certain mobile electronic devices such as mobile phones that have, among other things, a
touchscreen and an actuator. The '710 patent teaches, among other things, systems and methods
in which the mobile electronic device receives user input via the touchscreen. The mobile
electronic device generates an actuator signal to the actuator to output a first haptic effect if the
user input is recognized and a command is determined, confirming receipt of the command.
Otherwise, the mobile electronic device generates an actuator signal to the actuator to output a
second haptic effect. The second haptic effect may indicate that the user input is unrecognized
or that a corresponding command was not found. Further details of the invention are described
in the specification and the claims.
46.
47.
Plus.
6251724
- 17 -
otherwise.
48.
Apple iPhone 6s and Apple iPhone 6s Plus. For example, when a user of the accused devices
attempts to access the "Quick Actions" associated with an application by applying pressure on
the touchscreen, a first haptic effect will be output if the application supports "Quick Actions."
Otherwise, a second haptic effect will be output to the housing of the device, indicating that the
application does not support "Quick Actions." Additional details relating to the accused devices
and their infringement are in the possession of Defendants.
49.
patents
on
its
website
(formerly
located
at

to
Defendants.
See,
e.g.,
("When
it
6251724
- 18 -
50.
others.

others.
See,
e.g.,

51.
6251724
- 19 -
52.
53.
FOURTH CLAIM FOR RELIEF FOR PATENT INFRINGEMENT
(AGAINST DEFENDANT APPLE)
(U.S. Patent No. 7,336,260)
54.
55.
in full.
No. 7,336,260 (the "'260 patent"), entitled "Method and Apparatus for Providing Tactile
Sensations," which was duly issued on Feb. 26, 2008. A copy of the '260 patent is attached as
Exhibit D.
56.
The '260 patent, in general, describes advances in providing haptic feedback in a
variety of electronic devices such as mobile phones and laptop computers that, among other
things, detect pressures and provide tactile sensations. The '260 patent teaches, among other
6251724
- 20 -
things, systems in which the electronic device detects different levels of pressure on the device
and provide tactile sensations in response. Further details of the invention are described in the
specification and the claims.
57.
Apple has infringed and is currently infringing the '260 patent, in violation of 35
U.S.C. 271, by, among other things, making, using, offering for sale, selling, and/or importing
within this judicial district and elsewhere in the United States, without license or authority,
products and/or processes falling within the scope of one or more claims of the '260 patent,
including without limitation the Apple MacBook, Apple MacBook Pro 13-inch, and Apple
MacBook Pro 15-inch.
58.
infringe or are covered by claims of the '260 patent: the Apple MacBook, Apple MacBook Pro
13-inch, and Apple MacBook Pro 15-inch. Immersion makes this preliminary identification of
infringing apparatuses, products, devices, processes, methods, acts, or other instrumentalities
without the benefit of discovery or claim construction in this action, and expressly reserves the
right to augment, supplement, and revise its identifications based on additional information
obtained through discovery or otherwise.
59.
Apple and its customers directly infringe the '260 patent using the Apple
MacBook, Apple MacBook Pro 13-inch, and Apple MacBook Pro 15-inch. The accused devices
include a Force Touch trackpad. The Force Touch trackpad is capable of detecting three or more
different pressure levels associated with a finger touching the trackpad, and the actuator
connected to the Force Touch trackpad can provide tactile sensations based on the three or more
different measured pressure levels.
6251724
- 21 -
60.
Apple also indirectly infringes claims of the '260 patent. Apple is aware of the
Apple was aware of the '260 patent before the date of this Complaint, including without
limitation through Apple's knowledge of Immersion, and Immersion's disclosure of its patents on
its
website
(formerly
located
at

to
Apple.
See,
e.g.,
("When
it
comes to haptics, Immersion is usually the first company that comes to mind."). Apple knew
that the Apple MacBook, Apple MacBook Pro 13-inch, and Apple MacBook Pro 15-inch
infringed the '260 patent, or at a minimum believed there was a high probability that the Apple
MacBook, Apple MacBook Pro 13-inch, and Apple MacBook Pro 15-inch were covered by the
'260 patent, but willfully blinded itself to Immersion's patent and the infringing nature of the
Apple MacBook, Apple MacBook Pro 13-inch, and Apple MacBook Pro 15-inch.
61.
Apple has induced and is currently actively inducing infringement of the '260
patent, in violation of 35 U.S.C. 271(b), by, among other things, actively and knowingly aiding
and abetting others to directly make, use, offer for sale, sell, and/or import within this judicial
district and elsewhere in the United States, without license or authority, products and/or
processes falling within the scope of one or more claims of the '260 patent, including without
limitation the Apple MacBook, Apple MacBook Pro 13-inch, and Apple MacBook Pro 15-inch.
For example, Apple provides directions, instruction manuals, guides, and/or other materials that
6251724
- 22 -
encourage and facilitate infringing use by others. See, e.g., http://www.apple.com/macbook/;

http://www.apple.com/macbook-pro/. Apple has sold and is selling these products with the
knowledge and intent that customers who buy the products will use the products for their
infringing use and therefore that customers have been and are directly infringing the '260 patent.
62.
Apple has contributorily infringed and is currently contributorily infringing the
'260 patent, in violation of 35 U.S.C. 271(c), by, among other things, selling, offering for sale,
or authority, products or components of products which constitute a material part of the '260
patent, knowing that such products and/or components are especially made or especially adapted
for use in the infringement of the '260 patent, and not staple articles or commodities of
commerce suitable for substantial noninfringing use.
63.
Apple's infringement of the '260 patent has been and continues to be willful and
deliberate. Despite knowledge of the '260 patent, Apple has acted and is acting despite an
objectively high likelihood that its actions constitute patent infringement. This objective risk
was and is known to Apple, and is also so obvious that it should have been known to Apple.
64.
As a direct and proximate result of Apple's infringement of the '260 patent,
PRAYER FOR RELIEF
WHEREFORE, Immersion prays for judgment as follows:
A.
That U.S. Patent Nos. 8,749,507; 7,808,488; 8,581,710; and 7,336,260
(collectively, the "Patents-In-Suit") are valid and enforceable;
6251724
- 23 -
B.
That Apple has directly infringed the Patents-In-Suit;
C.
That AT&T has directly infringed U.S. Patent Nos. 8,749,507; 7,808,488; and
8,581,710;
D.
That Apple has induced the infringement of the Patents-In-Suit;
E.
That AT&T has induced the infringement of U.S. Patent Nos. 8,749,507;
7,808,488; and 8,581,710;

F.
That Apple has contributorily infringed the Patents-In-Suit;
G.
That AT&T has contributorily infringed U.S. Patent Nos. 8,749,507; 7,808,488;
and 8,581,710;
H.
That Apple and its subsidiaries, parents, officers, directors, agents, servants,
employees, affiliates, attorneys, and all others in active concert or participation with any of the
foregoing be preliminarily and permanently enjoined from further acts of infringement of the
Patents-In-Suit;
I.
That AT&T and its subsidiaries, parents, officers, directors, agents, servants,
employees, affiliates, attorneys, and all others in active concert or participation with any of the
foregoing be preliminarily and permanently enjoined from further acts of infringement of U.S.
Patent Nos. 8,749,507; 7,808,488; and 8,581,710;
J.
That Defendants be ordered to pay compensatory damages to Plaintiff, together
with pre and post judgment interest;

K.
That Defendants be ordered to provide an accounting;
L.
That Defendants be ordered to pay supplemental damages to Plaintiff, and pre and
post judgment interest thereon;

M.
6251724
That Defendants be ordered to pay Plaintiff's costs;
- 24 -
N.
That the infringement by Defendants be adjudged willful;
O.
That the damages be enhanced under 35 U.S.C. 284 to three times the amount
found or measured;
P.
That this be adjudged an exceptional case and that Plaintiff be awarded their
attorneys' fees, costs, and expenses in this action; and

Q.
That Plaintiff be awarded such other and further relief as the Court may deem
appropriate.
DEMAND FOR JURY TRIAL
Immersion hereby demands trial by jury on all issues.
Dated: May 5, 2016
Respectfully submitted,
Of Counsel:
FARNAN LLP
Morgan Chu
Richard M. Birnholz
Jason G. Sheasby
IRELL & MANELLA LLP
1800 Avenue of the Stars, Suite 900
Los Angeles, CA 90067
Phone: (310) 277-1010
Fax: (310) 203-7199
mchu@irell.com
rbirnholz@irell.com
jsheasby@irell.com
/s/ Brian Farnan

Brian E. Farnan (Bar No. 4089)
Michael J. Farnan (Bar No. 5165)
919 North Market Street
12th Floor
Wilmington, DE 19801
(302) 777-0300
(302) 777-0301
bfarnan@farnanlaw.com
mfarnan@farnanlaw.com
Lisa Glasser
Babak Redjaian
IRELL & MANELLA LLP
840 Newport Center Drive, Suite 400
Newport Beach, CA 92660
Phone: (949) 760-0991
Fax: (949) 760-5200
lglasser@irell.com
bredjaian@irell.com
Attorneys for Plaintiff Immersion Corporation
6251724
- 25 -
Case 1:16-cv-00325-UNA Document 1-1 Filed 05/05/16 Page 1 of 15 PageID #: 26
EXHIBIT A

US008749507B2
(12) United States Patent
(10) Patent N0.:

(45) Date of Patent:
DaCosta et a].
(54)
SYSTEMS AND METHODS FOR ADAPTIVE

INTERPRETATION OF INPUT FROM A
TOUCH-SENSITIVE INPUT DEVICE
(56)
References Cited
2,972,140
3,157,853
3,220,121
3,497,668
3,517,446
Christophe Ramstein, San Francisco,

CA (US); Danny Grant, Laval (CA)
(73) Assignee: Immersion Corporation, San Jose, CA

(Us)
Notice:
Subject to any disclaimer, the term of this

patent is extended or adjusted under 35
(21) Appl. No.: 13/441,108

(22) Filed:
3,903,614 A
3,911,416 A
9/1975 Diamond et al.

10/1975 Feder
4,127,752 A
11/1978 Lowthorp
(Continued)
FOREIGN PATENT DOCUMENTS
EP
EP
0349086
0536715 A2
Aug. 2, 2012
(63)
Continuation of application No. 10/723,778, ?led on

Nov. 26, 2003, noW Pat. No. 8,164,573.
(51)
Int. Cl.
G09G 5/00
US. Cl.
USPC
(58)
1/1990
4/1993
(Continued)
Adel stein, A Virtual Environment System For The Study of Human
Arm Tremor, Ph.D. Dissertation, Dept. of Mechanical Engineering,
MIT, Jun. 1989.
(Continued)
Related US. Application Data
(52)
Hirsch
Hirsch
Cutler
Hirsch
Carlyon et a1.
OTHER PUBLICATIONS
Prior Publication Data
US 2012/0194472 A1
2/1961
11/1964
11/1965
2/1970
6/1970
11/1971 Kagan
9/1975 Hightower
Apr. 6, 2012
(65)
A
A
A
A
A
3,623,064 A
3,902,687 A
U.S.C. 154(b) by 0 days.

This patent is subject to a terminal dis
claimer.
*Jun. 10, 2014
U.S. PATENT DOCUMENTS
(75) Inventors: Henry DaCosta, Montreal (CA);
(*)
US 8,749,507 B2
(2006.01)
...................................... .. 345/173; 178/1801
Field of Classi?cation Search

CPC ..... .. G06F 3/016; G06F 3/017; G06F 3/0487;
G06F 3/0488; G06F 2203/014

USPC ....... .. 345/173,174; 178/1801, 18.03, 18.06;
715/701, 702
See application ?le for complete search history.
Primary Examiner * Regina Liang

(74) Attorney, Agent, or Firm * Kilpatrick Townsend &
Stockton LLP
(57)
ABSTRACT
Systems and methods for adaptively interpreting a users

intent based on parameters supplied by a touch-sensitive
input device are described. In one of the methods described, a
processor is programmed for receiving contact data from an
input device; determining Whether to output a haptic effect

based on the contact data; and outputting the haptic effect
based on the contact data.
18 Claims, 4 Drawing Sheets
Processor
>
Various
Interfaces
US 8,749,507 B2
Page 2
References Cited
6,307,465
6,326,901
6,337,678
6,373,463
B1
B1
B1
B1
4,160,508 A
7/1979 831is
4,236,325 A
12/1980 Hall et al
6,388,999 B1
4,262,549
4,333,070
4,464,117
4,484,191
4,513,235
4,581,491
4,599,070
4,708,656
4/1981
6/1982
8/1984
11/1984
4/1985
4/1986
7/1986
11/1987
6,414,674
6,422,941
6,429,846
6,445,284
6,469,695
6,487,421
6,492,979
6,509,847
A
A
A
A
A
A
A
4,713,007
4,758,165 A
4,772,205 A
4,794,392
4,885,565 A
4,926,879
4,930,770
4,934,694
5,019,761
A
A
A
A
6,388,655 B1
Schwellenbach
Barnes
Forest
Vavra
Acklam e131.
Boothroyd
Hladky etal,
De Vries e131.
12/1987
7/1988 Tieman et al
9/1988 Chlumsky e131.

12/1988
12/1989 Embach
Selinko
5/1990
6/1990
6/1990
5/1991
Sevrain e131.
Baker
Mcintosh
Kraft
B1
B1
B2
B1
B1
B2
B1
B1
2/2003
1/2003 Cooper et et 31.
6,590,568
6,610,917 B1
B2
7/2003
8/2003 Ludwig
Astala et
B1
B2
B1
B1
6/1991 Freels
6,730,863 B1
6/1991 Horch et al
6,801,191 B2
5,035,242 A
7/1991
7,030,860 B1
5,038,089
5,078,152
5,121,091
5,159,159
5,165,897
5,175,459
5,186,695
5,212,473
5,240,417
8/1991 Szakaly
1/1992 Bond
6/1992 Fujiyama
A
A
5/1993 Louis
8/1993 Smithson e131.
7,046,235
2001/0035854
2002/0033795
2002/0149561
2002/0149570
2002/0156807
2003/0006892
2003/0016211
2003/0022701
2/1994
1/1994 Cook
Aigner
Pierce
Corder
Everett, Jr et al
Wherlock
Asano et 31
Gutman et al
Taylor
Wang et 31
Hogan
Mead
Yamagaki
2003/0048260
2003/0058265
2003/0063073
2003/0067449
2003/0071795
2003/0095105
2003/0128191
2003/0128192
2003/0151597
2003/0174121
2003/0179190
10/1992
11/1992
12/1992
2/1993
Asher
Johnson
Danial etal,
Mangseth et al
5,299,810
5,302,132
5,309,140
5,334,027
5,389,849
5,436,622
5,437,607
5,461,711
5,466,213
5,488,204
5,547,382
5,275,174
5,283,970 A
A
A
A
A
A
A
A
A
A
A
A
4/1994
4/1994
5/1994
8/1994
2/1995
7/1995
8/1995
10/1995
11/1995
1/1996
8/1996
5,575,761 A
5,600,777 A
11/1996 Hajianpour
2/1997 Wang et a1.
5,638,060
5,719,561
5,736,978
5,766,016
5,785,630
A
A
A
A
A
5,880,411 A
5,887,995
5,889,511
5,917,906
5,977,867
6,008,800
6,067,081
6,081,536
6,111,577
A
A
A
A
A
A
A
A
6,118,435 A
6,128,007 A
6,140,987 A
6,151,332 A
6/1997
2/1998
4/1998
6/1998
7/1998
Kataoka et al.
Gonzales
Hasser et al.
Sinclair
Bobick et al.
3/1999 Gillespie et al. ............ .. 345/173
3/1999
3/1999
6/1999
11/1999
12/1999
5/2000
6/2000
8/2000
Holehan
Ong
Thronton
Blouin
Pryor
Hahlganss et al.
Gorsuch et al.
Zilles et al.
9/2000 Fujita et al. ................. .. 345/173
10/2000 Seybold
10/2000 Stein et al.
11/2000 Gorsuch et al.
6,154,210 A
11/2000
6,160,489 A
6,198,206 B1
12/2000 Perry et al.

3/ 2001 Saarmaa et al.
6,219,034
6,236,647
6,243,080
6,262,717
B1
B1
B1
B1
4/2001
5/2001
6/ 2001
7/2001
K21HIPeret 4
Thorner
Rosenberg
et et 31.
Cruz-Hernandez et al.
Whlte
Hess et 31.
Kent 61 al.
Anderson
3/2003 CPeretal~
5,022,407 A
A
A
A
A
5/2002 Gorsuch et 31.
7/2002
7/2002
8/2002
9/2002
10/2002
11/2002
12/2002
V2003
6,509,892
6,518,958 B1
5,022,384 A
A
A
A
Kayrna et 31.
Gonzales
Fish
Beeks
5/2002 Leung
6,535,201 B1
6,628,195
6,636,202
6,639,582
6,647,145
Franklin
10/2001
12/2001
1/2002
4/2002
B2
A1
A1
9/2003
10/2003
10/2003
11/2003
5/2004 Gerpheide e131. .......... ~~ 345/174
10/2004
et 31.
4/2006 Hsll e131. ~~~~~~~~~~~~~~~~~~~~ ~~ 345/173
A1
A1
5/2006
11/2001
3/2002
10/2002
10/2002
10/2002
1/2003
1/2003
V2003
2003/0030628
2003/0038776 A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
3/2003
3/2003
4/2003
4/2003
4/2003
5/2003
7/2003
7/2003
8/2003
9/2003
9/2003
A1
A1
A1
A1
Coudon
Ishmael, Jr. et 31.
Shrader
Gay
Katoh
Rosenberg et 31.
Shahoia 61 al.
Fukumoto et 31.
Knowles et 31.
Dieberger
Church
Woolley
Gupta
2/2003 Rosenberg
S310 et
et 31.
Matusis
Robinson et 31.
Geaghan et 31.
Yoshikawa et al.
Baldauf et al.
Vaananen
Strasser et 31.
van Os
Roberts et 31.
Poupyrev et 31.
Franzen

EP
EP
JP
JP
0556999 B1
1182851 A1
01-003664
02-109714
5/1998
2/2002
7/1990
1/1992
JP
04-007371
8/1993
JP
JP
JP
JP
JP
JP
JP
JP
06-282369
05-193862
08-044493
09-016318
11506559
2001-350592 A
2002-259059 A
2003-337659
10/1994
1/1995
2/1996
1/1997
6/1999
12/2001
9/2002
11/2003
W0
WO 97/18546 A1
5/1997
W0
W0
W0
WO 02/12991 A1
WO 02/27645
WO 02/31807 A1
2/2002
4/2002
4/2002
Anderson ................... .. 345/173
OTHER PUBLICATIONS
162151331121; 31
Adel-stein, Design and Implementation of anForce Re?ecting
Elbing et al
Amal?tano
Moine
Donohue et al.
Manipulandum for Manual Control research, DSC- vol. 42,

Advances in Robotics, Edited by H. Kazerooni, pp. 1-12, 1992.
Aukstakalnis et al., Silicon Mirage: The Art and Science of Virtual
Reality, ISBN 0-938151-82-7, pp. 129-180, 1992.
US 8,749,507 B2
Page 3
(56)
References Cited
Kaczmarek et al., Tactile Displays, Virtual Environment Technolo
OTHER PUBLICATIONS
Kontarinis et al., Tactile Display of Vibratory Information in
gies.
Baigrie, Electric Control Loading:A Low Cost. High Performance
Alternative, Proceedings, pp. 247-254, Nov. 6-8, 1990.
Bejczy et al. A Laboratory Breadboard System For Dual-Arm
Teleoperation, SOAR 89 Workshop, J SC, Houston, TX, Jul. 25-27,
Teleoperation and Virtual Environments, Presence, 4(4):387-402,

1995.
Lake, Cyberman from Logitech, GameBytes, l994.

Kontarinis et al., Display of High-Frequency Tactile Information to
1989.
Teleoperators, Telemanipulator Technology and Space Telerobot
Bejczy et al., Kinesthetic Coupling Between Operator and Remote
ics, Won S. Kim, Editor, Proc. SPIE vol. 2057, pp. 40-50, Sep. 7-9,
Manipulator, International Computer Technology Conference, The
1993.
American Society of Mechanical Engineers. San Francisco, CA,

Aug. 12-15, 1980.
Bejczy, Sensors, Controls, and Man-Machine Interface for
Advanced Teleoperation, Science, vol. 208, No. 4450, pp. 1327
1335, 1980.
Bejczy, et al., Universal Computer Control System (UCCS) For
Marcus, Touch Feedback in Surgery. Proceedings of Virtual Real

ity and Medicine The Cutting Edge, Sep. 8-1 l, 1994.
Space Telerobots. CH 2413-3/87/0000/031650100 1987 IEEE,

1987.
Bejczy, Generalization of Bilateral Force-Re?ecting Control of
Manipulators, Proceedings Of Fourth CISM' IFToMM, Sep. 8-l2,

1981.
Bliss, Optical-to-Tactile Image Conversion for the Blind, IEEE

Transactions on Man- Machine Systems. vol. MMS-ll, No. 1, Mar.
1970.
Brooks et al., Hand Controllers for Teleoperation- A State-of-the
Art Technology Survey and Evaluation, JPL Publication 85-ll;

NASA-CR-l75890; N85-28559, pp. 1-84, Mar. 1, 1985.
Burdea et al., Distributed Virtual Force Feedback, Lecture Notes for
Workshop on Force Display in Virtual Environments and its Appli
cation to Robotic Teleoperation, 1993 IEEE International Confer
ence on Robotics and Automation, pp. 25-44, May 2, 1993.
Cadler, Design of A Force-Feedback Touch-Introducing Actuator

For Teleoperator Robot Control, Bachelor of Science Thesis, MIT,
Jun. 23, 1983.
Caldwell et al., Enhanced Tactile Feedback (Tele-Taction) Using a
Multi-Functional Sensory System, 1050-4729/93, pp. 955-960,

1993.
Cyberman Technical Speci?cation, Logitech Cyberman SWIFT

Supplement, Apr. 5, 1994.
Eberhardt et al., Including Dynamic Haptic Perception by The
Hand: System Description and Some Results, DSC-vol. 55-1,
Dynamic Systems and Control: vol. I, ASME l994.
Eberhardt et al., OMAR-A Haptic display for speech perception by
McAffee, Teleoperator Subsystem/Telerobot Demonsdtrator: Force

Re?ecting Hand Controller Equipment Manual, JPL D-5172, pp.
1-50, Al-A36, 8l-B5, Cl-C36, Jan. 1988.
Minsky. Computational Haptics: The Sandpaper System for Syn

thesiZing Texture for a Force-Feedback Display, Ph.D. Dissertation.
MIT, Jun. 1995.
Ouh-Young et al., The Development of A Low-Cost Force Feedback
Joystick and Its Use in the Virtual Reality Environment, Proceedings

of the Third Paci?c Conference on Computer Graphics and Applica
tions. Paci?c Graphics 95, Seoul, Korea, Aug. 21-24, 1995.
Ouh-Young, A Low-Co st Force Feedback Joystick and Its Use in PC
Video Games, IEEE Transactions on Consumer Electronics, vol. 4 I,
No. 3, Aug. 1995.

Ouh-Young,Force Display in Molecular Docking, Order No.
9034744, p. 1-369, 1990.
Patrick et al., Design and Testing of A Non-reactive. Fingertip,
Tactile Display for Interaction with Remote Environments, Coop
erative Intelligent Robotics in Space, Rui J. deFigueiredo et al.,
Editor, Proc. SPIE vol. 1387, pp. 215-222, 1990.
Patrick, Design, Construction, and Testing of a Fingertip Tactile
Display for Interaction with Virtual and Remote Environments,
Master of Science Thesis, MIT, Nov. 8, 1990.
Pimentel et al., Virtual Reality: through the new looking glass, 2"
Edition; McGraw-Hill, ISBN 0-07-050167-X, pp. 41-202, 1994.
RabinowitZ et al., Multidimensional tactile displays: Identi?cation
of vibratory intensity, frequency, and contactor area, Journal of The
Acoustical Society ofAmerica, vol. 82, No. 4, Oct. 1987.
Russo,Controlling Dissipative Magnetic Particle Brakes'in Force

Re?ective Devices, DSC-vol. 42, Advances in Robotics, pp. 63-70,
deaf and deaf-blind individuals, IEEE Virtual Reality Annual Inter
ASME 1992.
national Symposium, Seattle, WA, Sep. 18-22, 1993.
Russo, The Design and Implementation of a Three Degree of Free

dom Force Output Joystick, MIT libraries Archives Aug. 14, 1990,
pp. 1-131, May 1990.
Safe Flight Instruments Corporation, Coaxial Control Shaker, Part
Fokumoto, Active Click: Tactile Feedback For Touch Panels, ACM
CHI2001, Extended Abstracts, pp. 121-122, Apr. 2001.

Force Feedback Touch Panel, Represented by CSC Division, Sales
Department, SIXIK Corporation, Tokyo, Japan, www.smk.co.jp.

Gobel et al., Tactile Feedback Applied to Computer Mice, Interna
tional Journal of Human-Computer Interaction, vol. 7, No. I, pp.
1-24, 1995.
Gotow et al.,Controlled Impedance Test Apparatus for Studying
Human Interpretation of Kinesthetic Feedback, WAll-ll:00, pp.
332-337.
Howe. A Force-Re?ecting Teleoperated Hand System for the Study

of Tactile Sensing in Precision Manipulation, Proceedings of the
1992 IEEE International Conference on Robotics and Automation,
Nice, France, May 1992.

IBM Technical Disclosure Bulletin. Mouse Ball-Actuating Device
With Force and Tactile Feedback, vol. 32, No. 98, Feb. 1990.
No. C-25502, Jul. 1, 1967.

Scannell. Taking a Joystick Ride, Computer Currents. Boston Edi
tion. vol. 9, No. 11. Nov. 1994.
Shimoga, Finger Force and Touch Feedback Issues in Dexterous
Telemanipulation, Proceedings of Fourth Annual Conference on
Intelligent Robotic Systems for Space Exploration, Rensselaer Poly

technic Institute, Sep. 30-Oct. l, 1992.
Snow et al.,Model-X Force-Re?ecting-Hand-Controller, NT Con
trol No. MPO-l7851; JPL Case No. 5348, pp. 1-4, Jun. 15, 1989.
SMK Corporation, Multi-Functional Touch Panel, Force-Feedback
Type, Developed: A Touch Panel Providing a Clicking Feeling,

http://www.smk.cojp/whatsnewe/628csc.com/html, Sep. 30, 2002.
Iwata, Pen-based Haptic Virtual Environment, 0-7803-l363-l/93

IEEE, pp. 287-292, 1993.
Jacobsen et at., High Performance, Dextrous Telerobotic Manipu
lator With Force Re?ection, Intervention/ROV9l Conference &
SMK Corporation, Force Feedback Type Optical Touch Panel

Developed, SMK Corporation Website, Oct. 30, 2002.
Stanley et al., Computer Simulation of Interacting Dynamic
Mechanical Systems Using Distributed Memory Parallel Proces
Exposition, Hollywood, Florida. May 21-23, 1991.

Johnson, Shape-Memory Alloy Tactile Feedback Actuator,
Armstrong Aerospace Medical Research Laboratory, AAMRL-TR
90-039, Aug. 1990.
sors, DSC-vol. 42, Advances in Robotics, pp. 55-61. ASME l992.
Synaptics, Inc., Synaptics Touchpad Interfacing Guide, Second

Edition, Downloaded Aug. 26, 2003.
Tadros, Control System Design for a Three Degree of Freedom
Springer International (Springer-Verlag); Experimental Brain
Virtual Environment Simulator Using Motor/Brake Pair Actuators,

MIT Archive Massachusetts Institute of Technology, pp. 1-88, Feb.
Research, vol. 79, No. I, pp. 150-156, 1990.
1990.
Jones et al., A perceptual analysis of stiffness, ISSN 0014-4819
US 8,749,507 B2
Page 4
(56)
References Cited
OTHER PUBLICATIONS
Terry et al., Tactile Feedback In A Computer Mouse, Proceedings

of Fourteenth Annual Northeast Bioengineering Conference. Univer
sity ofNeW Hampshire, Mar. 10-11, 1988.

Wiker, Teletouch Display Development: Phase 1 Report, Technical
Report 1230, Naval Ocean Systems Center, San Diego, Apr. 17, 1989.
European Patent Of?ce, European Search Report, European Appli

cation No. 10010754, dated Nov. 8, 2010.
Japanese Patent Of?ce, Notice of Reasons for Rejection, Application

No. 2006-541126, dated Oct. 13, 2009.

No. 2006-541126, dated Oct. 7, 2008.
European Patent Of?ce, Supplemental European Search Report,

European Application No. 04779422, dated Jan. 22, 2007.
European Patent Of?ce, Communication Pursuant to Article 96(2)
EPC Application No. 04779422, dated Aug. 22, 2007.
European Patent Of?ce, Communication Pursuant to Article 96(2)
EPC, Application No. 04779422, dated Nov. 9, 2010.
Chinese State Intellectual Property Of?ce, Of?ce Action, Application
No. 200480040718, dated Dec. 25, 2007.
No. 200480040718, dated Sep. 11,2008.
No. 200480040718, dated Mar. 25, 2009.
German Patent and Trademark Of?ce, Of?ce Action, Application No.
11 2004 000918, dated Apr. 20, 2012.

No. 2010-091566, dated Mar. 13, 2012.
* cited by examiner
US. Patent
Jun. 10, 2014
Sheet 1 0f 4
US 8,749,507 B2
$253, $0thc
on
Hemwo i
.05H
US. Patent
Jun. 10, 2014
Sheet 3 0f 4
US 8,749,507 B2
m.05
gm
US. Patent
Jun. 10, 2014
Sheet 4 0f4
US 8,749,507 B2
US 8,749,507 B2
1
SYSTEMS AND METHODS FOR ADAPTIVE

INTERPRETATION OF INPUT FROMA
TOUCH-SENSITIVE INPUT DEVICE
Thus, a method and system are needed for accurately deter

mining a users intent based on data supplied by a touch
CROSS-REFERENCE TO RELATED
APPLICATIONS
SUMMARY
sensitive input device.
An embodiment of the present invention provides systems

and methods for adaptive interpretation of input received
This application is a continuation of US. patent applica

tion Ser. No. 10/723,778, ?led Nov. 26, 2003, entitled Sys
tems and Methods for Adaptive Interpretation of Input from a
Touch-Sensitive Device, now US. Pat. No. 8,164,573, the
from a touch-sensitive input device by receiving a pressure
signal indicating a pressure from the input device, comparing

the pseudo pressure signal to an adaptive pressure threshold
value, and outputting a signal if the pseudo pressure signal is

greater than the adaptive pressure threshold value. Further
details and advantages of embodiments of the present inven
entirety of which is hereby incorporated by reference.

NOTICE OF COPYRIGHT PROTECTION
tion are set forth below.
A section of the disclosure of this patent document and its

BRIEF DESCRIPTION OF THE FIGURES
?gures contain material subject to copyright protection. The

copyright owner has no objection to the facsimile reproduc
tion by anyone of the patent document, but otherwise reserves
all copyright rights whatsoever.
These and other features, aspects, and advantages of the

20
present invention are better understood when the following

Detailed Description is read with reference to the accompa
nying drawings, wherein:
FIELD OF THE INVENTION
ticularly relates to adaptive interpretation of input received
FIG. 1 illustrates an exemplary environment for implemen

tation of one embodiment of the present invention;
FIG. 2 is a ?ow chart illustrating a process or algorithm for
detecting ?nger presses on a touchpad in one embodiment of
from a touch-sensitive input device.
the present invention;
The present invention generally relates to receiving input

from a touch-sensitive input device. This invention more par
25
FIG. 3 is a ?ow chart illustrating a process for detecting a

?nger press on a touchpad in another embodiment of the
BACKGROUND
30
A variety of input devices may be used to provide position
FIG. 4 is a group of charts illustrating various ?lters that
and control data to programs executing on computers, cell
may be utilized in embodiments of the present invention.
phones, and other processor-equipped devices. These input

devices include mice, trackballs, touchpads, touch screens,
touch panels, and various other devices. While the mouse and
present invention; and
DETAILED DESCRIPTION
35
trackball provide distinct control elements for performing

positioning and other control actions, the touchpad combines
positioning and control.
Referring now to the drawings in which like numerals

indicate like elements throughout the several ?gures, FIG. 1
illustrates an exemplary environment for implementation of
a control function, such as selecting a graphical representa
an embodiment of the present invention. The embodiment

shown includes a touch-sensitive device commonly called a
touchpad 102. Touchpad 102 senses the position of a conduc
tor, such as a ?nger, on the surface of the touchpad (102). The
tion on a screen. In these systems, a users intent to make a
touchpad (102) is further able to provide a position, compris
For example, a conventional mouse includes a ball or opti
cal sensor for determining changes in position of the mouse.
40
The mouse also includes one or more buttons for performing
positional change or provide control input is apparent to the
system.
ing X andY parameters, as well as a pressure, Z parameter, as

45
an output signal. Conventional touchpads are very accurate in
In contrast, conventional touchpads combine the position
determining and providing the position of the conductor. For
and control functionality in a way that often masks the user s
50
example, some conventional touchpads have resolutions

greater than 1000 dpi. However, conventional touchpads are
less accurate in determining and providing the pressure
exerted on the touchpad. Other embodiments of the present
intent to make a positional change to provide control input. A

user moves a ?nger along a touchpad to reposition a cursor. A
user may also perform gestures to simulate functions of the

buttons of a mouse, such as drag, click, and double-click. In
either case, the users ?nger is in contact with the surface of
invention may use other touch- sensitive input devices, such as

a touch panel or touch screen.
the touchpad. Changes in position on the touchpad and in the
The touchpad 102 shown does not sense an actual pressure.
pressure exerted on the surface of the touchpad must be used

to determine the user s intent. Because of the variety of users
that may interact with a touchpad and the variety of functions
that may be performed, determining the user s intent based on
a gesture on a touchpad is dif?cult. Variables affecting the
ability of a program to determine what a user is attempting to
Instead, the pressure reading from the touchpad 102 is a
do include the following: the physical difference between
55
pseudo pressure. Touchpads work by utilizing resistance,

capacitance, or membrane switches. The touchpad 102 shown
in FIG. 1 utilizes capacitance, however, an embodiment of the
60
present invention may be implemented in conjunction with

any touch-sensitive input device, including resistive and
membrane-switch touchpads. In other embodiments, actual
users; the different angles at which a user may place their
pressure may be sensed. For example, in one embodiment, a
?nger while using a touchpad; the variance in pressure
touch screen with an attached explicit pressure sensor is uti
between different users and between the same user; the move
lized.
Capacitance-based touchpads are well known to those
ment of the ?nger across the touchpad while simultaneously

attempting to perform actions on the touchpad. US. Pat. No.
6,414,671 to Gillespie, et al. describes one conventional
method for recognizing a users gesture as a drag gesture.
65
skilled in the art, and therefore, only a basic description of
their function is provided herein. A capacitance touchpad,

such as touchpad 102 shown in FIG. 1, includes two sets of
US 8,749,507 B2
3
wires, which are perpendicular to one another and con?gured
connected via wires or a wireless connection. For example,
so that a gap is formed between them. When a user places a
the touchpad 102 may utilize the PS/ 2, Serial, Apple Desktop
conductor, such as a ?nger, on the touchpad 102, wires of the

two perpendicular sets are brought together and form a
capacitance. The touchpad 102 measures which of the wires
in each of the two sets has the most capacitance to determine
Bus (ADB), or other communication protocol in communi

cating with the processor. The processor 106 is capable of
executing program code stored on a computer-readable
medium. Although the processor shown is separate from the

touchpad 102, some conventional touchpads include a pro
cessor, such as an Application Speci?c Integrated Circuit
(ASIC). AnASIC may provide some processing of the move
where the conductor is touching the touchpad 102 and, based

on this information, provides the X andY coordinates of the
position of the conductor on the touchpad 102.
The touchpad 102 also provides a pseudo pressure, Z. The
pseudo pressure is based on the amount of capacitance result
ments on the touchpad 102 to determine whether or not the
user is making gestures. This integrated processor may be
ing from the conductor touching the touchpad 102. Accord
utilized alone or in combination with the processor 106
ingly, the amount of capacitance is not a direct measure of

pressure but rather a pseudo pressure.
In other words, the pseudo pressure or Z parameter pro
vided by the touchpad 102 is not a measure of the actual
vertical displacement by a conductor at a single point on the
touchpad 102, but rather an estimation of the vertical dis
according to the present invention.

Processor 106 may include, for example, digital logic pro
placement based on the size of the capacitance change. The

pseudo pressure may not accurately represent the amount of
pressure actually exerted on the touchpad 102. For example,
the larger the surface of the conductor used on the touchpad
102, e. g., a users ?nger, the larger the change in capacitance
cessors capable of processing input, executing algorithms,

and generating output as necessary in response to the inputs
received from the touch-sensitive input device. Such proces
sors may include a microprocessor, the aforementioned
ASIC, and state machines. Such processors include, or may
20
readable media, which stores instructions that, when

executed by the processor 106, cause the processor 106 to
perform the steps described herein.
per amount of pressure exerted. As would be expected, if a
user presses heavily against the touchpad 102 with a ?eshy

part of the ?nger, the amount of touchpad 102 area covered by
the ?nger is greater than then when the same part of the ?nger
is touching lightly. However, what is less obvious is that the
area covered, and the corresponding pseudo pressure, is also
greater than when the user presses heavily with a bony part of
be in communication with, media, for example computer
25
Embodiments of computer-readable media include, but are

not limited to, an electronic, optical, magnetic, or other stor
age or transmission device capable of providing a processor,
such as the processor 106 in communication with a touch
sensitive input device, with computer-readable instructions.

Other examples of suitable media include, but are not limited
30
a ?nger.
to, a ?oppy disk, CD-ROM, magnetic disk, memory chip,

ROM, RAM, an ASIC, a con?gured processor, all optical
media, all magnetic tape or other magnetic media, or any
Additionally, the difference in the features of different

conductors, for instance the size or makeup of different users
other medium from which a computer processor can read
?ngers, affects the capacitance change for any given change
instructions. Also, various other forms of computer-readable
in pressure. For example, if a ?rst user with a large ?nger
35
media may transmit or carry instructions to a computer,
the pseudo pressure signal output by the touchpad 102 is
including a router, private or public network, or other trans

mission device or channel, both wired and wireless. The
greater for the ?rst person than for the second person for the
instructions may comprise code from any computer-pro
applies the same pressure as a second user with a small ?nger,
same amount of applied pressure.
The dif?culty in determining a users intent by evaluating

the data provided by the touchpad 102 is compounded by the
different ways in which a conductor may be utilized. For
example, the pressure exerted across the surface of the touch
pad may vary as the users ?nger moves in relation to the
hand. The users ?nger covers a larger area of the touchpad
40
The embodiment shown in FIG. 1 may be implemented in

a variety of devices. Such devices include personal comput
ers, many of which include an integrated touchpad. Such
devices may also include handheld devices, such as handheld
45
when the ?nger is extended horizontally away from the hand

on the touchpad 102 than when the ?nger is close to the hand.
Similarly, a pointing device held vertical in relation to the
50
Referring again to FIG. 1, the touchpad 102 transmits the

X, Y, and Z parameters 104 to a processor 106. The touchpad
102 in various embodiments of the present invention may be
capable of sending several types of coordinate information.

For example, a Synaptics TouchPad is able to send either
relative or ab solute coordinates. Relative coordinates provide
the movement of the conductor on the touchpad 102 since the
55
60
Referring again to FIG. 1, the processor 106 and touchpad

102 may be connected directly or indirectly and may be
device in communication with the touch-sensitive input

device.
Embodiments of the present invention address the dif?cul
ties faced in attempting to determine the intent of a user based
when a user is tapping or pressing on a speci?c portion of a
touch-sensitive input device that corresponds to a control

displayed on the input device or displayed on a separate,
synchronized display.
acter of a contact with the touchpad, such as accidental. An

embodiment of the present invention may utilize such a
parameter to accurately determine a users intent.
to implement haptic effects in devices such as those men

tioned above. In such an embodiment, the haptic effects result
from various actions by a user interfacing with a touch-sen
sitive input device, and the effects may be based on the user s
intent as determined by the processor 106. Haptic effects may
also result from interaction with software executing on a
on the X, Y, and Z parameters supplied by the touchpad 102.

Examples of determining a users intent include determining
last coordinates were transferred. Absolute coordinates pro
vide the position of the conductor on the touchpad 102 at that

moment. An embodiment of the present invention may utilize
additional parameters as well. For example, the Synaptics
TouchPad provides a W parameter, which reports the char
organizers, cellular telephones, handheld communicators,

MP3 players, GPS receivers, and the like.
Embodiments of the present invention may also be utilized
touchpad 102 may cover a smaller surface area than one held
at an angle to the touchpad 102.
gramming language, including, for example, C, C#, Visual

Basic, Java, and JavaScript.
Embodiments of the present invention provide systems and

65
methods for adaptive interpretation of the intent of a user of a

touch-sensitive input device. In one embodiment of the
present invention, a processor receives a pressure signal indi
US 8,749,507 B2
5
eating a pressure from the input device, compares the pres
the user s ?nger is touching. If not, the algorithm repeats step
sure signal to an adaptive pressure threshold value, and out
204 until a ?nger is detected. If a ?nger is on the touchpad

(102), the processor (106) determines whether a ?nger was
puts a signal if the pressure signal is greater than the adaptive
previously on the touchpad (102) 206. The processor (106)

may accomplish this in several ways. For example, the pro
pressure threshold value. The pressure may be a pseudo pres

sure or an explicit pres sure. Also, the pres sure may be ?ltered.
Embodiments of the present invention may also utilize the

velocity of the conductor across the touchpad in determining
cessor (106) may store the current or previous state of the
touchpad (102) in memory and from that data, deduce

whether the ?nger was previously on the touchpad (102).
a users intent. Additionally, an embodiment may utilize
adaptive thresholds alone or in combination with digital ?l
If the ?nger was not previously on the touchpad (102), the
tering to more accurately determine a users intent.
processor (106) starts a ?rst tick count 208. The ?rst tick
Thresholds for pressure, pseudo pressure, pseudo-pressure
count is used to determine the length of time the ?nger

remains on the key and is used in other parts of the algorithm
for gesture recognition. If the ?nger was not on the touchpad
(102) or after the ?rst tick count is started, the processor (106)
determines where the ?nger is positioned 210. The processor
(106) makes this determination based on the X andY coordi
change, velocity, and other measures may be stored in a
computer-readable medium when the device is manufac

tured. Alternatively, software executed by a processor may
provide settings for the thresholds. Thresholds set by soft

ware may be static or adaptive. Adaptive thresholds may rely
on various parameters, including, for example, the length of

time the input device has been active, the placement of the
nates provided by the touchpad (102).
conductor on the surface of the input device, and the current
the coordinates to determine whether the ?nger is on a key

212. Each key displayed on the touchpad (102) or corre
In the embodiment shown, the processor (1 06) then utilizes
user of the device.

20
sponding display is associated with numerous attributes.
FIG. 2 is a ?ow chart illustrating a process or algorithm for
detecting and interpreting ?nger presses on the touchpad

(102) according to the present invention. In the embodiment
These attributes include characteristics of the key, such as the
size, position and behavior of the key. The processor (106)

determines if the ?nger is on the key by comparing the X and
shown, a keypad is displayed on the touchpad (102) or on a
corresponding display. In various embodiments, the keypad
25
Y position data reported by the touchpad (102) to the charac
may be virtual or physical, and may be displayed or not

displayed. A processor executing the process shown com
teristics of the key. If the ?nger is not on a key, the processor
pares the pseudo pressure against a minimum threshold value
is on a key, the processor (106) determines whether the

release tick count has elapsed 214. If the release tick count has
(106) repeats the process beginning at step 204. If the ?nger
and compares changes in pseudo pressure against additional

minimum thresholds.
The processor (106) may use adaptive thresholds. For
30
not elapsed, then the processor (106) repeats the process

beginning at step 204. If the release tick count has elapsed, the
example, the processor (106) may utilize different threshold
processor (106) determines whether or not the ?rst tick count
values based on the position of the conductor on the touchpad
has elapsed 216.
(102). The processor (106) may also vary the thresholds based
on the speci?c user who is touching the touchpad (102). The
35
processor (106) may also vary the threshold when the user
initially touches the touchpad (102) to account for the large
change in pseudo pressure typically encountered during ini

tial contact. For example, in one embodiment, the processor
(106) varies the thresholds during the ?rst one-half second

that a pseudo pressure is detected, because the pseudo pres
sure value tends to vary drastically during the ?rst one-half
second of input. The variances may be based on activity of the
user or upon the passage of time. The processor (106) may
update the threshold stored in memory, store a separate set of
40
then compared to the change in pseudo pressure 222. If the

change inpseudo pressure does not exceed the threshold set in
steps 218 and 220, the process repeats beginning at step 204.
If the change in pseudo pressure exceeds the threshold value,
the pseudo pressure, i.e., the current value of Z, is compared
to an absolute threshold 224. If the pseudo pressure does not
exceed the absolute threshold, the process repeats beginning

45
adaptive thresholds, or calculate and apply the adaptive
at step 204. If the pseudo pressure exceeds the absolute

threshold, then the processor (106) determines that the user is
pressing the key 226. The processor (106) generates and
thresholds on an ongoing basis.

In one embodiment in which an adaptive threshold is based
on a speci?c user, the processor (106) executing the software

is able to identify the user. A user identi?er is stored in a
In the embodiment shown, if the ?rst tick count has

elapsed, the threshold is set to the move threshold for the key
218. If the ?rst tick count has not elapsed, the threshold is set
to the ?rst threshold for the key 220. The threshold value is
50
sends a signal indicating that a press has been made. This

signal is used by other software to control the ?ow of a
program. For example, a word processing program may
computer-readable medium, and retrieved based on input
receive the signal, and in response, display a number, high
received from a user, such as a user name, password, or other
light a word, or perform some other action.

Once the determination that a press has occurred is made,
the processor (106) starts the release tick count 228, and the
user identi?er. Multiple user identi?ers and threshold sets
may be stored. The threshold may also depend on the orien

tation of the users grip on a device used for pointing. For
55
grip orientation on the stylus.

Referring again to FIG. 2, the processor (106) executes
whenever the touchpad (102) is active 202. The touchpad
(102) reports data continuously to the processor (106) at
approximately 80 Hz. The processor (106) receives this data
process repeats beginning at step 204. As described above, the

process continues to iterate for as long as the touchpad (102)
is active.
In the process shown in FIG. 2, the ?rst tick count is set
when the ?nger goes from a non-touching to a touching state
example, a stylus may incorporate a sensor to sense a users
60
and is used to measure a time interval during which a different
and uses it to determine the users intent based on gestures
(higher) set of thresholds is used because users typically push

harder when they ?rst touch a touchpad (102). The release
made on the touchpad (102). The processor (106) ?rst deter
tick count is used to measure a time interval following the
detection of a press during which the ?nger is deemed to be
mines whether or not a ?nger or other conductor is on the
touchpad (102) 204. The processor (106) determines that the
?nger is on the touchpad (102) by evaluating the pseudo

pressure (Z) parameter. If the Z parameter is greater than zero,
65
pressing. During this interval, the processor (106) does not

perform further press detection. In other words, the user can
not press again if the user is already pressing and the user
US 8,749,507 B2
7
cannot press any faster than some predetermined rate. Once

the release tick count expires, even if the user is still pressing
hard, the algorithm detects a press if the user presses even
harder (provided there is still room to press harder). The use
cycle (s:\/(AX)2+(AY)2), is also ?ltered and then compared

against a maximum speed threshold. Until the speed falls
below the maximum speed threshold, the processor (106) will
not recognize a press.
of these tick counts provides for the adaptability of the algo
In the embodiment shown in FIG. 3, a processor (106)

executing program code ?rst compares the pseudo pressure to
rithm.
FIG. 3 is a ?ow chart illustrating another process according
to the present invention for detecting a ?nger press on a
an upper threshold value 302. If the pseudo pressure exceeds

the upper threshold value, the process continues at step 314. If
not, the processor (106) determines whether the user was
touchpad (102). Similar to the embodiment shown in FIG. 2,
previously touching, for example by checking the value of a
in the process shown in FIG. 3, a processor (106) compares

the pseudo pressure against a minimum threshold value and
compares the change in pseudo pressure against a minimum
threshold value. Also similar to the process above the thresh
olds may vary depending on where the ?nger touches the
stored ?ag 304. If so, the processor (106) compares the

pseudo pres sure to a lower threshold value 3 06. If the user was
not previously touching or if the pseudo pressure is equal to or

below the lower threshold, the processor (106) determines
whether the ?rst tick counter has elapsed 308.
touchpad (102).
However, the process shown in FIG. 3 differs from the
process shown in FIG. 2 in several ways. In the embodiment
shown in FIG. 2, the processor (106) compares the pseudo

pressure against both lower and upper thresholds to determine
20
whether the ?nger is touching. If the ?nger was not previously
touching the touchpad (102), the processor (106) requires that

the pseudo pressure exceed the upper threshold before the
processor (106) can conclude that the ?nger is currently
touching the touchpad (102). If the ?nger was previously

touching the touchpad (102), the processor (106) requires that
25
30
unwanted noise, which results from extraneous contact with
the touchpad (102), such as sliding of a ?nger.

35
data sent from a device, perform a numerical calculation on
40
nals in the transition band. A low-pass ?lter allows low
pass ?lter allows high-frequency signals to pass. A band-pass

?lter allows frequencies to pass that are at some de?ned
45
from passing.
A recursive or non-recursive ?lter may be utilized in an
embodiment of the present invention. A non-recursive or
?nite impulse response (FIR) ?lter utilizes only current input

values for calculating an output value. A non-recursive ?lter
does not use previous output values from the ?lter in comput
ing the current output. In contrast, a recursive or in?nite
50
impulse response (IIR) ?lter utilizes both current input values

and past output values in calculating the current output value.
In one embodiment, the ?lter performs as a sliding window,
placing more weight on recent values than on previous values.
A digital ?lter has order. The order of a non-recursive
55
recognized.
pseudo pressure is compared to a threshold value 322. If the
change in pseudo pressure is less than or equal to the thresh
old, the processor (106) returns to step 302 in the process. If
the change in pseudo pressure is greater than the threshold,

the processor (106) determines whether the ?rst interval has
elapsed 324. If so, the processor (106) concludes that the user
is pressing 326 and the processor (106) returns to step 302 in
the process.
Embodiments of the invention may use ?ltering to reduce
the effects of unwanted noise. In one such embodiment, three
variables are ?ltered: (l) the speed at which the ?nger moves
pressure (Z), and (3) the change in pseudo-pressure (AZ). The

?ltering of each of the above quantities may be performed
example, in one embodiment, the quantities are ?ltered using
a low-pass ?rst-order recursive digital ?lter based on the
60
current output calculation. A non-recursive ?lter can be a zero
following formula:
ym)
order ?lter. A recursive ?lter must by de?nition be at least a

?rst order ?lter.
In the embodiment shown in FIG. 3, the speed at which the
to signify a press, because the ?nger is moving across the

touchpad (102), the user does not intend for a press to be
using the same type of ?lter or different types of ?lters. For
used in the current calculation. The order of a recursive digital
?nger is moving over the surface of the touchpad (102), i.e.,

the change in X and Y position on the touchpad (102) per
processor (106) returns to step 302 in the process. In compar
across the surface of the touchpad (102), (2) the pseudo
digital ?lter is equal to the number of previous input values

?lter is the greater of either the number of (i) previous input
values and (ii) previous output values that are used in the
the speed is greater than or equal to the speed threshold, the
If the speed is less than the speed threshold, the change in
frequency (de?ned by the ?lter parameters) to pass. A high

frequency, and a band-reject ?lter prevents certain signals
318. If the pseudo pressure is greater than the lower threshold

306 and the user was previously touching 304, the processor
(106) restarts the tick count 316.
In any event, in the embodiment shown, once the processor
(106) concludes that the user is touching 318, the processor
(106) compares the speed to a speed threshold value 320. If
ing the speed to the speed threshold, the processor (106) may
determine that although the user is exerting enough pressure
the data received, and provide the ?ltered data as output. The
digital ?lter is programmable, allowing some signals to pass
unaltered (passband) and blocks other signals (stopband).

The signals between the passband and the stopband are sig
cludes that the user is touching 318. If the user was not
?rst tick counter 316 and concludes that the user is touching
concluding that the ?nger is not touching the touchpad (102).
One digital ?lter useful in an embodiment of the present

invention comprises software executing on a processor (106),
such as a digital signal processor (DSP), to receive samples of
step 302.
At step 302, if the processor (106) determines that the
pseudo pressure exceeds the upper threshold, the processor
(106) determines whether the user was previously touching
314. If so, the processor (106) bypasses step 316 and con
previously touching at step 314, the processor (106) starts the
the pseudo pressure fall below the lower threshold before
Also, in the embodiment shown in FIG. 3, the change in

pseudo pressure is digitally ?ltered to reduce the effects of
If the ?rst tick counter has elapsed, the process resumes at

step 302. If the tick counter has not elapsed, then the proces
sor (106) concludes that the user is tapping 310. The proces
sor (106) clears the ?rst tick count, and the process returns to
[Equation 1]
65
N is a parameter affecting the cut-off frequency of the ?lter.

For example, in one embodiment, N is set to 10 in ?ltering
US 8,749,507 B2
10
9
speed and pseudo pressure, and N is set to 5 in ?ltering the
change in pseudo-pressure. These values are related to the
sampling frequency of the touchpad (102), which, in the
? = N i- 12 Am)
[Equation 2]
W") = 4(n) ?
[Equation 3]
[Equation 4]
embodiment shown, is about 80 HZ. Such a ?lter computes a
rolling average using a weighting function emphasiZing more

recent samples. This ?lter requires minimal computational
and storage requirements. In one embodiment, the following
v(n), v(n) > 0
PM) _ {0,
thresholds are used to detect ?nger presses:
p = 2 pm)
[Equation 5]
PM)
[Equation 6]
TABLE 1
Variable
Threshold
32
lower = 16
User-dependent and locationdependent.
where:
B is the bias among Mn),
N(n) are the unbiased coe?icients,
Thresholds typically range from 2 to 8.
P(n) are the positive coef?cients extracted from v(n),
upper = 32
AZ
v(n) s O
Ramp type ?lters work very well in ?ltering out unwanted

noise. However, various types of ?lters, utiliZing a variety of
waveforms, a combination of ?lters, and other processing
20
sure is computed by subtracting the ?ltered (average) pseudo
The foregoing description of the preferred embodiments of

the invention has been presented only for the purpose of
illustration and description and is not intended to be exhaus
tive or to limit the invention to the precise forms disclosed.
means may be used as part of a process for determining a
user s intent. In one embodiment, the change in pseudo pres
M is the sum of the positive coef?cients, and

A(n) are the ?nal ?lter coef?cients.
25
pressure from the current pseudo pressure. In another
Numerous modi?cations and adaptations thereof will be

apparent to those skilled in the art without departing from the
spirit and scope of the present invention.
embodiment, the previous ?ltered pseudo pressure is sub

tracted from the current ?ltered pseudo pressure.
In one embodiment, the user adjusts the threshold values
for each key to attain maximum accuracy in intent determi
nation. In another embodiment, the threshold is based on the
That which is claimed:

30
determining an interaction with a displayed object on a

screen based on the contact data;
standard deviation of the pseudo-pressure change. In yet

another embodiment, more sophisticated ?ltering techniques
are utilized. FIG. 4 is a group of charts illustrating various
responsive to determining the interaction, determining a

35
?lters that may be utilized in embodiments of the present
invention. The following table provides the waveforms used

for each ?lter shown:
Waveform
402
Step
gesture based on the contact data comprising:

determining a pressure and a change in pressure based
on the contact data, and
determining a press if:
the pressure is greater than a pressure threshold,
the change in pressure is greater than a change in

pressure threshold, and
a ?rst interval has elapsed; and
TABLE 2
Reference
1. A method comprising:
receiving contact data from an input device;
Formula
responsive to determining the gesture, outputting the hap

1,
n<N/2
Mn)={0, n=N/2
45
l, n > N / 2
404
Pulw
l,
406
Ramp
408
Triangle
3. The method of claim 2, wherein the pseudo pressure is

based on a change in capacitance resulting from the contact.
4. The method of claim 3, wherein the contact data is based
-1, n<N/4,n>3N/4
Mn) ={0, n=N/4, n: 3N/4

N/4 < n < 3N/4
tic effect.
2. The method of claim 1, wherein the contact data com
prises an actual pressure and a pseudo pressure.
50
on a contact on a speci?c portion of a touch-sensitive input
device.
5. The method of claim 4, wherein the contact information
Mn) = l 2n/N
comprises position data, and further comprising determining

4n/Nl, nsN/Z
Mn) =
410
412
414
416
418
Quarter Cosine
Quarter Sine
Half Cosine
Half Sine
Full Cosine
Mn)
Mn)
Mn)
Mn)
Mn)
=
=
=
=
=
34n/N, n>N/2
sin(nn/2N)
cos(nn/2N)
cos(nn/N)
sin(mn/N)
cos(2nn/N)
whether a contact is made on a softkey based on the position

55
ture comprises determining a tapping gesture.

7. The method of claim 1, wherein determining the gesture
further comprises:
60
In various embodiments, the coef?cients of these wave

forms are further biased so their average is zero and scaled so
the sum of the positive coef?cients is one. More formally, the

?lter coef?cients a(n) are computed from the above coef?
cients Mn) by the following equations:
data.
6. The method of claim 1, wherein the determining a ges
65
calculating a ?rst value associated with a speed of move

ment of a contact across the input device;
comparing the ?rst value to a speed threshold value; and

outputting a signal if the ?rst value is less than the speed
threshold value.
8. The method of claim 7, further comprising applying a
speed ?lter to the ?rst value before comparing the speed to the
speed threshold value.
US 8,749,507 B2
11
12
9. A non-transitory computer-readable medium compris

ing program code for causing a processor to execute a
14. A system comprising:

a computer-readable medium; and
method, the program code comprising:
a processor in communication With the computer-readable
program code for receiving contact data from an input
medium, the processor con?gured to:
device;
receive contact data from an input device;
program code for determining an interaction With a dis

played object on a screen based on the contact data;
determine an interaction With a displayed object on a

screen based on the contact data;
program code for, responsive to determining the interac
responsive to determining the interaction, determine a
tion, determining a gesture based on the contact data
comprising:
program code for determining a pres sure and a change in
pressure based on the contact data, and
program code for determining a press if:
gesture based on the contact data comprising:

determining a pres sure and a change in pressure based
on the contact data, and
determining a press if:


program code for, responsive to determining the gesture,

outputting the haptic effect.
10. The non-transitory computer-readable medium of
claim 9, Wherein the contact data comprises an actual pres
responsive to determining the gesture, output the haptic

20
sure and a pseudo pressure.
16. The system of claim 15, Wherein the pseudo pressure is

based on a change in capacitance resulting from the contact.
17. The system of claim 16, Wherein the contact data is

claim 10, Wherein the pseudo pressure is based on a change in
capacitance resulting from the contact.

claim 11, Wherein the contact data is based on a contact on a
speci?c portion of a touch-sensitive input device.

claim 9, Wherein the program code for determining a gesture
comprises program code for determining a tapping gesture.
effect.
15. The system of claim 14, Wherein the contact data com
prises an actual pressure and a pseudo pressure.
25
based on a contact on a speci?c portion of a touch-sensitive
input device.
18. The system of claim 14, Wherein the processor is con
?gured to determine a gesture by determining a tapping ges

ture.
EXHIBIT B

US007808488B2
(12) United States Patent
(10) Patent N0.:

Martin et al.
(54)
(75)
(56)
METHOD AND APPARATUS FOR

PROVIDING TACTILE SENSATIONS
References Cited
2,972,140 A
(US); Steven P. Vassallo, Redwood City,

CA (US); Alex S. Goldenberg, Portola
Valley, CA (US); Alexander Jasso, Los
Altos, CA (U S); Kollin Tierling,
(Continued)

(Us)
USC 154(b) by 431 days.
0349086
1/1990
(Continued)
OTHER PUBLICATIONS
Adel stein, A Virtual Environment System For The Study of Human
Arm Tremor, Ph.D. Dissertation, Dept. of Mechanical Engineering,
MIT, Jun. 1989.
This patent is subject to a terminal dis

claimer.
(21) App1.No.: 11/693,117

Filed:
2/1961 Hirsch

EP
Milpitas, CA (US)
(22)
*Oct. 5, 2010
Inventors: Kenneth M. Martin, Los Gatos, CA
Notice:
US 7,808,488 B2
(Continued)
Primary ExamineriRicardo L Osorio
(74) Attorney, Agent, or FirmiKilpatrick Stockton LLP
(57)
Mar. 29, 2007
ABSTRACT
Oct. 4, 2007
Products and processes for providing tactile sensations to

input devices or electronic devices are provided. Input
devices include mechanical input devices (such as, for
devices (such as, for example, touchpads). Tactile feedback is
(65)
US 2007/0229455 A1
example, mechanical switches) and non-mechanical input
(63)
Continuation of application No. 10/285,450, ?led on

Nov. 1, 2002, noW Pat. No. 7,336,260.
(60)
Provisional application No. 60/335,493, ?led on Nov.
1, 2001, provisional application No. 60/399,883, ?led

on Jul. 31, 2002.
(51)
(52)
(58)
may be employed to receive signals from the input devices

and control the actuator. Tactile feedback to an input device or
electronic device may be provided in response to one or more
events or situations. Such an event or situation may be any one
designated. Examples of such events and situations include

the level of pres sure placed on an input device; the availability
Int. Cl.
G09G 5/00
provided by using an actuator or other means in communica

tion With the input device or electronic device. A controller
or lack of availability of a function associated With an input
(2006.01)
US. Cl. ..................................... ..
345/169; 345/156
Field of Classi?cation Search ............... ..
345/156,
device; and the function, menu, or mode of operation associ

ated With an input devices activation. A variety of feedback
types and combinations may be selected.
345/161,163,173,167*169
See application ?le for complete search history.
US 7,808,488 B2
Page 2
US. PATENT DOCUMENTS
3,157,853
3,220,121
3,497,668
3,517,446
A
A
A
A
11/1964
11/1965
2/1970
6/1970
Hirsch
Cutler
Hirsch
C9r1y9neta1~
3,623,064 A
3902687 A
11/1971 K9899
9/1975 Hlghtower
3,903,614
3,911,416
4,127,752
4,160,508
4,236,325
4,262,549
4311980
4,333,070
4,362,408
4,464,117
4,484,191
4,513,235
4,581,491
4,581,972
4,599,070
4,692,756
4,708,656
4,713,007
4,725,817
4,791,416
4794392
4,795,296
4,798,919
4,821,030
4,823,106
9/1975
10/1975
11/1978
7/1979
12/1980
4/1981
V1982
6/1982
12/1982
8/1984
11/1984
4/1985
4/1986
4/1986
7/1986
9/1987
11/1987
12/1987
2/1988
12/1988
12/1988
1/1989
1/1989
4/1989
4/1989
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
4,840,634 A
4,885,565
4,891,764
4930770
4,934,694
4982918
4,983,786
51019761
5,022,384
5,022,407
5,035,242
5,038,089
A
A
A
A
A
A
A
A
A
A
A
5,461,711
5,466,213
5,489,812
5,496,174
5,514,150
5,521,336
5,547,382
5,575,761
5,631,861
5,684,722
5,691,747
5,709,219
5,729,249
5,766,016
5,767,457
5,785,630
5,791,992
5,844,392
5,857,986
5,887,995
5,889,670
5,889,672
5,917,906
5,943,044
5,945,772
5,977,867
5,988,902
6,059,506
6,078,126
Diamondetal.
Peder
Lowthorp
Salisbury, Jr. et 31.
Hall et a1.
Schwellenbach
Prudenziati
Barnes
Cordes et a1.
Forest
Vavra
Acklametal.
Boothroyd
Hoshino
Hladky eta1~
Clark
DeVries etal.
Alban
Wihlborg
Adler
Selinko
Jau
Miessleretal.
Batsonetal.
Lovell
6,097,964 A
6,111,577 A
6/1989 Mulleretal.
12/1989
V1990
6/1990
6/1990
l/199l
1/1991
5/1991
6/1991
6/1991
7/1991
8/1991
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
Embach
McIntosh
Baker
McIntosh
Kaye
Stevens
Kraft
Freels
Horch er 91Franklin etal.
Szakaly
6,118,435
6,131,097
6,160,439
6,195,592
A
A
A
B1
6,198,206
6,218,966
6,219,034
6,225,976
6,292,173
B1
B1
B1
B1
B1
6,307,465 B1
6,344,791 B1
6,369,803 B2
6,373,463 B1
6,374,255 B1
10/1995
11/1995
2/1996
3/1996
5/1996
5/1996
8/1996
11/1996
5/1997
11/1997
11/1997
V1998
3/1998
6/199g
6/1998
7/1998
g/199g
12/1998
1/1999
3/1999
3/1999
3/1999
6/1999
8/1999
8/1999
11/1999
11/1999
5/2()()()
6/2000
Wang et a1.
Hogan etal.
Furuhata et a1.
Gamer
Rostoker
Bnohanan etal.
Yamasaki et a1.
Hajianponr
Kramer
Thorneretal.
Anlan0
Chen et a1
Yasutake
Sinclair etal
Gerpheide et a1.
Bobick etal.
Crump et a1
Peurach etal.
Moriyasu
Holehan
Schuleretal.
Schuler et a1.
Thornton
Martinellietal.
Macnaket 31
Bouin
Holehan
Kramer
Rollins etal.
8/2000 Nuovo et a1.

g/ZOOO Zines etal
9/2000 pujitaetal
10/2000 Peurach etal.
12/2000 Perryet 31,

2/2001
3/2001
4/2001
4/2001
5/2001
9/2001
10/2001
2/2002
4/2002
4/2002
4/2002
Schuleretal.
saarrnaa et al.
Goodwin et a1.
Elbing et a1.
Yates et a1.
Rambaldi et al.
Kaymaetal.
Armstrong
Brisebois et a1.
Beeks
Peurach etal.
5,053,585 A
10/1991 Yaniger
6,388,655 B1
5/2002 Leung
5,078,152 A
1/1992 Bond
6,422,941 B1
7/2002 Thorneretal.
5,116,051 A
5,165,897
5,175,459
5,182,557
5,186,695
A
A
A
A
5/1992 Moncriefetal-
11/1992
12/1992
V1993
2/1993
6,429,846 B2*
Johnson
Danialet al.
Lang
Mangseth er 91-
6,543,487
6,597,347
6,657,617
6,781,569
B2
B1
B2
B1
8/2002 Rosenberg et a1. ........ .. 345/156
4/2003
7/2003
12/2003
8/2004
Bazinet
Yasutake
Paolini et a1.
Gregorio et al.
5212473 A
5/1993 L<>uiS_
6,801,191 B2
10/2004 Mukaietal.
5,223,658 A
6/1993 Suzukl
6,976,562 B1
12/2005 Perret, Jr. etal.
5237327 A
5,240,417 A
5241308 A
8/1993 sai?oh er 918/1993 smlthson eta1~
5246316 A
8/1993 Young
9/1993 Smith
7,202,851 B2*
2002/0033795 A1
2002/0128048 A1
2002/0149561 A1
4/2007 Cunninghametal. ..... .. 345/156

3/2002 shahoian etal.
9/2002 Aaltonen etal.

10/2002 Fukumoto et a1.
5,271,290 A
12/1993 Flscher
2002/0171621 A1
11/2002 Johnson
5,275,174 A
V1994 (390k
2002/0177471 A1
11/2002 Kaaresojaetal.
5,283,970 A
2/1994 Algner
2005/0099393 A1
5/2005 Johnson
5289273 A
2/1994 Lang
2008/0068350 A1*
3/2008 Rosenberg et a1. ........ .. 345/173
5,299,810
5,309,140
5,334,027
5,355,148
5,390,128
5,390,296
5,402,499
5,402,680
5,436,622
5,437,607
5,451,924
A
A
A
A
A
A
A
A
A
A
A
4/1994
5/1994
8/1994
10/1994
2/1995
2/1995
3/1995
4/1995
7/1995
8/1995
9/1995
Pierce et a1.
Everett
Wherlock
Anderson
Ryan et a1.
crandall et a1.
Robison etal.
Korenaga
Gutman et a1.
Taylor
Massimino etal.

EP
GB
JP
JP
JP
JP
JP
JP
0817110
2180342
01003664
H2-185278
02-109714
H4-8381
04007371
H5-192449
1/1998
3/1987
7/1990
7/1990
1/1992
1/1992
8/1993
8/1993
US 7,808,488 B2
Page 3
JP
JP
JP
JP
JP
JP
JP
JP
JP
JP
JP
JP
JP
KR
W0
W0
W0
W0
WO
W0
W0
05-193862
H7-24147
8221173
10171586
1124834
11-085400
2001-222379
2001-265485
2001-290572
2001-296950
2001-350592
2002-259059
2002-528831
2001-0028369
W0 9520787
WO 97/18546
WO 99/49443
WO 01/54109
WO01/91100
WO 02/27645
W0 0231807
A
A
A1
A1
A1
1/1995
1/1995
8/1996
6/1998
1/1999
3/1999
8/2001
9/2001
10/2001
10/2001
12/2001
9/2002
9/2002
4/2001
8/1995
5/1997
9/ 1999
7/2001
11/2001
4/2002
4/2002
Howe, A Force-Re?ecting Teleoperated Hand System for the Study

of Tactile Sensing in Precision Manipulation, Proceedings of the
1992 IEEE International Conference on Robotics and Automation,
Nice, France, May 1992.

IBM Technical Disclosure Bullein, Mouse Ball-Actuating Device
With Force and Tactile Feedback, vol. 32, No. 9B, Feb. 1990.
International Search Report ?led Nov. 1, 2002 corresponding to
PCT/US02/35016.
Iwata, Pen-based Haptic Virtual Environment, 0-7803-1363-1/93

IEEE, pp. 287-292, 1993.
Jacobsen et al., High Performance, Dextrous Telerobotic Manipu
lator With Force Re?ection, Intervention/ROV 91 Conference &
Exposition, Hollywood, Florida, May 21-23, 1991.

90-039, Aug. 1990.
Springer International (Springer-VeItag); Experimental Brain

Research, vol. 79, No. 1, pp. 150-156, 1990.
KacZmarek et al., Tactile Displays, Virtual Environment Technolo
gies.
OTHER PUBLICATIONS
Adelstein, Design and Implementation of a Force Re?ecting

Manipulandum for Manual Control research, DSC-vol. 42,
Advances in Robotics, Edited by H. KaZerooni, pp. 1-12, 1992.
Reality, ISBN 0-938151-82-7, pp. 129-180, 1992.

Baigrie, Electric Control LoadingiA Low Cost, High Performance
Alternative, Proceedings, pp. 247-254, Nov. 6-8, 1990.
BejcZy, Sensors, Controls, and Man-Machine Interface for
1335, 1980.
BejcZy, Generalization of Bilateral Force-Re?ecting Control of
Manipulators, Proceedings Of Fourth CISM-IFToMM, Sep. 8-12,

1993.
Teleoperation and Virtual Environments, Presence, 4(4):387-402,

1995.
Lake, Cyberman from Logitech, GameBytes, 1994.

Marcus, Touch Feedback in Surgery, Proceedings of Virtual Real
ity and Medicine The Cutting Edge, Sep. 8-11, 1994.

McAffee, Teleoperator Subsystem/Telerobot Demonsdtrator: Force
Re?ecting Hand Controller Equipment Manual, JPL D-5172, pp.
1-50, A1-A36, B1-B5, C1-C36, Jan. 1988.
Minsky, Computational Haptics: The Sandpaper System for Syn
1970.
thesiZing Textue for a Force-Feedback Display, Ph.D. Dissertation,

MIT, Jun. 1995.
Ouh-Young, Force Display in Molecular Docking, Order No.
9034744, p. 1-369, 1990.
Ouh-Young, A Low-Co st Force Feedback Joystick and Its Use in PC
Video Games, IEEE Transactions on Consumer Electronics, vol. 41,
No. 3, Aug. 1995.
Ouhyoung et al., The Development of A Low-Cost Force Feedback
Joystick and Its Use in the Virtual Reality Environment, Proceedings
of the Third Paci?c Conference on Computer Graphics and Applica
Brooks et al., Hand Controllers for TeleoperationiA State-of-the
tions, Paci?c Graphics 95, Seoul, Korea, Aug. 21-24, 1995.
Art Technology Survey and Evaluation, JPL Publication 85-11;

NASA-CR-175890; N85-28559, pp. 1-84, Mar. 1, 1985.
Patrick et al., Design and Testing of a Non-reactive, Fingertip,

Pimentel et al., Virtual Reality: through the new looking glass, 2
RabinowitZ et al., Multidimensional tactile displays: Identi?cation
of vibratory intensity, frequency, and contractor area, Journal of The
Accoustical Society ofAmerica, vol. 82, No. 4, Oct. 1987.
Russo, The Design and Implementation of a Three Degree of Free
dom Force Output Joystick, MIT Libraries Archives Aug. 14, 1990,
pp. 1-131, May 1990.
Russo, Controlling Dissipative Magnetic Particle Brakes in Force
1981.
BejcZy, et al., Universal Computer Control System (UCCS) For

Space Telerobots, CH2413-3/87/0000/0318501.00 1987 IEEE,
1987.
BejcZy et al., A Laboratory Breadboard System For Dual-Arm

Teleoperation, SOAR 89 Workshop, J SC, Houston, TX, Jul. 25-27,
1989.

Transactions on Man-Machine Systems, vol. MMS-11, No. 1, Mar.
Burdea et al., Distributed Virtual Force Feedback, Lecture Notes for

Workshop on Force Display in Virtual Environments and its Appli
cation to Robotic Teleoperation, 1993 IEEE International Confer
ence on Robotics and Automation, pp. 25-44, May 2, 1993.

Jun. 23, 1983.
Caldwell et al., Enhanced Tactile Feedback (Tele-Traction) Using a
Multi-Functional Sensory System, 1050-4729/93, pp. 955-960,

1993.
Eberhardt et al., OMARiA Haptic display for speech perception

by deaf and def-blind individuals, IEEE Virtual Reality Annual
International Symposium, Seattle, WA, Sep. 18-22, 1993.
Dynamic Systems and Control: vol. 1, ASME 1994.
Fokumoto, Active Click: Tactile Feedback For Touch Panels, ACM
CHI2001 Extended Abstracts, pp. 121-122, Apr. 2001.
Gobel et al., Tactile Feedback Applied to Computer Mice, Interna
tional Journal of Human-Computer Interaction, vol. 7, No. 1, pp.
1-24, 1995.
Gotow et al, Controlled Impedance Test Apparatus for Studying

Human Interpretation of Kinesthetic Feedback, WA11-11:00, pp.
332-337.
ASME 1992.
Scannell, Taking a Joystick Ride, Computer Currents, Nov. 1994,

Boston Edition, vol. 9 No. 11.
Intelligent Robotic Systems for Space Exploration, Rensselaer Poly

technic Institute, Sep. 30-Oct. 1, 1992.
http://www.smk.cojp/whatsnewie/628cscie.html, Sep. 30, 2002.

US 7,808,488 B2
Page 4
Snow et al., Model-X Force-Re?ecting-Hand-Controller, NT Con
trol No. MPO-17851; JPL Case No. 5348, pp. 1-4, Jun. 15, 1989.

sors, DSV-vol. 42, Advances in Robotics, pp. 55-61, ASME 1992.
MIT Archive Massachusetts Institute of Technolo gy, pp. 1-88, Feb.
1990.

of Fourteenth Annual Northeast Bioengineering Conference, Univer
sity ofNeW Hampshire, Mar. 10-11, 1988.
Wakiwaka, et al., In?uence of Mover Support Structure on Linear
Oscillatory Actuator for Cellular Phones, The Third International
Symposium on Linear Drives for Industry Applications, 2001, p.
260-263, Nagano, Japan.

Wiker, Teletouch Display Development: Phase 1 Report, Technical
Report 1230, Naval Ocean Systems Center, San Diego, Apr. 17, 1989.
US. Of?ce of China Sinda Intellectual Property Ltd., Noti?cation of
First Of?ce Action, Chinese Application No. 0282l854.X, mailed
Sep. 5, 2007.
Japanese Patent Of?ce, Response to Notice of Reasons for Rejection
for Jan. 29, 2008, Japanese Patent Application No. 2003-540973,
mailed Feb. 20, 2008.
Japanese Patent Of?ce, Notice of Reasons for Rejection of Sep. 11,

2007, Japanese Patent Application No. 2003 -540973, mailed Sep. 26,
2007.
United States Patent and Trademark Of?ce, Of?ce Action, U. S. Appl.

No. 10/285,450, mailed Dec. 23, 2005.
No. 10/285,450, mailed May 18, 2006.
No. 10/285,450, mailed Nov. 15, 2006.
No. 10/285,450, mailed Jun. 1,2007.
European Patent Of?ce, Supplemental Search Report, European

Application No. 027739606, mailed Jul. 1, 2008.
KIPO Notice of Preliminary Rejection, Korean Patent Application
No. 10-2004-7006627, mailed Apr. 16, 2009.
Of?ce Action mailed Nov. 25, 2009 for corresponding Korean Appli
cation No. 10-2009-7017838.
Of?ce Action mailed Dec. 4, 2009 for corresponding Korean Appli

cation No. 10-2004-7006627.
Of?ce Action dated Jun. 19, 2009 for corresponding Chinese Appli
cation No. 200810008845.X.
* cited by examiner
US. Patent
0a. 5, 2010
Sheet 3 0f 11
FIG.3
US 7,808,488 B2
US. Patent
0a. 5, 2010
Sheet 4 0f 11
US 7,808,488 B2
\-.
AYUE
f
2
US. Patent
0a. 5, 2010
Sheet 5 0f 11
37
36k
320
\
32b
FIG.5
US 7,808,488 B2
US. Patent
0a. 5, 2010
Sheet 7 of 11
Input
US 7,808,488 B2
Electronic
40/ Devices
'
\43
_/39
41
\ Controller
Control
45/ Circuitry
Actuator
46/
.
Dlsploy
/ 44
Memory r42
US. Patent
Oct. 5, 2010
US 7,808,488 B2
Sheet 8 0f 11
Access
Memory
Monitor
Output
Device
72
Obtain
Ambiguous
Feedback
Info.
70
/59
Obtain
Failure
Feedback
Info.
|
Pressure
Data?
Produce
Tactile
Sensation \55
Obtain
Pressure
Data
US. Patent
0a. 5, 2010
Sheet 11 0f 11
US 7,808,488 B2
100
1 10~
f
'
I] New
112
Qlose
tr|+N
\ El ove
Ctr+S
_ 6 Erint
Ctr|+P
Send To
Properties
Exit
102- /_
5|Muil Recipient
63 Mgil Recipient (as Attachment)
5 Microsoft EowerPoint
\ 114
US 7,808,488 B2
1

may be dif?cult or impossible for a user to simultaneously use
CROSS-REFERENCES TO RELATED
APPLICATION
interacting With the device. In addition, electronic devices

relying on softkeys can be dif?cult to read in bright light
environments such as in bright sunlight and can contain very
This application is a continuation of co-pending US.

patent application Ser. No. 10/285,450 entitled Method and
Apparatus for Providing Tactile Sensations ?led Nov. 1,
2002, Which claims priority to US. Provisional Application
No. 60/335,493, ?led Nov. 1, 2001, and US. Provisional
small fonts and graphics that are dif?cult to read and select.
to the user of the touchpad. US. Pat. No. 5,977,867 is one
Application No. 60/399,883, ?led Jul. 31, 2002, the entirety

of all of Which is hereby incorporated by reference.
to a user interacting With an electronic device. Moreover, such
such an electronic device because such devices typically

require the user to look at the device, at least brie?y, When
Some conventional touchpads include vibratory feedback

example. Such conventional systems and methods are lim
ited, though. They lack a full range of functionality assistance

systems and methods still require considerable visual atten
tion from the user.
SUMMARY
The present invention relates to methods and apparatus for
providing tactile sensations.
The present invention comprises products and processes

BACKGROUND
20
Conventional electronic devices, such as mobile tele
for providing tactile sensations to input devices or electronic
devices. Input devices include mechanical input devices

(such as, for example, mechanical sWitches) and non-me
phones and Personal Digital Assistants (PDAs), include
chanical input devices (such as, for example, touchpads).
visual displays. A user of such devices interacts With the

visual display using any one of a number of input devices.
Tactile feedback is provided by using an actuator or other

25
sticks, trackballs, steering Wheels, stylus, tablets, pressure

sensitive spheres, scroll Wheels, keyboards, and keypads. The
response to one or more events or situations. Such an event or
user provides instructions, responses, and other input to the
device using such input devices.
30
In conventional mobile telephones and PDAs, con?rma

tion of the input provided by the user is primarily limited to
situation may be any one designated. Examples of such events

and situations include the level of pressure placed on an input
device; the availability or lack of availability of a function
visual or audible con?rmation. In some such devices, physi
cal feedback is provided by conventional mechanical

sWitches in the form of the conventional mechanical feedback
means in communication With the input device or electronic
device. A controller may be employed to receive signals from

the input devices and to control the actuator. Tactile feedback
to an input device or electronic device may be provided in
Examples of such input devices include computer mice, joy
35
associated With an input device; and the function, menu, or

mode of operation associated With an input devices activa
tion. A variety of feedback types and combinations may be
of sWitches, for example the sWitch closure force-displace

ment pro?le. Typically, in such devices, the mechanical feed
back provided by each button is identical. In addition, in such
selected. Mobile telephones and PDAs bene?t from employ

ing such products and processes, but other devices bene?t as
Well. The advantages offered by the various embodiments of
conventional devices, for those buttons that serve multiple
the present invention may be understood by examining this
functions, the mechanical feedback generally remains the
40
speci?cation.
same regardless of the current function of the button.

BRIEF DESCRIPTION OF THE DRAWINGS
In addition to providing extremely limited and rudimentary

mechanical con?rmation of button selection, conventional
buttons as used, for example, in keypads for mobile tele
phones and PDAs, provide simple passive touch cues regard
FIG. 1 is a perspective vieW of an embodiment of an appa

45
ing the alignment of keys. Such cues include raised bumps on
FIG. 2 is a plan vieW of another embodiment of an appa
the center key of a telephone keypad or on the F and G

keys of a keyboard that assist a user in orienting to the pattern
ratus according to the present invention;

FIG. 3 is a plan vieW of an electronic device including an
of keys in the keypad and keyboard. Again, these physical

queues are very limited, and users typically need to vieW a
50
keypad or keypad for visual con?rmation that the correct

instructions or information is being entered.
When a ?at surface interface device is used, such as a
55
bined With ?at-panel display screens that display one or more
graphically generated buttons or softkeys. Normally, the soft

60
FIG. 5 is a plan vieW of another electronic device including

another embodiment of the present invention;
FIG. 6 is a schematic representations through line 6-6 of
FIG. 5;
present invention;
FIG. 9 is a table illustrating a ?rst set of data to be used in
The use of electronic devices using such conventional
one embodiment of the present invention;
mechanical buttons and touchpad arrangements are particu
FIG. 10 is a table illustrating a second set of data to be used
larly dif?cult to use in distracting environments or When the
user is attempting to perform another task simultaneously

Withusing the electronic device. For example, if the other task
involves operating a motor vehicle or heavy machinery, it
FIG. 7 is a block diagram illustrating an embodiment of the

apparatus in an electronic device;
FIG. 8 is a How chart illustrating a method according to the
keys are visible through the touchpad. A users contact With

the touchpad in an area de?ned by a softkey provides the
electronic device having the touchpad With the input associ

ated With that softkey.
embodiment of the present invention;

FIG. 3;
touchpad for a computer or PDA, these simple mechanical

cues are unavailable to the user. Often, touchpads are com
ratus according to the present invention;
65
in another embodiment of the present invention; and

FIG. 11 shoWs a device for providing tactile sensations
according to one embodiment of the present invention.
US 7,808,488 B2
3
invention. The entire disclosure of the application Ser. No.

09/ 585,741 is incorporated herein by reference. Tactile sen
The present invention includes methods and systems for

providing tactile sensations. One embodiment includes meth
ods and systems for providing tactile sensations to input
sations can also be delivered to the input device 2 from a

speaker included With an electronic device into Which the
apparatus is placed, for example the speaker in a mobile
devices, both mechanical and non-mechanical (for example
telephone or in a personal computer.

Although the embodiment shoWn in FIG. 1 includes one
input device 2 and one actuator 6, other embodiments include
a plurality of input devices, all in communication With a
soft-keys that are computer generated and displayed on a

screen). Embodiments of the present invention can be utiliZed
in Wide variety of electronic devices including telephones,

mobile telephones, remote controls, gamepads, joystick
handles, automotive controls (radios, Compact Disc (CD)
single actuator. Alternatively, an embodiment can include a

plurality of actuators each in communication With at least one
input device. Various arrangements of actuators in combina
players, automobile functions, etc.), consumer electronics

devices, Personal Digital Assistants (PDAs), personal com
tion With input devices are suitable for use in the present
puters, laptop computers, portable gaming devices, pagers,
invention. For example, US. patent application Ser. No.

09/263,263, ?led Jul. 26, 2001, published on Mar. 21, 2002,
I-pagers, audio equipment, televisions, security or alarm sys

tems, Automated Teller Machines (ATM), calculators, home
as US. Patent Pub. No. US2002/ 0033795 illustrates actuators
appliances, and White goods.
in combination With input devices that may be used in

embodiments of the present invention. The entire disclosure
ofapplication Ser. No. 09/263,263, Pub. No. 2002/0033795 is
FIG. 1 shoWs one embodiment of the present invention.

The apparatus 1 shoWn in FIG. 1 includes an input device 2
having multiple positions for communicating a plurality of
20
input signals. The input device 2 can be any device capable of

transmitting an input signal. In the embodiment shoWn, the
input device 2 is a rocker-type sWitch. The rocker sWitch 2
As mentioned, the actuator 6 is in communication With the

input device 2. In the embodiment shoWn in FIG. 1, the
actuator 6 is in communication With the input device 2
shoWn can pivot or rock betWeen tWo positions in Which the

rocker sWitch contacts and activates one of tWo rubber
through a cantilevered beam or lever arms 7 attached to the

25
sWitches 3 containing conductive pads. The use of rubber

sWitches 3 provides the advantage of alloWing the user to still
feel a substantial vibration or force through the input device 2
When the user had fully depressed the sWitch. Suitable rubber
sWitches are available and knoWn in the art. In other embodi
ments, the input device may include an analog sWitch, a force

sending resistor, a strain gauge based sensor, a capacitative
touch sWitch, a scroll Wheel, a mini-joystick, a touchpad, a
touch screen, a 3-Way sWitch, a 4-Way sWitch, a 5-Way sWitch,
or other input device. Each position of the input device 2
corresponds to one of the input signals.
incorporated herein by reference.
pivoting rocker, amplifying the effective forces of the actua

tor 6 felt by the user. The tactile sensations generated by the
actuator 6 propagate through the lever arm 7 to the input
device 2. Suitable materials for the lever arm 7 are capable of
transmitting the tactile sensations and can be, for example,

30 metal. The lever arm 7 shoWn includes one or more bends 8 to
?t Within the electronic device in Which the apparatus 1 is
disposed. Different shapes of bends may be used to ?t Within
35
the electronic device. In another embodiment, the actuator 6

is mounted directly to the input device 2 or to any component
of the input device. Alternatively, the actuator 6 is mounted to
the PCB 4 to Which the input device is attached, communi
The input device 2 and rubber sWitches 3 are mounted on a
cating tactile sensations to the input device through the PCB.
Printed Circuit Board (PCB) 4 in the embodiment shoWn to

facilitate electrical communication betWeen the input device
2 and an electronic device (not shoWn). The PCB 4 can be
custom shaped according to the device into Which the appa
ratus 1 is placed. The PCB 4 also provides for secure mount
In another embodiment, the actuator is an existing eccentric

40
mass motor as is used, for example, as a vibrating ringer in a

pager or mobile telephone.
The vibrotactile actuator 6 can also be mounted to a portion
of the case or housing of the electronic device in Which the
ing Within the device by including, for example, a plurality of
apparatus 1 is disposed, communicating the tactile sensations
holes 5 to accept fasteners for securing to the electronic

device. In another embodiment, the input device 2 can be
directly connected or mounted in the electronic device.
The apparatus 1 shoWn in FIG. 1 also includes a vibrotac
tile actuator 6 in communication With the input device 2.
Preferably, the actuator 6 is con?gured to output a plurality of
distinct tactile feedback sensations to the input device 2.
to the entire electronic device. In one embodiment, tWo actua

tors can be incorporated in the case or back of an electronic
device, for example the case of a mobile phone in an area that
Suitable tactile sensations include vibrations, for example,

jolts and textures, and a plurality of distinct tactile sensations
can be created by varying the frequency, amplitude and Wave
form output by the actuator 6. The actuator 6 is selected to
deliver the desired tactile sensations to the input device 2 The
45
contacts the users hand. This arrangement effectively

doubles the amplitude of the tactile sensation, and the users
50
?ngers do not tend to attenuate the tactile sensation.

The apparatus 1 also includes a controller 9 in communi
cation With the input device 2 to receive the input signals

therefrom. The controller 9 can also receive additional infor
55
mation from the input device 2 including the position of the

input device 2 and the amount of pressure applied to the input
device 2. In one embodiment, the input signal includes infor
actuator 6 shoWn in FIG. 1 is a voice coil actuator. Other
mation related to the amount of pressure applied to the input
suitable actuators include, for example, pieZo-electric actua

tors, eccentric mass actuators, moving magnet actuators, and
device 2, or a combination of information about pressure and
device 2, information related to the position of the input
position. In addition to being in communication With the input
friction brakes in contact With metal shafts. In addition, the

actuator can include a ?exure, for example an arrangement of
60
?exible material, coupled to the rotating shaft of a DC motor
device 2, the controller 9 is in communication With the actua

tor 6 to produce a tactile sensation in the actuator 6 corre
or step motor to transform the rotation of the motor shaft into
sponding to the input or input signal received by the controller
vibrations or other haptic sensations. Various arrangements of
9 from the input device 2.

The controller 9 is located in a suitable location according
to the needs of the device in Which the apparatus 1 is placed.
a ?exure coupled to a motor may be used as an actuator. For
example, US. patent application Ser. No. 09/585,741, ?led

Jun. 2, 2000, illustrates suitable arrangements of ?exures and
motors for use as an actuator in embodiments of the present
65
In one embodiment, the controller 9 is attached to the PCB 4
as shoWn in FIG. 1. Suitable controllers, include, for example,
US 7,808,488 B2
5
digital logical processors capable of processing input,
placed on the button (greater than no pressure placed by the

user), the third level being a second magnitude of pressure
placed on the button (Where the second magnitude of pres sure
is different from or greater than the ?rst magnitude), the
fourth level being a third magnitude of pressure placed on the
button (Where the third magnitude is different from or greater
than the second magnitude), and the ?fth level being a fourth
magnitude of pressure placed on the button (Where the fourth
magnitude is different from or greater than the third).
In button 101', each of levels tWo through ?ve is associated
With a distinct input signal. When the button 101' is in its ?rst
state, then the button 101' does not transmit an input signal.
execute algorithms, and generate output as necessary to cre
ated the desired tactile sensations in the input device in

response to the inputs received from that input device. Such
controllers may include a microprocessor, an Application
Speci?c Integrated Circuit (ASIC), and state machines. Such

controllers include, or may be in communication With, media,
for example computer readable media, Which stores instruc
tions that, When executed by the controller, cause the control
ler to perform the steps described herein as carried out, or
assisted, by a controller. On embodiments of a suitable com
puter-readable medium includes an electronic, optical, mag

netic, or other storage or transmission device capable of pro
When pressure is applied to the button 101' by a user that

exceeds the ?rst magnitude of pressure, the button 101' trans
mits a ?rst signal to the controller 9. When greater pressure is
viding a processor, such as the processor in a Web server, With
computer-readable instructions. Other examples of suitable

media include, but are not limited to, a ?oppy disk, CD-ROM,
applied to the button 101' that exceeds the second magnitude of
magnetic disk, memory chip, ROM, RAM, ASIC, con?gured
pressure, the button 101' transmits a second signal to the con
processor, all optical media, all magnetic tape or other mag
troller. When still greater pressure is applied to the button 101'

that exceeds the third magnitude of pressure, the button 101'
netic media, or any other medium from Which a computer

processor can read. Also, various other forms of computer
20
readable media may transmit or carry instructions to a com
puter, including a router, private or public netWork, or other

transmission device or channel.
In one embodiment, the apparatus 1 includes a dedicated
controller 9 for use speci?cally With the apparatus 1. This
25
embodiment is particularly Well suited for applications Where

the apparatus 1 is retro?tted into an existing electrical or
electronic device. In another embodiment, the controller 9 is
the microprocessor or Central Processing Unit (CPU) of the

electronic device in Which the apparatus 1 is disposed. The
30
apparatus 1 can also include additional circuitry such as the
drive circuitry (not shown) necessary to create the tactile

sensations in the actuator 6 in response to input from the
controller 9 and a memory medium for storing data to be
accessed by the controller 9 for example a correlation

betWeen the tactile sensations to be produced in the actuator
6 and the input information or input signal received from the
35
input device 2.
FIG. 2 shoWs another embodiment of the present invention.
An apparatus 60 shoWn in FIG. 2 includes multiple input
devices. These multiple input devices include tWelve ?xed or
40
assigned function buttons 1111-0, and three assignable func

45
50
55
The pre-assigned function buttons 11a-c of the apparatus 1

are keypad push buttons. Each of the buttons 11a-c is capable
of resolving three levels of pressure placed on the buttons
none), and a second magnitude of pressure (greater than the

?rst magnitude). Examples of functions carried out by such
pre-assigned function buttons 11a-c include Send 11a,
Power 11b, and End Call 110.
In the embodiment shoWn, each of the pre-assigned func
tion buttons 11a-c is con?gured such that the ?rst magnitude
of pressure is an amount of pressure that signi?es that a user s
The buttons of FIG. 2 are illustrative of such an embodi
level is a state in Which no pressure is placed on the button by

a user, the second level being a ?rst magnitude of pressure
ber 2 on the keypad) is capable of resolving four levels of

pressure placed on the button 10b (the ?rst level being no
pressure placed on the button). Like button 101', the levels
resolved by button 1 0b in the alternative embodiment are each
11a-c no pressure, a ?rst magnitude of pressure (greater than
user (higher/loWer), the magnitude of pressure placed on a

touchpad-type input device, or other means.
ment. Each of the alphanumeric input buttons 10 shoWn in

FIG. 2 is a keypad button. Each of the buttons 10 is capable of
resolving multiple levels of pressure placed on the buttons 10.
For example, the button 101' (corresponding to the number 9
on the keypad) is capable of resolving ?ve levels of pressure
placed on the button 101'. In the embodiment shoWn, the ?rst
In the embodiment shoWn, the alphanumeric buttons 1 0 are

all capable of resolving ?ve levels of pressure. In alternative
embodiments, the various buttons are capable of resolving
differing levels of pressure. For example, in an alternative
embodiment, While the button 101' is capable of resolving ?ve
associated With a distinct input signal, and are each associated

With a distinct letter of the alphabet, A-C.
The input device is capable of resolving multiple levels of

pressure placed on the input device, and of transmitting a
signal associated With the level of pressure placed on the input
device. These multiple levels of pressure may be de?ned by,
for example, the physical location of, or distance traveled by,
a sWitch-type input device in the x-plane When pressed by a
number 9, and the letters W, X, Y, and Z.
levels of pressure, the button 10b (corresponding to the num
pre-assigned alphanumeric input buttons 10a-l, three pre

tion buttons 12a-c. The plurality of inputs devices are
arranged according to the electronic device in Which the
apparatus 60 is situated. In the embodiment shoWn, the plu
rality of input devices are arranged as the keys in a key pad for
a telephone or mobile telephone.
Embodiments of the present invention include an input
device having a means for determining or sensing pressure.
transmits a third signal to the controller. When even greater

pressure is applied to the button 101' that exceeds the fourth
magnitude of pressure, the button 101' transmits a fourth signal
to the controller. The structural arrangement of the commu
nication by the button 101' to the controller 6 of an input signal
is further illustrated in FIG. 4, described beloW.
Each of the levels tWo through ?ve of button 101' (and thus
each of their associated signals) is associated With a letter,
W-Z. The second level is associated With the letter W, the third
level is associated With the letter X, and so on. The second
level is associated With the letter W, the third level is associ
ated With the letter X, and so on. In the embodiment shoWn,
the key 101' has ?ve positions corresponding to no pressure,
and the letters W, X, Y, and Z. In an alternative embodiment,
the key 101' has six positions corresponding to no pressure, the
60
?nger is hovering over, or touching With more than passing

force, the button. Each is also con?gured such that the second
magnitude of pressure is an amount of pressure that signi?es
that a user s ?nger applies When the user Wishes to activate the
button.
65
Thus, in the embodiment shoWn, When a users ?nger

hovers over the Send button 110, a ?rst signal is trans
mitted by the button 110 to the controller. And, When a user s
US 7,808,488 B2
7
?nger activates the Send button 110, a second signal is

transmitted by the button 110 to the controller.
The assignable-function buttons 12a, 12b, 120 are buttons
Whose function depends upon the mode of operation of the
device With Which the apparatus 1 is associated. For example,
shoWn in FIG. 4. The actuator 61 is in communication With
the various input devices, and is con?gured to provide vibra

tions of varying frequencies, magnitudes, and Wave-forms to
the input devices. The actuator 61 is also in communication
With the controller 9. Further description of embodiments of
such communication and con?guration is provided beloW.
When such an apparatus 1 is associated With a mobile tele
phone, such buttons 12a, 12b, 120 may be used to navigate the
In the embodiment shoWn, the controller 9 receives an
menus displayed to carry out various functions, such as
input signal from one of the input devices. The controller 9

then analyZes the input signal received to determine a signal
scrolling through an address book, selecting a number to dial,

editing a number, re-setting the time displayed, and similar
to transmit to the actuator 61.
functions.
In addition, the assignable-function buttons 12a-c are con
For example, the controller 9 of FIG. 2 is con?gured such

that When the controller 9 receives a signal associated With the
?gured similarly to the pre-assigned buttons 11a, 11b, 110, in
second level from button 101' (the 9 key), the controller 9

sends a ?rst control output signal to the actuator, and When the
controller receives a signal associated With the third level
that the buttons 12a, 12b are con?gured such that the ?rst
that a user s ?nger is hovering over, or touching With more
than passing force, the button, and such that the second mag
nitude of pressure is an amount of pressure that signi?es that
a users ?nger applies When the user Wishes to activate the
button. Preferably, the buttons 11a, 11b, 110, 12a, 12b, 120
are con?gured such that they receive and analyZe other data in
determining Whether the user is merely hovering or, instead,
Wishes to activate the button (such as type of, and duration of,
contact With the button). Any suitable input-device may be
used an assignable-function input device. Examples of such
input-devices include rocker-sWitches and scroll Wheels.
In an alternative embodiment (not shoWn), the middle
assignable-function button 120, includes the input device of
FIG. 1. It is in communication With the actuator 6 (not shoWn)
shoWn in FIG. 1 as Well, and operates in the manner described
With reference to FIG. 1. In such an embodiment, the PCB 62
20
frequency.
25
apparatus 60 ofFIG. 2 are encased in a case 18 ofthe mobile
screen 15 capable of displaying graphic objects 16 and alpha

30
Referring again to FIG. 2, although in the embodiment

35
function buttons 12a, 12b, 120 are con?gured to indicate

hover/activation signals, in other embodiments, other con
disposed on the PCB 62. The sWitches 19 are in communica

40
meric keys 10 have four or ?ve available states, thus alloWing

them to be associated With three or four (or more) input
signals, such keys 10 may be con?gured to provide input

45
tion With the controller 9 (not shoWn). Suitable sWitches 19

include any analog or digital sWitch, for example rubber
sWitches, snap dome-type sWitches, and pressure sensitive
sWitches. Preferably, the sWitches 19 are capable of produc
ing distinct input signals to the controller. Even more prefer
ably, the sWitches 19 are capable of producing such signals
for tWo or more positions. In the embodiment shoWn, the keys
buttons 11a, 11b, 11c, and assignable-function buttons 12a,
10 contact a plurality of digital sWitches, each capable of

producing four distinct input signals to the controller 9 to
correspond to the four levels at Which pressure is applied to
12b, 120 in the embodiment shoWn in FIG. 2.

Moreover, in the embodiment shoWn, the levels for the
alphanumeric input devices 10 correspond to magnitudes of
able function buttons 12a-12c.

FIG. 4 is a schematic representation through line 4-4 of
FIG. 3 illustrating that the alpha-numeric input buttons or

keys 10 in the mobile telephone pass through the case 18 of
the mobile telephone and contact a plurality of sWitches 19
the pre-assigned buttons 11a, 11b, 11c, and the assignable
signals at, for example, only tWo of the states. In this Way,
such keys 10 may be con?gured to provide hover/activation
signals similar to that Which is provided in the pre-assigned
numeric information 17. The alpha-numeric information 17

that may be displayed includes phone numbers and lists, for
example of list of entries in a phone book, that are input by the

alpha-numeric input buttons 10 and accessed by the assign
contact With PCB62.
?gurations may be used. Moreover, although the alphanu
FIG. 3 shoWs another embodiment of the present invention,

in the form of a mobile telephone 14 having the apparatus of
FIG. 2. The controller 9, actuator 61, and the PCB 62 of the
telephone 14. The mobile telephone 14 also includes a display
is separated at line 62a, such that the PCB4 of FIG. 1 is not in

shoWn there the alphanumeric keys have four or ?ve available
states (embodying an alphanumeric-character selection), and
from the button 101', the controller sends a second control

output signal to the actuator 61, and so on. The ?rst control
output signal is one that causes the actuator to provide a
vibration of a certain, ?rst frequency. The second control
vibration of a certain, higher frequency, and so on. In other
embodiments, the vibrations provided may be of the same
pressure, but in other embodiments the levels resolved can be
the buttons 10 by the user.

The PCB 62, on a side opposite the sWitches, is in commu
type of touch, magnitude, physical position of the sWitch and
nication With the actuator 61 . As illustrated, the actuator 61 is
50
other attributes of contact With the button, or some combina
a pieZo-electric actuator having a metal diaphragm 20 in
tion thereof. The input signals provided by such input devices

may be con?gured accordingly.
contact With the PCB 62 through one or more spacers 21 and

55
In the embodiment shoWn in FIG. 1, the input signals that

are transmitted by the input devices are transmitted to a con
troller 9. In the embodiment shoWn, the controller is in com
munication With storage memory (not shoWn). Examples of

such memory includes Read Only Memory (ROM). The stor
A biasing member arrangement (not shoWn) as is available

60
ciated With various haptic feedback signals. This is explained
2, and not to scale or in physical placement. An alternate

actuator 61 and physical placement of the actuator 61 is
and understood in the art is used to hold the keys in the rest
position 23. An object 24, for example the users ?nger or a

stylus, is used to select one or more of the keys 10 by applying
age memory includes a table in Which input signals are asso
more fully in relation to FIGS. 9-10.

The apparatus 1 shoWn in FIG. 2 also includes an actuator
61. The actuator 61 is shoWn in representative fashion in FIG.
a pieZo ceramic element 22 in contact With the metal dia

phragm 20. Alternative actuator embodiments include a ?ex
ure coupled to the shaft of a motor, secured to the PCB 62.
As illustrated, the keys 10 are initially in a rest position 23.
pressure in the direction of arroW A. This pressure causes the

65
selected key to progress through a plurality of positions. As

illustrated, after leaving the rest position 23, the keys pass
sequentially through a second position 25, a third position,
26, a fourth position 27, and a ?fth position, 28 as greater and
US 7,808,488 B2
10
greater pressure is applied to the button 10. The distance of
travel between each position does not have to be equal, and
the amount of pressure required to move betWeen each posi
tion can vary. In addition, for a given key, the number of
positions can vary from tWo (no pressure and activated) up to
to cause the button to move to its second level 25, the ?rst
signal is transmitted to the controller 6 continuously While the
the number of input signals assigned to a given key. There
button receives pressure to push it at or past the second level

25 but not suf?cient pres sure to push the button 1 01' to the third
level 26. The controller 9 determines the length of time the
button is maintained at the second level 25 by monitoring the

length of time the ?rst signal is transmitted to the controller 9.
If the ?rst signal is received for greater than a pre-determined
length of time, the controller determines that the user Wishes
to select the function associated With the second level 25 by
fore, in the embodiment shoWn, a key 101' is moveable from a

?rst level (rest) 23 to a second level 25 upon the application of
a suf?cient amount of pressure to the input device. In the
embodiment shoWn in FIG. 3, the amount of pressure neces
sary to move the key 101' from rest 23 to the secondposition 25
is about equal to the amount of pressure that users ?nger
Would exert upon contact With the key Without actually select
the fact that the user dWelled at that level for the pre
determined time. Upon so determining, the controller 9 trans

mits a signal to a processor (not shoWn) indicating that the
ing the key.
user has selected the function associated With the second level
of button 101', in this case the selection of the letter W. In
Accordingly, in one method of using the embodiment

shoWn in FIG. 3, When a user of the mobile telephone 14
embodiments, the controller 9, upon so determining, also
shoWn in FIG. 3 presses the 9 key 101' using a relatively light
transmits a signal to the actuator 61 to cause the actuator 61 to
amount of pressure, the button 101' moves from rest state 23 to

its second level 25. Such movement causes the button 101' to
vibrate at a frequency, magnitude, and/or Wave-form indica

tive of selection of the function.
In one embodiment, in addition to providing haptic feed
back to the input device, the controller 9 also sends a signal to
the display 17 to cause the alphanumeric character associated
With the input signal to be displayed. For example, in one
embodiment, upon detecting the presence of a ?rst pressure
apply pressure to sWitch 19a, Which is received by sWitch
20
19a. The sWitch 19a is in communication With the controller

9. The sWitch 19a is con?gured to transmit a ?rst signal to the
controller 9 upon receiving a pressure of magnitude indicat
ing that suf?cient pressure has been placed on button 101' to
move from its ?rst level 23 to its second level 25. The con
25
(through receipt of a ?rst input signal) and suf?cient dWell
troller 9 receives this ?rst signal. The controller 9 is con?g
time to indicate a selection, the controller sends a signal to the
ured to transmit a ?rst controller output signal to the actuator
display 17 indicating that the display should display the letter
61 upon receipt of this ?rst signal from the sWitch 19a. The
controller transmits the ?rst controller output signal to the
X. Upon detecting the presence of a second pressure
actuator 61. The actuator 61 is con?gured to provide a vibra
tion of a ?rst pre-selected frequency to the metal diaphragm

20 of a pre-selected duration upon receipt of such a ?rst
signal. In the embodiment shoWn, the actuator 61 provides a
side-to-side vibration to the diaphragm. The diaphragm 20
thus vibrates at the pre-selected frequency, in turn causing the
(through receipt of a second input signal) and suf?cient dWell

30
display 17 indicating that the display should display the letter

Y. Upon detecting the presence of a third pressure (through
receipt of a third input signal) and suf?cient dWell time to
indicate a selection, the controller sends a signal to the display
35
PCB 62 to vibrate at that same frequency, and thus in turn
causing the sWitches 19 to vibrate at that frequency. The

sWitch 19a is in communication With the button 101', thus
causing the button 101' to vibrate at that frequency.
When the user applies further pressure to the button 101'
time to indicate a selection, the controller sends a signal to the
40
17 indicating that the display should display the letter Z.

The display 17 then displays each of these characters, X, Y, Z.
Various other embodiments may be employed. For
example, instead of having a single actuator to provide feed
back to all input devices receiving such feedback, like the
embodiments shoWn in FIGS. 2-3, other embodiments have
suf?cient to cause the button to move from the second level 25
tWo or more actuators. These tWo or more actuators may be in
to a third level 26, the buttons force is applied to sWitch 19a.

Switch 1911 receives the force and is con?gured to transmit a
second signal to the controller 9 Whenever it receives force to
indicate that the button 101' has moved from the second level
25 to the third level 26. The sWitch 1911 does so, and the
controller 9 receives the second signal. The controller 9 is
con?gured to transmit a second controller output signal to the
communication With all or part of the input devices that

provide tactile feedback. The tWo actuators may each provide
signi?cantly different types of feedback to the same set of
input devices, or each may be in communication With a dif
actuator 61 upon receipt of this second signal from the sWitch

19a. The controller 61 transmits the second controller output
signal to the actuator 61. The actuator 6 is con?gured to
provide a vibration of a second pre-selected frequency, dif
ferent from the ?rst pre-selected frequency, for a pre-deter
mined duration to the metal diaphragm 20 upon receipt of
such a second signal. In other embodiments, the ?rst and
second pre-selected frequencies are the same. As above, the
45
ferent group of input devices to provide the same or different
types of feedback. As another example, the actuator and input

devices may be con?gured to provide vibration to only the
button that is receiving pressure from the user, or they may be
50
con?gured to provide vibration to all buttons or at least more
actuator 61 provides a side-to-side vibration to the dia
buttons than the one(s) receiving pressure from the user.

Moreover, although the actuator 61 is shoWn as disposed
beloW the PCB 62 in FIG. 4, in other embodiments the actua
tor 61 may be disposed at other locations Within the device
having such apparatus, Whether the device is a mobile tele
phone, PDA, or other device. Preferably, the actuator is dis
posed Within the housing of the device. Preferably, it is com
phragm, Which is communicated through the PCB 62 and
munication With the PCB 62, but is placed anyWhere in
sWitches 19 to the button 101'.

When a user applies pressure to the button 101', Which is
55
communication With the PCB 62 as the siZe and space restric
communicated to the sWitch 1911, at each level 25, 26, 27, 28,
tions of the application Will alloW. In other embodiments, the

actuator 61 is located outside the housing of the device (such
a distinct signal is transmitted by the sWitch 19a to the con
as beside it). In still other embodiments, the actuator 61 is in
60
troller 9. Thus, in the embodiment shoWn, different signals
communication With the input devices other than through the
are transmitted by the sWitch 19a for each pressure-applied
PCB 62.
In the embodiment shoWn, a distinct tactile sensation is
levels 25, 26, 27, 28.

In the embodiment shoWn, a dWell to select function is
employed. For example, When a user provides suf?cient input
65
produced for each of the various levels at each of the various

keys. In other embodiments, the controller 6 selects one of a
EXHIBIT C

US008581710B2
(12) Ulllted States Patent
(10) Patent N0.:
Heubel et a].
(54)
SYSTEMS AND METHODS FOR HAPTIC

CONFIRMATION OF COMMANDS
(75)
Inventors: Robert W. Heubel, San Leandro, CA
7/2002 Zirul et al.

8/2002 Roth et al.
2011/0061017 A1
3/2011 Ullrich et al.
2011/0102160 A1
5/2011 Heubel et 31'
Assignee: Immersion Corporation, San Jose, CA

(US)
(*)
Notice:
*Nov. 12, 2013
2002/0098874 A1
2002/0107695 A1
(Us); Erin Ramsay Montreal (CA)

(73)
US 8,581,710 B2
W0
W0 99/045531
9/1999
W0
WO 2004/053830
6/2004
W0
WO 2004/102527
11/2004

OTHER PUBLICATIONS
U'S'C' 154(1)) by 0 days
Patent Cooperation Treaty, International Search Report, Application
This patent is Subject to a terminal dis_
No. PCT/US2010/054967, mailed- Feb. 24, 2011.
Claimem
I I
Patent Cooperation Treaty, Preliminary report on Patentability,

Application No. PCT/US2010/054967, mailed May 18, 2012.
(21) Appl. No.: 13/603,578
* Cited by examiner
(22)
Primary Examiner * Toan N Pham
Flled:
sepl 5 2012
(65)
prior publication Data
US 2013/0088340 A1
(74) Attorney, Agent, or Firm * Kilpatrick Townsend &

Stockton LLP
Apr. 11, 2013
(57)
_

_
_
_
ABSTRACT
_
_
Systems and methods for haptic con?rmation of commands
(63)
Commuanon of aPPheatlOn No- 12/612,230, ?led On

N0V~ 4, 2009, HOW Pat N0~ 8,279,052
are disclosed. For example, one system includes a micro

phone; a housing con?gured to be contacted by a user, and an
actuator in communication With the housing, the actuator
(51)
Int Cl
con?gured to output a haptic effect to the housing. The system
G08B 6/00
(200601)
also includes a processor in communication With the micro
(52)
U-s- Cl-
phone and the actuator, the processor con?gured to receive
(58)
USPC .................... .. 340/4071; 340/407.2; 715/702

Field of Classi?cation Search
USPC .................. .. 340/4071, 407.2; 704/275, 246;
Speech information from the microphone; recognize the

speech information and determine a command associated
with the speech information, If the speech information is
715/780, 702, 816

See application ?le for Complete Search history.
recognized and the command is determined, the processor is

con?gured to generate an actuator signal con?gured to cause
(56)
the actuator to output a haptic effect, and transmit the actuator
References Clted
signal to the actuator. Otherwise, the processor is con?gured

generate a second actuator signal con?gured to cause the
actuator to output a second haptic effect; and transmit the
second actuator signal to the actuator.

3,736,551 A
5,767,796 A
8,279,052 B2 *
5/1973 Hirsch
6/1998 Van Roekel et a1.
10/2012
Heubel et al. ............ .. 340/4072
It
Memory
115
105
Processor
F
Actuator
13
l1_0
L Microphone
1 5
US. Patent
Nov. 12, 2013
Sheet 1 0f 7
2
2
8.5
US 8,581,710 B2
m
<2
10/
5
(I)
+'_"
1Figure
E
E
g
E
m
P
Actuaor
US. Patent
Nov. 12, 2013
on\
?
Sheet 2 017
US 8,581,710 B2
68391
20:62
moSmE
com\>
ll-v
1|
a
;>
E082
6382a
Awconbi
m?
US. Patent
Nov. 12, 2013
Sheet 3 017
US 8,581,710 B2
1.
Elma
lmlwm
Fa
com\>
0|ax96h3o:8i20m5z41.-v
m23ml
v.-n.2oI.082
6305
305 -2
"Qt\\
-.
go_mcwzumoh>gwo<
> |L
EHa
.3a
50:62
m6{o8035t9omazgv.An-
US. Patent
Nov. 12, 2013
Sheet 6 017
US 8,581,710 B2
>
E
o.
.9
D
o
In
co
600f
2:5
5.51
61K0
Memory _6_2_
3:
Proces
529.
Actuaor
O
3
(0
LL
US. Patent
Nov. 12, 2013
Sheet 7 017
US 8,581,710 B2
cow\
hmozwc m 6&6
wucmEo
or/
own
@8659 _5m3c29_4w
0m/
23096 5:25
om/
N959m
US 8,581,710 B2
1
SYSTEMS AND METHODS FOR HAPTIC

CONFIRMATION OF COMMANDS
FIGS. 1-4 shoW block diagrams of systems for haptic con

?rmation of commands according to embodiments of the
present invention;
CROSS-REFERENCE TO RELATED
APPLICATIONS
FIG. 5 shoWs a How diagram of a computer-implemented
method for haptic con?rmation of commands according to

one embodiment of the present invention;
FIG. 6 shoWs a block diagram of a system for haptic con
?rmation of commands according to embodiments of the
This application is a continuation of US. patent applica

tion Ser. No. 12/612,230, entitled System and Methods for
Haptic Con?rmation of Commands, ?led Nov. 4, 2009, now
present invention; and
US. Pat. No. 8,279,052, Which is hereby incorporated by

reference in its entirety.
method for haptic con?rmation of commands according to
FIG. 7 shoWs a How diagram of a computer-implemented

one embodiment of the present invention.
The present invention generally relates to haptic feedback

methods for haptic con?rmation of commands. For example,
and more particularly to haptic con?rmation of commands.
in one embodiment, a user may speak a command into a
BACKGROUND
Commands to electronic devices have typically been

issued by pressing a button or ?ipping a sWitch. HoWever,
voice and other types of commands are becoming more preva
lent in user interfaces, such as the voice-commanded dialing
of cell phones. In such systems, a user may speak a command
into a microphone to dial a friends phone number, and the
microphone on a cellular telephone, such as call home. The

20
successful, it generates a haptic effect to provide a tactile

indication to the user that the spoken command Was recog
niZed and that the phone Will perform the requested function.
The phone may then attempt to make a phone call to a tele
25
user may hear a beep or see a light ?ash to indicate that the
phone is dialing the number. But if the cell phones speaker is

already being used, such as because the phone is playing a
song, the cell phone may interrupt the music to play the beep,
phone attempts to recogniZe the spoken command, and if it is
phone number corresponding to home. HoWever, if the

phone does not recogniZe the command, it generates a second
haptic effect to provide a tactile indication to the user that the
spoken command Was not recogniZed and that no function
Would be executed.
30
In another embodiment, a ?rst user and a second user may
35
be in communication With each other using Wireless commu

nication devices, such as radios. Each user has a headset With
a microphone and a radio in communication With the micro
phone. The headset also includes an actuator for outputting
haptic effects. The ?rst user may issue a verbal command to
or, if sounds have been muted, it may not provide an acknoWl

edgement to the user at all. Thus, it may be desirable to
provide other mechanisms for providing responses to the

user.
the second user over the radio. The second user may be unable
SUMMARY
to provide a verbal response to the ?rst user. For example, the

methods for haptic con?rmation of voice commands. For
example, in one embodiment, a method for haptic con?rma
second user may be located in a combat Zone near enemy

40
or on the radio to cause a haptic message to be transmitted to
tion of commands comprises receiving speech information

from a microphone, recognizing the speech information and
determining a command associated With the speech informa
tion, and if the speech information is recogniZed and the
command is determined, generating a ?rst actuator signal
combatants. The second user may respond to the verbal com

mand by pressing a button or series of buttons on his head set
the ?rst user. The ?rst users radio may receive the haptic
message, and transmit a signal to the actuator in the ?rst user s
headset to cause the actuator to output a haptic effect to the
45 ?rst user.
con?gured to cause an actuator to output a ?rst haptic effect,
Illustrative System for Haptic Con?rmation of
and transmitting the ?rst actuator signal to the actuator. Oth

erWise, generating a second actuator signal con?gured to
Commands
cause the actuator to output a second haptic effect, and trans
50
mitting the second actuator signal to the actuator. Another

embodiment comprises a computer-readable medium com
prising program code for executing such a method.
These illustrative embodiments are mentioned not to limit
or de?ne the invention, but to provide examples to aid under

standing thereof. Illustrative embodiments are discussed in
55
prises a processor 110, a memory 115, an actuator 130, and a
60
These and other features, aspects, and advantages of the
nying ?gures, Wherein:
the housing 105. In the embodiment shoWn in FIG. 1, the

device 100 is a cell phone and the housing 105 is con?gured
to be grasped by a user, Who can speak into the microphone
135. In some embodiments, the device 100 may be a headset
comprising a housing 105, a processor 110, a memory 115, an
actuator 130, and a microphone 135. In some embodiments,
present invention are better understood When the folloWing

Detailed Description is read With reference to the accompa
a block diagram of a system for haptic con?rmation of com

mands according to one embodiment of the present invention.
In the embodiment shoWn in FIG. 1, the system comprises a
device 100 having a housing 105. The device further com
microphone 135, each of Which is disposed in, or coupled to,
the Detailed Description, and further description of the inven

tion is provided therein. Advantages offered by various
embodiments of this invention may be further understood by
examining this speci?cation.
Referring noW to the ?gures in Which like numbers refer to
like elements throughout the several draWings, FIG. 1 shoWs
65
the device 100 may comprise a system having a plurality of

devices. For example, FIG. 2, Which Will be discussed in
greater detail beloW, comprises tWo devices 110, 150 in com
munication With each other.
US 8,581,710 B2
3
Referring again to the embodiment shown in FIG. 1, the

processor 110 is in communication With the memory 115,
actuator 130, and microphone 135. The microphone 135
may generate a third actuator signal and transmit the third

actuator signal to the actuator to cause the actuator to output
a haptic effect to indicate that the voice command Was par
tially recognized. In such an embodiment, the haptic effect
encodes speech information received from the user into one

or more microphone signals that are transmitted to the pro
may indicate not only that the command Was partially recog
nized, but Which part of the command Was not recognized. For
example, if the date Was not recognized, the device may
output a fourth haptic effect, While if the time Was not recog
cessor 110. The processor 110 receives the microphone sig

nals and executes voice recognition software stored in
memory 115 to attempt to recognize the speech information
encoded Within the received microphone signals. If the pro
cessor 110 is able to recognize the speech information, it
executes softWare stored in memory 115 that attempts to
nized, the device may output a ?fth haptic effect. Using haptic
signaling, a user may be apprised of di?iculties understand
ing the spoken command Without needing to resort to vieWing
identify a command associated With the recognized speech
a display to determine the source of the problem.

In a similar embodiment, the processor 110 may generate
actuator signals as parts of a speech command are received
and recognized or not recognized. For example, in one
embodiment, a user may say add appointment to calendar for
information. If the processor 110 identi?es a command asso
ciated With the recognized speech information, it generates a

?rst actuator signal con?gured to cause the actuator to output
a ?rst haptic effect. The ?rst haptic effect is con?gured to
Tuesday, September 2 at 3 pm.The processor may recognize
indicate to the user that the voice command Was recognized.

The processor 110 then transmits the ?rst actuator signal to
the actuator 130, Which outputs the ?rst haptic effect based on
the ?rst actuator signal.
HoWever, if the speech information is not recognized, or if
a command corresponding to the speech information is not
found, the processor 110 generates a second actuator signal
con?gured to cause the actuator 130 to output a second haptic
effect. The processor 110 then transmits the second actuator
20
add appointment to calendar and generate a ?rst actuator

signal to cause the actuator to output a ?rst haptic effect to
indicate that a portion of the command Was recognized. The
?rst haptic effect is then output very shortly after the user has
said add appointment to calendar to indicate to the user that
25
this portion of the speech command Was recognized. Another

haptic effect may then be output after the date is recognized or
not recognized, and a third haptic effect may then be output
signal to the actuator 130, Which outputs the second haptic
after the time is recognized or not recognized. The user may
effect based on the second actuator signal. The second haptic

effect is con?gured to indicate that the speech information
then re-state the unrecognized portion of the command. Hap

tic effects that are output corresponding to parts of a speech
Was not recognized, or that a corresponding command Was
not found. In one embodiment, hoWever, different haptic

effects may be output for a failure to recognize the speech
30
information or failure to ?nd a command corresponding to the
recognized speech information. Note that the identi?ers

?rst and second are used here to differentiate betWeen
different signals and effects, rather than a sequence of par

ticular signals or effects. For example, if the voice command
ments described above in Which a user attempts to add an

35
40
vehicle (eg a car, a Wheelchair, etc.), a non-portable device

(e. g. a desktop computer), or other devices capable of receiv
ing voice commands, processing them, and generating haptic
appointment to a calendar on a PDA, additional haptic effects
may be used. For example, if the processor 110 recognizes the

command, but determines that the commanded date and time
Would present a con?ict With an appointment already stored
is not recognized, only the second actuator signal is generated

and transmitted to the actuator; the ?rst signal, indicating that
the voice command Was recognized, is not generated or trans
mitted to the actuator.
The embodiment in FIG. 1 can comprise any of a number
of devices, such as a handheld device, a Wearable device, a
command may indicate to the user Which portion of the com

mand needs to be repeated rather than requiring the user to
restate the entire command.
Still further haptic effects may be output to a user in
response to a voice command. For example, in the embodi
45
in the calendar, the processor 110 may generate an actuator

signal to cause the actuator to output a haptic effect to indicate
that the command Was recognized, but that a potential con?ict
exists. In a similar embodiment, if the processor 110 recog
nizes the command, but determines that the commanded date
and time are adjacent to an existing appointment, the proces
sor 110 may generate an actuator signal to cause the actuator
effects.
For example, in one embodiment, the device 100 com
to output a haptic effect to indicate that the command Was
prises a personal digital assistant (PDA). In such an embodi
existing appointment. In such an embodiment, the haptic

effect indicating a potential con?ict is different than the hap
tic effect indicating that the appointment time is adjacent to
ment, a user may speak a voice command into the microphone

135 to cause the PDA to perform a function. For example, the
user may instruct the PDA to add an appointment to the
calendar for Tuesday at 3 pm. The processor 110 attempts to
recognize the voice command and if the processor 110 rec
recognized, but that the desired date and time is adjacent to an
50
another appointment.
As described above, some embodiments of the present
invention may attempt to recognize speech information as it is
ognizes the voice command, the processor 110 then generates

an actuator signal and transmits the actuator signal to the
received, such as in real-time. HoWever, even in some

55
actuator to cause the actuator to output a haptic effect to
indicate that the voice command Was recognized. Alterna

tively, if the voice command Was not recognized, the proces
sor 110 may generate a second actuator signal and transmit
the second actuator signal to the actuator to cause the actuator
60
begins to recognize the speech information, it may recognize
to output a haptic effect to indicate that the voice command

Was not recognized.
In a similar embodiment, the processor 110 may partially
recognize the command. For example, the processor 110 may

recognize that the speech command Was to add an appoint
ment to the calendar, but may not have recognized the time of
the appointment. In such an embodiment, the processor 110
embodiments that do not attempt to recognize speech infor

mation as it is received, it may be advantageous to perform
other functions When speech information is received. For
example, in one embodiment, a device 100 receives speech
information from the microphone and attempts to recognize
the speech information as it is received. As the device 100
a Word or phrase that corresponds to a command. For
example, if the user says call home, the device 100 may
recognize the Word call and determine that it corresponds to
65
a command to start a phone call. This may indicate to the
device that the user is issuing a voice command, and the

device may prepare to output a haptic effect before receiving
US 8,581,710 B2
5
the entire command. For example, the device 100 may Warm
up an ampli?er (not shown) that supplies current to the actua

tor 130 in anticipation of generating haptic effect. The device
less connection may comprise an infrared link, an RF link, a

Wi? link, or other Wireless connection. In another embodi
ment, the tWo processors 110, 160 are in communication over
100 may select one or more potential haptic effects that may
a Wired connection, such as a serial or parallel connection or
be output, depending on What additional speech information
Ethernet connection. Using the connection 170, processor

110 is able to transmit signals to and receive signals from
processor 160, though in some embodiments the connection
may be received. In one embodiment in Which the device 100
does not attempt to recogniZe speech information as it is

received, the device 100 may begin to Warm-up an ampli?er
When it detects that it is receiving speech information. Such
an embodiment may determine that speech information is
being received and the speech information may comprise a
voice command. Thus, it may be advantageous for the device
100 to Warm-up the ampli?er While it is receiving the speech
information to ensure a haptic effect may be output shortly
170 may be a one-Way connection.
For example, in one embodiment, device 100 comprises a

Bluetooth headset and device 150 comprises a cell phone. In
such an embodiment, The microphone 135 encodes speech
information received from the user into one or more micro
phone signals that are transmitted to the processor 110. The

processor 110 in the headset then causes the microphone
signals to be transmitted to the processor 160 in the cell
phone. Processor 160 then executes voice recognition soft
Ware stored in memory 165 to attempt to recogniZe the speech
after a voice command has been recogniZed or not recog
niZed.
In addition to handheld embodiments, other devices may
provide haptic con?rmation of voice commands. For

example, in one embodiment, the device 100 comprises a
desktop computer. A user may issue a command to cause the
computer 100 to perform a function that takes an extended
period of time to complete, such as to execute a softWare
20
information encoded Within the received microphone signals.

If the processor 160 is able to recogniZe the speech informa
tion, it executes softWare stored in memory 165 that attempts
to identify a command associated With the recogniZed speech
build. If the processor 110 recogniZes the command, it may
information. If the processor 160 identi?es a command asso
generate a ?rst actuator signal con?gured to cause the actua

tor 130 to output a haptic effect to the user, such as through an
actuator 130 embedded Within a keyboard or mouse (not
ciated With the recogniZed speech information, it generates a
shoWn) coupled to the computer 100. Then, as the processor

110 executes the function, it may periodically transmit a
second actuator signal to the actuator 130 Within the keyboard
or mouse to indicate that the function is still executing. Such
a haptic effect may be a loW magnitude vibration designed to
provide unobtrusive status information to the user. In another
25
?rst actuator signal con?gured to cause the actuator to output

a ?rst haptic effect. The processor 160 transmits the ?rst
actuator signal to processor 1 10, Which then transmits the ?rst
actuator signal to the actuator 130. In some embodiments, the
processor 110 transmits a signal to the actuator to cause the
30
actuator to poWer-up in preparation for outputting a haptic

effect. In another embodiment, processor 160 transmits the
?rst actuator signal to the actuator 130.
HoWever, if the speech information or a corresponding
command is not recognized, the processor 160 generates a
For example, the magnitude of the vibration may increase, or
35
the frequency may increase or decrease. When the processor

110 completes the function, it may generate and transmit a
third actuator signal to the actuator 130 Within the keyboard
or mouse to indicate that the function has completed. Alter
natively, if an error occurs during execution of the function,
40
second actuator signal con?gured to cause the actuator 130 to

output a second haptic effect. The processor 160 transmits the
second actuator signal to processor 110, Which then transmits
the second actuator signal to the actuator 130. In another
embodiment, processor 160 transmits the second actuator
signal to the actuator 130. After receiving the second actuator
signal, the actuator 130 outputs a second haptic effect based
on the second actuator signal.
In some embodiments, processor 160 may transmit high
level actuator signals to processor 110. Processor 110 may
then generate a loW-level actuator signal based on the high
level actuator signal. Processor 110 may then transmit the
embodiment, the vibration may initially be unobtrusive, but

may get stronger as the function gets nearer to completion.
such as a build error, a haptic effect may be output to indicate
that the function terminated unsuccessfully. Such an embodi

ment may be useful in contexts Where a user begins a function
on a device 100 and alloWs the function to execute in the
background, but the device 100 is able to keep the user

apprised of the status of the function. For example, a user may
45
issue a voice command to his cell phone to initiate a doWnload
loW-level actuator signal to the actuator 130 to cause the
of an application from a remote server over a cellular net
doWnload to the user, such as percentage of the doWnload
actuator 130 to output a haptic effect. More detailed descrip

tions of high-level and loW-level actuator signals may be
found inU.S. Pat. No. 5,734,373, entitled Method andAppa
ratus for Controlling Force Feedback Interface Systems Uti
liZing a Host Computer, ?led Dec. 1, 1995 and issued Mar.
completed, a problem With the doWnload, or a haptic indica

tion When the doWnload has completed.
Referring noW to FIG. 2, FIG. 2 shoWs a block diagram of
a system 200 for haptic con?rmation of commands according
to one embodiment of the present invention. The system 200
reference.
Referring again to FIG. 2, one embodiment of the present
invention may provide haptic effects that emulate the com
mand determined by the processor 110, 160 or the function
Work. The user may then put the cell phone back in his pocket,
initiate a phone call, or perform some other task. HoWever, the
cell phone may still be able to provide haptic status of the
50
31, 1998, the entirety of Which is incorporated herein by

55
performed by the processor 110, 160. For example, in one
shoWn in FIG. 2 comprises a ?rst device 100 and a second
device 150 Where each device 100, 150 is in communication

With the other device using connection 170. The ?rst device
100 comprises the components described With respect to FIG.
1. The second device comprises a processor 160 in commu
nication With a memory 165. The processor 110 in the ?rst
device 140 is in communication With the processor 160 in the
second device 150. For example, in one embodiment, the tWo
processors 110, 160 communicate over a Wireless connec
tion, such as via Bluetooth. In some embodiments, the Wire
60
embodiment, the ?rst device 1 00 comprises an earbud and the

second device 150 comprises a cell phone. In such an embodi
ment, a user may issue a voice command, such as call home.
65
The processor 110 transmits the speech information received

from the microphone 135 to the processor 160 in the second
device 150. The processor 160 recogniZes the voice command
and generates a series of haptic effects corresponding to each
number dialed and each ring of the phone being called. Actua
tor signals corresponding to the haptic effects are transmitted
US 8,581,710 B2
7
to the processor 110 in the ?rst device 100, Which outputs the
haptic effects, such as at a time corresponding to each button
prises a device 100 as shoWn in FIG. 1. Device 425 also
press and each ring.
and an actuator 440 as described With respect to FIG. 1.
comprises a processor 430, a memory 115, a microphone 435,
HoWever, each device 410, 412 also comprises a netWork

interface 315. Each netWork interface 315 is in communica
tion With its respective processor 110, 430 and is con?gured
to communicate With netWork 480, thus enabling the devices
In other embodiments, a haptic command may be received

from a processor associated With a remote device. For
example, FIG. 3 shoWs a block diagram of a system 300 for

haptic con?rmation of commands according to one embodi
ment of the present invention. In the embodiment shoWn in
FIG. 3, a system 300 for haptic con?rmation of commands
comprises a haptically-enabled medical alert device 100 in
410, 412 to communicate With each other over netWork 480.

In such an embodiment, device 410 is capable of receiving a
voice command from its microphone 435 and transmitting the

voice command to device 412. For example, in one embodi
communication With a communications device 150. The

communications device is in communication With a remote
ment, the speech information received from the microphone
system 170. In this embodiment, the medical alert device 100

is in communication With the communication device 150
using an RF link, such as an RF link using standard cordless
135 may be transmitted over the netWork 480 so that a user of
device 412 can hear the voice command, such as via a speaker
(not shoWn). The user may con?rm that the command Was
recogniZed by pressing a button on device 425 to send a signal
indicating an acknoWledgment that the voice command Was
phone frequencies. The medical alert device 100 is con?g

ured to be Worn by a user, such as around the users neck or
received. Device 410 may receive the signal, and processor

110 may generate an actuator signal and transmit the actuator
Wrist. The medical alert device 100 can be used by the user to
indicate a medical emergency, such as a fall or heart attack.
For example, the user may speak into a microphone on the
20
device, such as by saying help or by pressing a button and

speaking a command, such as help or emergency. In the
embodiment shoWn in FIG. 3, the processor 110 in the medi
cal alert device 100 then transmits one or more signals to the
processor 160 in the communications device 150. The com
munications device 150 then initiates a connection to the
remote system 170 and transmits one or more signals to the
remote system 170 indicating that the user has signaled a
medical emergency. In such an embodiment, the remote sys
tem 170 may be monitored by a dispatcher Who can dispatch
25
In another embodiment, a user may respond to a received
ing a button or tapping on a touch-sensitive device. Each of
the haptic inputs may be correlated to a response, Which may

30
command into a touch-sensitive input device coupled to a
35
Was received and that help is on the Way. The processor 160
may then transmit a high-level actuator signal to the processor
110 in the medical alert device to cause a haptic effect to be
output. The processor 110 in the medical alert device may
then generate a loW-level actuator signal based on the
received signal from the processor 1 60 and transmit the actua
tor signal to the actuator to cause a haptic effect to be output.
40
spoken message. In such an embodiment, the tactile response

may correspond to a pre-recorded message or to pre-deter
mined text that may be converted to speech by the device 412.
Such an embodiment, Would alloW for a silent acknoWledge
ment by a user that results in a spoken reply to the recipient.
45
The system shoWn in FIG. 4 may also be advantageously

used by multiple users to perform coordinated tasks or pro
cedures. For example, in one embodiment, devices 410 and
412 may comprise computers in different locations Within a
bank. Each computer 410, 412 is in communication With a
security system that controls access to a bank vault. Users at
the tWo computers may need to perform functions, or provide

50
such as a Word or phrase, to verify authorization to open the
vault. After the command is recogniZed, a haptic con?rmation

may be sent to a second user at the second computer 412 to
55
patched. The dispatcher may periodically transmit such hap

tic signals to indicate that he is still paying attention to the
user and their request is being responded to. Such an embodi
ment may be advantageous for a person Who has dif?culty
hearing or seeing. It may also provide assurance to the user
that their request for help has been received and that the
indicate that they need to speak a command, such as another

Word or phrase, to complete authentication to open the vault.
If the second users command is recognized, a haptic con?r
mation may be output at each of the tWo devices 410, 412 to
indicate to the users that the command Was successful. At
60
such a time, the lock on the bank vault may be disengaged.

HoWever, if one of the commands Was not recogniZed, one
or both users may receive a haptic indication that the authen
tication failed. In one embodiment, the user Whose command
dispatcher is still handling their needs.


shoWn in FIG. 4 comprises tWo devices 410, 412 in commu
nication With each other via netWork 480. Device 410 com
authentication to open the bank vault. For example, a user at
the ?rst computer 410 may be required to speak a command,
For example, the dispatcher may transmit a long vibration to

indicate that the request Was received, folloWed later by addi
tional vibrations to indicate that an ambulance has been dis
Will comply. The device 412 may then transmit the tactile
response to another device, such as device 410, Which then
converts the tactile response into an audible signal, such as a
ted across the netWork 380 to the processor 160 in the com
munications device 150. For example, the dispatcher can

provide haptic signals to the user to indicate that their request
be converted to a spoken command and transmitted to a

recipient. For example, a soldier may tap a response to a
communication device, such as device 412, in a sequence to

indicate that the command Was received and that the soldier
communication device 150 may send a message to the medi

cal alert device 100 that causes the medical alert device 100 to
output a haptic effect to indicate that an emergency call has
been initiated. After the call has been ansWered, the commu
nication device 150 may cause a second haptic effect to be
output to indicate that the call Was ansWered. Alternatively, or
in addition, the dispatcher may cause a signal to be transmit
by providing a haptic indication that the command Was not

received, or that the command Was garbled or unintelligible.
In one such embodiment, the user may also provide a haptic
indication to request a re-transmission of the message.
voice command by providing haptic inputs, such as by press
an ambulance or other services to aid the user.
During the transmission of messages betWeen the medical

alert device 100, the communication device 150, and the
remote system 170, additional haptic effects may be gener
ated and output. For example, in the embodiment shoWn, the
signal to the actuator to cause the actuator to output a haptic

effect to the user. In some embodiments, the user may respond
65
failed may be given a second opportunity to provide authen

tication. In such an embodiment, a haptic effect indicating
that the authentication failed but that the command may be
retried may be output to one or both users. HoWever, if the
US 8,581,710 B2
10
information. HoWever, if no additional speech information is
received, the method 500 proceeds to step 536.
authentication has ?nally failed such that no retries are avail

able, a different haptic effect may be output to one or both
users to indicate that authentication failed.
At block 536, the processor 110 executes a function asso
ciated With the recogniZed speech information and the com
Still further embodiments including different or more com
plicated sequences of events may be used. In such embodi
mand. For example, if the speech information comprised call
ments, haptic effects may be output to one or more users to
home and the determined command to dial a phone number

associated With a contact called home, the processor Would
then execute a dialing function to dial the number associated
With home. After the function has begun, the method pro
ceeds to block 540 Which is a test condition for a loop based
on Whether the function has completed or not.
In block 540, the processor 110 determines Whether the
function has completed or not. If the function has not com
indicate that action on their part is necessary to complete the
sequence. For example, in one embodiment, multiple

mechanics may be repairing a large machine. In such a sce
nario, each user may Wear a device, such as device 410 or 412.
As users complete portions of the repair or need assistance,

they may transmit commands to trigger haptic effects on other
users devices 410, 412. By doing so, the next user may be
noti?ed that a task is complete or that they need to perform
some action to continue the repair process. The recipient of
the haptic effect may provide a haptic con?rmation back to
the sender of the command to indicate that the recipient Will
pleted, the method proceeds to step 542 to output a haptic

effect to indicate that the function is still executing. HoWever,
if the function has completed, the method proceeds to step
550, Where another haptic effect is output to indicate that the
function has completed.
take the necessary action or some other indication.
Referring noW to FIG. 5, FIG. 5 shoWs a How diagram of a
computer-implemented method 500 for haptic con?rmation
20
of voice commands according to one embodiment of the

present invention. The method 500 shoWn in FIG. 5 Will be
discussed With reference to the device 100 shoWn in FIG. 1.
The method 500 begins in block 510 When a processor 100
receives speech information. For example, the processor 110

may receive speech information encoded in a microphone
signal from a microphone 135 or it may receive the speech
haptic effect that indicates that the function is still executing.

In one embodiment, the processor 110 may generate the same
25
information from another processor, such as over a Bluetooth
connection. After receiving speech information, the method

proceeds to block 520.
30
At block 520, after receiving the speech information, the

processor 100 then executes speech recognition software to
recogniZe the speech information. If the processor recogniZes
the speech information, it attempts to determine a command
associated With the speech information. If the processor 110
determines a command associated With the speech informa
tion, the method proceeds to step 530. HoWever, if the pro
cessor is unable to recogniZe the speech information, or is
unable to determine a command associated With the speech
information, the method proceeds to block 522.
In block 522, the processor 110 generates a second actuator
35
40
45
50
user that the speech information Was recogniZed and that a
55
to block 534.
60
provide haptic effects to indicate that parts of the speech

information Were recogniZed. For example, a user may
attempt to add an event to calendar, and the processor may
perform steps 510-532 or steps 510-524 for each component

of the event, such as the date, time, and location of the event.
If additional speech information is available to be received,
the method returns to step 51 0 to receive the additional speech
signal associated With a haptic effect that indicates that the
After generating an actuator signal in block 542, in block

544 the processor 110 transmits the actuator signal to the
actuator signal to the actuator 130 as shoWn in block 524.

In block 530, the processor 110 generates a ?rst actuator
signal con?gured to cause an actuator to output a ?rst haptic
In block 534, the processor 110 determines Whether addi

tional speech information is available. For example, as
described above, the processor 110 may attempt to recogniZe
parts of the speech information as it is received and then
signals to generate increasingly stronger haptic effects for
function encountered an error.
cate to the user that the speech information Was not recog
command corresponding to the speech information Was

found. The processor 110 then transmits the actuator signal to
the actuator 130 as shoWn in block 532. After transmitting the
?rst actuator signal to the actuator 130, the method proceeds
each time the haptic effect is output. Thus, a user Would feel
the same haptic effect about every 30 seconds. HoWever, in
some embodiments, different haptic effects may be output
depending on the status of the function. For example, in one
embodiment, the processor 110 may generate different actua
tor signals each time the method returns to block 542. In such
an embodiment, the processor 110 may generate actuator
the function is still executing, but encounters an error. In such

an embodiment, the processor 110 may generate an actuator
signal con?gured to cause an actuator to output a second
effect. The ?rst haptic effect is con?gured to indicate to the
actuator signal regardless of hoW many times the haptic effect

has been output. For example, if the function requires ten
minutes to complete and a haptic effect is output every 30
seconds, the processor may generate the same actuator signal
each iteration through blocks 542-546. In one embodiment,

the processor 110 may generate a different actuator signal if
haptic effect. The second haptic effect is con?gured to indi

niZed, or that no command corresponding to the speech infor
mation Was found. The processor 110 then transmits the
Blocks 542 through 546 are steps to output a haptic effect

to indicate that a function is still executing. In block 542, the
processor 110 generates an actuator signal corresponding to a
65
actuator 130 to cause the actuator to output a haptic effect. In

some embodiments, the processor may transmit the actuator
signal to another device comprising an actuator. For example,

in the system shoWn in FIG. 2, processor 1 60 may transmit the
actuator signal to processor 110. Processor 110 may then
transmit the actuator signal to the actuator 130. As described
above, processor 160 may transmit a high-level actuator sig
nal to processor 110, Which may then generate a loW-level
actuator signal that the processor 110 then transmits to the
actuator 130. After the actuator signal has been transmitted,
the method proceeds to block 546.
In block 546, the method delays for a period of time before
returning to block 540.A delay may be advantageous to alloW
time to pass betWeen haptic effects. For example, a delay may
alloW haptic effects to be output once every 30 seconds so that
the user is not feeling haptic effects repeatedly over very short
periods of time, Which may be distracting to the user. HoW
ever, in some embodiments, the delay time may be set to 0,
Which may alloW haptic effects to be output as needed. After
the delay has elapsed, the method returns to block 540.
After the function has completed, the method proceeds to
block 550 at Which time the processor 110 generates an actua
tor signal corresponding to a haptic effect that indicates that

the function has completed. For example, in one embodiment,
US 8,581,710 B2
11
12
the processor 110 may generate an actuator signal corre

sponding to a haptic effect that indicates the function com
activities from another location, such as on a ship or at another
military installation. In such an embodiment, each of the

other locations may receive communications to and from the
pleted successfully. Alternatively, the processor 110 may gen

indicates the function encountered an error and terminated
squad, including haptic commands and con?rmations from

the squad.
before running to completion. After the actuator signal has

been generated, the method proceeds to block 552 Where the
Referring again to FIG. 6, one embodiment of the present

invention may output haptic effects that emulate the com
processor 110 transmits the actuator signal to the actuator 130

to cause the actuator 130 to output a haptic effect. After
mand issued or the function executed. For example, in one

embodiment, a device 610 may be con?gured to control a
vehicle, such as a Wheelchair. In such an embodiment, it may
erate an actuator signal corresponding to a haptic effect that
Which, the method returns to block 510 and the processor 110
attempts to receive more speech information.
be advantageous to provide haptic effects that emulate com

mands issued to the vehicle. For example, if a user selects a

comprises a device 610 having a housing 605. The device 610
further comprises a processor 620, a memory 625, an actuator
630, and a sensor 640. The processor 620, the memory, 625,
the actuator 630, and the sensor are each disposed Within or
coupled to the housing 605. The processor 620 is in commu
nication With each of the memory 625, the actuator 630, and
the sensor 640. The processor is further in communication
command to turn the vehicle to the right, the processor 620

may generate an actuator signal to cause the actuator to output
a haptic effect the emulates a right turn. For example, in a
Wheelchair embodiment, the actuator may comprise a plural
20
actuators are activated in succession. The user may thus per
With a display 650, Which may coupled to the housing, dis

posed Within the housing, or a separate device. The device
610 is con?gured to receive commands based on sensor sig
nals received from the sensor and to output haptic effects
ceive the vibration moving to the right across their back and
therefore receive feedback indicating that the command Was
recogniZed.
25
30
sponding to commands, such as commands to control a

vehicle (eg a Wheelchair) or commands to send to another
detects the orientation of the users eye is looking and trans
40
haptic effect. The processor 620 then transmits the actuator

signal to the actuator 630. In another embodiment, the device
610 further comprises a netWork interface, such as netWork
interface 315 shoWn in FIG. 4. In such an embodiment, the
processor 620 may receive the command from a second
device, such as device 412 shoWn in FIG. 4, across a netWork
480. The device 610 may output a haptic effect to the user to
indicate a command has been received and may display pos
sible responses to the command on the display 650. The user
of the device 610 may select a response from the available
responses to send to the second device 412 across the net
Work. Such an embodiment may be useful in a covert military
operation Where spoken commands or responses may not be

desirable.
In a related embodiment, the selected response may be sent
to a plurality of recipients. For example, a military unit, such
interface element at a location corresponding to a sensed
sor 620 determines a command corresponding to information

45
50
55
60
con?gured to indicate that a command Was not recogniZed

based on the input received from the sensor. After generating
the actuator signal, the method proceeds to step 724 When the
processor transmits the actuator signal to the actuator 630 to
If the method proceeds to block 730, the processor 620

generates an actuator signal corresponding to a haptic effect
con?gured to indicate that a command Was recogniZed based
on the input received from the sensor. After generating the
actuator signal, the method proceeds to step 750 When the
processor transmits the actuator signal to the actuator 630 to
cause the actuator to output a haptic effect. After outputting
the haptic effect, the method proceeds to block 750 the pro
cessor 610 executes a function associated With the command,
edgements from each soldier. A further embodiment may

include multiple command centers to receive responses from
each of the soldiers. For example, a commander of the squad
may be in a base camp near the squad s operations or may be
received in the sensor signal, the method 700 proceeds to

block 730. OtherWise, the method 700 proceeds to block 722.
If the method proceeds to block 722, the processor 620
generates an actuator signal corresponding to a haptic effect
cause the actuator to output a haptic effect.
command center and provide haptic responses to received
With the squad but separated due to the nature of the opera
tion. In addition, other locations may monitor the squads
a touch-sensitive display, such as display 650. In another

embodiment, the sensor signal may correspond to an orien
tation of a users eye. After receiving the sensor signal, the
method 700 proceeds to block 720.
In block 720, the processor 620 attempts to identify a
command corresponding to the received sensor signal. For
example, the processor 620 may identify a button or user
position on the touch-sensitive display screen. If the proces
as a squad of soldiers, may be in communication With a

voice commands. In such an embodiment, the command cen
ter may transmit voice or haptic commands to each of the
soldiers in the squad and may receive responses or acknoWl
commands according to one embodiment of the present

invention. FIG. 7 Will be described With respect to the system
600 shoWn in FIG. 6.
In the embodiment shoWn in FIG. 7, the method 700 begins
in block 710 When the processor 620 receives a sensor signal
from the sensor 640. For example, in one embodiment the
sensor signal may correspond to a user touching a location on
person, such as an order to a soldier. To issue a command, a 35

user may look at the desired command. The sensor 640
mits a sensor signal to the processor 620. The processor 620

determines a command associated With the orientation and
generates an actuator signal to cause the actuator to output a
Referring noW to FIG. 7, FIG. 7 shoWs a How diagram of a
computer-implemented method for haptic con?rmation of
based on the received sensor signals.

In one embodiment the device 610 may be carried or Worn
by a user and the sensor 640 may comprise an optical sensor
con?gured to detect a location a users eye is vieWing. For
example, display 650 may comprise a display Worn by a user,

such as goggles With an integrated display. The processor 620
may cause the display 650 to display images or text corre
ity of actuators. The actuators may then output a vibration that

begins in the middle of the back of the Wheelchair and moves
to the right. For example, the actuators may be placed along a
horiZontal line in the back of the Wheelchair, such that the
though in some embodiments, the processor 610 may begin
executing the function before generating the actuator signal

65
or transmitting the actuator signal to the actuator.

While the methods and systems herein are described in
terms of softWare executing on various machines, the meth
ods and systems may also be implemented as speci?cally
US 8,581,710 B2
14
13
con?gured hardware, such a ?eld-programmable gate array
(FPGA) speci?cally to execute the various methods. For
example, referring again to FIGS. 1-4 and 6, embodiments
an actuator con?gured to output a haptic effect to the hous
ing; and
a processor in communication With the sensor and the
can be implemented in digital electronic circuitry, or in com
actuator, the processor con?gured to:
puter hardWare, ?rmWare, software, or in combination of
receive, from the sensor, a sensor signal associated With
them. In one embodiment, a computer may comprise a pro

cessor or processors. The processor comprises a computer
readable medium, such as a random access memory (RAM)
recogniZe the user input and determine a command asso

ciated With the user input;
a user input;
if the user input is recogniZed and the command is deter

mined:
generate a ?rst actuator signal con?gured to cause the
actuator to output a ?rst haptic effect; and
transmit the ?rst actuator signal to the actuator;
otherWise:
coupled to the processor. The processor executes computer

executable program instructions stored in memory, such as
executing one or more computer programs for editing an
image. Such processors may comprise a microprocessor, a
digital signal processor (DSP), an application-speci?c inte

grated circuit (ASIC), ?eld programmable gate arrays (FP
generate a second actuator signal con?gured to cause

the actuator to output a second haptic effect; and
transmit the second actuator signal to the actuator.
2. The system of claim 1, Wherein the sensor comprises an
optical sensor.
GAs), and state machines. Such processors may further com
prise programmable electronic devices such as PLCs,
programmable interrupt controllers (PlCs), programmable

logic devices (PLDs), programmable read-only memories
(PROMs), electronically programmable read-only memories
(EPROMs or EEPROMs), or other similar devices.
Such processors may comprise, or may be in communica
20
tion With, media, for example computer-readable media, that

may store instructions that, When executed by the processor,
can cause the processor to perform the steps described herein
as carried out, or assisted, by a processor. Embodiments of
25
computer-readable media may comprise, but are not limited

to, an electronic, optical, magnetic, or other storage device
capable of providing a processor, such as the processor in a
Web server, With computer-readable instructions. Other

examples of media comprise, but are not limited to, a ?oppy
30
disk, CD-ROM, magnetic disk, memory chip, ROM, RAM,

ASlC, con?gured processor, all optical media, all magnetic
tape or other magnetic media, or any other medium from
Which a computer processor can read. The processor, and the
processing, described may be in one or more structures, and
may be dispersed through one or more structures. The pro
cessor may comprise code for carrying out one or more of the
35
40
The foregoing description of some embodiments of the

invention has been presented only for the purpose of illustra
45
50
phrase in one embodiment or in an embodiment in vari
55
a housing con?gured to be contacted by a user;
phone, a personal digital assistant, a touch-sensitive surface,

a medical alert device, a mouse, or a keyboard.
11. The system of claim 1, Wherein the user input com

prises an indication of a medical emergency.
12. The system of claim 1, Wherein command is a request

associated With an operation, and the ?rst haptic effect is
con?gured to provide information associated With the status
of the operation.
13. A computer-implemented method comprising the steps
of:
60
receiving, from a sensor, a sensor signal associated With a
user input;
recogniZing the user input and determining a command
associated With the speech information;
That Which is claimed is:
a sensor;
8. The system of claim 1, Wherein the ?rst haptic effect is

con?gured to emulate the command.
9. The system of claim 1, Wherein the sensor, actuator, and
processor are disposed Within the housing.
one of: a device con?gured to be Worn by the user, a mobile
ment means that a particular feature, structure, operation, or

other characteristic described in connection With the embodi
ment may be included in at least one implementation of the
invention. The invention is not restricted to the particular
embodiments described as such. The appearance of the
1. A system for generating haptic effects to con?rm receipt

of a command, the system comprising:
communication With the processor, and Wherein the proces

sor is further con?gured to:
generate a display signal con?gured to cause the display to
10. The system of claim 1, Wherein the housing comprises
Reference herein to one embodiment or an embodi
ous places in the speci?cation does not necessarily refer to the

same embodiment. Any particular feature, structure, opera
tion, or other characteristic described in this speci?cation in
relation to one embodiment may be combined With other
features, structures, operations, or other characteristics
described in respect of any other embodiment.
communication With the processor, and Wherein the proces

sor is further con?gured to:
generate an audio signal con?gured to cause the speaker to
output a sound; and
display an image; and

transmit the display signal to the display.
tion and description and is not intended to be exhaustive or to
limit the invention to the precise forms disclosed. Numerous

modi?cations and adaptations thereof Will be apparent to
those skilled in the art Without departing from the spirit and
scope of the invention.
completion of execution of the function.

5. The system of claim 4, Wherein the third haptic effect is
con?gured to emulate the function.
6. The system of claim 1, further comprising a speaker in
transmit the audio signal to the speaker.

7. The system of claim 1, further comprising a display in
methods (or parts of methods) described herein.

General
3. The system of claim 2, Wherein the user input comprises

a focal point ofa users eyes.
4. The system of claim 1, Wherein the processor is further
con?gured to, if the user input is recogniZed and the com
mand is determined:
execute a function associated With the user input;
generate a third actuator signal con?gured to cause the
actuator to output a third haptic effect; and
transmit the third actuator signal to the actuator upon
65
if the user input is recogniZed and the command is deter

mined:
generating a ?rst actuator signal con?gured to cause an
actuator to output a ?rst haptic effect; and
US 8,581,710 B2
15
16
transmitting the ?rst actuator signal to the actuator;

otherwise:
17. The computer-implemented method of claim 16,

Wherein the third haptic effect is con?gured to emulate the
generating a second actuator signal con?gured to cause

the actuator to output a second haptic effect; and
transmitting the second actuator signal to the actuator.

Wherein the sensor comprises an optical sensor.

Wherein the user input comprises a focal point of a users
eyes.

Wherein if the user input is recogniZed and the command is
determined:
executing a function associated With the command;
generating a third actuator signal con?gured to cause the
actuator to output a third haptic effect; and
transmitting the third actuator signal to the actuator upon
completion of execution of the function.
function.
18. The computer-implemented method of claim 13, fur
ther comprising:
generating an audio signal con?gured to cause a speaker to
output a sound; and
transmitting the audio signal to the speaker.

19. The computer-implemented method of claim 13, fur
ther comprising:
generating a display signal con?gured to cause a display to
display an image; and
transmitting the display signal to the display.

Wherein the ?rst haptic effect is con?gured to emulate the
command.
EXHIBIT D

US007336260B2
(12)
(54)
United States Patent
(10) Patent N0.:
Martin et al.

3,220,121 A
3,497,688 A
US 7,336,260 B2
Feb. 26, 2008
11/1965 Cutler
2/1970 Hirsch
(75) Inventors: Kenneth M. Martin, Los Gatos, CA
(Continued)
City, CA
Steven
(US);P-Alex
Vassallo,
S. Goldenberg,
Redwood
Portola Valley, CA (US); Alexander
Jasso, Los Altos, CA (US); Kollin
EP
0349086
Tierling, Milpitas, CA (US)
(Con?rmed)
OTHER PUBLICATIONS
(US)
(*)
Notice:
I/ 1990
_
Adelstein, A Virtual Environments System For The Study of

Human Arm Tremor, Ph.D. Dissertation, Dept. of Mechanical
Engineering, MIT, JllI1~ 1989

(Continued)
U.S.C. 154(b) by 730 days.

Primary ExamineriRicardo Osorio
(21) APP1~ NOJ 10/285,450
(74) Attorney, Agent, or FirmiKilpatrick Stockton LLP
(22) Filed:
(57)
(65)
Nov. 1, 2002

Us 2003/0122779 A1
]u1_ 3, 2003
ABSTRACT
Products and processes for providing tactile sensations to

input devices or electronic devices are provided. Input
devices include mechanical input devices (such as, for
example, mechanical switches) and non-mechanical input
(60) Provisional application No. 60/335,493, ?led on Nov.
fievlces-sgcl as -for example touchp?dsy Tacnl-e feedback
(51)
52
1 2001 provisional application No 60/399 883 ?led
15- pro-vl e - y uslpg an acu-lator or 0t er means n Commu
J 1 151 2002
On u '
'
n1cat1on W1th the 1nput device or electromc device. A con

troller may be employed to receive signals from the input
1e
'
""
0
(56)
""
assl ca Ion
P
device or electronic device may be provided in response to
(2006 01)
U 5 Cl
)
devices and control the actuator. Tactile feedback to an in ut
Int Cl
GobG /00
(58) F: l'd
(
'
345/169 3 4 5 /1 5 6
one or more events or situations. Such an event or situation
345/156
may be any one designated. Examples of such events and
"""
situations include the level of pressure placed on an input
345/173 157 161 163} 167T169
earc
""""""" "
device; the availability or lack of availability of a function
See apphcanon ?le for Complete Search hlstory'

References Cited
associated With an input device; and the function, menu, or

mode of operation associated With an input devices activa
tion. A variety of feedback types and combinations may be
selected.

2,972,140 A
2/1961 Hirsch
3,157,853 A
11/1964 Hirsch

24
l8
18
35
W Z)
s
34
\_
W A
\
\Q
30
j '4
\_ 34
33\
21
21
20
/
64
13
/
22
US 7,336,260 B2
Page 2
US. PATENT DOCUMENTS
3,517,446 A
3,623,064 A
3902687 A
5,496,174
5,514,150
5521336
5,547,382
6/1970 Corlyon et al.

11/1971 Kagen
3/1996
5/1996
5/1996
8/1996
Garner
Rostoker
Buchanan
Yamasaki et a1.
5,575,761 A
11/1996 Hajianponr
3,903,614 A
3,911,416 A
9/1975 Diamond et a1.

10/1975 Peder
5,631,861 A
5,684,722 A
4,127,752
4,160,508
4,236,325
4,262,549
4,311,980
4,333,070
11/1978
7/1979
12/1980
4/1981
1/1982
6/1982
5,691,747 A
5,709,219 A
571997
11/1997
l1/1997
1/1998
3/1998
6/1998
6/199g
7/1998
A
A
A
A
A
A
9/1975 Hightower
A
A
A
A
Lowthorp
Salsbury
Hall et 31.
Schwellenbach
Prudenziati
Barnes
5,729,249 A
5,766,016 A
5,767,457 A
5,785,630 A
4,362,408 A
12/1982 Cordes et a1. ............ .. 400/481
5,791,992 A
4,464,117
4,484,191
4,513,235
4,581,491
4,581,972
4,599,070
4692756
4,708,656
4,713,007
4,725,817
4,791,416
4,794,392
4,795,296
4,798,919
4,821,030
4,823,106
4,840,634
4,885,565
4,891,764
4,930,770
4,934,694
4982916
8/1984
11/1984
4/1985
4/1986
4/1986
7/1986
9/1987
11/1987
12/1987
2/1988
12/1988
12/1988
V1989
V1989
4/1989
4/1989
6/1989
12/1989
1/1990
6/1990
6/1990
l/199l
5,844,392
5,857,986
5,887,995
5,889,670
5,889,672
5,917,906
5,943,044
5,945,772
5,977,867
5,988,902
6,059,506
6,078,126
6,097,964
6,111,577
6,118,435
6,131,097
6,160,489
6,195,592
6,198,206
6,218,966
6,219,034
6,225,976
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
Foerst
Vavra
Acklam et 31.
Boothroyd
Hoshino
Hladky er 91
Clark
De Vries et 31.
Alba
Wihlborg
Adler
Selinko
Jau
Suita
Batson
Pope
Muller
Embach
McIntosh
Baker
McIntosh
DuPOIlt
4,983,786 A
1/1991
5,019,761 A
5/1991 Kraft
6,307,465 B1
5,022,384 A
5,022,407 A
6/1991 Freels
6/ 1991 Horch er 91-
6,344,791 B1*
6,369,803 B2*
5,035,242 A
7/1991 Franklin et a1.
5,038,089
5,053,585
5,078,152
5,116,051
5,165,897
A
A
A
A
A
5,175,459 A
8/1991
10/1991
1/1992
5/1992
11/1992
Stevens
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
B1
B1
B1
B1
B1
6,292,173 B1*
Szakaly
Yaniger
Bond
Moncrief
Johnson
12/1992 Danial @1111
Kramer
Thorner
Anlan0
Chen
Yasutake
Sinclair et 31
Gerpheide
Bobick eta1.
8/1998 clump
12/1998
1/1999
3/1999
3/1999
3/1999
6/1999
8/1999
g/1999
11/1999
11/1999
5/2000
6/2000
8/2000
8/2000
9/2000
10/2000
12/2000
2/2001
3/2001
4/2001
4/2001
5/2001
9/2001
Peurach
Moriyasn
Holehan
Schuler
Schuler
Thornton
Martine11i
Macnak
Bouin
Holehan
Kramer
Rollins
Nuovo
Zilles eta1.
Fujita et a1.
Peurach
Perry eta1.
Schuler
Saarmaa
Goodwin
Elbing et a1.
Yates
Rarnhaldi eta1. ......... .. 345/157
10/2001 Kayarna eta1.

2/2002 Arnistrong ..
. 338/114
4/2002 Brisebois et a1. ......... .. 245/173
6,373,463 B1
4/2002 Beeks
6,374,255
6,388,655
6,422,941
6,429,846
6,543,487
4/2002
5/2002
7/2002
8/2002
5/2003
B1
B1
B1
B2
B2
6,597,347 B1*
Peurach
Leung
Thornereta1.
Rosenberg
Bazinet
7/2003 Yasutake .................. .. 345/173
5,182,557 A
V1993 Lang
6,657,617 B2* 12/2003 Pao1inieta1. ..
345/173
5,186,685 A
5212473 A
2/1993 Grossmann @1111

5/1993 Louis
6,781,569 B1* 8/2004 Gregorio eta1.

6,801,191 B2* 10/2004 Mukai et a1.
345/157
345/179
5223658 A
6/1993
5,237,327 A
8/1993 Saitoh
2002/0177471 A1
5,240,417 A
5,241,308 A
5,246,316 A
8/1993 Smithson er 918/1993 Young

9/1993 Smith
2002/0033795 A1
2002/0128048 A1
2002/0149561 A1
3/2002 shahoian
9/2002 Aa1tonen
10/2002 Fukumoto eta1.
2002/0171621 A1*
2005/0099393 A1*
11/2002 Johnson .................... .. 345/156

5/2005 Johnson .................... .. 345/163
Suzuki
5,271,290 A
5,275,174 A
12/1993 Fischer
V1994 (399k
5,283,970 A
5,289,273 A
2/1994 Aigner
2/ 1994 Lang
5,299,810
5,309,140
5,334,027
5,355,148
5,390,128
5,390,296
5,402,499
5,402,680
5,436,622
5,437,607
5,451,924
5,461,711
5,466,213
5,489,812
A
A
A
A
A
A
A
A
A
A
A
A
A
A
4/1994
5/1994
8/1994
10/1994
2/1995
2/1995
3/1995
4/1995
7/1995
8/1995
9/1995
10/1995
11/1995
2/1996
Pierce et a1.
Everett
Wherlock
Anderson
Ryan
Crandall
Robinson
Korenaga
Gntrnan eta1.
Taylor
Massimino
Wang
Hogan eta1.
Furuhata
6,976,562 B1*
12/2005
Perreteta1. ........... .. 188/24.11
1/2002 Kaaresoja

JP
JP
JP
JP
JP
JP
JP
JP
JP
JP
WO
W0
W0
01003664
H2-185278
02109714
H4-8381
04007371
H5-192449
05193862
H7-24147
2001350592 A
2002259059 A
W09520787
W0 09718546 A1
W0 02/27645
7/1990
7/1990
1/1992
1/1992
8/1993
8/1993
1/1995
1/1995
12/2001
9/2002
8/1995
5/1997
4/2002
US 7,336,260 B2
Page 3
W0
W0
W0 0231807 A1
W0 019110A1 A1
4/2002
11/2002

OTHER PUBLICATIONS
Adelstein, Design and Implementation of a Force Re?ecting

Manipulandum for Manual Control research, DSC-vol. 42,
Advances in Robotics, Edited by H. KaZerooni, pp. 1-12, 1992.
Reality, ISBN 0-938151-82-7, pp. 129-180, 1992.

Baigrie, Electric Control LoadingiA Low Cost, High Perfor
mance Alternative, Proceedings, pp. 247-254, Nov. 6-8, 1990.
BejcZy, Sensors, Controls, and Man-Machine Interface for
1335, 1980.
BejcZy, Generalization of Bilateral Force-Re?ecting Control of
Manipulators, Proceedings Of Fourth CISM-IFToMM, Sep. 8-12,

1981.
BejcZy, et al., Universal Computer Control System (UCCS) For

Space Telerobots, CH2413-3/87/0000/0318501.00 1987 IEEE,
1987.
BejcZy et al., A Laboratory Breadboard System For Dual-Arm

Teleoperation, SOAR 89 Workshop, J SC, Houston, TX Jul. 25-27,
1989.

Transactions on Man-Machine Systems, vol. MMS-11, No. 1, Mar.
1970.
Brooks et al., Hand Controllers for TeleoperationiA State-of-the
Art Technology Survey and Evaluation, JPL Publication 85-11;

NASA-CR-175890; N85-28559, pp. 1-84, Mar. 1, 1985.
Burdea et al., Distributed Virtual Force Feedback, Lecture Notes
for Workshop on Force display in Virtual Environments and its
Application to Robotic Teleoperation, 1993 IEEE International
Conference on Robotics and Automation, pp. 25-44, May 2, 1993.

Jun. 23, 1983.
Caldwell et al., Enhanced Tactile Feedback (Tele-Traction) Using

a Multi-Functional Sensory System, 1050-4729/93. pp. 955-960,
1993.
Eberhardt et al., OMARiA Haptic display for speech perception

by deaf and def-blind individuals, IEEE Virtual Reality Annual
International Symposium, Seattle, WA, Sep. 18-22, 1983.
Dynamic Systems and Control: vol. 1, ASME 1994.
Fokumoto, Active Click: Tactile Feedback for Touch Panels,
ACM CHI2001 Extended Abstracts, pp. 121-122, Apr. 2001.
Gobel et al., Tactile Feedback Applied to Computer Mice, Inter
national Journal of Human-Computar Interaction, vol. 7, No. 1, pp.
1-24, 1995.
Gotow et a1, Controlled Impedance Test Apparatus for Studying
Human Interpretation of Kinesthetic Feedback, WA11-11:00. pp.
332-337.
Howe, A Force-Re?ecting Teleoperated Hand System for the

Study of Tactile Sensing in Precision Manipulation, Proceedings of
the 1992 IEEE International Conference on Robotics and Automa
tion, Nice, France, May 1992.

IBM Technical Disclosure Bulletin, Mouse Ball-Actuating Device
With Force and Tactile Feedback, vol. 32, No. 98, Feb. 1990.
International Search Report ?led Nov. 1, 2002 corresponding to
PCT/US02/35016.
IWATA, Pen-based Haptic Virtual Environment, 0-7803-1363-1/

93 IEEE, pp. 287-292, 1993.
Jacobsen et al., High Performance, Dextrous Telerobotic Manipu

lator With Force Re?ection, Intervention/ROV 91 Conference &
Exposition, Hollywood, Florida, May 21-23, 1991.

90-039, Aug. 1990.
1993.
Teleoperation and Virtual Environments, PRESENCE, 4(4):387

402, 1995.
Lake, Cyberman from Logitech, GameBytes, 1994.

Marcus, Touch Feedback in Surgery, Proceedings of Virtual
Reality and Medicine The Cutting Edge, Sep. 8-11, 1994.
McAffee, Teleoperator Subsystem/Telerobot Demonsdtrator:

Force Re?ecting Hand Controller Equipment Manual, JPL
D-5172, pp. 1-50, A1-A36, B1-B5, C1-C36, Jan. 1988.
Minsky, Computational Haptics: The Sandpaper System for Syn

thesiZing Texture for a Force-Feedback Display, Ph.D. Disserta
tion, MIT, Jun. 1995.
Ouh-Young, Force Display in Molecular Docking, Order No.

9034744, p. 1-369, 1990.
Ouh-Young, A Low-Cost Force Feedback Joystick and Its Use in
PC Video Games, IEEE Transactions on Consumer Electronics,
vol. 41, No. 3, Aug. 1995.

OuhYoung et al., The Development of a Low-Cost Force Feedback
Joystick and Its Use in the Virtual Reality Environment, Proceed
ings of the Third Paci?c Conference on Computer Graphics and
Applications, Paci?c Graphics 95, Seoul, Korea, Aug. 21-24, 1995.

Patrick et al., Design and Testing of A Non-reactive, Fingertip,
Pimentel et al., Virtual Reality: through the new looking glass, 2nd
RabinowitZ et al., Multidimensional tactile displays: identi?cation
of vibratory intensity, frequency, and contractor area, Journal of
The Accoustical Society of America, vol. 82, No. 4, Oct. 1987.
Russo, The Design and Implementation of a Three Degree of
Freedom Force Output Joystick, MIT Libraries Archives Aug. 14,
1990, pp. 1-131, May 1990.
Russo, Controlling Dissipative Magnetic Particle Brakes in Force
ASME 1992.
Scannell, Taking a Joystick Ride, Computer Currents, Nov. 1994,

Boston Edition, vol. 9, No. 11.
Intelligent Robotic Systems for Space Exploration, Rensselaer

Polytechnic Institute, Sep. 30-Oct. 1, 1992.
http://www.smk.cojp/whatsnewie/628csciehtml, Sep. 30, 2002.
Snow et al., Model-X Force-Re?ecting-Hand-Controller, NT
Control No. MPO-17851; JPL Case No. 5348, pp. 1-4, Jun. 15,
1989.

sors, DSV-vol. 42, Advances in Robotics, pp. 55-61, ASME 1992.
MIT Archive Massachusetts Institute of Technology, pp. 1-88,
Feb. 1990.

of Fourteenth Annual Northeast Bioengineering Conference, Uni
versity of New Hampshire, Mar. 10-11, 1988.
Wiker, Teletouch Display Development: Phase 1 Report, Tech
nical Report 1230, Naval Ocean Systems Center, San Diego, Apr.
17, 1989.
Wakiwaka, et al., In?uence of Mover Support Structure on Linear
Oscillatory Actuator for Cellular Phones, The Third International
Springer International (Springer-VeItag); Experimental Brain
Symposium on Linear Drives for Industry Applications, 2001, p.
Research, vol. 79, No. 1, pp. 150-156, 1990.

KacZmarek et al., Tactile Displays, Virtual Environment Tech
260-263, Nagano, Japan.
nologies.
* cited by examiner
U.S. Patent
Feb. 26, 2008
Sheet 1 6f 11
US 7,336,260 B2
U.S. Patent
Feb. 26, 2008
Sheet 2 6f 11
US 7,336,260 B2
U.S. Patent
Feb. 26, 2008
Sheet 3 6f 11
US 7,336,260 B2
U.S. Patent
Feb. 26, 2008
Sheet 5 6f 11
FIG.5
US 7,336,260 B2
U.S. Patent
Feb. 26, 2008
Sheet 6 6f 11
US 7,336,260 B2
owl/r 31/!
v
i /
\g\
aw.
7/la\|
\6.
\\
x
U.S. Patent
Feb. 26, 2008
Sheet 7 6f 11
Input
US 7,336,260 B2
Electronic
40/ Delnoes
'
(3:283? \43
f3?!
41
_
Drsploy
\ Controller
Control
45 j Circultry
45f
/ 44
Memory f42
Actuator
FIG.7
U.S. Patent
Feb. 26,2008
US 7,336,260 B2
Sheet 8 0f 11
Access
Memory
72
Obtain
Ambiguous
Feedback
Info.
Pressure
Data?
Obtain
Pressure
Data
Produce
Tactile
Sensation
FIG.8
U.S. Patent
Feb. 26, 2008
Sheet 11 6f 11
US 7,336,260 B2
110-_\/
[ ||e dit y?ew
D New
E Qpen
112*
Ctrl+N
Ctr|+0
Qlose
\ *5LEW
Erint
Ctrl+P
Send To
102/
/}
//
Properties
63 Hgii Recipient (as Attachment)
Exit
l? Microsoft EowerPoini
\-| 14
US 7,336,260 B2
1

require the user to look at the device, at least brie?y, When
interacting With the device. In addition, electronic devices

relying on softkeys can be dif?cult to read in bright light
environments such as in bright sunlight and can contain very
RELATED APPLICATIONS
small fonts and graphics that are dif?cult to read and select.
Some conventional touchpads include vibratory feedback
This application claims priority to US. Provisional Appli

cation No. 60/335,493, ?led Nov. 1, 2001, and US. Provi
sional Application No. 60/399,883 ?led Jul. 31, 2002. The
to the user of the touchpad. US. Pat. No. 5,977,867 is one
example. Such conventional systems and methods are lim
entire disclosure of both applications are incorporated herein
ited, though. They lack a full range of functionality assis
by reference.
tance to a user interacting With an electronic device. More
over, such systems and methods still require considerable

visual attention from the user.
The present invention relates to methods and apparatus
SUMMARY OF THE INVENTION
for providing tactile sensations.
The present invention comprises products and processes

BACKGROUND
for providing tactile sensations to input devices or electronic
devices. Input devices include mechanical input devices

(such as, for example, mechanical sWitches) and non-me
Conventional electronic devices, such as mobile tele
phones and Personal Digital Assistants (PDAs), include
20
visual displays. A user of such devices interacts With the

visual display using any one of a number of input devices.
means in communication With the input device or electronic
Examples of such input devices include computer mice,
joysticks, trackballs, steering Wheels, stylus, tablets, pres

sure-sensitive spheres, scroll Wheels, keyboards, and key
device. A controller may be employed to receive signals

from the input devices and to control the actuator. Tactile
25
pads. The user provides instructions, responses, and other

input to the device using such input devices.
In conventional mobile telephones and PDAs, con?rma
tion of the input provided by the user is primarily limited to
visual or audible con?rmation. In some such devices, physi
Such an event or situation may be any one designated.
30
displacement pro?le. Typically, in such devices, the

35

40
FIG. 1 is a perspective vieW of an embodiment of an
apparatus according to the present invention;

FIG. 2 is a plan vieW of another embodiment of an
telephones and PDAs, provide simple passive touch cues
apparatus according to the present invention;
regarding the alignment of keys. Such cues include raised

bumps on the center key of a telephone keypad or on the F
and G keys of a keyboard that assist a user in orienting to
45
FIG. 3 is a plan vieW of an electronic device including an
embodiment of the present invention;

the pattern of keys in the keypad and keyboard. Again, these

physical queues are very limited, and users typically need to
vieW a keypad or keypad for visual con?rmation that the
correct instructions or information is being entered.
Examples of such events and situations include the level of

pressure placed on an input device; the availability or lack
of availability of a function associated With an input device;
and the function, menu, or mode of operation associated
With an input devices activation. A variety of feedback
types and combinations may be selected. Mobile telephones
and PDAs bene?t from employing such products and pro
cesses, but other devices bene?t as Well. The advantages
offered by the various embodiments of the present invention
may be understood by examining this speci?cation.
remains the same regardless of the current function of the

button.
In addition to providing extremely limited and rudimen

tary mechanical con?rmation of button selection, conven
tional buttons as used, for example, in keypads for mobile
feedback to an input device or electronic device may be

provided in response to one or more events or situations.
cal feedback is provided by conventional mechanical

sWitches in the form of the conventional mechanical feed
back of sWitches, for example the sWitch closure force
mechanical feedback provided by each button is identical. In
addition, in such conventional devices, for those buttons that
serve multiple functions, the mechanical feedback generally
chanical input devices (such as, for example, touchpads).

Tactile feedback is provided by using an actuator or other
FIG. 3;
FIG. 5 is a plan vieW of another electronic device includ
50
ing another embodiment of the present invention;

When a ?at surface interface device is used, such as a
touchpad for a computer or PDA, these simple mechanical
FIG. 5;
cues are unavailable to the user. Often, touchpads are
FIG. 7 is a block diagram illustrating an embodiment of

the apparatus in an electronic device;
FIG. 8 is a How chart illustrating a method according to
combined With ?at-panel display screens that display one or
more graphically generated buttons or softkeys. Normally,

the softkeys are visible through the touchpad. A users
55
the present invention;

FIG. 9 is a table illustrating a ?rst set of data to be used
contact With the touchpad in an area de?ned by a softkey
provides the electronic device having the touchpad With the

input associated With that softkey.
in one embodiment of the present invention; and

FIG. 10 is a table illustrating a second set of data to be
mechanical buttons and touchpad arrangements are particu
used in another embodiment of the present invention.

FIG. 11 is a plan vieW of another embodiment of the
larly difficult to use in distracting environments or When the
present invention.
The use of electronic devices using such conventional
user is attempting to perform another task simultaneously

With using the electronic device. For example, if the other
task involves operating a motor vehicle or heavy machinery,
it may be dif?cult or impossible for a user to simultaneously
use such an electronic device because such devices typically
60
65
The present invention includes methods and systems for

providing tactile sensations. One embodiment includes
US 7,336,260 B2
3
methods and systems for providing tactile sensations to
input device 2 from a speaker included With an electronic
input devices, both mechanical and non-mechanical (for

example soft-keys that are computer generated and dis
device into Which the apparatus is placed, for example the

speaker in a mobile telephone or in a personal computer.
Although the embodiment shoWn in FIG. 1 includes one
input device 2 and one actuator 6, other embodiments
played on a screen). Embodiments of the present invention

can be utilized in Wide variety of electronic devices includ
ing telephones, mobile telephones, remote controls, game

pads, joystick handles, automotive controls (radios, Com
pact Disc (CD) players, automobile functions, etc.),
include a plurality of input devices, all in communication

With a single actuator. Alternatively, an embodiment can
include a plurality of actuators each in communication With

at least one input device. Various arrangements of actuators
in combination With input devices are suitable for use in the
consumer electronics devices, Personal Digital Assistants
(PDAs), personal computers, laptop computers, portable

gaming devices, pagers, I-pagers, audio equipment, televi
present invention. For example, US. patent application Ser.

No. 09/263,263, ?led Jul. 26, 2001, published on Mar. 21,
sions, security or alarm systems, Automated Teller Machines
(ATM), calculators, home appliances, and White goods.
2002, as US. Patent Pub. No. US2002/0033795 illustrates

actuators in combination With input devices that may be
FIG. 1 shoWs one embodiment of the present invention.

The apparatus 1 shoWn in FIG. 1 includes an input device 2
used in embodiments of the present invention. The entire

disclosure of application Ser. No. 09/263,263, Pub. No.
having multiple positions for communicating a plurality of
2002/0033795 is incorporated herein by reference.
input signals. The input device 2 can be any device capable

of transmitting an input signal. In the embodiment shoWn,
the input device 2 is a rocker-type sWitch. The rocker sWitch
2 shoWn can pivot or rock betWeen tWo positions in Which
20
As mentioned, the actuator 6 is in communication With the

input device 2. In the embodiment shoWn in FIG. 1, the
actuator 6 is in communication With the input device 2
the rocker sWitch contacts and activates one of tWo rubber
through a cantilevered beam or lever arms 7 attached to the
sWitches 3 containing conductive pads. The use of rubber

sWitches 3 provides the advantage of alloWing the user to
still feel a substantial vibration or force through the input
device 2 When the user had fully depressed the sWitch.
pivoting rocker, amplifying the e?ective forces of the actua

tor 6 felt by the user. The tactile sensations generated by the
actuator 6 propagate through the lever arm 7 to the input
25
device 2. Suitable materials for the lever arm 7 are capable
Suitable rubber sWitches are available and knoWn in the art.
of transmitting the tactile sensations and can be, for
In other embodiments, the input device may include an

analog sWitch, a force sending resistor, a strain gauge based
bends 8 to ?t Within the electronic device in Which the
sensor, a capacitative touch sWitch, a scroll Wheel, a mini

joystick, a touchpad, a touch screen, a 3-Way sWitch, a 4-Way
example, metal. The lever arm 7 shoWn includes one or more
apparatus 1 is disposed. Di?erent shapes of bends may be

30
used to ?t Within the electronic device. In another embodi
sWitch, a 5-Way sWitch, or other input device. Each position

of the input device 2 corresponds to one of the input signals.
ment, the actuator 6 is mounted directly to the input device

2 or to any component of the input device. Alternatively, the
The input device 2 and rubber sWitches 3 are mounted on

a Printed Circuit Board (PCB) 4 in the embodiment shoWn
to facilitate electrical communication betWeen the input
device 2 and an electronic device (not shoWn). The PCB 4
can be custom shaped according to the device into Which the
apparatus 1 is placed. The PCB 4 also provides for secure
actuator 6 is mounted to the PCB 4 to Which the input device
is attached, communicating tactile sensations to the input

35
tor is an existing eccentric mass motor as is used, for

example, as a vibrating ringer in a pager or mobile tele
phone.
mounting Within the device by including, for example, a

plurality of holes 5 to accept fasteners for securing to the
electronic device. In another embodiment, the input device
device through the PCB. In another embodiment, the actua
The vibrotactile actuator 6 can also be mounted to a

40
portion of the case or housing of the electronic device in
Which the apparatus 1 is disposed, communicating the tactile
2 can be directly connected or mounted in the electronic

device.
The apparatus 1 shoWn in FIG. 1 also includes a vibro
sensations to the entire electronic device. In one embodi

ment, tWo actuators can be incorporated in the case or back
of an electronic device, for example the case of a mobile
tactile actuator 6 in communication With the input device 2. 45 phone in an area that contacts the users hand. This arrange
Preferably, the actuator 6 is con?gured to output a plurality

of distinct tactile feedback sensations to the input device 2.
ment e?cectively doubles the amplitude of the tactile sensa

tion, and the users ?ngers do not tend to attenuate the tactile
sensation.
Suitable tactile sensations include vibrations, for example,

jolts and textures, and a plurality of distinct tactile sensations
can be created by varying the frequency, amplitude and
The apparatus 1 also includes a controller 9 in commu

50
nication With the input device 2 to receive the input signals
Waveform output by the actuator 6. The actuator 6 is selected

to deliver the desired tactile sensations to the input device 2
therefrom. The controller 9 can also receive additional
The actuator 6 shoWn in FIG. 1 is a voice coil actuator. Other
the input device 2 and the amount of pressure applied to the
suitable actuators include, for example, pieZo-electric actua

tors, eccentric mass actuators, moving magnet actuators, and
friction brakes in contact With metal shafts. In addition, the
information from the input device 2 including the position of

input device 2. In one embodiment, the input signal includes
55
information related to the amount of pressure applied to the
input device 2, information related to the position of the
actuator can include a ?exure, for example an arrangement
input device 2, or a combination of information about
of ?exible material, couple to the rotating shaft of a DC
pressure and position. In addition to being in communication

With the input device 2, the controller 9 is in communication
motor or step motor to transform the rotation of the motor
shaft into vibrations or other haptic sensations. Various
60
With the actuator 6 to produce a tactile sensation in the
arrangements of a ?exure coupled to a motor may be used

as an actuator. For example, US. patent application Ser. No.
actuator 6 corresponding to the input or input signal received

by the controller 9 from the input device 2.
09/585,741, ?led Jun. 2, 2000, illustrates suitable arrange
The controller 9 is located in a suitable location according

to the needs of the device in Which the apparatus 1 is placed.
ments of ?exures and motors for use as an actuator in
embodiments of the present invention. The entire disclosure
of the application Ser. No. 09/ 585,741 is incorporated herein
In one embodiment, the controller 9 is attached to the PCB

4 as shoWn in FIG. 1. Suitable controllers, include, for
by reference. Tactile sensations can also be delivered to the
example, digital logical processors capable of processing
65
US 7,336,260 B2
5
input, execute algorithms, and generate output as necessary

to created the desired tactile sensations in the input device in
response to the inputs received from that input device. Such
pressure placed by the user), the third level being a second

magnitude of pressure placed on the button (Where the
second magnitude of pressure is different from or greater
than the ?rst magnitude), the fourth level being a third
magnitude of pressure placed on the button (Where the third
magnitude is different from or greater than the second
magnitude), and the ?fth level being a fourth magnitude of
pressure placed on the button (Where the fourth magnitude
is different from or greater than the third).
In button 101', each of levels tWo through ?ve is associated
With a distinct input signal. When the button 101' is in its ?rst
state, then the button 101' does not transmit an input signal.
controllers may include a microprocessor, an Application
Speci?c Integrated Circuit (ASIC), and state machines. Such

controllers include, or may be in communication With,
media, for example computer readable media, Which stores

instructions that, When executed by the controller, cause the
controller to perform the steps described herein as carried
out, or assisted, by a controller. On embodiments of a
suitable computer-readable medium includes an electronic,

optical, magnetic, or other storage or transmission device
capable of providing a processor, such as the processor in a
When pressure is applied to the button 101' by a user that
Web server, With computer-readable instructions. Other
exceeds the ?rst magnitude of pressure, the button 101'

transmits a ?rst signal to the controller 9. When greater
pressure is applied to the button 101' that exceeds the second
magnitude of pressure, the button 101' transmits a second
signal to the controller. When still greater pressure is applied
to the button 101' that exceeds the third magnitude of
examples of suitable media include, but are not limited to, a
?oppy disk, CD-ROM, magnetic disk, memory chip, ROM,

RAM, ASIC, con?gured processor, all optical media, all
magnetic tape or other magnetic media, or any other medium
from Which a computer processor can read. Also, various
other forms of computer-readable media may transmit or
carry instructions to a computer, including a router, private
or public netWork, or other transmission device or channel.
In one embodiment, the apparatus 1 includes a dedicated
controller 9 for use speci?cally With the apparatus 1. This
embodiment is particularly Well suited for applications
20
25
Where the apparatus 1 is retro-?tted into an existing electri

cal or electronic device. In another embodiment, the con
troller 9 is the microprocessor or Central Processing Unit

(CPU) of the electronic device in Which the apparatus 1 is
disposed. The apparatus 1 can also include additional cir
cuitry such as the drive circuitry (not shoWn) necessary to
create the tactile sensations in the actuator 6 in response to
input from the controller 9 and a memory medium for
storing data to be accessed by the controller 9 for example
a correlation betWeen the tactile sensations to be produced in
the actuator 6 and the input information or input signal
received from the input device 2.
FIG. 2 shoWs another embodiment of the present inven
tion. An apparatus 60 shoWn in FIG. 2 includes multiple
input devices. These multiple input devices include tWelve

?xed or pre-assigned alphanumeric input buttons 10a-l,
three pre-assigned function buttons 1111-0, and three assign
able function buttons 12a-c. The plurality of inputs devices
are arranged according to the electronic device in Which the
apparatus 60 is situated. In the embodiment shoWn, the
plurality of input devices are arranged as the keys in a key
pad for a telephone or mobile telephone.
Embodiments of the present invention include an input
device having a means for determining or sensing pressure.
The input device is capable of resolving multiple levels of

pressure placed on the input device, and of transmitting a
signal associated With the level of pressure placed on the
input device. These multiple levels of pressure may be
de?ned by, for example, the physical location of, or distance
traveled by, a sWitch-type input device in the x-plane When
pressed by a user (higher/loWer), the magnitude of pressure
30
of pressure placed on the button 101'. In the embodiment

shoWn, the ?rst level is a state in Which no pressure is placed
on the button by a user, the second level being a ?rst
magnitude of pressure placed on the button (greater than no
button 101' that exceeds the fourth magnitude of pressure, the

button 101' transmits a fourth signal to the controller. The
structural arrangement of the communication by the button
101' to the controller 6 of an input signal is further illustrated
in FIG. 4, described beloW.
Each of the levels tWo through ?ve of button 101' (and thus
each of their associated signals) is associated With a letter,
W-Z. The second level is associated With the letter W, the
third level is associated With the letter X, and so on. The
second level is associated With the letter W, the third level

is associated With the letter X, and so on. In the embodiment
shoWn, the key 101' has ?ve positions corresponding to no

pressure, and the letters W, X, Y, and Z. In an alternative
35
40
45
embodiment, the key 101' has six positions corresponding to

no pressure, the number 9, and the letters W, X, Y, and Z.
In the embodiment shoWn, the alphanumeric buttons 10
are all capable of resolving ?ve levels of pressure. In
alternative embodiments, the various buttons are capable of
resolving differing levels of pressure. For example, in an
alternative embodiment, While the button 101' is capable of
resolving ?ve levels of pressure, the button 10b (correspond
ing to the number 2 on the keypad) is capable of resolving
four levels of pressure placed on the button 10b (the ?rst
level being no pressure placed on the button). Like button
101', the levels resolved by button 10b in the alternative
embodiment are each associated With a distinct input signal,
and are each associated With a distinct letter of the alphabet,
A-C.
50
55
The pre-assigned function buttons 11a-c of the apparatus

1 are keypad push buttons. Each of the buttons 11a-c is
capable of resolving three levels of pressure placed on the
buttons lla-cino pressure, a ?rst magnitude of pressure
(greater than none), and a second magnitude of pressure
(greater than the ?rst magnitude). Examples of functions

carried out by such pre-assigned function buttons 11 11-0
include Send 1111, Power 11b, and End Call 110.
placed on a touchpad-type input device, or other means.

The buttons of FIG. 2 are illustrative of such an embodi
ment. Each of the alphanumeric input buttons 10 shoWn in

FIG. 2 is a keypad button. Each of the buttons 10 is capable
of resolving multiple levels of pressure placed on the buttons
10. For example, the button 101' (corresponding to the
number 9 on the keypad) is capable of resolving ?ve levels
pressure, the button 101' transmits a third signal to the

controller. When even greater pressure is applied to the
In the embodiment shoWn, each of the pre-assigned

60
function buttons 11 11-0 is con?gured such that the ?rst

that a users ?nger is hovering over, or touching With more
than passing force, the button. Each is also con?gured such

that the second magnitude of pressure is an amount of
pressure that signi?es that a users ?nger applies When the

65
user Wishes to activate the button.
Thus, in the embodiment shoWn, When a users ?nger

hovers over the Send button 110, a ?rst signal is
US 7,336,260 B2
7
transmitted by the button 110 to the controller. And, When a

users ?nger activates the Send button 110, a second signal
is transmitted by the button 110 to the controller.
The assignable-function buttons 12a, 12b, 120 are buttons
alternate actuator 61 and physical placement of the actuator

61 is shoWn in FIG. 4. The actuator 61 is in communication
With the various input devices, and is con?gured to provide

vibrations of varying frequencies, magnitudes, and Wave
Whose function depends upon the mode of operation of the

device With Which the apparatus 1 is associated. For
example, When such an apparatus 1 is associated With a
forms to the input devices. The actuator 61 is also in

communication With the controller 9. Further description of
embodiments of such communication and con?guration is
mobile telephone, such buttons 12a, 12b, 120 may be used
provided beloW.
to navigate the menus displayed to carry out various func
In the embodiment shoWn, the controller 9 receives an
tions, such as scrolling through an address book, selecting a

number to dial, editing a number, re-setting the time dis
input signal from one of the input devices. The controller 9

then analyZes the input signal received to determine a signal
played, and similar functions. In addition, the assignable
to transmit to the actuator 61.
For example, the controller 9 of FIG. 2 is con?gured such
function buttons 12a-c are con?gured similarly to the pre
assigned buttons 11a, 11b, 11 c, in that the buttons 12a, 12b
that When the controller 9 receives a signal associated With
are con?gured such that the ?rst magnitude of pressure is an

amount of pressure that signi?es that a users ?nger is
hovering over, or touching With more than passing force,
the button, and such that the second magnitude of pressure
is an amount of pressure that signi?es that a users ?nger
applies When the user Wishes to activate the button. Prefer
the second level from button 101' (the 9 key), the controller
9 sends a ?rst control output signal to the actuator, and When
20
ably, the buttons 11a, 11b, 110, 12a, 12b, 120 are con?gured
such that they receive and analyZe other data in determining
Whether the user is merely hovering or, instead, Wishes to
activate the button (such as type of, and duration of, contact
With the button). Any suitable input-device may be used an
25
assignable-function input device. Examples of such input

assignable-function button 120, includes the input device of
PCB 62 of the apparatus 60 of FIG. 2 are encased in a case

30
18 of the mobile telephone 14. The mobile telephone 14 also

includes a display screen 15 capable of displaying graphic
objects 16 and alpha-numeric information 17. The alpha

numeric information 17 that may be displayed includes
phone numbers and lists, for example of list of entries in a
FIG. 1 is not in contact With PCB62.
Referring again to FIG. 2, although in the embodiment
frequency.
tion, in the form of a mobile telephone 14 having the
apparatus of FIG. 2. The controller 9, actuator 61, and the
devices include rocker-sWitches and scroll Wheels.

In an alternative embodiment (not shoWn), the middle
FIG. 1. It is in communication With the actuator 6 (not
shoWn) shoWn in FIG. 1 as Well, and operates in the manner
described With reference to FIG. 1. In such an embodiment,
the PCB 62 is separated at line 62a, such that the PCB4 of
the controller receives a signal associated With the third level

from the button 101', the controller sends a second control
output signal to the actuator 61, and so on. The ?rst control
vibration of a certain, ?rst frequency. The second control
vibration of a certain, higher frequency, and so on. In other
embodiments, the vibrations provided may be of the same
35
phone book, that are input by the alpha-numeric input
shoWn there the alphanumeric keys have four or ?ve avail

able states (embodying an alphanumeric-character selec
buttons 10 and accessed by the assignable function buttons
tion), and the pre-assigned buttons 11a, 11b, 11c, and the
FIG. 4 is a schematic representation through line 4-4 of

FIG. 3 illustrating that the alpha-numeric input buttons or
keys 10 in the mobile telephone pass through the case 18 of
the mobile telephone and contact a plurality of sWitches 19
assignable-function buttons 12a, 12b, 120 are con?gured to

indicate hover/activation signals, in other embodiments,
other con?gurations may be used. Moreover, although the
alphanumeric keys 10 have four or ?ve available states, thus
alloWing them to be associated With three or four (or more)
input signals, such keys 10 may be con?gured to provide

input signals at, for example, only tWo of the states. In this
1211-120.
40
disposed on the PCB 62. The sWitches 19 are in communi
45
Way, such keys 10 may be con?gured to provide hover/

activation signals similar to that Which is provided in the
pre-assigned buttons 11a, 11b, 11c, and assignable-function

buttons 12a, 12b, 120 in the embodiment shoWn in FIG. 2.
Moreover, in the embodiment shoWn, the levels for the
cation With the controller 9 (not shoWn). Suitable sWitches

19 include any analog or digital sWitch, for example rubber
sWitches, snap dome-type sWitches, and pressure sensitive
sWitches. Preferably, the sWitches 19 are capable of produc
ing distinct input signals to the controller. Even more
preferably, the sWitches 19 are capable of producing such
signals for tWo or more positions. In the embodiment shoWn,
50
the keys 10 contact a plurality of digital sWitches, each
alphanumeric input devices 10 correspond to magnitudes of
capable of producing four distinct input signals to the
pressure, but in other embodiments the levels resolved can
controller 9 to correspond to the four levels at Which

pressure is applied to the buttons 10 by the user.
The PCB 62, on a side opposite the sWitches, is in
communication With the actuator 61. As illustrated, the
actuator 61 is a pieZo-electric actuator having a metal
diaphragm 20 in contact With the PCB 62 through one or
more spacers 21 and a pieZo ceramic element 22 in contact
With the metal diaphragm 20. Alternative actuator embodi
ments include a ?exure coupled to the shaft of a motor,
secured to the PCB 62.
As illustrated, the keys 10 are initially in a rest position
be type of touch, magnitude, physical position of the sWitch

and other attributes of contact With the button, or some
combination thereof. The input signals provided by such

input devices may be con?gured accordingly.
55
In the embodiment shoWn in FIG. 1, the input signals that

are transmitted by the input devices are transmitted to a
controller 9. In the embodiment shoWn, the controller is in
communication With storage memory (not shoWn).

Examples of such memory includes Read Only Memory
60
(ROM). The storage memory includes a table in Which input

signals are associated With various haptic feedback signals.
This is explained more fully in relation to FIGS. 9-10.
The apparatus 1 shoWn in FIG. 2 also includes an actuator
61. The actuator 61 is shoWn in representative fashion in
FIG. 2, and not to scale or in physical placement. An
23. A biasing member arrangement (not shoWn) as is avail

able and understood in the art is used to hold the keys in the
65
rest position 23. An object 24, for example the users ?nger
or a stylus, is used to select one or more of the keys 10 by
applying pressure in the direction of arroW A. This pressure
US 7,336,260 B2
10
controller 9. Thus, in the embodiment shoWn, different
signals are transmitted by the sWitch 19a for each pressure
causes the selected key to progress through a plurality of
positions. As illustrated, after leaving the rest position 23,

the keys pass sequentially through a second position 25, a
third position, 26, a fourth position 27, and a ?fth position,
applied levels 25, 26, 27, 28.
28 as greater and greater pressure is applied to the button 10.

The distance of travel betWeen each position does not have
to be equal, and the amount of pressure required to move
betWeen each position can vary. In addition, for a given key,
the number of positions can vary from tWo (no pressure and
activated) up to the number of input signals assigned to a
employed. For example, When a user provides suf?cient
In the embodiment shoWn, a dWell to select function is

input to cause the button to move to its second level 25, the
?rst signal is transmitted to the controller 6 continuously

While the button receives pressure to push it at or past the
second level 25 but not suf?cient pressure to push the button
101' to the third level 26. The controller 9 determines the
length of time the button is maintained at the second level 25
given key. Therefore, in the embodiment shoWn, a key 101'
by monitoring the length of time the ?rst signal is transmit
is moveable from a ?rst level (rest) 23 to a second level 25

upon the application of a suf?cient amount of pressure to the
ted to the controller 9. If the ?rst signal is received for
input device. In the embodiment shoWn in FIG. 3, the
greater than a pre-determined length of time, the controller
amount of pressure necessary to move the key 101' from rest
determines that the user Wishes to select the function
23 to the second position 25 is about equal to the amount of

pressure that user s ?nger Would exert upon contact With the
associated With the second level 25 by the fact that the user
dWelled at that level for the pre-determined time. Upon so
determining, the controller 9 transmits a signal to a proces
sor (not shoWn) indicating that the user has selected the
function associated With the second level of button 101', in
this case the selection of the letter W. In embodiments, the
controller 9, upon so determining, also transmits a signal to
key Without actually selecting the key.

Accordingly, in one method of using the embodiment
shoWn in FIG. 3, When a user of the mobile telephone 14
20
shoWn in FIG. 3 presses the 9 key 101' using a relatively

light amount of pressure, the button 101' moves from rest
state 23 to its second level 25. Such movement causes the
button 101' to apply pressure to sWitch 19a, Which is received

by sWitch 19a. The sWitch 19a is in communication With the
controller 9. The sWitch 19a is con?gured to transmit a ?rst
signal to the controller 9 upon receiving a pressure of
the actuator 61 to cause the actuator 61 to vibrate at a
frequency, magnitude, and/or Wave-form indicative of selec

25
magnitude indicating that suf?cient pressure has been placed

on button 101' to move from its ?rst level 23 to its second
level 25. The controller 9 receives this ?rst signal. The

controller 9 is con?gured to transmit a ?rst controller output
30
tion of the function.

In one embodiment, in addition to providing haptic feed
back to the input device, the controller 9 also sends a signal
to the display 17 to cause the alphanumeric character
associated With the input signal to be displayed. For

example, in one embodiment, upon detecting the presence of
a ?rst pressure (through receipt of a ?rst input signal) and
signal to the actuator 61 upon receipt of this ?rst signal from
su?icient dWell time to indicate a selection, the controller
the sWitch 19a. The controller transmits the ?rst controller

output signal to the actuator 61. The actuator 61 is con?g
ured to provide a vibration of a ?rst pre-selected frequency
to the metal diaphragm 20 of a pre-selected duration upon
receipt of such a ?rst signal. In the embodiment shoWn, the
actuator 61 provides a side-to-side vibration to the dia
sends a signal to the display 17 indicating that the display

should display the letter X. Upon detecting the presence of
a second pressure (through receipt of a second input signal)
35
and suf?cient dWell time to indicate a selection, the control
ler sends a signal to the display 17 indicating that the display

should display the letter Y. Upon detecting the presence of
a third pressure (through receipt of a third input signal) and
phragm. The diaphragm 20 thus vibrates at the pre-selected

frequency, in turn causing the PCB 62 to vibrate at that same
frequency, and thus in turn causing the sWitches 19 to vibrate
at that frequency. The sWitch 19a is in communication With
the button 101', thus causing the button 101' to vibrate at that
40
frequency.
When the user applies further pressure to the button 101'
45
su?icient to cause the button to move from the second level
25 to a third level 26, the buttons force is applied to sWitch

1911. Switch 1911 receives the force and is con?gured to
transmit a second signal to the controller 9 Whenever it
receives force to indicate that the button 101' has moved from
the second level 25 to the third level 26. The sWitch 1911 does
so, and the controller 9 receives the second signal. The
controller 9 is con?gured to transmit a second controller
50
output signal to the actuator 61 upon receipt of this second

signal from the sWitch 19a. The controller 61 transmits the
second controller output signal to the actuator 61. The
55
suf?cient dWell time to indicate a selection, the controller
sends a signal to the display 17 indicating that the display

should display the letter Z. The display 17 then displays
each of these characters, X, Y, Z.
Various other embodiments may be employed. For
example, instead of having a single actuator to provide
feedback to all input devices receiving such feedback, like
the embodiments shoWn in FIGS. 2-3, other embodiments
have tWo or more actuators. These tWo or more actuators
may be in communication With all or part of the input

devices that provide tactile feedback. The tWo actuators may
each provide signi?cantly different types of feedback to the

same set of input devices, or each may be in communication
With a different group of input devices to provide the same
or different types of feedback. As another example, the
actuator 6 is con?gured to provide a vibration of a second
actuator and input devices may be con?gured to provide

vibration to only the button that is receiving pressure from
the user, or they may be con?gured to provide vibration to
pre-selected frequency, different from the ?rst pre-selected
all buttons or at least more buttons than the one(s) receiving
frequency, for a pre-determined duration to the metal dia

phragm 20 upon receipt of such a second signal. In other
pressure from the user.

60
Moreover, although the actuator 61 is shoWn as disposed

beloW the PCB 62 in FIG. 4, in other embodiments the
actuator 61 may be disposed at other locations Within the
device having such apparatus, Whether the device is a mobile
telephone, PDA, or other device. Preferably, the actuator is
65
disposed Within the housing of the device. Preferably, it is

communication With the PCB 62, but is placed anyWhere in
embodiments, the ?rst and second pre-selected frequencies

are the same. As above, the actuator 61 provides a side-to
side vibration to the diaphragm, Which is communicated

through the PCB 62 and sWitches 19 to the button 101'.
When a user applies pressure to the button 101', Which is
communicated to the sWitch 19a, at each level 25, 26, 27, 28,
a distinct signal is transmitted by the sWitch 19a to the
communication With the PCB 62 as the siZe and space
US 7,336,260 B2
11
12
restrictions of the application Will allow. In other embodi

ments, the actuator 61 is located outside the housing of the
device (such as beside it). In still other embodiments, the
actuator 61 is in communication With the input devices other
than through the PCB 62.
In the embodiment shoWn, a distinct tactile sensation is
produced for each of the various levels at each of the various
keys. In other embodiments, the controller 6 selects one of
a pre-selected group of sensations to provide in response to
to the surface 35 of the touchpad 30. The controller also

determines When an input is ambiguous, such as When tWo
or more softkeys are simultaneously selected or When an
area of the display containing no graphics 38 is touched, and

causes the actuator to output an appropriate tactile sensation.
Preferably, the same controller that controls the displayed

softkeys 36 also controls the tactile feedback sensations
produced by the actuator 64.
Certain softkeys 36b-i represent multiple positions or

multiple inputs, each input or position corresponding to a
distinct amount of pressure applied to the softkey 36b-i. This
the various signals received by the controller.

tion. Referring to FIG. 5, a PDA 31 having an input device
in the form of a pressure-sensitive touchpad 30 is shoWn.
The PDA 31 also includes a plurality of mechanical type
buttons 32. The PDA 31 also includes a display panel 33
distinct amount of pressure is detected by the controller in
communication With the touchpad 30. Alternatively, the

apparatus can include a separate pressure calculator to
measure the amount of pressure applied to the touchpad 30.

In the embodiment shoWn, the amount of pressure applied to
capable of displaying computer generated graphics. Suitable

display panels include ?at-panel type displays including a
Liquid Crystal Display (LCD), plasma displays, Thin Film
Transistor (TFT) type displays or other ?at displays, such as
are found in laptops and color PDAs, and conventional
cathode ray tube displays.
the touchpad 30 is calculated by the controller based upon
20
FIG. 6 shoWs a cross-sectional vieW of the PDA 31 of
FIG. 5 along line 6-6. As is best displayed in FIG. 6, the

display 33 is underneath the touchpad 30 and is in commu
nication With the touchpad 30 to transmit tactile sensations
25
the amount of area of the object 24 used to select the softkey

that is in contact With the surface 35 of the touchpad 30.
The amount of pressure applied to the input device or to
the softkey 36a-i on the touchpad can be determined by
reading or determining the siZe or area of the contact patch
created by the object 24, such as the users ?nger, on the
input device or softkey 36a-i. In addition to reading the
current siZe of the contact patch, the rate of change of the
thereto. The display 33 is also in communication With an
contact patch can also be determined, using dynamic thresh
actuator 64 to receive a tactile sensation therefrom for
olds and to look at hoW fast the users pressure is changing.

If the contact patch area changes at a suf?ciently large rate,
the controller can determine that the corresponding input
device or softkey 36a-i is being selected.
The functionality of the softkeys shoWn in FIG. 5 is
similar to the mechanical key counter parts described in
communication to the touchpad 30. Other arrangements of

the touchpad 30, display 33 and actuator 64 are also possible
including arrangements in Which the actuator 64 is in direct
contact With the touchpad 30. The display 33 is in commu
nication With the touchpad 30 through tWo spacers 34.
30
Suitable spacers are constructed of a material that can
transmit the tactile sensations betWeen the display 33 and the
touchpad 30. In other embodiments, the touchpad 30 and

display 33 are in direct physical contact, and the touchpad 30
35
relation to FIGS. 2 and 3. Therefore, the pressure level of a

selected softkey may be moveable from a ?rst position to a
second position upon the application of a suf?cient amount
of pressure. The amount of pressure necessary to move the
and display are not in communication. The tactile sensations
softkey 36a to the second position (the ?rst position being at
produced in the touchpad 30 are transmitted to the object 24

When the object 24 is brought into contact With a surface 35
of the touchpad 30.
rest or no contact) input device to the ?rst position is about

equal to the amount of pressure that users ?nger Would
exert upon contact With the touchpad surface and sliding
40
Referring again to FIG. 5, the display 33 displays a

plurality of softWare-generated buttons or keys, called soft
keys 36a-i. The softkeys 36a-i provide a graphical user
lightly along the surface. In this embodiment, the controller

is con?gured to cause the actuator to produce a ?rst tactile
interface for the PDA 31 and are arranged in a desired
pattern or grid. Each softkey 36 occupies a distinct location

on the display panel. As illustrated, the PDA 31 can function
45
sensation When the softkey 36a is in the second position or

When the applied pressure is less than the amount of pressure
necessary to indicate that the softkey has been selected, that
is the third position. The controller Would then cause the
as a mobile telephone, and the softkeys 36 are arranged as
actuator 64 to produce a second tactile sensation upon
a telephone keypad to provide the same functionality as the

mechanical keys on a conventional telephone keypad. The
display 33 of the PDA 31 also includes additional graphical
outputs 37 and areas 38 Without graphical output. The
receipt of the input signal associated With the third position
displayed softkeys 36 are vieWable through the touchpad 30

and represent corresponding unique positions on the touch
pad 30.
An object 24, for example a human ?nger, selects a
desired softkey 36a-i by contacting the touchpad 30 at the
appropriate location. A controller (not shoWn) is in commu
nication With the touchpad 30. The controller of this embodi
ment is similar in structure and functionality to the controller
described in relation to the embodiment of FIG. 3. The
controller is capable of determining the location on the
or upon detection of a sufficient amount of pressure applied

50
associated With four distinct applied pressures and no pres
sure at the softkey 361', and corresponding to the input

signals for the letters W, X, Y, and Z. A dWell to select
feature can be used to determine the desired position and
55
can cause vibrations in the touchpad 35 in a direction
parallel to the surface 35 of the touch paid or perpendicular
associated input signal.

This functionality facilitates a user moving an object over
60
display screen 33 that is touched by the object 24, and the

softkey 36 corresponding to the touched location. Based
upon this information, the controller causes the actuator 64
to provide a corresponding tactile sensation. The actuator 64
to the softkey 36a. The softkey 361' has ?ve positions
the various softkeys displayed on the input device and

receiving a speci?c frequency or tactile sensation to signal
that a particular softkey has been touched. As the object 24
contacts other softkeys in the display matrix, additional
distinct tactile sensations unique to these other softkeys are
produced. With continued use, the user can quickly become
accustomed to the various distinct tactile sensations and the
65
associations betWeen sensations and speci?c softkeys, per

mitting identi?cation and selection of softkeys or buttons by
touch alone. In fact, distinct tactile sensations can be used
With the same button regardless of the electronic device,
-65HY
CIVIL COVER SHEET
7KH-6FLYLOFRYHUVKHHWDQGWKHLQIRUPDWLRQFRQWDLQHGKHUHLQQHLWKHUUHSODFHQRUVXSSOHPHQWWKHILOLQJDQGVHUYLFHRISOHDGLQJVRURWKHUSDSHUVDVUHTXLUHGE\ODZH[FHSWDV
SURYLGHGE\ORFDOUXOHVRIFRXUW7KLVIRUPDSSURYHGE\WKH-XGLFLDO&RQIHUHQFHRIWKH8QLWHG6WDWHVLQ6HSWHPEHULVUHTXLUHGIRUWKHXVHRIWKH&OHUNRI&RXUWIRUWKH
SXUSRVHRILQLWLDWLQJWKHFLYLOGRFNHWVKHHW(SEE INSTRUCTIONS ON NEXT PAGE OF THIS FORM.)
I. (a) PLAINTIFFS
DEFENDANTS
Immersion Corporation
Apple Inc., AT&T Inc., and AT&T Mobility LLC

&RXQW\RI5HVLGHQFHRI)LUVW/LVWHG'HIHQGDQW
(b)&RXQW\RI5HVLGHQFHRI)LUVW/LVWHG3ODLQWLII
(EXCEPT IN U.S. PLAINTIFF CASES)
(IN U.S. PLAINTIFF CASES ONLY)

,1/$1'&21'(01$7,21&$6(686(7+(/2&$7,212)
7+(75$&72)/$1',192/9('

$WWRUQH\V(If Known)
127(
(c)$WWRUQH\V(Firm Name, Address, and Telephone Number)

Brian E. Farnan, Farnan LLP
919 N. Market Street, 12th Floor
Wilmington, DE 19801
II. BASIS OF JURISDICTION(Place an X in One Box Only)

u 86*RYHUQPHQW
3ODLQWLII
u )HGHUDO4XHVWLRQ
(U.S. Government Not a Party)
u 86*RYHUQPHQW
'HIHQGDQW
u 'LYHUVLW\
(Indicate Citizenship of Parties in Item III)
III. CITIZENSHIP OF PRINCIPAL PARTIES (Place an X in One Box for Plaintiff

(For Diversity Cases Only)
PTF
&LWL]HQRI7KLV6WDWH
u
DEF
u
and One Box for Defendant)

PTF
DEF
,QFRUSRUDWHGor3ULQFLSDO3ODFH
u
u
RI%XVLQHVV,Q7KLV6WDWH
&LWL]HQRI$QRWKHU6WDWH
u
u
,QFRUSRUDWHGand3ULQFLSDO3ODFH
RI%XVLQHVV,Q$QRWKHU6WDWH
u
u
&LWL]HQRU6XEMHFWRID
)RUHLJQ&RXQWU\
u
u
)RUHLJQ1DWLRQ
u
u
IV. NATURE OF SUIT(Place an X in One Box Only)

CONTRACT
TORTS
u
u
u
u
,QVXUDQFH
0DULQH
0LOOHU$FW
1HJRWLDEOH,QVWUXPHQW
5HFRYHU\RI2YHUSD\PHQW
(QIRUFHPHQWRI-XGJPHQW
0HGLFDUH$FW
5HFRYHU\RI'HIDXOWHG
6WXGHQW/RDQV
([FOXGHV9HWHUDQV
5HFRYHU\RI2YHUSD\PHQW
RI9HWHUDQV%HQHILWV
6WRFNKROGHUV6XLWV
2WKHU&RQWUDFW
&RQWUDFW3URGXFW/LDELOLW\
)UDQFKLVH
u
u
u
u
u
u
REAL PROPERTY
/DQG&RQGHPQDWLRQ
)RUHFORVXUH
5HQW/HDVH (MHFWPHQW
7RUWVWR/DQG
7RUW3URGXFW/LDELOLW\
$OO2WKHU5HDO3URSHUW\
u
u
u
u
u
u
u
PERSONAL INJURY

u $LUSODQH
u $LUSODQH3URGXFW
/LDELOLW\
u $VVDXOW/LEHO
6ODQGHU
u )HGHUDO(PSOR\HUV
/LDELOLW\
u 0DULQH
u 0DULQH3URGXFW
/LDELOLW\
u 0RWRU9HKLFOH
u 0RWRU9HKLFOH
3URGXFW/LDELOLW\
u 2WKHU3HUVRQDO
,QMXU\
u 3HUVRQDO,QMXU\
0HGLFDO0DOSUDFWLFH
CIVIL RIGHTS
u 2WKHU&LYLO5LJKWV
u 9RWLQJ
u (PSOR\PHQW
u +RXVLQJ
$FFRPPRGDWLRQV
u $PHUZ'LVDELOLWLHV
(PSOR\PHQW
u $PHUZ'LVDELOLWLHV
2WKHU
u (GXFDWLRQ
FORFEITURE/PENALTY
PERSONAL INJURY
u 3HUVRQDO,QMXU\
3URGXFW/LDELOLW\
u +HDOWK&DUH
3KDUPDFHXWLFDO
3HUVRQDO,QMXU\
3URGXFW/LDELOLW\
u $VEHVWRV3HUVRQDO
,QMXU\3URGXFW
/LDELOLW\
PERSONAL PROPERTY
u 2WKHU)UDXG
u 7UXWKLQ/HQGLQJ
u 2WKHU3HUVRQDO
3URSHUW\'DPDJH
u 3URSHUW\'DPDJH
3URGXFW/LDELOLW\
PRISONER PETITIONS
Habeas Corpus:
u $OLHQ'HWDLQHH
u 0RWLRQVWR9DFDWH
6HQWHQFH
u *HQHUDO
u 'HDWK3HQDOW\
Other:
u 0DQGDPXV 2WKHU
u &LYLO5LJKWV
u 3ULVRQ&RQGLWLRQ
u &LYLO'HWDLQHH
&RQGLWLRQVRI
&RQILQHPHQW
u 'UXJ5HODWHG6HL]XUH
RI3URSHUW\86&
u 2WKHU
BANKRUPTCY
u $SSHDO86&
u :LWKGUDZDO
86&
PROPERTY RIGHTS
u &RS\ULJKWV
u 3DWHQW
u 7UDGHPDUN
u
u
u
u
u
u
LABOR
)DLU/DERU6WDQGDUGV
$FW
/DERU0DQDJHPHQW
5HODWLRQV
5DLOZD\/DERU$FW
)DPLO\DQG0HGLFDO
/HDYH$FW
2WKHU/DERU/LWLJDWLRQ
(PSOR\HH5HWLUHPHQW
,QFRPH6HFXULW\$FW
u
u
u
u
u
SOCIAL SECURITY
+,$II
%ODFN/XQJ
',:&',::J
66,'7LWOH;9,
56,J
FEDERAL TAX SUITS

u 7D[HV863ODLQWLII
RU'HIHQGDQW
u ,567KLUG3DUW\
86&
OTHER STATUTES
u
u
u
u
u
u
u
u
u
u
u
u
u
u
u
u
)DOVH&ODLPV$FW
6WDWH5HDSSRUWLRQPHQW
$QWLWUXVW
%DQNVDQG%DQNLQJ
&RPPHUFH
'HSRUWDWLRQ
5DFNHWHHU,QIOXHQFHGDQG
&RUUXSW2UJDQL]DWLRQV
&RQVXPHU&UHGLW
&DEOH6DW79
6HFXULWLHV&RPPRGLWLHV
([FKDQJH
2WKHU6WDWXWRU\$FWLRQV
$JULFXOWXUDO$FWV
(QYLURQPHQWDO0DWWHUV
)UHHGRPRI,QIRUPDWLRQ
$FW
$UELWUDWLRQ
$GPLQLVWUDWLYH3URFHGXUH
$FW5HYLHZRU$SSHDORI
$JHQF\'HFLVLRQ
&RQVWLWXWLRQDOLW\RI
6WDWH6WDWXWHV
IMMIGRATION
u 1DWXUDOL]DWLRQ$SSOLFDWLRQ
u 2WKHU,PPLJUDWLRQ
$FWLRQV
V. ORIGIN(Place an X in One Box Only)

u 2ULJLQDO
3URFHHGLQJ
u 5HPRYHGIURP
6WDWH&RXUW
u
5HPDQGHGIURP
$SSHOODWH&RXUW
u 5HLQVWDWHGRU
5HRSHQHG
u 7UDQVIHUUHGIURP
$QRWKHU'LVWULFW
u 0XOWLGLVWULFW
/LWLJDWLRQ
(specify)
VI. CAUSE OF ACTION
&LWHWKH86&LYLO6WDWXWHXQGHUZKLFK\RXDUHILOLQJ(Do not cite jurisdictional statutes unless diversity)

35 USC 271
%ULHIGHVFULSWLRQRIFDXVH
Infringement of U.S. Patents US 8,749,507 B2; US 7,808,488 B2; US 8,581,710 B2; and US 7,336,260 B2
DEMAND $
u &+(&.,)7+,6,6$CLASS ACTION
VII. REQUESTED IN
81'(558/()5&Y3
COMPLAINT:
VIII. RELATED CASE(S)
(See instructions):
The Honorable Richard G. Andrews
IF ANY
-8'*(
'$7(
&+(&.<(6RQO\LIGHPDQGHGLQFRPSODLQW
u <HV
u 1R
JURY DEMAND:
'2&.(7180%(5 16-cv-77-RGA
6,*1$785(2)$77251(<2)5(&25'
/s/ Brian E. Farnan
05/05/2016
FOR OFFICE USE ONLY
5(&(,37
$02817
$33/<,1*,)3
-8'*(
0$*-8'*(

Second Delaware Immersion Suit

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Second Delaware Immersion Suit

Uploaded by

Copyright:

Available Formats

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 1 of 25 PageID #: 1

IN THE UNITED STATES DISTRICT COURT

JURY TRIAL DEMANDED

COMPLAINT FOR PATENT INFRINGEMENT

Immersion is a leading innovator and developer of haptic technology. Since its

"Haptics" refers to the science of touch. Haptics in consumer electronic devices

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 2 of 25 PageID #: 2

When haptics is implemented on a mobile phone (for example, through

In electronic devices, haptic effects are typically managed and controlled by

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 3 of 25 PageID #: 3

Pressure-enabled haptics is a form of haptics. For example, pressure-enabled

Haptics, including pressure-enabled haptics, can be incorporated into many

features of a smartphone or laptop. For example, it is especially useful in electronic devices

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 4 of 25 PageID #: 4

Haptic effects, including pressure-enabled haptics, may also be used to confirm

Haptics play a central role in Defendants' products. For example, important

These features of the Apple iPhone 6s and 6s Plus provide haptic

See, e.g., Apple, iPhone 6s

3DTouch, http://www.apple.com/iphone-6s/3d-touch/ (3D Touch):

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 5 of 25 PageID #: 5

See also, e.g., http://www.apple.com/iphone-6s/technology/ (Taptic Engine):

See also, e.g., https://www.att.com/cellphones/iphone/iphone-6s.html (3D Touch advertised by

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 6 of 25 PageID #: 6

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 7 of 25 PageID #: 7

Defendants are capitalizing on Immersion's innovation and success by selling

Immersion is a Delaware corporation with its principal place of business located

of business at 1 Infinite Loop, Cupertino, CA 95014.

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 8 of 25 PageID #: 8

Defendant AT&T Inc. is a Delaware corporation with its principal place of

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 9 of 25 PageID #: 9

Venue is proper in this District pursuant to 28 U.S.C. 1400(b) and 1391(b)-(c)

Joinder of Defendants in this action is proper under 35 U.S.C. 299 because,

Immersion incorporates by reference paragraphs 1 through 20 as if set forth here

The '507 patent, in general, describes advances in providing haptic feedback in

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 10 of 25 PageID #: 10

Immersion makes this preliminary identification of infringing apparatuses, products,

The accused devices determine

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 11 of 25 PageID #: 11

technology/patents/index.html and currently located at http://www.immersion.com/legal/

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 12 of 25 PageID #: 12

See, e.g., http://www.apple.com/iphone-6s/3d-touch/; http://www.apple.com/iphone-

6s/technology/; http://help.apple.com/iphone/9/. Defendant AT&T similarly provides directions,

iphone/iphone-6s.html; http://www.att.com/esupport/iphonefaqs.jsp. The Defendants have sold

Defendants have contributorily infringed and are currently contributorily

As a direct and proximate result of Defendants' infringement of the '507 patent,

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 13 of 25 PageID #: 13

Immersion incorporates by reference paragraphs 1 through 20 as if set forth here

The '488 patent, in general, describes advances in providing haptic feedback in

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 14 of 25 PageID #: 14

Immersion makes this preliminary identification of infringing apparatuses, products,

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 15 of 25 PageID #: 15

technology/patents/index.html and currently located at http://www.immersion.com/legal/

See, e.g., http://www.apple.com/iphone-6s/3d-touch/; http://www.apple.com/iphone-

6s/technology/; http://help.apple.com/iphone/9/. Defendant AT&T similarly provides directions,

iphone/iphone-6s.html; http://www.att.com/esupport/iphonefaqs.jsp. The Defendants have sold

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 16 of 25 PageID #: 16

Defendants have contributorily infringed and are currently contributorily

As a direct and proximate result of Defendants' infringement of the '488 patent,

Immersion incorporates by reference paragraphs 1 through 20 as if set forth here

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 17 of 25 PageID #: 17

The '710 patent, in general, describes advances in providing haptic feedback in

Immersion makes this preliminary identification of infringing apparatuses, products,

Case 1:16-cv-00325-UNA Document 1 Filed 05/05/16 Page 18 of 25 PageID #: 18

technology/patents/index.html and currently located at http://www.immersion.com/legal/

4/2006 Hsll e131. ~~~~~~~~~~~~~~~~~~ 345/173