Downloaded from http://bjsm.bmj.com/ on November 8, 2015 - Published by group.bmj.

com

Review

Smartphone and tablet apps for concussion road

warriors (team clinicians): a systematic review for
practical users
Hopin Lee,1,2,3 S John Sullivan,1 Anthony G Schneiders,1 Osman Hassan Ahmed,1,4
Arun Prasad Balasundaram,1 David Williams,1 Willem H Meeuwisse,5 Paul McCrory6
1

Centre for Health, Activity and

Rehabilitation Research, School
of Physiotherapy, University of
Otago, Dunedin, New Zealand
2
Neuroscience Research
Australia, Sydney, Australia
3
School of Medical Sciences,
University of New South Wales,
Sydney, Australia
4
The FA Centre for Disability
Football Research, St Georges
Park, Burton-Upon-Trent,
Burton-Upon-Trent,
Staffordshire, UK
5
Faculty of Kinesiology,
University of Calgary, Calgary,
Alberta, Canada
6
The Florey Institute of
Neuroscience and Mental
HealthMelbourne Brain
Centre, University of
Melbourne, Heidelberg,
Australia
Correspondence to
Professor S John Sullivan,
Centre for Health, Activity and
Rehabilitation Research, School
of Physiotherapy, University of
Otago, PO Box 56, Dunedin
9054, New Zealand;
sjohn.sullivan@otago.ac.nz
Accepted 24 February 2014
Published Online First
25 March 2014

ABSTRACT
Background Mobile technologies are steadily replacing
traditional assessment approaches for the recognition
and assessment of a sports concussion. Their ease of
access, while facilitating the early identication of a
concussion, also raises issues regarding the content of
the applications (apps) and their suitability for different
user groups.
Aim To locate and review apps that assist in the
recognition and assessment of a sports concussion and
to assess their content with respect to that of
internationally accepted best-practice instruments.
Methods A search of international app stores and of
the web using key terms such as concussion, sports
concussion and variants was conducted. For those apps
meeting the inclusion criteria, data were extracted on the
platform, intended users and price. The content of each
app was benchmarked to the Sport Concussion
Assessment Tool 2 (SCAT2) and Pocket SCAT2 using a
custom scoring scheme to generate a percentage
compliance statistic.
Results 18 of the 155 apps identied met the
inclusion criteria. Almost all (16/18) were available on
an iOS platform and only ve required a payment to
purchase. The apps were marketed for a wide range of
intended users from medical professionals to the general
public. The content of the apps varied from 0% to
100% compliance with the selected standard, and
symptom evaluation components demonstrated the
highest level of compliance.
Conclusions The surge in availability of apps in an
unregulated market raises concerns as to the
appropriateness of their content for different groups of
end users. The consolidation of best-practice concussion
instruments now provides a framework to inform the
development of future apps.
INTRODUCTION

To cite: Lee H, Sullivan SJ,

Schneiders AG, et al. Br J
Sports Med 2015;49:
499505.

The introduction of mobile technologies in the

form of smartphones and tablets has created a new
landscape for innovative health delivery strategies.
Since 2007, there has been a rapid growth in smartphone usage, with an estimated one billion users in
2012.1 This gure is expected to rise to two billion
by 2015,2 meaning that more than one-third of the
global population potentially has access to this
technology.
Collectively, this demonstrates the important role
that mobile devices play in our daily lives. These
devices are becoming essential not only for interpersonal communications but also as a vehicle for

conducting business, and for running a wide range

of specically developed software applications
(apps). In parallel with the increase in the number
of smartphones and tablets has been the exponential rise in the number of apps available, often at
minimal or no cost to the user. At the time of
writing, there were more than 800 000 apps available from the iTunes store3 and 700 000 available
from Google Play.4
Apps are available for a wide range of purposes
including uses in health, sports and medicine.5
There has been a rapid surge in the number of
medical and healthcare apps with approximately
40 000 apps available for downloading for smartphones and tablets.6 Recent reports have also suggested that a large sector of the US population is
using health-related apps.7 With the proliferation in
the number of health-related apps and with no
associated regulatory control over their content, the
health practitioner and consumer are often challenged as to which app to choose and whether the
content of the app is based on current best-practice
information.8 9 With the recent release of guidelines for the regulation of medical apps by the US
Food and Drug Administration (FDA) agency,10 it is
likely that the regulatory control over mobile
health devices will gain more public attention.
However, the selection of apps that are proposed
to undergo appropriate review still remains
ambiguous, especially for those apps that pose a
lower risk to the public.
The quality of apps is dependent on the knowledge, experience and motivations of the developers
and whether suitably informed health professionals
are involved in the development process. Recently,
there has been the emergence of a number of systematic reviews of apps conducted in a range of
health areas including: orthopaedic surgery,11 pain
management,12 smoking cessation,8 the selfmanagement of asthma,9 weight loss,13 sports
injury prevention14 and neurology.15 While there
have been individual evaluations16 and news
releases17 of specic apps for the recognition and
assessment for a sports concussion, until now there
have been no systematic reviews of concussionrelated apps.
With the development of recent consensus18 and
position statements19 on the best-practice management of a suspected concussion, there is a content
basis on which the apps can be developed. The
Concussion in Sport (CIS) Group that published
the 2008 Zurich guidelines have made available a

Lee H, et al. Br J Sports Med 2015;49:499505. doi:10.1136/bjsports-2013-092930

1 of 8

Downloaded from http://bjsm.bmj.com/ on November 8, 2015 - Published by group.bmj.com

Review
free iPhone and iPad app with their assessment tools provided
in an electronic format, while others have developed a range of
independent apps (eg, Hockey Canada Concussion
Awareness20). Collectively, the use of these apps represent an
important opportunity in translating21 this essential information
not only to sports medicine professionals but also to coaches,
teachers, team managers and the parents of young athletes. The
empowering of these individuals through the availability of
concussion-related apps is a valuable step in improving the early
diagnosis of a possible concussion, particularly at the community sports level where trained professionals may not be present.
Paramount is the necessity for these apps to deliver current bestpractice information and assessment procedures, contained in
foundation documents such as the recent CIS consensus statements.18 22
With the increased availability of concussion-related apps in
the consumer marketplace, it is timely to review these, in order
to provide the consumer and health professional with comparative information to inform their choice of app and whether the
content of the app is in keeping with current international bestpractice concussion management strategies. Thus, the purpose
of this review was to identify and appraise smartphone/tablet
apps related to the recognition and assessment of a sports
concussion.

METHODS
The key elements of a traditional systematic review methodology were used to locate all available apps relating to the
assessment of a sports concussion at the time of the search (10
April 2013). In addition, a systematic approach was taken to the
extraction of data and appraisal of content to reduce reviewer
bias. Although the apps were all evaluated with respect to their
content, no personal data were collected as part of this review.
Thus, ethical approval from a recognised ethics committee was
not sought.

app stores. The search strategy and selection process are presented in gure 1. The apps identied by the individual searches
were pooled and duplicates removed. Where the same app was
available on different platforms (iOS or Android), only one
version of the app was retained for analysis, with the iOS platform being preferred. Those apps meeting the inclusion criteria
were downloaded onto an Apple iPad2 (iOS 6) if they were
available through the Apple iTunes store, and onto a Samsung
Galaxy SII smartphone device (Android V.2.3.3) if available
through Google play for further exploration and analysis.
Downloading to the Apple iPad allowed the maximum exibility
in reviewing the apps.

Data extraction
A wide range of data about the apps was extracted from the
smartphone app stores description and by reviewing the apps in
operation. This information included: the purpose of the app,
the date of release or update, pricing, scientic underpinnings,
platform availability, intended user and whether it contained
social networking site/email sharing capabilities.
The data were extracted in a systematic manner by the principal investigator (HL) and veried by at least one other member
of the research team prior to entry into an Excel spreadsheet
(Microsoft 2010, Redmond, Washington, USA).

Appraisal of the app content

A systematic search strategy was used to identify smartphone

apps available at the time of the search (10 April 2013). The
iTunes stores (http://www.apple.com/itunes) from Australia,
Canada, New Zealand, the UK and the USA were searched
using the Power Search (iTunes V.10.7) function using the key
words; concussion, sport concussion, sports concussion,
sport-concussion, sports-concussion, mild traumatic brain
injury or mTBI. Searches using the same strategy were also
conducted in other smartphone app stores including: Google
play, Blackberry App World, Ovi Store (for Nokia/Symbian
brands), Windows Mobile Marketplace and Samsung apps. A
supplementary search was also conducted by browsing Google
with the terms; apps, apps for concussion and apps for
sports concussion to capture any further apps which may not
have been classied appropriately or were emerging into the

The content of each app was reviewed and appraised with reference to the content of either the Sport Concussion Assessment
Tool 2 (SCAT2)23 or the Pocket SCAT224 depending on the
intended user. Those apps which indicated in their description
or within the product that they were for use by medical personnel were reviewed with reference to SCAT2, which was considered as the gold standard in concussion recognition,
assessment and management at the time the apps were developed. This tool has been systematically developed by an international group of experts (the CIS group) and widely
disseminated internationally, cited in the research literature25
and adopted by major sporting organisations (eg, FIFA, IOC,
IIHF, AFL and the IRB) and modifed and adopted by others
(NFL, NHL and MLB). It was developed for use with athletes
10 years of age and over and for use by medical personnel or
persons working under the supervision of such. As no formal
appraisal tool was available, we developed a checklist and decision rule using an approach similar to that in a previous study
evaluating the content of concussion websites26 and to that used
in the content appraisal of other health-related apps.8 9 This
checklist considered whether the seven key elements (eg,
Symptoms, Physical Signs, Balance Assessment, etc) contained in the SCAT2 were included in the app. The checklist
also contained items relating to accuracy of measurement (eg, if
the symptoms were assessed on a seven-point Likert scale) and
whether the SCAT2 Score (overall score) was generated, to
provide a total of 10 items of interest. The content of each app
was initially examined with reference to the checklist by the
principal investigator (HL) and where there was doubt in the
scoring, this was veried by at least one member of the research
team who discussed the scoring before arriving at a joint decision. The presence of items was scored using a three-point categorical scoring system where 2=present and complete,
1=present but incomplete and 0=not included/not specied.
The score of the 10 items was summed (maximum score=20)
and presented as a SCAT2 compliance score (%) as an indication
of how closely the contents followed that of SCAT2.

2 of 8

Lee H, et al. Br J Sports Med 2015;49:499505. doi:10.1136/bjsports-2013-092930

Inclusion criteria
Apps were considered for inclusion in the study if they were
produced in English; designed and/or marketed to be used in
the recognition and assessment of concussion; available to the
sports medicine professional and/or the general public and
were a self-contained product. Apps were excluded if their
primary function was unrelated to concussion assessment (eg,
games and brain training apps), if they were essentially
scoring apps for particular concussion tests, or if they were
designed purely for the dissemination of information on
concussion.

Search strategy

Downloaded from http://bjsm.bmj.com/ on November 8, 2015 - Published by group.bmj.com

Review
Figure 1 Search ow diagram.

The search strategy shown in gure 1 identied 18 apps as

meeting the studys inclusion criteria (table 1). Of these, 17
were identied by searching the app stores and one via the supplementary search (Google browsing) search. The major reason
for apps not meeting the studys criteria included those which
were not in English language, designed for gaming purposes,
designed to raise awareness and/or educate about concussion,
or which did not include elements of assessment. In all cases,
the developers of the apps were clearly identied. Almost all
apps were available on the Apple iTunes store and in some
cases were offered on multiple platforms (eg, iOS and
Android). All apps were compatible on both smartphone and

tablet devices, except the King-Devick Concussion Screening

Test V.2.3.0 app27 which was kept exclusive to the iPad. Most
(13/18) were free to download, with the others ranging from
US$0.99 to US$44.99. The apps were identied as being suitable for a wide range of intended user groups including:
doctors, healthcare professionals, athletic trainers, parents and
coaches. In some instances, apps were targeted at more than
one potential user group. Generally, the apps referred to a
recognised standard or provided some research evidence to
support the content of the product; however, in several
instances, this was absent. The potential to transmit the results
was a common feature of 14 of the apps, with email being the
most frequent mode. Four of the 18 apps provided the opportunity to transmit the results of the assessment to a central
database hosted by the developers.
Table 2 documents the content of each of the apps which
were targeted at medically qualied persons with respect to the
core elements of SCAT2. The SCAT2 Compliance scores
ranged from 0% to 100%. Of note here is the general lack of
compliance with the SCAT2 content, with the majority of apps
not including many of the core items. The content of those
apps which indicated that they were for use by lay (ie, nonmedical) groups is shown in table 3, where a similar spectrum
of compliance to the three core elements of the Pocket
SCAT224 is seen.

Lee H, et al. Br J Sports Med 2015;49:499505. doi:10.1136/bjsports-2013-092930

3 of 8

Those apps targeted at non-medically qualied individuals

such as athletic trainers, parents or coaches were assessed with
respect to contents of the Pocket SCAT224 using a similar
approach. The Pocket SCAT2 is a simplied version of SCAT2
developed by the same CIS group with its elements targeted at
the recognition of a potential concussion by laypersons with
minimal training. Thus, apps which were suggested to be appropriate for medical and non-medical user groups were reviewed
against both SCAT2 and Pocket SCAT2. The data are presented
in a descriptive format to allow comparisons between apps and,
where applicable, as frequency counts.

RESULTS

Downloaded from http://bjsm.bmj.com/ on November 8, 2015 - Published by group.bmj.com

Review
Table 1 App descriptors

App titleversion

Date for most

recent update

Price
(US$)

Platform
(s)

Cognit1.0.0

04-07-2009

Free

iOS

Concussion1.2.0

10-05-2011

iOS

Concussion Assessment & Response:

Sport Version (CARE)1.1.0
Concussion Recognition & Response:
Coach & Parent Version1.2.2
Concussion Signs and Symptoms
2.3.0
Concussion Quick Check1.0.1

18-03-2013

US
$0.99*
US$4.99

03-01-2012

US$0.99

20-07-2012

Free

15-03-2013

Free

Concussion22.0.0
EmergencyConsult1.0.0

01-12-2012
19-03-2013

Free
Free

First Responder1.0.0

12-03-2013

Free

ImPACT Concussion Awareness Tool

(ImCAT)1.1.0
King-Devick Concussion Screening
Test2.3.0

07-07-2012

Free

11-09-2012

US
$44.99

PLAY IT SAFE Concussion

Assessment1.3.0
Pocket SCAT21.0.0

17-11-2011

Free

iOS

03-08-2011

Free

iOS

ReturnToPlay1.0.0

19-03-2013

Free

Return2Play1.3.0

09-04-2013

Free

SACTool Beta1.0.0
SCAT21.0.0
SCAT2Sport Concussion
Assessment Tool1.3.0

19-08-2011
28-04-2011
16-02-2012

Free
US$3.99
Free

iOS
Android
iOS
Android
Android
iOS
iOS

iOS
Android
iOS
Android
Android
iOS
Android
iOS
iOS
Android
iOS
Android
iOS
Android
iOS

Sharing
capabilities

Reference to a
standard or
guidelines

Link to a
central
database

AT, coaches and

HCPs
Medical, coaches
and parents
Medical and ATs

Email

NS

Email

NS

Email

Parents and
coaches
NS

Email

NA

NS

NS

General

NA

NS

NS

Coaches
Medical

Email
Email

NS
NS

NS
NS

Coaches, parents
and teachers
General

Email

NS

NS

Email

NS

King-Devick
online system
Email
Email

NS

NA

NS

Intended user

Medical, AT,
coaches and
parents
HCPs, coaches and
parents
Coaches and
parents
Medical and
healthcare team
Patients and HCPs

Email

NS

NS

Email

NS

AT
Medical and HCPs
Medical and HCPs

NA
Email
Email

Y
Y
Y

NS
NS
Y

*Additional diagnostic module available.

Additional costs for more than three participants.
Exclusive to the US iTunes store.
App, application; ATs, athletic trainers; HCPs, healthcare professionals; iOS, i operating system; NA, not available; NS, not specified; Y, yes.

There has been a major international effort to consolidate the

comprehensive understanding of what constitutes a concussion
in a sports situation and how to manage the player to safely
return to activity. Successive meetings of the CIS group have led
this initiative which has resulted in the development of the concussion consensus18 22 28 29 as well as specic assessment tools
(SCAT,29 SCAT223 and Pocket SCAT224) and, more recently,
SCAT3,30 Child SCAT331 and Pocket CRT.32 Although not
without criticism,33 these instruments have gone on to become
the backbone of concussion assessment worldwide. The advent
of smartphones and mobile technologies has created opportunities for the packaging of these and other assessment instruments into the growing market of apps and facilitating their use
by a much wider audience.34
The growth in the availability of mobile apps related to concussion recognition and assessment has been noteworthy. The
rst concussion-related app appeared in the marketplace in
mid-2009, with 18 apps being identied in the search process
for this study. It can be expected that this number will increase
as new developers enter the market (eg, Concussion Toolbox,35

X2CMS36), and as apps are developed/revised to deliver SCAT

3,30 Child SCAT 331 and the Pocket CRT.32 This increased
choice of apps will present the sports medicine practitioner and
the consumer with the dilemma as to which app to adopt for
their particular need and circumstances. It is thus important that
all users are provided with a reference to some criteria beyond
populist ratings or blogs from other consumers to inform their
selection.
In order to provide a reference standard, we chose SCAT223
and the Pocket SCAT224 as benchmarks for examining the
content of the apps. While this decision can be debated, it does
provide an internationally recognised standard with which to
compare the content of each app. This is an approach analogous
to that used in the examination of apps relating to smoking cessation8 and asthma self-management,9 which referenced the app
content to widely accepted guidelines using coding systems
similar to that used in this study. It is acknowledged that not all
the apps were designed to provide a comprehensive assessment
for a concussion, with some37 focusing on specic tests such as
the Standardized Assessment of Concussion38 which are well
established and embedded within the SCAT2 tool.

4 of 8

Lee H, et al. Br J Sports Med 2015;49:499505. doi:10.1136/bjsports-2013-092930

DISCUSSION

Downloaded from http://bjsm.bmj.com/ on November 8, 2015 - Published by group.bmj.com

Review
Table 2 Apps benchmarked to SCAT2 content
Symptoms

App titleversion
Concussion1.2.0
Concussion Assessment &
Response: Sport Version
(CARE)1.1.0
Emergency Consult1.0.0
King-Devick Concussion
Screening Test2.3.0
ReturnToPlay1.0.0
SCAT21.0.0
SCAT2Sport Concussion
Assessment Tool1.3.0

Physical signs

Sideline

22
Seven-point
items Likert scale

Worse
with
physical/
mental
activity

LOC/
unresponsiveness
Balance/
Maddocks
unsteadiness
GCS score

2
2

2
2

2
2

2
1

2
0

1
0

2
0

0
0

2
0

1
2
2

2
2
2

0
2
2

0
2
2

Cognitive Balance

Coordination

SAC

Modified
BESS
FTN

SCAT2 SCAT2
total compliance
score score (%)

2
0

2
2

2
2

2
1

2
0

100
60

0
0

2
0

2
0

0
0

2
0

0
0

55
0

0
2
2

0
2
2

2
2
2

0
2
2

2
2
2

0
2
2

35
100
100

2, present and complete; 1, present but incomplete; 0, not included/not specified; apps, applications; BESS, Balance Error Scoring System; FTN, finger to nose; GCS, Glasgow Coma
Scale; LOC, loss of consciousness; SAC, Standardized Assessment of Concussion; SCAT2, Sport Concussion Assessment Tool 2.

Among those apps which provided a comprehensive

approach, only the original SCAT2 Sport Concussion
Assessment Tool app (developed by the CIS group) and the
SCAT2 app (which was independently developed) showed a
perfect compliance with the SCAT2 reference criteria adopted
for this study. There was considerable variability among the
content of the other apps. For instance, some apps would alter
the wording of a particular item while most probably still retaining the overall objective of the item itself, or provide a reduced
set of symptoms. The failure to obtain a high SCAT2 compliance score does not necessarily imply that the app does not
include appropriate content for the recognition and/or assessment of a concussion; rather, it indicates that the app does not
comply with the core element of the SCAT2 reference standard.

Table 3

As some apps were clearly designed to be used by nonmedical personnel such as coaches or parents, these were referenced to the Pocket SCAT2 (the tool designed for this purpose).
As might be expected, many of the apps met these minimum
expected criteria for the recognition of a concussion and hence
demonstrated a high level of compliance to the core information
contained in the Pocket SCAT2; however, there was considerable variation in the content of these apps. In many instances,
this subset of apps is more important as it is directed at users
who may not be expected to have a sophisticated knowledge of
concussion or medical training, and may rely on the app to
provide prompts and an early indication of whether a player
might be concussed and thus in need of a medical consultation.
The public availability of apps allows anybody to download and

Apps for use by laypersons benchmarked to Pocket SCAT2

App titleversion

Symptoms
22 items

Memory function
Maddocks score

Balance testing
Modified BESS

Pocket SCAT2
compliance score (%)

Cognit1.0.0
Concussion1.2.0
Concussion Assessment & Response: Sport Version (CARE)1.1.0
Concussion Recognition & Response: Coach & Parent Version1.2.2
Concussion Signs and Symptoms2.3.0
Concussion Quick Check1.0.1
Concussion22.0.0
EmergencyConsult1.0.0
First Responder1.0.0
ImPACT Concussion Awareness Tool (ImCAT)1.1.0
King-Devick Concussion Screening Test2.3.0
PLAY IT SAFE Concussion Assessment (USA only)1.3.0
Pocket SCAT21.0.0
ReturnToPlay1.0.0
Return2Play1.3.0
SACTool Beta1.0.0
SCAT21.0.0
SCAT2Sport Concussion Assessment Tool1.3.0

1
2
2
1
1
1
1
1
1
1
0
1
2
1
1
0
2
2

0
2
0
0
0
1
2
2
2
0
0
0
2
0
0
0
2
2

0
1
2
0
0
0
1
0
0
0
0
1
1
0
0
0
2
2

17
83
67
17
17
33
67
50
50
17
0
33
83
17
17
0
100
100

2, present and complete; 1, present but incomplete; 0, not included/not specified; apps, applications; BESS, Balance Error Scoring System; SCAT2, Sport Concussion Assessment Tool 2.

Lee H, et al. Br J Sports Med 2015;49:499505. doi:10.1136/bjsports-2013-092930

5 of 8

Downloaded from http://bjsm.bmj.com/ on November 8, 2015 - Published by group.bmj.com

Review
use an app. While this may be seen as widening the opportunity
to identify and manage a greater number of concussions, it also
provides the potential for less qualied individuals to use such
apps inappropriately. Fundamental to this point of availability is
whether there is an actual need for an app. While it can be
argued that they may represent nothing more than a hi-tech
way of presenting and managing information, they offer a range
of functions beyond this. The use of mobile technologies is now
becoming an integral component of everyday lifestyle, and thus
the availability of concussion apps promotes the global access to
information and facilitates documentation. Although not essential to the recognition and management of a concussion, mobile
apps provide a new dimension in concussion management.
Currently, not all health-related apps are formally regulated by
the FDA or any other agency, although this is likely to change in
the coming years.39 The purpose of such regulation is to provide
the consumer with the condence that the product can be used
safely by the intended user. In the interim, the onus is on the
developers of apps to provide the consumer with a welldocumented product, and a clear indication as to the intended
user group of the app. Of the apps identied in this study, it was
sometimes difcult to ascertain who the app was designed to be
used by; thus, these apps were evaluated both against SCAT223
and the Pocket SCAT2.24 Developers of future apps need to
ensure that their product is based on a current gold standard information (such as the SCAT3,30 Pocket CRT32 and the Child
SCAT331 assessment tools18); in addition, they need to be clearly
targeted at the appropriate user group (eg, medical doctors; athletic trainers, physiotherapists or the general public).
A feature of many of the apps identied in this study was the
ability to transmit information, either via email or to a host
database for research purposes. The ability to transmit this
information via email to a doctor or a medical clinic is a useful
feature; however, its usefulness is predicated on the recipient of
this information having the necessary background to understand
and interpret the information transmitted. A number of apps
provide the option of sending the individuals data to a database
for subsequent research purposes. This is of value to the database hosts and to the wider research community, although the

Table 4

potential security of this information and biased data retrieval is

of concern. As consumers are required to make a choice
between a growing number of apps, often without a sophisticated understanding of the eld of concussion assessment and
management, we have developed a brief consumer checklist
(table 4) to assist them in making their choice. This easy-to-use
guide provides a number of questions which an individual
should ask as part of the process of comparing apps and making
an informed choice. The checklist does not make recommendations with respect to a specic app, but rather empowers the
consumer with key information to consider when choosing an
app that meets their intended need. This checklist may also
provide guidance to developers of sports concussion-related
apps to ensure that their apps provide the necessary information
to allow consumers to make an informed choice.
This study provides an initial insight into the properties and
content of the apps currently available for use by professionals
and consumers to aid in the determination of whether a player
has been concussed. A key strength of this study was the utilisation of the key elements of a systematic review, including a systematic search strategy and transparent inclusion criteria to
identify the apps to be considered. Key data on the content of
each app were benchmarked to the core components of
SCAT2,23 a tool which can be considered as the international
reference standard in concussion assessment. This provides a
recognised point of comparison for assessing the quality of the
currently available concussion-related apps and those that will
be released in the coming years. The core criteria used to assess
the content of the apps will need to be updated in keeping with
the recent publication of SCAT3.30
The study is not without a number of limitations. While the
apps were downloaded and information was extracted, no
scoring of the usability of the apps was undertaken (eg, user
interface and customer satisfaction). The usability is likely to be
dependent on the background and experience of the individuals
using the apps, and such an investigation was beyond the scope
of this paper. Although the appraisal process and data extraction
were conducted in an objective manner to reduce the risk of
bias, the checklist developed to appraise the apps could have

Consumer checklist to assist in choosing a sports concussion app

Characteristics

Questions to ask

Intended use

What was the app developed to do?

Who is the intended user (medical or lay person) and is that appropriate for this consumer?
When was the latest update?
Was the app developed as a commercial product?
Is the app endorsed/recommended by a recognised agency or organisation?
Was a medical doctor or a qualified health professional or group involved in the development of the app? Are they clearly identified?
Does the app reference a best-practice standard?
Does the app indicate that it is based on/or conforms with recognised international best-practice concussion management
statements?
Does the app contain advertising or commercial messages?
Are there any consumer satisfaction/ratings or reports available on the ease of use of the app?
Can you contact the producer if you have questions or require support?
Are there any resources or pathways to further information about a sports concussion?

Authority evidence

Commercialisation
Users comments
Producers support
Additional information and
resources
Data management
Research
Price

Can the data be shared via email, printed or saved?

If so, are privacy issues clearly identified?
Are the data retained by the developer for research or other intended purposes?
Is this clearly stated?
Are there additional or hidden costs associated with the use of the app or interpretation of the data?

App, application.

6 of 8

Lee H, et al. Br J Sports Med 2015;49:499505. doi:10.1136/bjsports-2013-092930

Downloaded from http://bjsm.bmj.com/ on November 8, 2015 - Published by group.bmj.com

Review
What are the new ndings?
This study provides a comparative review of mobile and
tablet applications (apps) available to be downloaded and
used in the recognition and assessment of a sports-related
concussion.
There was considerable diversity in the content of the apps
indicating the need for consideration as to the most
appropriate app for the designated purpose.
This review provides the developers of new apps, or those
upgrading current apps, with key information to include in
their products.
A checklist is provided to assist consumers in making a
choice as to the most appropriate app for their needs.

and the development of educational material. He is a cofounder and shareholder in

two biomedical companies (involved in eHealth and Compression garment
technologies) but does not hold any individual shares in any company related to
concussion or brain injury assessment or technology. WHM has received research
grant support through the University of Calgary from the Canadian Institutes of
Health Research, Alberta Innovates Health Solutions, FIFA, Alberta Childrens
Hospital Research Institute and the Hotchkiss Brain Institute. He is Co-Chair of the
Sport Injury Prevention Research Centre, which is one of the International Research
Centres for Prevention of Injury and Protection of Athlete Health supported by IOC.
He has a clinical and consulting practice in sport medicine at the University of
Calgary Sport Medicine Centre with a focus on sport-related concussion. He has
received travel funding from FIFA and the Medical Commission of the IOC. From
1998 to 2012 he was the Editor-in-Chief of the Clinical Journal of Sport Medicine
and received editorial support funding from Lippincott-Wolters Kluwer. He is
co-chair of the NHL/NHLPA Joint Health and Safety Committee and is a shareholder
of PrivIT Inc.
Provenance and peer review Not commissioned; externally peer reviewed.

REFERENCES
1

been further rened to include the verication of the exact

details of each component of all the appraised items rather than
their global inclusion. However, this approach was not deemed
necessary for the purpose of this exploratory investigation.
This study provides a comparative evaluation of the characteristics and content of apps designed to be used in the recognition
and assessment of sports-related concussion. With apps being
readily available to the profession and the public alike, there is a
need to examine what these various apps have to offer in their
content and thus whether they meet the needs of the user.
Although readily available, it must be remembered that the
content of SCAT2 was originally developed for use by medically
trained personnel. Thus, it is important that they are not seen as
a do-it-yourself solution and that the information is used to
guide a player with a suspected concussion to seek medical
review.

CONCLUSION
The study provides an initial systematic review of the apps
developed to recognise and assess a sports-related concussion.
The ease of availability of these apps raises issues of how they
might be used and by whom. It is important that the sports
medicine practitioners and the public are assured that the
content of the apps represents the best-practice information and
is presented in an appropriate manner for the target audience.
As mobile technologies become an everyday part of our lives,
apps such as those reviewed here have a major role to play in
safeguarding the health of sports persons via the early recognition of a possible concussion.
Contributors OHA, HL, PM and SJS designed the study and prepared the
preliminary draft. HL, APB, AGS and SJS conducted the search and data extraction.
HL, SJS, PM and WHM developed the analysis strategy and criteria. All authors
contributed to the analysis, drafting and approved the nal version of the
manuscript.
Competing interests PM is a coinvestigator, collaborator or consultant on grants
relating to mild TBI funded by several governmental organisations. He is Co-Chair of
the Australian Centre for Research into Sports Injury and its Prevention (ACRISP),
which is one of the International Research Centres for Prevention of Injury and
Protection of Athlete Health supported by the International Olympic Committee
(IOC). He has a clinical and consulting practice in general and sports neurology. He
receives book royalties from McGraw-Hill and was employed in an editorial capacity
by the British Medical Journal Publishing Group from 2001 to 2008. He has been
reimbursed by the government, professional scientic bodies and sporting bodies for
presenting research relating to mild TBI and sports-related concussion at meetings,
scientic conferences and symposiums. He received consultancy fees in 2010 from
Axon Sports (USA) for the development of educational material (which was not
renewed) and has received support since 2001 from CogState Inc for research costs
Lee H, et al. Br J Sports Med 2015;49:499505. doi:10.1136/bjsports-2013-092930

2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

TechCrunch.com. Mobile milestone: the number of smartphones in use passed 1

billion in Q3, says strategy analytics 2012 [cited 18 May 2013]. http://techcrunch.
com/2012/10/16/mobile-milestone-the-number-of-smartphones-in-use-passed-1billion-in-q3-says-strategy-analytics/
Bloomberg.com. Smartphones in use surpass 1 billion, will double by 2015 2012
[cited 18 May 2013]. http://www.bloomberg.com/news/2012-10-17/smartphonesin-use-surpass-1-billion-will-double-by-2015.html
Apple Inc. App Store 2013 [cited 14 May 2013]. https://www.apple.com/uk/iphone/
from-the-app-store/
Wired.com. Report: over 1 million apps have been submitted to iOS app store 2012
[cited 14 May 2013]. http://www.wired.com/gadgetlab/2012/11/report-1-millionapps/
iMedicalApps. iMedicalAppsthe leading physician publication on mobile medicine
2013 [cited 18 May 2013]. http://www.imedicalapps.com/
Behrman D. A health expert at your ngertips-the latest medical apps 2012. [cited
14 May 2013]. http://www.telegraph.co.uk/health/9621612/A-health-expert-atyour-ngertips-the-latest-medical-apps.html
Pew Internet & American Life Project. Half of adult cell phone owners have apps on
their phones 2011. [cited 14 May 2013]. http://pewinternet.org/Reports/2011/
Apps-update.aspx
Abroms LC, Padmanabhan N, Thaweethai L, et al. iPhone apps for smoking
cessation: a content analysis. Am J Prev Med 2011;40:27985.
Huckvale K, Car M, Morrison C, et al. Apps for asthma self-management: a
systematic assessment of content and tools. BMC Med 2012;10:144.
U.S. Food and Drug Administration. Mobile Medical Applications 2011. [cited 26
Jan 2014]. http://www.fda.gov/downloads/MedicalDevices/
DeviceRegulationandGuidance/GuidanceDocuments/UCM263366.pdf
Franko OI. Smartphone apps for orthopaedic surgeons. Clin Orthop Relat Res
2011;469:20428.
Rosser BA, Eccleston C. Smartphone applications for pain management. J Telemed
Telecare 2011;17:30812.
Breton ER, Fuemmeler B, Abroms LC. Weight lossthere is an app for that! But
does it adhere to evidence-informed practices? Transl Behav Med 2011;1:5239.
van Mechelen DM, van Mechelen W, Verhagen EALM. Sports injury prevention in your
pocket?! Prevention apps assessed against the available scientic evidence: a review. Br J
Sports Med. Published Online First: 10 May 2013. doi:10.1136/bjsports-2012-092136
Cohen AB, Nahed BV, Sheth KN. Mobile medical applications in neurology. Neurol
Clin Pract 2013;3:5260.
Curaudeau GA, Sharma N, Rovin RA. Development of an iPhone application for
sideline concussion testing. Neurosurg Focus 2011;31:E4.
Walkinshaw E. iPhone app an aid in diagnosing concussions. CMAJ 2011;183:E10478.
McCrory P, Meeuwisse WH, Aubry M, et al. Consensus statement on concussion in
sport: the 4th International Conference on Concussion in Sport held in Zurich,
November 2012. Br J Sports Med 2013;47:2508.
Harmon KG, Drezner JA, Gammons M, et al. American Medical Society for Sports
Medicine position statement: concussion in sport. Br J Sports Med 2013;47:1526.
Hockey Canada. Hockey Canada launches free concussion awareness apps for
smartphones and tablets 2012. [cited 19 May 2013]. http://www.hockeycanada.ca/
en-ca/news/2012-nr-130-en
Finch CF, McCrory P, Ewing MT, et al. Concussion guidelines need to move from
only expert content to also include implementation and dissemination strategies. Br
J Sports Med 2013;47:1214.
McCrory P, Meeuwisse W, Johnston K, et al. Consensus statement on concussion in
sport: the 3rd International Conference on Concussion in Sport held in Zurich,
November 2008. Br J Sports Med 2009;43(Suppl 1):i7684.
SCAT2. Br J Sports Med 2009;43(Suppl 1):i858.
Pocket SCAT2. Br J Sports Med 2009;43(Suppl 1):i8990.

7 of 8

Downloaded from http://bjsm.bmj.com/ on November 8, 2015 - Published by group.bmj.com

Review
25
26
27
28
29
30
31
32

Alla S, Sullivan SJ, McCrory P, et al. Spreading the word on sports concussion:
citation analysis of summary and agreement, position and consensus statements on
sports concussion. Br J Sports Med 2011;45:1325.
Ahmed OH, Sullivan SJ, Schneiders AG, et al. Concussion information online:
evaluation of information quality, content and readability of concussion-related
websites. Br J Sports Med 2012;46:67583.
King-Devick Test LLC. King-Devick Concussion Screening Test-iTunes App Store
2013. [cited 14 May 2013]. https://itunes.apple.com/us/app/king-devickconcussion-screening/id424793542?mt=8
Aubry M, Cantu R, Dvorak J, et al. Summary and agreement statement of the rst
International Conference on Concussion in Sport, Vienna 2001. Br J Sports Med
2002;36:610.
McCrory P, Johnston K, Meeuwisse W, et al. Summary and agreement statement of
the 2nd International Conference on Concussion in Sport, Prague 2004. Br J Sports
Med 2005;39:196204.
SCAT3. Br J Sports Med 2013;47:25962.
ChildSCAT3. Br J Sports Med 2013;47:2636.
Pocket CRT. Br J Sports Med 2013;47:267.

8 of 8

33
34
35
36

37
38
39

Patricios J, Collins R, Braneld A, et al. The sports concussion note: should SCAT
become SCOAT? Br J Sports Med 2012;46:198201.
Ahmed OH, Pulman A. Concussion information on the move: the role of
mobile technology in concussion management. J Community Inform 2013;9.
TheConcussionToolbox.com. The Concussion Toolbox App 2013 [cited 19 May
2013]. http://theconcussiontoolbox.com/concussion-toolbox-app/
NFLevolution.com. X2 Biosystems app to help NFL doctors with concussions
next season 2013 [cited 19 May 2013]. http://www.nevolution.com/article/
X2-Biosystems-app-to-help-NFL-doctors-with-concussions-next-season?
ref=6015
Google play. SACTool BetaAndroid Apps on Google play 2013 [cited 14 May
2013]. https://play.google.com/store/apps/details?id=net.nutecht.sactool&hl=en
McCrea M, Kelly JP, Randolph C, et al. Standardized Assessment of Concussion
(SAC): on-site mental status evaluation of the athlete. J Head Trauma Rehabil
1998;13:2735.
U.S. Food and Drug Administration. Health Information Technologies: Administration
Perspectives on Innovation and Regulation 2013 [cited 14 May 2013]. http://www.
fda.gov/NewsEvents/Testimony/ucm344395.htm

Lee H, et al. Br J Sports Med 2015;49:499505. doi:10.1136/bjsports-2013-092930

Downloaded from http://bjsm.bmj.com/ on November 8, 2015 - Published by group.bmj.com

Smartphone and tablet apps for concussion

road warriors (team clinicians): a systematic
review for practical users
Hopin Lee, S John Sullivan, Anthony G Schneiders, Osman Hassan
Ahmed, Arun Prasad Balasundaram, David Williams, Willem H
Meeuwisse and Paul McCrory
Br J Sports Med 2015 49: 499-505 originally published online March 25,
2014

doi: 10.1136/bjsports-2013-092930
Updated information and services can be found at:
http://bjsm.bmj.com/content/49/8/499

These include:

References
Email alerting
service

Topic
Collections

This article cites 23 articles, 17 of which you can access for free at:
http://bjsm.bmj.com/content/49/8/499#BIBL
Receive free email alerts when new articles cite this article. Sign up in the
box at the top right corner of the online article.

Articles on similar topics can be found in the following collections

BJSM Reviews with MCQs (123)

Notes

To request permissions go to:

http://group.bmj.com/group/rights-licensing/permissions
To order reprints go to:
http://journals.bmj.com/cgi/reprintform
To subscribe to BMJ go to:
http://group.bmj.com/subscribe/