Professional Documents
Culture Documents
TABLE OF CONTENT
WHY YOU NEED AN IN-HOUSE TEST LAB
REQUIRED INFRASTRUCTURE 14
Introduction
Building a large-scale in-house test lab from scratch can be a big challenge. It takes time
and efforts to figure out the right equipment and tools that are needed for a successful
deployment. In addition, it can be trickier if something is not properly configured.
At Testdroid, we understand how daunting and time-consuming it can be. And we have
experienced it ourselves when we were building Testdroid Cloud. To help customers
with different needs, we have been building various in-house test labs ranging from 20
devices to several hundred devices with Testdroid Enterprise.
Here, weve gathered and put together a guide that discusses about why you need to
build an in-house test lab, what and how many devices you need, what infrastructure
is required, how to create enterprise grade reliability, and how to operate your test lab
without failure. In fact, the tips we cover in this ebook are mainly influenced by our own
setup at Testdroid Cloud.
With the help of this ebook, you will be able to create the best offline test lab with the
right setup that maximizes your return on investment, instead of spending tons of time
correcting the wrong configuration back and forth.
Chapter 1
The Reasons
for an In-House
Test Lab
Share this eBook!
SPECIAL SETUPS
inuous
t
n
o
c
g
ualin
i
q
t
e
p
h
o
t
d
g
a
By
mprovin
i
y
l
n
o
your
t
o
g
n
n
i
e
k
r
a
a
t also m
ion you
u
t
b
a
r
,
s
g
s
e
t
e
c
n
i
pro
er, more
t
d
s
n
a
f
a
h
e
c
d
r co
ffort mu
e
g
n
ty of you
i
t
s
asure.
and te
e
t
n
m
e
o
t
m
p
r
o
e
devel
nd easi
a
t
n
e
t
s
i
cons
Chapter 2
Devices for
In-House
Test Lab
Share this eBook!
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
14.
15.
LG G2 (1.1%)
16.
17.
18.
19.
20.
21.
LG Nexus 5 (<1%)
22.
11.
23.
12.
24.
LG Nexus 4 (<1%)
13.
25.
HTC Thunderbolt
Respectively if you want to cover the most used iOS devices, the list looks like this:
1.
2.
3.
4.
5.
iPhone 5S
iPhone 5C
iPhone 5
iPhone 4S
iPhone 4
8.
iPad Air
9.
10.
11.
12.
6.
iPad Mini
13.
7.
14.
Chapter 3
Required
Infrastructure
Share this eBook!
HARD DISKS
You need to have enough disk space for all test artifacts (logs,
screenshots, videos, performance data, memory dumps, network dumps etc.) and make sure that there is a mechanism that
either deletes the data created by previous test runs or transfers it to somewhere where there is enough storage space.
Running out of disk space is one of the most common failure
points in large-scale test automation. You also may want to
consider investing in SSD disks to make sure that writing the
test results to storage does not become a bottleneck
under heavy load.
RAM
You need enough RAM to run several virtual
machines in one physical device control server
as that is the best way to ensure that each test
run starts from identical starting point. Also running virtual machines is required to run several iOS
Instruments sessions on one Mac.
ENERGY EFFICIENCY
If you are building a large device lab, you will
potentially running tens of device control servers as well. So keep in mind that the more power your hardware draws, the more heat those
will generate and more cooling you will need.
USB Hubs
As all devices will be connected to the device control servers over USB, you need to pay
attention to the quality and capabilities of your USB hardware. This is one of the most
error-prone parts of the system. While there are no fixed hard limits on how many
devices you can plug into one USB, our experiments show that you should not connect over 16 devices per USB hub. If you connect more than 16, the devices may start
disconnecting randomly especially when a lot of data is being transferred either at the
beginning or end of each test.
The other important reason why good quality USB hubs are essential is that many devices (especially ones with a large display) have
problems with charging if the charging current is not high enough. The USB specification
mandates that the output current is limited to
500mA when the USB port is on data mode but
the best USB hubs on the market allow you to
programmatically switch between data and
charging modes allowing you to implement automatic charging cycles for devices that cannot
be charged in USB data mode.
QUICKTIPS!
To make sure that there are as
few USB related problems as
possible we prefer professional
USB hub hardware by company
called Cambrionix. Those are sold
under several names and at least
ones sold under Datamation brand
are great. Just be careful when
selecting those as some models are
just for charging and
some models are
just for iOS device.
WIFI Infrastructure
WIFI infrastructure is another very important area that is often overlooked when creating large-scale test lab. You can get to about 10 maybe even 15 devices with any WIFI infrastructure without any problems. But as the number of devices in your WIFI network
adds up, so do the problems. Quite often all devices are able to connect to the WIFI
without problems but the problems appear when all of those start transferring data
at the same time. Most WIFI access points are not designed for this kind of throughput
and you will start observing different kinds of timeouts on server responses etc. Network throughput on typical WIFI router is around 100-150 Mbit/s. But when you have
several dozens of wireless devices trying to download at tens of Mbits/s, you can easily
see where the bottleneck is.
QUICKTIPS!
There are several brands on the market
selling enterprise grade WIFI access
points. At Testdroid, we have tried
at least Aerohive, Cirrus and
Cisco Meraki. Our current
favorite is Cisco Meraki MR24 which has
three separate radios each capable
of working on either
2.4Ghz or 5Ghz frequency band and each
of them can transfer up to
300 Mbit/s giving whopping
900 Mbit/s from your devices to
the back end servers. Another nice thing
about Cisco Meraki WIFI infrastructure is
their cloud based management UI that has
features like throttling the speed of individual WIFI connection which can be useful for instance when simulating different
network conditions.
Continuous
integration
Source Management
USER INTERFACE
API
application
server
API
UI
Test
execution
DEVICES
Test
management
reporting
admin
monitoring
users &
roles
DATABASE
Chapter 4
Enterprise Grade
Reliability
Share this eBook!
Create Intelligent
Retry Mechanisms
Some of the test execution related failures are
not actually related to the application under test
but are caused by some test infrastructure related
reason. The good news is that this type of failures are
easy to identify and tests that have failed because of
such reasons can be automatically retried as many times
as needed to have a test run that is free from infrastructure
related failures. The most common reasons for test infrastructure
related failures are: The connection between the device control server
and device fails, the connection between the device and your back end server fails,
either the device or device control server runs out of storage space, device runs out
of power in the middle of test session, and etc. All of these can be identified during
the test session and the tests can be automatically retried so that the only remaining
failures are real failures related to the application under test or the test scripts.
Share this eBook!
DISK SPACES
Automated tests create a lot of data in the form of logs, screenshots, videos, memory
dumps, and network dumps. When you are running tens or even hundreds of devices
sometimes 24/7, you may run out of disk space just out of the blue.
POWER
Disruptions on power supply (even very short ones) can cause very strange problems
when the system has lots of moving parts and all of them are dependent on steady
power supply. Use UPS backed power whenever you can and also make sure that your
servers close down and come up in a controlled way when longer power disruption
happens.
You can use any generic monitoring software for this. For Testdroid Cloud we are currently using AWS monitoring, Loggly and New Relic to handle our monitoring.
Chapter 5
Operating
In-House
Test Lab
Share this eBook!
g
o
L
l
a
n
o
i
t
a
r
e
p
O
t up
Once youve se
cftware infrastru
so
d
n
a
re
a
w
rd
ha
lab is the
g your device
n
ti
ra
e
p
o
,
re
tu
ke care of. The
ta
to
d
e
e
n
u
o
opnext thing y
s is to save all
n
o
ti
ra
e
p
o
f
o
docfirst aspect
teams should
n
o
ti
ra
e
p
O
s.
to
eration log
s that happen
ce
ie
p
d
n
a
s
it
b
runs.
ument all
vices and test
e
d
t,
n
e
m
n
o
ir
cs,
the env
ework with do
m
o
h
t
a
re
g
g
in
By do
and
le to maintain
b
a
e
b
l
il
w
s
m
tea
hen
nvironment w
e
e
th
e
d
ra
g
p
u
.
needed easily
Keep You
Devices C r
oo
Everyt
power
hin
n
o
waday g is about
its con
s. T
ant th sumption is he power a
ing to
contrib the most im nd
of the
po
u
tempe system. Eve te the tempe rtns
rat
ra
malfun ure can cau maller chan ture
ge
se
ct
estima ion (This sho that lots of s in
device
te
u
s
up the d!) and go o ld never be
u
ffl
device
n
i
d
n
e
e. Whe
poo
enoug
n you rh spac l, make sure
set
e for t
y
o
u
l
e
ho
a
breat se devices t ve
h.
o
e
h
t
r
o
t
i
n
o
M
s
e
c
i
v
e
D
f
o
g
Chargin
ke
need to ma
y
ll
a
r
tu
a
n
You
ugh power
o
n
e
t
e
g
s
evice
4/7
vices are 2
sure your d
e
d
e
s
o
th
evic. Once
gly, those d
to perform
in
is
r
p
r
u
s
, maybe
if they get
d
e
r
o
it
n
o
plugged in
tly m
are compo
be constan
w
d
r
a
h
s
a
y
es need to
atter
sily
wer. The b
ing and ea
o
g
r
p
a
h
h
g
c
u
ll
o
fu
n
e
ow
g
t always all
t in chargin
p
e
k
y
tl
n
ta
nent doesn
ns
es!
hen it is co
those devic
r
o
it
n
o
gives up w
m
e sure you
mode. Mak
Use Staging
Environment
USB Tips
Standardize Your
Test Lab Hardwar
e
It is a
to use the lways a good idea
setting up same hardware w
hen
o
house lab r configuring a lar
ger in.
certain de This means that if
vice setup
you use
for serve
your ope
rs, it mak
rational t
es
eams life
very sam
easier. Th
e hardwa
e
re is cons
across th
tantly use
e environ
d
ment. Th
naturally
o
se device
differ, bu
s
t then ag
use the s
ain you c
ame way
an
trol and m s to connect, conaintain th
ose.
number of
m
u
im
x
a
m
h
Find the
ected to eac
n
n
o
c
e
b
n
a
tc
experience,
r
devices tha
u
o
m
o
fr
ne
erally, and
nected to o
n
o
c
server. Gen
s
e
ic
v
e
nectumber of d
metime con
o
S
.
0
the typical n
-2
5
1
rver
bustness is
the same se
in
ts
lo
s
server for ro
B
S
U
in different
.
ing devices
tter stability
e
b
h
c
u
m
e
may giv
to
ely pays off
it
n
fi
e
d
it
o
vS
ork
USB ports w
t
a
h
w
t
u
o
find
ne
o, there is o
ls
A
t.
s
e
b
e
th
nt
ery importa
minor but v
uge setups:
h
f
o
e
s
a
c
trick in
ed USB cad
o
-c
r
lo
o
c
e
Us
t
u know righ
o
y
d
n
a
s
le
b
t devices
away in wha
t with.
they connec
Conclusion
Building a large-scale in-house test lab is not a one- or two-day work. It needs all appropriate devices, infrastructure and software as basic weapon for your on-premise testing environment. More than that, you also need to make sure that your test lab is well
developed and reliable enough to support your team running tests 24/7 with no flaws.
At Testdroid, we ourselves have been hosting an over 300-device cloud-based test lab
Testdroid Cloud for the past years. All of the tips and tricks mentioned above are exactly
what we have experienced when hosting Testdroid Cloud. When you decide and start to
build your own in-house test lab, it is of great significance to take into account all of the
aspects (hardware, software and operations) mentioned above.
The purpose of this ebook is to give you a complete guide for creating offline test lab
and help you inspect potentially weak parts in your current in-house test lab settings
if you already have one in place. In addition, with plenty of experiences on hosting a
large-scale test lab, we are also able to deliver to you such a testing solution Testdroid
Enterprise. Building your in-house test lab with Testdroid Enterprise will ease you at any
point and you can just sit back and relax.