Welcome to

The Next Level

BlackBerry Technical Seminar 2007

Architecting a Highly
Available BlackBerry
Environment

Jeff Holleran
Research In Motion
 Welcome to
The Next Level
Agenda BlackBerry Technical Seminar 2007

• Preparing for recovery

• Architecting for higher availability
• Recovering from a failure
• Future options for increased availability

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Preparing Your Environment The Next Level
Lotus Domino BlackBerry Technical Seminar 2007

What should I backup, how and why?

• SQL Database – configuration information, wireless
backups
– MSDE – BlackBerryDBBackup.exe
– SQL – third party backup software
• Users Specific Data
– State Databases, BBProfiles.nsf – third party tools
• BlackBerry® Enterprise Server
– Any custom changes in registry
– SRP ID and key
– License keys (these are also stored in the database)

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Preparing Your Environment The Next Level
Microsoft Exchange BlackBerry Technical Seminar 2007

What should I backup, how and why?

• SQL Database – configuration information, wireless
backups
– MSDE – BlackBerryDBBackup.exe
– SQL – third party backup software
• Users Specific Data
– Exchange hidden folders – BlackBerryBackup.exe, third party
backup tools
• BlackBerry Enterprise Server
– Any custom changes in registry
– SRP ID and key
– License keys (these are also stored in the database)

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Preparing Your Environment The Next Level
Novell GroupWise BlackBerry Technical Seminar 2007

What should I backup, how and why?

• SQL Database – configuration information, wireless
backups
– MSDE – BlackBerryDBBackup.exe
– SQL – third party backup software
• Server Specific Data
– Generate_GW_Trusted_App.ini – Trusted Application Key
• BlackBerry Enterprise Server
– Any custom changes in registry
– SRP ID and key
– License keys (these are also stored in the database)

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Preparing Your Environment The Next Level
BlackBerryDBBackup.exe BlackBerry Technical Seminar 2007

• Backing up the MSDE database

BlackBerryDBBackup.exe -d <db name> -f <path_for_backup_file> -U
<sqlusername> -P <sqlpassword>
<db name> - Configuration Database name (the typical name is BESMgmt)
<path_for_backup_file> - path to store backup file
<sqlusername> - SQL username for database access (SA)
<sqlpassword> - password for SQL account

• Other options:
– -E to use NT Authentication (will not need –U or –P)
– -S to set the server (if a remote server)

*This process can be scripted to run automatically

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Preparing Your Environment The Next Level
Documentation BlackBerry Technical Seminar 2007

• SRP IDs and Keys

• License Keys
• BlackBerry Enterprise Server Admin account details
– Account Name, Password, Permissions Configured
• Special Firewall configurations (proxy address, etc) in
place that are specific to your environment
– Note how the BlackBerry Enterprise Server is configured for the
connection as well
• Disaster Recovery process that has been tested
for your environment
• BlackBerry Enterprise Server name

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
The Next Level
Agenda BlackBerry Technical Seminar 2007

• Preparing your environment

• Architecting for higher availability
• Recovering from a failure
• Future options for increased availability

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
High Availability Strategy The Next Level
Current Options BlackBerry Technical Seminar 2007

• Active – Passive Architecture

• Manual intervention required to failover
• Planned additional capacity
– Users moved at time of failure
– New service books will be sent
• n+1 – single spare server supporting multiple BlackBerry
Enterprise Servers
– Requires one or more spare servers
– Own SRP ID, sourceless user move, new service books
– Same SRP ID, configured at time of failure, no new service books
• 2n – preconfigured servers for knife edge cutover
– Requires 2 physical servers for each logical server
– Uses same SRP ID
– No need for new service books

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Strategy for Availability The Next Level
Additional Capacity BlackBerry Technical Seminar 2007

• User density limited by strategy of BES01

Limit to 50% user load
failing over users
• Users from BES01 could be
moved upon failure to BES02
– BES02 would have to have
sufficient capacity to
accommodate the extra load BlackBerry
Srp.xx.
Enterprise
• Requires manual move and Server
BlackBerry.net

wireless update of the service Configuration

Database
books by the IT Policy service Messaging
Servers
• All users must have BlackBerry®
Device Software v4.0 – otherwise
users must re-synchronize and
generate updated service books BES02
at desktop Limit to 50% user load

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Strategy for Availability The Next Level
Additional Capacity - Platform Specific BlackBerry Technical Seminar 2007

• IBM® Lotus® Domino®

– State Databases must be replicated between the servers
– Must share configuration database
• Microsoft® Exchange
– Must share configuration database
• Novell® GroupWise®
– Must share configuration database
– Both BlackBerry Enterprise Servers must be
running v4.0.2 or higher

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Strategy for Availability The Next Level
N+1 Servers Available - capacity BlackBerry Technical Seminar 2007

• Additional online server (own

SRP) available for users to be BES01 – BES0n
Full user load
moved to

• Users from any BlackBerry

Enterprise Server could be
moved upon failure to BES0x

• Requires manual move and

update of the service books by BlackBerry
Srp.xx.
the IT Policy service Enterprise Server
Configuration
BlackBerry.net
Database

• All users must have BlackBerry Messaging

Device Software v4.0 – Servers
otherwise users must re-
synchronize and generate
updated service books
at desktop

BES0x
No active users

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Strategy for Availability The Next Level
N+1 Servers Available – knife edge BlackBerry Technical Seminar 2007

• Additional online server without

BlackBerry Enterprise Server BES01 – BES0n
Full user load
software installed and
configured

• BlackBerry Enterprise Server

Software installed at time of
failure

• Configure with same SRP ID BlackBerry

Srp.xx.
and BlackBerry Enterprise Enterprise Server
BlackBerry.net
Server name as failed server Configuration
Database

Messaging
• Connect to same database Servers

• Same as knife-edge cutover, no

new service books or user
actions required
BES0x
No BlackBerry Enterprise Server installed

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Strategy for Availability The Next Level
N+1 Servers Available - Platform Specific BlackBerry Technical Seminar 2007

• IBM Lotus Domino

– Not a recommended solution due to state database
replication requirements
– All state databases must be replicated to the single spare server
– Must share configuration database
• Microsoft Exchange
– Must share configuration database
– Could use spare server for a knife-edge of the server that is out of
service
• Novell GroupWise
– Must share configuration database
– Both BlackBerry Enterprise Servers must be running v4.0.2
or higher

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Strategy for Availability The Next Level
2n Servers Available BlackBerry Technical Seminar 2007

• One-to-one hardware (fastest) BES01

(Primary)
• Uses the “knife-edge cutover” All BlackBerry Enterprise Server services running
process
• Standby server to be configured
same as production server
– BlackBerry Enterprise Server name
– SRP ID and Key BlackBerry Srp.xx.
Enterprise BlackBerry.net
– BlackBerry Enterprise Server Server
Configuration
Admin account Database
Messaging
– Same database Servers

• Does not require BlackBerry Device

Software v4.0+ on all devices (no
synchronization)
BES02
• Does not generate new (Standby)
All BlackBerry Enterprise Server services disabled
service books

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Strategy for Availability The Next Level
2n Servers Available - Platform Specific BlackBerry Technical Seminar 2007

• IBM Lotus Domino

– Configure for replication of both state databases and profiles database
– Must share configuration database
– DomBESCutover.exe available from support
• Updates BlackBerry Configuration Database with new Domino server info
• Microsoft Exchange
– Must share configuration database
• Novell GroupWise
– Must share configuration database
– Have copy of Generate_GW_Trusted_App.ini on spare server
– Both BlackBerry Enterprise Servers must be running v4.0.2 or higher

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Strategy for Availability The Next Level
Database BlackBerry Technical Seminar 2007

• One-way transactional replication (requires 2 servers)

• The following BES components will not be able to function

while Database is down:
– BESG Messaging Agent
– BESA Activation Agent
– Policy Server
– Sync Server
– MDSS
– BlackBerry Admin Server

• There are additional SQL HA options that are possible, but

not tested/certified by RIM

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Database The Next Level
One-Way Transactional Replication BlackBerry Technical Seminar 2007

BlackBerry Enterprise Server

• Manual failover Management Consoles

• Backup/DR strategy
• Either server can accept
a connection Primary
Database
Secondary
Database
One-Way
Replication

BlackBerry Enterprise
Server
* Requires BlackBerry Enterprise Server v4.1.2 or higher
© 2007 Research In Motion Limited. All rights reserved.
 Welcome to
Database The Next Level
One-Way Transactional Replication BlackBerry Technical Seminar 2007

BlackBerry Enterprise Server

• Manual failover Management Consoles

• Backup/DR strategy
• Either server can accept
a connection Primary
Database
Secondary
Database

• BlackBerry Enterprise
Server hosts and
management consoles
must be connected to
secondary database in
case of failure

BlackBerry Enterprise
Server

* Requires BlackBerry Enterprise Server v4.1.2 or higher

© 2007 Research In Motion Limited. All rights reserved.
 Welcome to
Database The Next Level
One-Way Transactional Replication BlackBerry Technical Seminar 2007

BlackBerry Enterprise Server

• Recommended recovery
Management Consoles

is to reverse server roles

• Reconfigure replication
in other direction Primary
Database
Secondary
Database
One-Way
Replication

BlackBerry Enterprise
Server
* Requires BlackBerry Enterprise Server v4.1.2 or higher

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
The Next Level
Agenda BlackBerry Technical Seminar 2007

• Preparing your environment

• Architecting for higher availability
• Recovering from a failure
• Future options for increased availability

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Recovering from a failure The Next Level
Rebuilding your BlackBerry Enterprise Server BlackBerry Technical Seminar 2007

• Install BlackBerry Enterprise Server software

– Same SRP ID and Key
– Same BlackBerry Enterprise Server name
• Point to database
– Easy if external database
– Create/restore database if local
• Conceptually the same as a “knife-edge cutover”,
except to the same hardware

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Recovering from a failure The Next Level
Configuration database failure BlackBerry Technical Seminar 2007

• BlackBerry Enterprise Server is considered to be in failure mode

without access to the configuration database
– You cannot start a BlackBerry Enterprise Server without the
database available
• Unable to manage BlackBerry Enterprise Server and users while
database is unavailable
• PIM items will sync when database is online again
• If you restore an older copy of the database
– Gap will exist in wireless backup data
– PIM items will re-sync based on database entries of last sync
– Re-create any administrative changes between backup and failure
• Starting with a new database
– Add server configuration
– Add users
– Will not have wireless backup data

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Site Disaster Recovery The Next Level
Starting Over with backups BlackBerry Technical Seminar 2007

• Complete Messaging Server DR

– Minimum of mailbox availability
• Install BlackBerry Enterprise Server
– Use same config (SRP, BlackBerry Enterprise Server name)
– Connect to restored database
– Restore BlackBerry specific data
• Domino – state databases, profiles database
• Exchange – hidden folder data to mailboxes
• GroupWise – Generate_GW_Trusted_App.ini

*Users will not need updated service books

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Site Disaster Recovery The Next Level
Starting Over – no backups BlackBerry Technical Seminar 2007

• Complete Messaging System DR

– Minimum of mailbox availability

• Install BlackBerry Enterprise Server

• Enterprise Activate Users

– PIM data from BlackBerry will sync to mailbox
– Users will only be able to forward data that exists on the
BlackBerry
• No attachments
• No ‘more’ data – existing messages orphaned on device

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
The Next Level
Agenda BlackBerry Technical Seminar 2007

• Preparing your environment

• Architecting for higher availability
• Recovering from a failure
• Future options for increased availability

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Future options for Higher Availability The Next Level
Overview
BlackBerry Technical Seminar 2007

• Multi-faceted strategy:
– Fault Tolerance via Controller
– Automated and Manual Failover modes between Primary and
Standby servers
• Administrator configurable failover thresholds
• Additional features
– SRP Lockout Avoidance
– Service Prioritization
– Prevention of failover looping
• No single point of failure when running High Availability
configuration

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Future options for Higher Availability The Next Level
Server Failover
BlackBerry Technical Seminar 2007

Primary BlackBerry Enterprise Server

• Primary BlackBerry Enterprise host running all services
Server host running all services
with live connections to BlackBerry
Network Operations Center (NOC),
Database and Mail Servers
• Standby BlackBerry Enterprise
Server running all services with live
BlackBerry
connection to Database and warm Enterprise Server
Srp.xx.
BlackBerry.net
connection to Messaging Servers Configuration
Database

Messaging
Servers

Standby BlackBerry Enterprise Server

host running all services

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Future options for Higher Availability The Next Level
Server Failover
BlackBerry Technical Seminar 2007

• Primary BlackBerry Enterprise

Server host running all services
with live connections to BlackBerry
Network Operations Center (NOC),
Database and Mail Servers
• Standby BlackBerry Enterprise
Server running all services with live
BlackBerry
connection to Database and warm Enterprise Server
Srp.xx.
BlackBerry.net
connection to Messaging Servers Configuration
Database
• Primary BlackBerry Enterprise
Messaging
Server host running all services Servers
fails
• Dispatcher on standby BlackBerry
Enterprise Server host identifies
failure and begins failover process
Standby BlackBerry Enterprise Server
host running all services

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Future options for Higher Availability The Next Level
Server Failover
BlackBerry Technical Seminar 2007

• Primary BlackBerry Enterprise

Server host running all services
with live connections to BlackBerry
Network Operations Center (NOC),
Database and Mail Servers
• Standby BlackBerry Enterprise
Server running all services with live
BlackBerry
connection to Database and warm Enterprise Server
Srp.xx.
BlackBerry.net
connection to Messaging Servers Configuration
Database
• Primary BlackBerry Enterprise
Messaging
Server host running all services Servers
fails
• Dispatcher on standby BlackBerry
Tentative connection
Enterprise Server host identifies
failure and begins failover process
• Standby BlackBerry Enterprise Standby BlackBerry Enterprise Server
Server host establishes tentative host running all services

connection to BlackBerry NOC

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Future options for Higher Availability The Next Level
Server Failover
BlackBerry Technical Seminar 2007

• Newly primary BlackBerry

Enterprise Server host validates it
is the only SRP connection and
create active connection
• Activates connections to mail
server and completes
startup tasks
BlackBerry Srp.xx.
Enterprise Server BlackBerry.net
Configuration
Database

Messaging
Servers

Active Connection

Primary BlackBerry Enterprise Server

host running all services

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Future options for Higher Availability The Next Level
Server Failover
BlackBerry Technical Seminar 2007

Standby BlackBerry Enterprise Server

• Newly primary BlackBerry Host running all services
Enterprise Server host validates it
is the only SRP connection and
create active connection
• Activates connections to mail
server and completes
startup tasks
BlackBerry Srp.xx.
• Once back online, original primary Enterprise Server BlackBerry.net
Configuration
BlackBerry Enterprise Server host Database
becomes
Messaging
standby host Servers
• Connects to Database Active Connection
• Establishes warm connections to
Messaging Servers

Primary BlackBerry Enterprise Server

Host running all services

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Future options for Higher Availability The Next Level
Database – SQL 2005 Mirroring
BlackBerry Technical Seminar 2007

BlackBerry Enterprise Server

• Automatic failover Management Consoles

• High availability strategy

• Witness server ensures that

only the principal server will Principal Replication Mirror
accept database connections

• BlackBerry Enterprise Server Witness

hosts and management
consoles configured with both
principal and mirror
server info

BlackBerry Enterprise
Server Host Pool

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Future options for Higher Availability The Next Level
Database – SQL 2005 Mirroring
BlackBerry Technical Seminar 2007

BlackBerry Enterprise Server

• Upon failure of database Management Consoles

principal, mirror will assume

principle role

• BlackBerry Enterprise Server

components will connect to new
Principal

principal server

Witness

BlackBerry Enterprise
Server Host Pool

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
Future options for Higher Availability The Next Level
Database – SQL 2005 Mirroring
BlackBerry Technical Seminar 2007

BlackBerry Enterprise Server

• Upon failure of database Management Consoles

principal, mirror will assume

principle role

• BlackBerry Enterprise
Server components will Mirror Replication
connect to new Principal

principal server

Witness
• When the failed server
returns, it will assume mirror
role and replication will begin
and sync the two databases

BlackBerry Enterprise
Server Host Pool

© 2007 Research In Motion Limited. All rights reserved.

 Welcome to
The Next Level
BlackBerry Technical Seminar 2007

Thank you!

Be sure to catch the Open Forums, coming to you live from

the Conference Hall!

This is your chance to ask technical questions and get live

answers from BlackBerry experts.

© 2007 Research In Motion Limited. All rights reserved.