HOL-PRT-1303

Table of Contents
Lab Overview .................................................................................................................... 2
HOL-PRT-1303 - EMC Lab - Using Puppet and the vSphere Web Client.................... 3
EMC and the vSphere Web Client...................................................................................... 5
EMC Virtual Storage Integrator (VSI) powered by ViPR Software Defined Storage .. 6
EMC Enterprise Backup and Recovery..................................................................... 9
Using Puppet to Deploy EMC Avamar and More .............................................................. 38
Setting up a Puppet Master................................................................................... 39
Facter and the Puppet Agent................................................................................. 41
Your First Puppet Class - hellovmworld.................................................................. 44
Create a Puppet Class - avamar ............................................................................ 55
Create a Puppet Class - avamar-win...................................................................... 61
Optional Puppet Classes - avamar-vm, apache2, mssql, avamar-sql, hyperic,
hyperic-win ........................................................................................................... 68
Puppet and VMware Guest Integration ........................................................................... 70
Using an External Node Classifier to Integrate Puppet and vSphere ..................... 71
VM Guest Facts...................................................................................................... 73
vCenter Hiera Lookup for Classes ......................................................................... 81
VM Image Backup Puppet Module ......................................................................... 87
Deploying PaaS with Puppet and vCO ............................................................................. 94
vCenter Orchestrator Workflows and Puppet ........................................................ 95
Review Provisioning Workflow ............................................................................... 96
Assign Workflows to VM Context Menu ............................................................... 103
Provisioning VMs and PaaS.................................................................................. 108
Review PaaS Deployment.................................................................................... 115
Monitoring PaaS and EMC Avamar with Hyperic ........................................................... 120
Configuring and Reviewing Hyperic Monitoring................................................... 121
vCOps and Conclusion ........................................................................................ 126

HOL-PRT-1303

Page 1

HOL-PRT-1303

Lab Overview

HOL-PRT-1303

Page 2

HOL-PRT-1303

HOL-PRT-1303 - EMC Lab - Using

Puppet and the vSphere Web Client
Lab Overview
Like many companies today, the management team at Rainpole has embraced
virtualized infrastructure to provide cloud based solutions. Up until now, the
administration team has been manually provisioning applications and visibility into
operations has been limited. Using this lab as a sandbox, you will help the team at
Rainpole transform their service offerings to accommodate greater scale, agility, and
more fluid operations. The CTO is asking you to leverage automation and workflow
management tools like Puppet and vCenter Orchestrator to pilot a PaaS (Platform as a
Service) offering for one of their biggest customers.
The first module includes a section on ViPR software defined storage and demonstrates
how the vCenter Web Client can be used to interface with ViPR to simplify storage
provisioning and infrastructure management. The module also uses the vCenter Web
Client to introduce you to EMC Backup and Recovery. In this section, you will backup
VM's directly from vCenter and perform both full and file level recovery without using a
separate element manager. And you experience the benefits of VMware's API's for Data
Protection (VADP) and VMware changed block tracking (CBT) for backup and restore
operations.
In the second module, you learn how to create a couple of simple Puppet modules. You
start with a simple "hellovmworld" module and then use the same concepts to automate
a simple EMC Avamar backup client installation for both Linux and Windows platforms.
There are also some optional modules for you to review and play with if you like.
The third module really gets the creative juices flowing. You'll learn about External Node
Classifiers and how Puppet can be configured to interact with information and attributes
(aka metadata) contained in vCenter. And you'll see how Puppet can be used to ensure
you never have a Virtual Machine that's without image protection older than 24-hours.
Next, you bring it all together and use vCenter Orchestrator to learn about workflows
and accomplish the lab's goal of providing Platform as a Service functionality.
And finally, there's a brief section on Hyperic monitoring goodness with some great
custom integration with the EMC Avamar backup platform.
The EMC team wants to thank you in advance for choosing this lab! Please don't
hesitate to raise your hand if you have any questions, lab feedback, or just want to talk
tech. Enjoy the lab and your experience here at VMworld 2013 - thanks again
- the EMC Lab Team

HOL-PRT-1303

Page 3

HOL-PRT-1303

NOTE: Would you like to be entered into a raffle for a chance to win an
unlocked iPhone? Following the lab please click on the Help button to get a
contest card from an EMC proctor. The proctors managing the raffle will have
Puppet Labs or EMC shirts on.

HOL-PRT-1303

Page 4

HOL-PRT-1303

EMC and the vSphere

Web Client

HOL-PRT-1303

Page 5

HOL-PRT-1303

EMC Virtual Storage Integrator (VSI)

powered by ViPR Software Defined
Storage
What is ViPR?
EMC ViPR is a lightweight software platform that abstracts existing storage arrays into a
single pool of virtual storage. EMC software adapters connect to the underlying arrays,
similar to how PCs and Macs use universal device drivers to connect to peripherals. This
approach creates an extensible plug and play storage environment that can
automatically discover and map to different arrays and understand their unique
attributes. In the first release, ViPR supports EMC VMAX, VNX, VPLEX, RecoverPoint,
Isilon, and Atmos storage, as well as third-party NetApp storage.
So whats the big difference with ViPR? It's the divide and conquer approach to the
Data Plane and the Control Plane. ViPR uniquely manages both storage infrastructure (in
the Control Plane) and the data stored within that infrastructure (in the Data Plane).
ViPR decouples the two, allowing the use of both together or enabling customers to use
only the Control Path to manage the underlying storage arrays.
By only abstracting the control plane, storage management operates at the virtual layer.
Storage administrators partition their storage pools into various virtual storage arrays
and manage them uniquely by policy just like a VMware vCenter administrator partitions
servers into many virtual machines. As a result, ViPR centralizes all storage
management tasks, yet allows applications to access file and block data directly. Block,
file and object storage run on top of ViPR as virtual services that still leverage the
unique capabilities of the underlying arrays. This means that for traditional file and
block storage platforms ViPR stays out of the data path, managing the underlying
platforms without introducing any performance overhead. In the case of ViPR Object
storage, it is implemented so that objects can be accessed as objects or as files, thus
removing a lot of the performance overhead traditionally associated with object storage.
In addition, ViPR object storage is supported on any underlying file-based platform
extending customers existing investments to support growing demand for object
storage in the enterprise.
Further differentiating ViPR is its extensibility. Any customer, service provider or vendor
can easily extend ViPR to support additional storage platforms and integrate with cloud
stacks such as VMware and OpenStack. ViPR also enables the development of new
global data services that can span arrays and support hybrid data types. The ViPR
object-on-file data service stores, accesses and manipulates objects on file-based
storage such as EMC VNX, EMC Isilon and NetApp without the need to rewrite existing
applications. Organizations can do more with existing file-based storage and increase
workflow efficiency by allowing data to be manipulated in place through different access

HOL-PRT-1303

Page 6

HOL-PRT-1303

methods. ViPR transforms existing file data into cloud data and cloud data into file data
without the need to move, copy or modify the data.

Virtual Storage Integrator (VSI) powered by ViPR

HOL-PRT-1303

Page 7

HOL-PRT-1303

VSI / ViPR Video

HOL-PRT-1303

Page 8

HOL-PRT-1303

EMC Enterprise Backup and Recovery

Overview
In this section, you are going to see how easy it is to perform backup and recovery
operations natively from the vSphere Web Client. The Backup and Recovery tasks that
you are going to be using take advantage of VMware API's for Data Protection (VADP)
and leverage changed block tracking (CBT) to provide high-performance and super
efficient data protection operations.

Login to the vSphere Web Client

To get started, from the controlcenter host click on the Internet Explorer icon. The home
page will default to the vSphere Web Client login page.
Choose the "Use Windows session authentication" check box under the Password field
and click the Login button.

HOL-PRT-1303

Page 9

HOL-PRT-1303

The EMC Backup and Recovery plugin

After logging into the vSphere Web Client, you will land at the home screen. On the left
hand side of the screen, you will see an icon labeled "EMC Backup and Recovery" - go
ahead and click it.

HOL-PRT-1303

Page 10

HOL-PRT-1303

Connect to avamar01
The plugin welcomes you to EMC Backup and Recovery. If you had multiple Avamar
servers in the environment, they would all be accessible from this page. In the case of
the lab, the only choice we've got is an Avamar Virtual Edition server called avamar01.
Go ahead and click the Connect button to get started.

HOL-PRT-1303

Page 11

HOL-PRT-1303

The getting started tab

After you connect to avamar01, you are presented with an introductory landing page.
From this page, you are able to create a backup job, restore a virtual machine, and see
the backup status for your virtual environment. Each option is really just a link to the
different tabs (Backup, Restore, and Reports) at the top of the EMC Backup and
Recovery window.
Go ahead and click on "See an Overview" to navigate to the reports tab.

The reports tab

From this view, you can see the server status, backup capacity utilization, and check for
successful and failed jobs in a clear and concise way. The Integrity Check Status is "out
of date" in this lab and that's by design - we're trying to keep the I/O levels as low as
possible and have intentionally disabled this check.
All of the VM's that are part of the vCenter environment are viewable in the Virtual
Machines section. You can use the filter options and the text box to search for specific
keywords and/or machines, and also sort by the different column headers. If you place
focus into Virtual Machines area and have a column sorted alphabetically, you can also
type a portion of a name to get you to the right place quickly. As an example, go ahead
and click on the virtual machine linux01.

HOL-PRT-1303

Page 12

HOL-PRT-1303

Once you have the Virtual Machine highlighted, additional details for that machines
protection status are available at the bottom of the window.

The configuration tab

Click on the Configuration tab in the upper area of the window. You will see server and
deduplication stats relative to the Avamar system that's currently connected.

HOL-PRT-1303

Page 13

HOL-PRT-1303

Also notice that you've got the ability to configure email alerting by clicking on the
Email button.
And you can view the backup server log by clicking on the Log button.

HOL-PRT-1303

Page 14

HOL-PRT-1303

The restore tab

Click on the restore tab. This is the page that's used to see what different restore points
are available for a given Virtual Machine, and also to initiate a full image restore if
necessary.
Expand each of the VM's represented here and you will notice that each has a single
restore point.
Also notice that you can filter by different criteria by using the Filter option.

The backup tab

Now go ahead and click on the tab labeled Backup. You will see (3) different jobs that
have already been created - Linux VM's, Puppet Server, and Windows VM's. Click on
each of the (3) jobs and look in the Backup Job Details section at the bottom of the
screen to see what Virtual Machines the given job is responsible for.

HOL-PRT-1303

Page 15

HOL-PRT-1303

Initiate a backup using CBT

Next, click the "Linux VMs" job so that it's highlighted and in focus. Then click the
"Backup Now" button and choose "Backup all sources" to get a current system backup.
You should receive a confirmation that the backup job has been submitted - click OK.

HOL-PRT-1303

Page 16

HOL-PRT-1303

Monitor the vCenter tasks

In order to view the Backup and Recovery tasks after the backup job starts, click on
"More Tasks" link in the Recent Tasks window.
Image based backups use this workflow:
1. Snapshot the VM to be backed up for a consistent point-in-time view
2. Present the snapshot to a proxy host using the VMware SCSI Hot Add functionality
3. Perform a CBT backup from the proxy host that captures only blocks that have
changed since the last backup
4. Unmount the snapshot from the proxy
5. Remove the snapshot
In very little time, you will see events corresponding to the above workflow occurring.
After the snapshot has been removed, you will see the parent task (EBR: Backup Job)
complete indicating that the entire backup job has been completed.

HOL-PRT-1303

Page 17

HOL-PRT-1303

Ever have one of those days?

It's not pleasant to think about, but it does happen. Every now and then a system gets
stuck in a state that renders it unusable. For the purposes of this lab, we're going to
cause one of those situations.
Start PuTTY (there's an icon on the desktop) and choose to connect to linux01 from the
list of available hosts.
Login using "root" for the username and "VMware1!" for the password.

Remove the /boot directory on linux01

Hopefully you never have this happen in your production environment, but a similar
catastrophe is possible for any number of different reasons. Let's go ahead and remove
the /boot directory rendering this host unbootable.
# rm -rf /boot
Everything seems fine, right? Well, go ahead and reboot it - you won't see it come back.

HOL-PRT-1303

Page 18

HOL-PRT-1303

# reboot

Open a console for linux01

Switch back into the vSphere Web Client and open a console for linux01. The easiest
way to do this is to enter "linux01" in the search bar in the upper right corner of the
window and click linux01 when it appears. After you click Virtual Machine: linux01 link,
there will be a "launch console" option available in the upper left corner of the screen.
The console will open (be patient) in another tab in the browser window. You will see
linux01 is no longer bootable. You can try typing "boot" at the grub prompt, but it's not
going to help.

HOL-PRT-1303

Page 19

HOL-PRT-1303

Power-Off linux01
Click back to your vSphere Web Client which should be the first tab in the browser (you
may need to press Ctrl-Alt to get out of the console window). Note that there is no need
to close the linux01 console tab as this will be used later in the lab.
In order to do an "in-place" restore, the VM that is going to be replaced needs to be in a
powered off state. Shutdown linux01 by right-clicking linux01 and selecting "Shut Down
Guest OS". Answer yes to powering the system down.

HOL-PRT-1303

Page 20

HOL-PRT-1303

Recover linux01
Right click on linux01 and choose "All EBR Actions" -> "Restore from Last Backup" (you
may need to wait a few seconds for the "All EBR Actions" choice to appear).

HOL-PRT-1303

Page 21

HOL-PRT-1303

Recover linux01 - Select Backup

The Restore a backup wizard is started and the most recent backup is chosen for you by
default. Expand linux01 if you want to drill deeper. Accept the default choice and click
Next.

HOL-PRT-1303

Page 22

HOL-PRT-1303

Recover linux01 - Set Restore Options

We're going to perform an in-place VM restoration, but as you can see by this dialog
there is also an out-of-place option that is set by default.
Choose the "Restore to Original Location" check box and also select the "Reconnect NIC"
option.
Click Next.

HOL-PRT-1303

Page 23

HOL-PRT-1303

Recover linux01 - Ready to Complete

Review the details here and click Finish when you are ready to begin the restore
process. A box indicating that the restore was initiated will confirm the action. Click OK.

HOL-PRT-1303

Page 24

HOL-PRT-1303

Incremental system restore using CBT

After the restore has been initiated, click on "More Tasks" to see a number of different
tasks kick off and complete in rapid succession. Because the restore is using the same
Changed Block Tracking mechanism to perform an incremental restore of the VM, the
workflow is similar to the one you saw for a backup:
1.
2.
3.
4.
5.
6.

Snapshot the VM to be restored.

Present the snapshot to a proxy host using SCSI Hot Add functionality
Perform a CBT restore writing only the blocks that differ from the last backup
Unmount the snapshot from the proxy
Merge/revert the snapshot into the original VM
Delete the snapshot

In very little time, you should see task events corresponding to the above workflow.
After the snapshot has been removed, you will see the parent task (EBR: Restore Job)
complete indicating that the entire restore has been completed.
The VM will then be powered on and connected to the network to boot.

HOL-PRT-1303

Page 25

HOL-PRT-1303

Check the console for linux01

The console for linux01 should still be open in the browser window (click the "Open
Console" link if you don't have a console window open). Click back to the console
window and depending on your timing, you will either watch the system boot, or see a
login prompt.
Thanks to EMC Backup and Recovery, VMware administrators are empowered with a
quick and easy system recovery mechanism directly from the vCenter Web Console.

File level recovery (FLR) - windows01

By now, you're probably starting to understand just how easy and convenient VM
backup and restore can be with EMC Backup and Recovery. But what if you don't want to
restore an entire VM just to get access to a few key files that may not need a full VM
restore?
As long as you've got network connectivity to the backup server and Adobe Flash
installed, file level recovery is no problem either.

HOL-PRT-1303

Page 26

HOL-PRT-1303

Before we get started, let's see what it would take to get a quick backup of windows01 after all, you can never be too careful and with CBT, backups are fast and easy.
If you're not already back in the vSphere Web Client, go ahead and navigate there (it
should be an open tab in your web browser). In the search bar (upper right corner), type
"windows01" and select Virtual Machine: windows01 when it appears.

Start a backup of windows01

Right click windows01 and select "All EBR Actions" -> " Backup Now" from the menu
(you may need to wait a few seconds for the "All EBR Actions" choice to appear).

HOL-PRT-1303

Page 27

HOL-PRT-1303

Start a backup of windows01 - Backup Now

You are presented with a dialog box where you are able to modify the backup
configuration if necessary. In this case, simply click cancel (you can click OK and
perform a backup if you so choose - an optional step).

HOL-PRT-1303

Page 28

HOL-PRT-1303

Optional - Monitor vCenter tasks if you started the backup

Please skip this step if you are not backing up windows01. Otherwise, if you want to see
the EBR tasks and backup workflow, click on "More Tasks."
Just like earlier, image backups use this workflow:
1. Snapshot the VM to be backed up for a consistent point-in-time view
2. Present the snapshot to a proxy host using the VMware SCSI Hot Add functionality
3. Perform a CBT backup from the proxy host that captures only blocks that have
changed since the last backup
4. Unmount the snapshot from the proxy
5. Remove the snapshot
In very little time, you see events corresponding to the backup workflow occurring. After
the snapshot has been removed, you will see the parent task (EBR: Backup Job)
complete indicating that the entire backup job has been completed.
Please wait for backup job to complete before moving on to the next step.

HOL-PRT-1303

Page 29

HOL-PRT-1303

Login to windows01
Open a remote desktop session to windows01. Click Start - Remote Desktop Connection
and choose windows01. Use "corp\administrator" for the username and "VMware1!"
as the password.

Browse remote backup sets - Login

From windows01, open Internet Explorer. The default homepage is already setup to
connect to the EMC Data Protection Restore Client, but the url syntax is easy to
remember for any environment:
https://avamar01:8543/flr
In this exercise, you are going to restore files from a machine that's different than the
one you use to connect to the restore client with and as such, you are required to use a
2-phase login. To enable this, click the "Advanced Login" link above the Login button.

HOL-PRT-1303

Page 30

HOL-PRT-1303

Next, enter the local backup administrator credentials into the "Local Credentials"
section:
Username: .\backup-admin
Password: VMware1!
And for the "vCenter Credentials", use any account with administrative rights:
Username: root
Password: VMware1!
Click the Login button to proceed.
* Note that the ".\backup-admin" is a local account on windows01 and you could have
used any account that has backup rights on that local machine (including domain
accounts). If we were restoring files using a backup set created from the host
windows01, no "Advanced Login" or vCenter administrative privileges are required.

HOL-PRT-1303

Page 31

HOL-PRT-1303

Browse remote backup sets - Manage Mounted Backups

Using the advanced login, you are able to see backup sets that were created from all
machines in the inventory. Expand the windows00 host and mount the available backup
set by highlighting the entry, clicking "Mount" and then "Close."

HOL-PRT-1303

Page 32

HOL-PRT-1303

Browse remote backup sets - Mounted Backups

We want to restore the "Important Data" directory, so navigate through the backup set,
into "Disk#1" and then select the directory called "Important Data" and choose the
button labeled "Restore selected files..."

HOL-PRT-1303

Page 33

HOL-PRT-1303

Restore selected files - Select Destination

Highlight the "C:" drive and click the Restore button.

Restore selected files - Initiate Restore

Choose "Yes" to initiate the file restoration. Then click OK.

HOL-PRT-1303

Page 34

HOL-PRT-1303

Restore selected files - Monitor Restores

Use the Monitor Restores button to check the file restore status. If the restore doesn't
show complete, use the Refresh button to force a status update.

HOL-PRT-1303

Page 35

HOL-PRT-1303

Check to see if files were restored

Use Windows Explorer to navigate to the Important Documents directory and verify that
the files are present on your local system disk.

HOL-PRT-1303

Page 36

HOL-PRT-1303

Logoff windows01
Click Start -> Log off to disconnect from windows01. This concludes the EMC Enterprise
Backup and Recovery module.

HOL-PRT-1303

Page 37

HOL-PRT-1303

Using Puppet to Deploy

EMC Avamar and More

HOL-PRT-1303

Page 38

HOL-PRT-1303

Setting up a Puppet Master

Connect to puppet
From the desktop, click the PuTTY icon and choose the session named puppet.
Login to puppet using the following credentials:
Username: root
Password: VMware1!

HOL-PRT-1303

Page 39

HOL-PRT-1303

Start the puppet master service

Under normal circumstances, you would need to install the package using a linux
installer (aka YAST, zypper, etc.), but to save some time we've already done that for
you. Once you've got the puppet master package installed, you need to start the puppet
master service (aka daemon) using the following command:
# service puppetmasterd start
The above command starts the puppet master service and sends the server process into
the background. But for troubleshooting purposes, you may want to have the daemon
running in the foreground so you can actively troubleshoot errors. For the purposes of
this lab we're going to recommend the background service option, but if you find you
need to review server side errors,try the following:
# service puppetmasterd stop
# puppet master --no-daemonize --verbose

HOL-PRT-1303

Page 40

HOL-PRT-1303

Facter and the Puppet Agent

What is facter?
Facter is a lightweight program written in Ruby that runs on a puppet client to gather
basic node information about the hardware and operating system. Facter is used to
retrieve information about the operating system, installed software, hardware
characteristics, and networking information.

HOL-PRT-1303

Page 41

HOL-PRT-1303

Connect to linux01
At this point, you may already have a PuTTY session open with linux01 - you can either
restart the session, or open a new session by clicking the PuTTY icon on the desktop and
selecting the linux01 configuration option.
Login using the following credentials:
Username: root
Password: VMware1!

HOL-PRT-1303

Page 42

HOL-PRT-1303

Execute the facter command

Inside the terminal window, type the command "facter" and observe the output. These
details will be used by puppet to understand the current state of the client. This
information will be checked against the current catalog and evaluated so that puppet
can provide instructions to get the client to the proper configuration.

HOL-PRT-1303

Page 43

HOL-PRT-1303

Your First Puppet Class - hellovmworld

Create the directory structure for hellovmworld
Bring the PuTTY session labeled "puppet" back into focus (or restart the session if
necessary).
In our configuration, the class files will be contained in the /etc/puppet/modules
directory. The first thing that needs to be done is to create the class path structure for
our first module that we're going to call hellovmworld. You can do this by using the
following command:
# mkdir -p /etc/puppet/modules/hellovmworld/manifests
This command creates a directory that's the class name (hellovmworld) and then a
subdirectory called manifests. You can execute the above command, but may notice
that the directory path already exists as we've taken the liberty to stage some things for
you in case you don't want to do as much typing.

Create the init.pp file for hellovmworld

Make sure you are in the /etc/puppet/modules/hellovmworld/manifests directory and use
the vi editor to create an init.pp file:
* Note that you can skip to the copy step at the bottom here if you don't want to type the lab guide formatting (or lack thereof) does make the syntax a little more difficult to
read.
# cd /etc/puppet/modules/hellovmworld/manifests
# vi init.pp
Once you are inside the vi editor, press the "i" key to insert the following text:

HOL-PRT-1303

Page 44

HOL-PRT-1303

class hellovmworld {
case $operatingsystem {
'windows': {
file { 'c:\\temp\\HelloVMworld.txt':
ensure => file,
content => "My First Puppet Class with Facters!"
}
}
default: {
file { '/tmp/HelloVMworld.txt':
ensure => file,
content => "My First Puppet Class with Facters!"
}
}
}
}
To exit vi, press the "Esc" key and then type ":wq" (without the quotes) to write the file
to disk and quit the editor.
If you aren't comfortable in vi, or don't want to compose the text feel free to copy the
template example using the following syntax:
# cp init.template init.pp

What did you just do?

Take a moment and look at the file you just created using the unix cat command:
# cat init.pp

HOL-PRT-1303

Page 45

HOL-PRT-1303

The first line "class hellovmworld {" is used to declare the class that you just built. You
can also use this section to tell this class to include other modules and/or classes that
are present in the environment.
The next line "case $operatingsystem {" checks the facter output and determines if the
host is or isn't running a Windows based operating system.
If the host is running Windows, then Puppet looks for a file in the temp directory of the
C:\ drive called HelloVMworld.txt. If the file isn't there, then it will create the file and
insert the content that you see above (My First Puppet Class with Facters!). Notice that
the backslash character is escaped with another backslash character (\\).
If the host is running anything other than Windows, then Puppet looks for a file in /tmp
called HelloVMworld.txt and creates the file with the appropriate content as necessary.
Let's go ahead and see if it works.

HOL-PRT-1303

Page 46

HOL-PRT-1303

Add hellovmworld as a default class

Next, we need to tell Puppet that all clients need to comply with the hellovmworld class.
To do this, edit the /etc/puppet/manifests/nodes_default.pp file:
# vi /etc/puppet/manifests/nodes_default.pp
Then use the "o" key to add a new line and type the following text:
include hellovmworld
Then use the "Esc" key followed by ":wq" to write the file and quit the vi editor. Your file
should now look like this.

Check linux01 for the HelloVMworld.txt file

Switch over to the white background putty window that is logged into linux01 (restart
putty and choose linux01 if necessary). Issue the following command to check for the
file:
# ls -la /tmp/HelloVMworld.txt
You should receive an error as the file isn't there yet.

HOL-PRT-1303

Page 47

HOL-PRT-1303

Manually run the puppet agent

From linux01, run the following command to start the puppet agent in test mode:
# puppet agent -t
If you got all of your syntax right, you will have seen the "Notice" event alerting you to
the fact that the HelloVMworld.txt file was created.

HOL-PRT-1303

Page 48

HOL-PRT-1303

Check the HelloVMworld.txt file contents

Let's make sure the file really exists. Just like before we started, let's use the "ls"
command to make sure the file exists:
# ls -la /tmp/HelloVMworld.txt
You can also check the file contents using the "cat" command:
# cat /tmp/HelloVMworld.txt

HOL-PRT-1303

Page 49

HOL-PRT-1303

Connect to windows01
When we created the hellovmworld class, we included a case statement to help
determine if the host checking into puppet was running a Windows operating system
and we need to verify that it's working.
Click Start -> Remote Desktop Connection and connect to windows01.
Login using corp\administrator with the VMware1! password.

HOL-PRT-1303

Page 50

HOL-PRT-1303

Check windows01 for the HelloVMworld.txt file

Using Windows Explorer (or a DOS prompt if you prefer), check the c:\temp directory for
the existence of the HelloVMworld.txt file.

HOL-PRT-1303

Page 51

HOL-PRT-1303

Manually run the puppet agent

From the desktop, click the icon labeled "Start Command Prompt with ..." A DOS window
opens, enter the following command:
run_puppet_interactive.bat
Observe the initial SSL key generation and exchange as well as the creation of the
HelloVMworld.txt file. If you don't see the "Notice" in white type, something is wrong.

HOL-PRT-1303

Page 52

HOL-PRT-1303

Check the HelloVMworld.txt file contents

Press any key to close the interactive window. Next, jump back to windows explorer and
you should now see the HelloVMworld.txt file. Double click the file to confirm the
contents.

Minimize the remote desktop connection to windows01

Click the minimize icon for the windows01 desktop session at the top center of the
screen.

HOL-PRT-1303

Page 53

HOL-PRT-1303

Congratulations
You've just completed writing your first puppet class, congratulations!

HOL-PRT-1303

Page 54

HOL-PRT-1303

Create a Puppet Class - avamar

Section overview
In this section, we're going to create a puppet module to install the Avamar backup
client and register with an Avamar server. We could have used a single module to
support both Windows and Linux clients, but decided it was best to keep it simple. As
such, there are (2) modules for each respective operating system: avamar and avamarwin.

The init.pp file (for linux clients)

The first thing we need to do is make sure you're back on the controlcenter host. If
necessary, minimize or close your remote desktop connection to windows01.
From controlcenter, open the PuTTY window that's connected to puppet (or start a new
session if necessary).
Next you need to change directories and get setup in the proper location, so let's
navigate using the following command:
cd /etc/puppet/modules/avamar/manifests
From this location, create the init.pp file and open it for editing:
vi init.pp
For those not familiar with vi, use the "i" key to place the editor into insert mode and
begin typing. If you don't feel like typing, go to the end of this step for bypass
directions.
The first line of the file will declare the class name (in this case avamar):
class avamar {
The next section of the file will be used to describe the installation package name and
the specifics about how the package should be handled if not already installed. Add this
section after the first line:
package { 'AvamarClient-7.0.100-379.x86_64':
provider => rpm,
ensure => installed,
source =>"/usr/local/installs/AvamarClient-linuxsles11-x86_64-7.0.100-379.rpm",

HOL-PRT-1303

Page 55

HOL-PRT-1303

}
-> notify {'AVAMAR client installed':}
The package that puppet is addressing is named AvamarClient-7.0.100-379.x86_64 and
we're indicating that this package is managed using the Linux RPM package
management facility. We're setting the "ensure" directive to installed to tell Puppet that
the avamar software should be present - if it's missing, puppet will manage the
installation using the rpm file that is specified in the source directive. The - notify
statement at the end of this section is called chaining and will create an event indicating
that the Avamar package was installed in the even that it was missing.
We're going to use the next section of the file to make sure that the agent service is
running. Continue entering the following text into the init.pp file:
service { 'avagent':
ensure => running,
enable => true,
subscribe => Package['AvamarClient-7.0.100-379.x86_64'],
require => Exec['register'],
}
This section describes the agent name (avagent) and tells Puppet to makes sure that
the agent is running and if it's not, start it up. And the subscribe directive tells Puppet to
wait until the activity that's subscribed to has completed (in this case, it's the software
installation that you just added moments ago). But before Puppet will try to take action
here, it needs to check in with something called register (hence the require directive).
At this point, you probably realize that register section that we're requiring is still
missing. Add this text as your final section:
exec { 'register':
command =>"/usr/local/avamar/etc/avagent.d register avamar01.rainpole.com
clients",
onlyif =>"test `/usr/local/avamar/etc/avagent.d status | grep -c avamar01` -ne
1",
provider => shell,
}
-> notify {'AVAMAR client registered':}
}
This last section handles the logic to ensure that the client we're installing the agent
onto has been registered with the Avamar backup server. In this case, we're telling
Puppet to run a "status" command and verify the agent has been registered. If no
registration is present, Puppet will ask the client run the command specified by the
command directive and create a logging event (notify directive). Otherwise, puppet will
go back to the service section and ensure that our backup agent is running.

HOL-PRT-1303

Page 56

HOL-PRT-1303

Go ahead and save the init.pp file by pressing the "Esc" key followed by a ":wq" (without
the quotes) to write the file and quit the editor.
Alternatively, you can use the following command to populate your init.pp file:
# cp init.template init.pp

Assign the module to linux01 - nodes.pp

Now that we've got a module put together that can install the Avamar agent, let's go
ahead and assign it to our Linux client linux01. Change directories to /etc/puppet/
manifests and create a nodes.pp file:
# cd /etc/puppet/manifests
# vi nodes.pp
In the vi editor, use the letter "i" to signal that you want to insert text. Next, insert the
following:
node 'linux01.corp.local' {
include avamar
}
Press the "Esc" key followed by the a ":wq" to tell vi that you want to write the file and
quit the editor. The information you added tells Puppet that linux01.corp.local needs to
have the avamar class is included in the inventory.

HOL-PRT-1303

Page 57

HOL-PRT-1303

Now let's see this thing in action.

Check to see that the Avamar package isn't installed

Toggle over to the putty window that is connected to linux01 (white background
window) or restart putty and choose linux01 and login as root/VMware1!. Issue the
following command to query the rpm package database for anything with avamar in the
name:
# rpm -qa | grep -i avamar
After a quick search, the prompt should return without any output.

HOL-PRT-1303

Page 58

HOL-PRT-1303

Manually run the puppet agent

From linux01, issue the following command to force the puppet agent to check back in
with the master:
# puppet agent -t
If the init.pp doesn't have any typo's, you should see the notify statements that were
added to indicate the Avamar client was installed and subsequently registered.

HOL-PRT-1303

Page 59

HOL-PRT-1303

Confirm the Avamar package installed and registered

Once again, query the rpm package database for anything with avamar in the name:
# rpm -qa | grep -i avamar
This time, there should be output that returns the package name
"AvamarClient-7.0.100-379.x86_64" that we asked Puppet to install.
Now let's make sure the client is registered with the Avamar server
avamar01.rainpole.com:
# /usr/local/avamar/etc/avagent.d status | grep -i avamar01
Assuming that everything worked correctly, you will see output from avagent.d
indicating that the client is activated with avamar01.rainpole.com.

Congratulations!
Now that you see how easy it is to install the Avamar agent using puppet, we can go
back and make the module more dynamic using variables instead of hard coded values.
We can also leverage NFS or HTTPS to transfer package files rather than having them
stored locally in a base operating system image.

HOL-PRT-1303

Page 60

HOL-PRT-1303

Create a Puppet Class - avamar-win

Section overview
In this section, we're going to create a puppet module to install the Avamar backup
client and register with an Avamar server in a use case that's specific to Windows
clients. Again, we could have used a single module to support both Windows and Linux
clients, but decided to keep it simple for the purposes of this lab. As such, there are (2)
independent modules to install the Avamar client: avamar (Linux based) and avamarwin (Windows based).

The init.pp file (for windows clients)

Switch back to the PuTTY window logged into puppet and change the current directory
using the following command:
# cd /etc/puppet/modules/avamar-win/manifests
Just like last time, you can read through this section and use the copy command to build
the init.pp file as an alternative to using vi.
Create the init.pp file and open it for editing:
# vi init.pp
For those not familiar with vi, use the "i" key to place the editor into insert mode and
begin typing. The first line of the file will declare the class name (in this case avamarwin):
class avamar-win {
Just like avamar module, the next section of the file will be used to describe the
installation package name and the specifics about how the package should be handled if
not already installed. Add this section after the first line:
package { 'EMC Avamar for Windows':
ensure => installed,
install_options => { 'AGREETOLICENSE' = 'yes' },
source =>"c:\\installs\\AvamarClient-windows-x86_64-7.0.100-379.msi",
} ->
The package that puppet is addressing is named EMC Avamar for Windows. We're
setting the "ensure" directive to installed to tell Puppet that the avamar software should
be present - if it's missing, puppet will manage the installation using the msi file that is
specified in the source directive. The "->" at the end of this section will ensure the first

HOL-PRT-1303

Page 61

HOL-PRT-1303

action (installing the package) is executed before the next action is called. This is called
chaining.
We're going to use the next section of the file to make sure that the agent service is
running. Continue entering the following text into the init.pp file:
service { 'avbackup':
ensure = 'running',
} ->
notify {'AVAMAR running':}
->
In the above section, we refer to the Windows service called avbackup and declare that
puppet needs to "ensure" that it's running. After that task is complete, we chain the
next event which is the AVAMAR running notification, at which point our final section can
then execute.
Add the following text to complete the init.pp file:
exec { 'register':
command => 'cmd.exe /c "c:\\program files\\avs\bin\avregister.bat"
avamar01.rainpole.com client',
unless =>"cmd.exe /c findstr.exe -i avamar c:\\program
files\\avs\\var\\avagent.cfg",
path => $::path
}
-> notify {'AVAMAR client registered':}
}
This final section called register is the logic behind performing the initial client
registration with the Avamar server. The declarative nature is pretty straightforward;
execute the "command" option "unless" Puppet finds that it's already registered (by
executing a command). And then create a notification event.
Go ahead and save the init.pp file by pressing the "Esc" key followed by a ":wq" (without
the quotes) to write the file and quit the editor.
If you didn't use vi to create your file, copy the init.template file to init.pp:
# cp init.template init.pp

HOL-PRT-1303

Page 62

HOL-PRT-1303

HOL-PRT-1303

Page 63

HOL-PRT-1303

Assign the module to windows01 - nodes.pp

Now that we've got a module put together that can install the Avamar agent for
Windows, let's go ahead and assign it to our Windows client windows01. Change
directories to /etc/puppet/manifests and edit the nodes.pp file:
# cd /etc/puppet/manifests
# vi nodes.pp
In the vi editor, use a "G" (shift-g) to signal that you want to jump to the bottom of the
file. Next, enter the letter "o" to indicate that you want to insert a new line.
Begin typing the following:
node 'windows01.corp.local' {
include avamar-win
}
Press the "Esc" key followed by the a ":wq" to tell vi that you want to write the file and
quit the editor. The information you added tells Puppet that windows01.corp.local needs
to have the avamar-win class included in the inventory.

Connect to windows01
Click the remote desktop icon for windows01. If you aren't still connected to windows01,
click Start -> Remote Desktop Connection and connect to windows01. Use corp\
administrator and VMware1! to login.

Check for the Avamar agent

The easiest way to check for the Avamar agent is to look in the toolbar at the bottom of
the screen. You can also navigate to add/remove programs to look for "EMC Avamar for
Windows" or even use the command "sc query avbackup" from a DOS window.

HOL-PRT-1303

Page 64

HOL-PRT-1303

Once you are convinced that the agent is not installed, proceed to the next step.

HOL-PRT-1303

Page 65

HOL-PRT-1303

Manually run the puppet agent

Toggle over to the "Administrator: Start Command Prompt with Puppet" DOS window, or
from the desktop, click the icon labeled "Start Command Prompt with ..." In the window,
enter the following command:
run_puppet_interactive.bat
After some time (~30 seconds), you should observe white "notice" text logging that
"EMC Avamar for Windows" was created. Wait until the routine completes (indicated by
the "Press any key to continue" message).

HOL-PRT-1303

Page 66

HOL-PRT-1303

Check for the Avamar agent

You may have seen the Avamar icon in the lower right corner of the screen after the
install completed. You can also run the same command as you did earlier "sc query
avbackup" to confirm the Avamar backup agent is now present.
The host windows01 is now able to perform both image based and in-guest based
backups.

Minimize the remote desktop connection to windows01

Click the minimize icon for the windows01 desktop session at the top center of the
screen.

HOL-PRT-1303

Page 67

HOL-PRT-1303

Optional Puppet Classes - avamar-vm,

apache2, mssql, avamar-sql, hyperic,
hyperic-win
avamar-vm
The avamar-vm module was created to demonstrate that Puppet could be used to
monitor the status of a VMware image based backup and ensure that there's always a
backup image that's less that 24-hours old. It uses a homegrown REST interface to
check backup status and initiate backup as necessary. More about this functionality will
be covered in module 3 of this lab.
This module can be found in /etc/puppet/modules/avamar-vm

apache2
The apache2 module will go out and use zypper to ensure that the necessary apache2
webserver packages are present on a linux host, start the web server, and transfer a
few files to establish an initial home page.
The module can be found in /etc/puppet/modules/apache2

mssql
The mssql module was downloaded from the Puppet Forge and MS-SQL software staged
into our base Windows image to demonstrate ease of database installation on the
windows platform.
The module can be found in /etc/puppet/modules/mssql

avamar-sql
The avamar-sql module was written to demonstrate how easy it can be to deploy the
different Avamar plug-in's to provide application consistent backups for almost any use
case.
The module can be found in /etc/puppet/modules/avamar-sql

HOL-PRT-1303

Page 68

HOL-PRT-1303

hyperic
The hyperic puppet module will install and configure the hyperic agent, auto discover
any supported monitoring components, and register the platform with Hyperic. This
agent is used in another module later in this lab.
The module can be found in /etc/puppet/modules/hyperic

hyperic-win
This module is the same as the hyperic module, except that it was written to support
installation and configuration on Windows clients. The hyperic and hyperic-win modules
could be combined with some additional work to have a common puppet module.
The module can be found in /etc/puppet/modules/hyperic-win

HOL-PRT-1303

Page 69

HOL-PRT-1303

Puppet and VMware

Guest Integration

HOL-PRT-1303

Page 70

HOL-PRT-1303

Using an External Node Classifier to

Integrate Puppet and vSphere
Introduction
So far we have demonstrated the most basic way that Puppet can leverage local files as
manifests that define which classes (or applications) get installed with which hosts.
There are plenty of other ways to work with flat files to make Puppet more dynamic in
how it applies classes to nodes (servers).
The next section will demonstrate how Puppet can be extensible through another option
called External Node Classifiers (ENC). The ENC can augment traditional node lookups
by allowing external databases, REST interfaces, or scripts as long as they return
appropriately formatted results. In the case of the lab, we will be using Hiera, an
embedded ENC for Puppet v3+, to lookup the classes appropriate for each Virtual
Machine. Notice we are mentioning Virtual Machines instead of servers or guest
operating systems.
This information may be held within an external database categorized by domain name,
but we will for this lab be storing the information alongside the VM in vCenter!

Review hiera configuration - /etc/puppet/hiera.yaml

Hiera requires that you configure what it calls the backend lookups. In this case we are
leveraging a Hiera vCenter lookup backend.
Review the configuration file at /etc/puppet/hiera.yaml as seen in the screenshot below.
This is where we specify the unique key to lookup (vminstanceuuid), as well as the
location and authentication to vCenter. This vCenter account can be Read-only. This
information will be leveraged by Hiera to retrieve the Custom Values per VM.

HOL-PRT-1303

Page 71

HOL-PRT-1303

Review hiera configuration - /etc/puppet/manifests/site.pp

In order to tell Puppet to use Hiera as a lookup, review the /etc/puppet/manifests/site.pp
file. The hiera_include statement ensures that the field "classes" are passed as a lookup
field to the Hiera vCenter backend lookup.

HOL-PRT-1303

Page 72

HOL-PRT-1303

VM Guest Facts
VM guest facts
In this lab the ENC returns the classes to be installed from based on information stored
per Virtual Machine in vCenter. However, in order to know which VM to lookup we need
to know something unique about the VM. In most cases, a VM can be addressed
uniquely by something unique or a combination of many things. Some these being but
not limited to: DNS FQDN, MAC Address, IP, or BIOS UUID.
This lab introduces a new VMware integration for Puppet which makes use of a specific
Virtual Machine guest fact that ensures uniqueness per vCenter instance. We are going
to leverage an identifier (instanceUuid) which is what vCenter tracks VMs by internally
and ensures uniqueness for Puppet! We are also going to leverage this instanceUuid for
a new Avamar-VM puppet module to manage image based backups.
Is it completely necessary to leverage the instanceUuid? No, Puppet to this point has
basically only been concerned with FQDN. But in order to future-proof and make Puppet
more consistent the instanceUuid may be a better long term approach to classifying
nodes.
The tricky part here is that the Puppet agent needs to have a fact that reports the
instanceUuid which then gets passed to the Puppet server. So how do we let a VM Guest
Operating Systems know a fact that is held at the VMware or Hypervisor level?

VM advanced settings
There are specific settings that a Virtual Machine guest can retrieve about itself from the
vCenter through VM Tools. This information can be custom if the name of the Advanced
Setting begins with "guestinfo." These advanced settings are persistently held with the
VM .vmx file.
In our case we are going to leverage a custom Advanced Setting called
"guestinfo.vminstanceuuid" which the guest can see.

Reviewing a custom fact

Let's review how to build a custom fact. In this case, we have "pluginsync" turned on
which means Puppet will deploy custom facts that we create automatically. Review the
/etc/puppet/modules/vmguest/lib/facter/vmguest.rb file.
Here you will see the structure that Puppet expects for custom facts. The goal of this
fact is to return the instanceUuid as a fact by using the VM Tools locally to run "info-get
guestinfo.vminstanceuuid" and return the output.

HOL-PRT-1303

Page 73

HOL-PRT-1303

Notice first that we have added some complexity here based on leveraging already
known facts of whether a guest is Linux or Windows (if Facter.value('operatingsystem')
== 'windows') to change the command.

Run the VM tools command #1

Open an SSH console to linux01 with PuTTY if you do not already have it open.
Run the following command.
# vmtoolsd --cmd "info-get guestinfo.vminstanceuuid"
The command should return a result of "No value found". This is because the VM does
not currently have its vminstanceuuid assigned.
Since this is a "custom" Advanced Setting per VM, we must assign it through one of
many different possibilities. We will demonstrate using a vCenter Orchestrator workflow.
*Further integration between Puppet and VMware could ensure that a guest can retrieve
it's own instanceUuid without this step.*

HOL-PRT-1303

Page 74

HOL-PRT-1303

Open vCenter Orchestrator

Open the vCenter Orchestrator Client shortcut from the start menu. Login wiith root as
the username and VMware1! as the password.

HOL-PRT-1303

Page 75

HOL-PRT-1303

HOL-PRT-1303

Page 76

HOL-PRT-1303

Navigate to a vCO workflow

When it opens select the workflow icon in the top left. Scroll through the hierarchy to
VMworld 2013 -> PaaS Helper Workflows.

HOL-PRT-1303

Page 77

HOL-PRT-1303

Submit vCO workflow

Right click the Update Adv Setting with vminstanceuuid workflow and press Start
Workflow.
Press the "not set" link and select linux01 from the hierarchy (vCenter Server -> https > Datacenters -> Datacenter Site A -> vm -> linux01). Press Select in the bottom right.
Press Submit in the bottom right.

HOL-PRT-1303

Page 78

HOL-PRT-1303

Workflow Success
You should then see a green arrow below the workflow indicating a successful run.

Run the VM tools command #2

Go back to the linux01 putty session. Re-enter the command from prior shown below.
# vmtoolsd --cmd "info-get guestinfo.vminstanceuuid"
Notice how the instanceUuid is being reported now!

Identify custom VM facts

Since we have autodeploy turned on for Puppet, the custom facts as part of the
VmGuest module are already deployed to the agent. You can now run facter manually to
see what the Puppet agent will be reporting to the Puppet Master about itself next time
it checks in.

HOL-PRT-1303

Page 79

HOL-PRT-1303

# facter --puppet
Notice how the vminstanceuuid is now populated at the end. We also follow the
command up with a command including "grep" to only include the vminstanceuuid.

HOL-PRT-1303

Page 80

HOL-PRT-1303

vCenter Hiera Lookup for Classes

Introduction
Here we will demonstrate how vCenter will be used as the External Node Classifier (ENC)
for our Virtual Machines. This is the case only if our VMs have "vminstanceuuid"
reported as a fact. Otherwise lookups happen as they did before (FQDN) by the
nodes.pp and then defaulting to nodes_default.pp file.

Clear nodes.pp file

Since we are using a new lookup for existing hosts that we previously configured let's
clear out the nodes.pp file.
# rm /etc/puppet/manifests/nodes.pp
# touch /etc/puppet/manifests/nodes.pp

HOL-PRT-1303

Page 81

HOL-PRT-1303

Run Puppet agent #1

Let's run the Puppet Agent again interactively. On this run at the end we should notice
that the "hellovmworld" class is applied. This is because we removed the classification
for linux01 from the nodes.pp file and the "hellovmworld" class is included in the
nodes_default.pp manfiest.
Switch over to the linux01 window and clear out the old HelloVMworld.txt file and rerun
the puppet agent using the following commands:
# rm /tmp/HelloVMworld.txt
# puppet agent -t

HOL-PRT-1303

Page 82

HOL-PRT-1303

Open vSphere client

Let's go ahead and open the legacy vSphere Client from the Start Menu. This client is
being used to review the Custom Values that are used per Virtual Machine to store the
Puppet class includes.
Login with the username of root and password of VMware1!

HOL-PRT-1303

Page 83

HOL-PRT-1303

Navigate to linux01
Select linux01 so that it's highlighted.
Notice the "puppet.classes" field that is listed under Annotations. This is considered a
custom value and is associated per virtual machine as a result of the Puppet integration
that we're demonstrating.

HOL-PRT-1303

Page 84

HOL-PRT-1303

Update custom value

Press the Edit button and double click the value area next to puppet.classes until the
value field is editable.
Enter "avamar" in this field.
Press the OK button at the bottom.

Run Puppet agent #2

Now let's run the Puppet Agent interactively again from linux01.
# puppet agent -t
Notice how the Avamar notices appeared stating that the Client is installed and
registered. Previously in Puppet Agent #1 we saw the "hellovmworld" class applied, here

HOL-PRT-1303

Page 85

HOL-PRT-1303

you can see now that we are looking to the Custom Value in vCenter the Puppet agent
now applies the "avamar" class as it is set in vCenter!

HOL-PRT-1303

Page 86

HOL-PRT-1303

VM Image Backup Puppet Module

Introduction
The concept of allowing a guest to report its underlying VM instanceUuid is important for
other reasons than simply looking up classes with vCenter as an ENC. It can also be
leveraged in Puppet modules that are VMware oriented or work at the Hypervisor level.
As a great example we are including an Avamar Puppet module that manages VM image
backups. Previous examples of Avamar in Puppet have been targeted at an in-guest
agent which is viable for both Virtual and Physically provisioned servers.
This module works by having the guest report its instanceUuid as a fact and the Puppet
module then leverages a REST provider from the VM guest Puppet agent in order to
allow the guest to request registration and image based protection. The Recovery Point
Objectives (RPO) are set as part of this Policy and default to a ensuring a backup every
24 hours.

Review avamar-vm facts #1

We finished the last module where you were looking at the new "vminstanceuuid" fact.
The facts were seen by running the following command from linux01.
# facter --puppet
As well as this fact, notice above this fact there are also facts that are listed as
vm_backup. These facts are being provided by the avamar-vm module.
Let's now look at the facts the guest knows about for the actual VM image backups
which are generated by the previous command.
vm_backup_clientlocation - The location of the Avamar VM Client account
vm_backup_current_jobstate - The current activity for the Avamar VM Client
vm_backup_last_hoursago - Hours since last image backup
vm_backup_lastcompletedjob_aftersourcededupe_kbyetessent - The actual bytes
after deduplication that were actually transmitted from the Avamar Image Proxy to the
Protection Storage. This represents previously "unseen" blocks.
vm_backup_lastcompletedjob_durationseconds - Seconds it took to backup the VM
on the last job
vm_backup_lastjob_state - Indicates whether the last backup job was successful or
not
vm_backup_recovery_point_date_range - The date range of backups that currently
exist for this VM
As you can see there is some awesome information for the server admins that can be
used to help them understand the current state of backups for a VM, which in turn

HOL-PRT-1303

Page 87

HOL-PRT-1303

serves as an important role for Puppet. Some of this information is also available from
the guest by leveraging the File Level Restore (FLR) GUI as shown in a previous module.

HOL-PRT-1303

Page 88

HOL-PRT-1303

Update linux01 VM class

In order to leverage the new module, let's update the Advanced Setting on the linux01
VM to include "avamar-vm" it as we did for "avamar" previously.
Note: This field permits multiple module names separated by commas.
From the vSphere Client, navigate to linux01. Look for Annotations in the Summary
page, and press Edit. Where we previously entered "avamar" for puppet.classes, let's
now enter avamar-vm.
Another note: When leveraging the avamar class we were actually installing a guest
agent into the virtual machine. The avamar-vm agent performs agent-less backups of
virtual machines and allows for granular file level recovery or image recovery. An agent
based protection mechanism can still make sense in certain scenarios where VMware
snapshots are not preferred. With an Avamar in-guest agent you still get the benefit of
File Level Recovery and Source-based deduplication, but do not have VMware Image
level recovery flexibility.

HOL-PRT-1303

Page 89

HOL-PRT-1303

Run Puppet agent #1

From linux01, let's run the Puppet agent interactively again to load the avamar-vm
module. As you saw above, the facts for this module were already loaded. This run of
the agent will actually take action with these facts to ensure the VM is registered and
backed up.
# puppet agent -t
If the new module is running then you will see notices that describe the VM as
registered and the status of backups for the VM.

HOL-PRT-1303

Page 90

HOL-PRT-1303

Remove Linux backups

One of the conditions of the avamar-vm module is to ensure a backup has occurred
within the last 24 hours. If there has not been a backup or a backup is not running or
queued, then one will be started. Since we previously took a backup of this VM, let's
remove it.
Open the vSphere web client. Go to EMC Backup and Recovery. Connect to avamar01 if
not connected already. Press the Restore tab. Expand the linux01 system and check the
backups within the past 24 hours. Press the Delete button. Confirm the delete as show
below.

HOL-PRT-1303

Page 91

HOL-PRT-1303

Review avamar-vm facts #2

Since we removed the backups you should see that the fact for
"vm_backup_last_hoursago" is now hundreds or thousands of hours. Run the following
command from linux01 to verify:
# facter --puppet | grep vm_backup

Run Puppet agent #2

Now lets run the Puppet agent manually again from linux01 (puppet agent -t). This will
ensure that the Virtual Machine has a current image based backup that's taken place in
the last 24 hours.

HOL-PRT-1303

Page 92

HOL-PRT-1303

Review avamar-vm facts #3

Once again, run facter to show the status of backups.
# facter --puppet | grep vm_backup
Depending on how quickly you run the facter command you may notice the job
RUNNING with the vm_backup_current_jobstate field. If not, then you will see the next
run of facter which shows IDLE for this field.
Continue to the next step once you see the job as IDLE. Review the duration and
updated RPO date ranges which reflect the date range for eligible recoveries.

HOL-PRT-1303

Page 93

HOL-PRT-1303

Deploying PaaS with

Puppet and vCO

HOL-PRT-1303

Page 94

HOL-PRT-1303

vCenter Orchestrator Workflows and

Puppet
Bringing it all together
We have been showing a lot about how Puppet can be used to not only ensure the
installation of applications but also help set VMware oriented policy for things like
backups. Everything so far has been manual to help show the fundamental workings
below the covers.
In this module we introduce vCenter Orchestrator to show how Rainpole can leverage
pre-built workflows in vCO that can provision, customize, set Puppet classes, and then
start the Puppet agent to bring the guest to an expected state.
Here we leverage vCO to do more of the VMware oriented (IaaS) tasks. We end the
workflows by issuing command to start the Puppet Agent which completes a (PaaS)
deployment and makes the Rainpole CTO a happy executive.

HOL-PRT-1303

Page 95

HOL-PRT-1303

Review Provisioning Workflow

In this section, you are going to take a few minutes and walk through a vCenter
Orchestrator workflow.

HOL-PRT-1303

Page 96

HOL-PRT-1303

Open vCO and PaaS workflow

We previously opened vCO to add a vminstanceuuid to a VM. If you do not have it open,
do so by going to the Start Menu and clicking vCenter Orchestrator. Login with root/
VMware1!.
Press the workflows icon at the top, and then drill down from root- VMworld2013 Lab PaaS VM Workflows.
Right click the EMC - New PaaS Linux VM workflow and click Edit.

HOL-PRT-1303

Page 97

HOL-PRT-1303

Review provisioning workflow schema

Press the Schema button. Here we can see the workflow visually that occurs during the
provisioning process. Right click anywhere in the schema window and press Zoom Out.

HOL-PRT-1303

Page 98

HOL-PRT-1303

Review workflow schema and start Puppet client

Out of the box, vCenter Orchestrator has workflows for provisioning in all types of
different scenarios. Here we are taking a provisioning workflow and customizing it for
the following purposes. We have included a screenshot that shows the additions for
review only.
1) Update Advanced Settings to include vminstanceuuid
2) Update Custom Values (attributes, annotations) to include Puppet classes chosen
3) Start Puppet agent after Guest Customization completes (hostname, IP, etc set)

HOL-PRT-1303

Page 99

HOL-PRT-1303

Review workflow presentation

Now you have seen the workflow, but there are questions that must be asked when
each workflow executes! For this there is a "presentation" tab.
Press the presentation tab at the top. Continue to drill through the hierarchy as shown in
the screenshot below. Each section is briefly explained below.
VMware Information - Basic VM Information
This is where we ask basic information such as the IP and name of the VM.
VMware Information - Puppet PaaS - Installs

HOL-PRT-1303

Page 100

HOL-PRT-1303

This is where we add the different applications or modules that we have for puppet. By
populating parameters here we are effectively adding classes that are set in the VM
Custom Values.
VMware Information - Puppet PaaS - Backups
Here we select guest, application, and VMware oriented backups.

HOL-PRT-1303

Page 101

HOL-PRT-1303

Close workflow
In the bottom of the workflow press the Cancel button followed by the Close Anyway
button that appears.

HOL-PRT-1303

Page 102

HOL-PRT-1303

Assign Workflows to VM Context Menu

Introduction
Now we could have executed the workflow from vCO, but let's make this even cooler
and run it from the new vCO plugin for vSphere Web Client!

Logout of vSphere Web Client

In order to leverage vCenter Orchestrator from the vSphere Web Client you must login
with different credentials based on the way we have configured this lab. We could have
granted the "Domain Administrators" group access to vCO workflows, but for purposes
of lab, we chose to use a different administrative user. Press the Logout button from the
"Administrator@corp.local" drop down at the top right of the vSphere Web Client.

HOL-PRT-1303

Page 103

HOL-PRT-1303

Login to vSphere Web Client

Login with the username of root and password of VMware1!

HOL-PRT-1303

Page 104

HOL-PRT-1303

Open vCO and vCO workflows in vSphere Web Client

Press the vCenter Orchestrator button on the left side.
Next, make sure you are on the "Manage" tab. Here you see workflows that are currently
assigned to different menu context options in the vSphere Web Client. For example,
when right clicking a Datastore, there is a workflow that is for converting disks to thin
provisioned.
Click the "+" icon at the top left of the same window to add a workflow.

Manage assigned workflows

Expand the vCO Servers list vCO Servers -> https -> VMworld 2013 Lab -> PaaS VM
Workflows.
Press EMC - New PaaS Linux VM.
In the bottom right, check the box next to Virtual Machine.

HOL-PRT-1303

Page 105

HOL-PRT-1303

Press the Add button at the top.

Repeat these steps for The Windows VM workflow.
Press OK.

HOL-PRT-1303

Page 106

HOL-PRT-1303

HOL-PRT-1303

Page 107

HOL-PRT-1303

Provisioning VMs and PaaS

Press the home button
Press the Home button to return to the vSphere Web Client to leverage the Workflows.

Go to template folder
From the vSphere web client, in the top right let's use the Search functionality to find
the Template folder with the appropriate VMs for provisioning. Enter templates, and wait
for the results to populate and press Templates. Select Templates.

HOL-PRT-1303

Page 108

HOL-PRT-1303

Provision new Windows VM

Click the Virtual Machines icon in the upper left window.
Right click the win-base-gold VM.
Wait for All vCenter Orchestrator Actions to appear and click this option.
Click the EMC - New PaaS Windows VM menu choice.

HOL-PRT-1303

Page 109

HOL-PRT-1303

Answer Windows VM questions

1. Enter the name of the virtual machine as "windows02" and press Next.

Installs Menu
1. Select the check box next to Microsoft SQL 2008 R2.
2. Select Hyperic Platform/Application Performance Monitoring.
Press Next

HOL-PRT-1303

Page 110

HOL-PRT-1303

Backups Menu
1. Select Avamar Backups for Microsoft SQL.
2. Select Virtual Machine Image Backups.
Press Finish

HOL-PRT-1303

Page 111

HOL-PRT-1303

Provision new Linux VM

Right click lin-base-gold and go to All vCenter Orchestrator Actions. Press the EMC - New
PaaS Linux VM button.
From there a window appears and similar questions will be asked from prior.
1. Enter the name of the virtual machine as "lin-sles-221"
2. Enter the IP as 192.168.110.221
Press Next

HOL-PRT-1303

Page 112

HOL-PRT-1303

Installs Menu
1. Select Apache2 Web Services
2. Select Hyperic Platform/Application Performance Monitoring
Press Next

Backups Menu
1. Select Virtual Machine Image Backups
Press Finish

Watch recent tasks

Notice in the top right corner there are two tasks that are progressing which represent
running workflows.
Based on recent explanations, you saw that there were three extra pieces to this
standard provisioning workflow So when these items complete, the VM will be ready and
the Puppet agent is then free to complete it's work.

HOL-PRT-1303

Page 113

HOL-PRT-1303

The expected time for the Linux VM is around 2 minutes. Once you see the workflow
complete, you should be able to open a PuTTY session to the name as entered (linsles-221) or Remote Desktop to the windows system (windows02). The windows system
will take a few more minutes since it needs to go through an Active Directory join and
sysprep process. As well the SQL server installation takes a few more minutes to
complete.

HOL-PRT-1303

Page 114

HOL-PRT-1303

Review PaaS Deployment

Wait for Linux workflow to complete
Watch the recent tasks on the right side, you should see a green check box next to the
EMC - New PaaS Linux VM task once it completes successfully.

HOL-PRT-1303

Page 115

HOL-PRT-1303

Check for Puppet agent status for Linux

Using PuTTY, SSH to the linux system you deployed (lin-sles-221) with root/VMware1!.
Check that the puppet agent process is running.
# ps auxw | grep puppet
If the Puppet agent was currently applying modules you would see this as a separate
process. If there is only only one process listed then it is idle.

Review installed classes

Each agent holds information about previous runs and installed classes locally. Since this
lab is focused more on the Puppet modules themselves, we will not be diving into the
different ways to manage Puppet as a whole. For now lets just look at the classes that
are installed.
# cat /var/lib/puppet/state/classes.txt
Here you can see that all of our classes that we requests are installed (apache2,
hyperic, avamar-vm).

Review apache2 web page

From the vSphere Web Client, go ahead and open a new tab by pressing the empty tab
on the right side.
Enter http://lin-sles-221 in the top and press enter.

HOL-PRT-1303

Page 116

HOL-PRT-1303

You should see the default web page that we distribute with Apache2 installed!

HOL-PRT-1303

Page 117

HOL-PRT-1303

Check VM image backup state

Along with Apache2 you should be able to see that VM Image level backups have
occurred due to the avamar-vm module. Execute the following command.
# facter --puppet | grep vm_backup
Here you should see the backup job status and last backup. Depending on how quickly
you were checking the status the job may still be running. In this case the
vm_backup_current_jobstate should report as RUNNING and NO_BACKUPS_EXIST should
be listed under vm_backup_last_hoursago.
If it is already finished then you should see it similar to the screenshot below.

HOL-PRT-1303

Page 118

HOL-PRT-1303

Checking the status of the Windows deployment

The Windows deployment will take a bit more time to complete. However, we are
probably running a bit short on time, so we are simply going to present what the
deployment should look like. Feel free to wait around for it, if the deployment is done
you should be able to see the verify the installs in a few places.
Control Panel -> Programs and Features - Shows the status of Avamar guest agent,
Avamar Windows SQL Agent, and Microsoft SQL 2008.
Control Panel -> Administrative Tools -> Services - The status of the Hyperic agent can
be seen here as Running
Start -> All Programs -> Puppet -> Start Command Prompt with Puppet ->
run_puppet_interactive.bat -> facter --puppet

HOL-PRT-1303

Page 119

HOL-PRT-1303

Monitoring PaaS and EMC

Avamar with Hyperic

HOL-PRT-1303

Page 120

HOL-PRT-1303

Configuring and Reviewing Hyperic

Monitoring
Introduction
Up to now we have leveraged Puppet to perform the installation and configuration steps
for our Platforms. As part of this we have included the Hyperic module which was
responsible for loading and configuring the Hyperic agent. This means we should have
representation in the Hyperic console for the platforms that included hyperic or hypericwin as classes!
This section is meant to be a brief look at Hyperic focusing on the application agent
auto-discovery and Avamar information.

Login to Hyperic
Open a new tab in Internet Explorer.
Use the shortcut within the favorites bar in Internet Explorer and enter hqadmin/
hqadmin for username and password.
Press the Sign in button.

HOL-PRT-1303

Page 121

HOL-PRT-1303

Review auto-discovery list

In the top right corner you should see platforms that have been detected and their
associated services. Press the green Add to Inventory button for each platform to start
monitoring it.
At this point the platform is now being monitored, and details about the services and
general operating system information is being collected.

Review Avamar resource

Press the Resources tab at the top and press the Avamar01 resource.

HOL-PRT-1303

Page 122

HOL-PRT-1303

EMC Avamar Backup Basic Stats

From there select the EMC Avamar Backup Basic Stats in the Platform Services Health
view.

HOL-PRT-1303

Page 123

HOL-PRT-1303

Review indicators
Initially you will be looking at the Indicators tab which is a method that Hyperic
leverages to graph metrics over time. Since we are pulling real information for this lab,
the graphs will not have much information.
Press the Metric Data tab to review that raw metrics values.

Review metric data

Here you can see that we are populating useful information for VMware admins about
the Avamar backup appliance. Certain things like backup, replication, and other key
stats are populated.
Performance - Backup-Rolling24Hr-Common - Reports the commonality percentage of
the data is being protected by Avamar.
Throughput - Backup-Rolling24Hr-Activities - How many backup activities occurred in the
past 24 hours

HOL-PRT-1303

Page 124

HOL-PRT-1303

Throughput - Backup-Rolling24Hr-New - New data seen in the past 24 hours

Throughput - Backup-Rolling24Hr-Scanned - Amount of provisioned space scanned in the
past 24 hours
Throughput - Backup-Rolling24Hr-SourceDedupe - Amount of space savings on backups
due to Avamar source side deduplication
Utilization - Data-Protected - Total data protected (VMs * Backups)

HOL-PRT-1303

Page 125

HOL-PRT-1303

vCOps and Conclusion

vCenter Operations
The stats that are collected by Hyperic can be very important to combine with VMware
stats. When looking at the broader stack, Hyperic can be targeted for the PaaS and up,
while vCenter Operations can be targeted at IaaS and below.
The combination of these two leads to some really killer views of your environment.
Although we do not show it in this lab, there is a natural integration between vCOps and
Hyperic that can be leveraged to combine the data collection capabilities with the
analytics of vCOps to provide a broader view of your Datacenter!

Summary
The Rainpole CTO is pleased with the work you've highlighted and is really excited to be
able to offer Platform as a Service that includes Data Protection to his biggest customer.
There's a new puppetmaster role that he's posting shortly and wants you to consider
applying for it...
In all seriousness, we hope you enjoyed the time you spent working in this lab and
found it both fun and educational. We realize that the lab wasn't 100% EMC product
focused and that was by design. As we continue to evolve into the next generation of
software defined everything, you should expect to see products like ViPR that work with
both EMC and other vendor's technologies, and EMC taking a more active role in helping
our customers use Open Source tools like Puppet to make the deployment of EMC
technologies even easier.
We welcome any and all feedback and once again encourage you to use the "raise your
hand" option in the lab if you have want to speak with us directly. Thanks again for
taking the time to complete this lab and we hope you enjoy the rest of the conference.
- the EMC Lab Team (Clint, Jeff, and Joel)
NOTE: Would you like to be entered into a raffle for a chance to win an
unlocked iPhone? Following the lab please click on the Help button to get a
contest card from an EMC proctor. The proctors managing the raffle will have
Puppet Labs or EMC shirts on.

HOL-PRT-1303

Page 126

HOL-PRT-1303

Conclusion
Thank you for participating in the VMware Hands-on Labs. Be sure to visit
http://hol.vmware.com/ to continue your lab experience online.
Lab SKU: HOL-PRT-1303
Version: 20140321-162110

HOL-PRT-1303

Page 127