Cloud Technology Partners

321 Summer Street, 5th Floor

Boston, MA 02210

Chef Development Process

Environment Setup
1. Access Version Control Contact the [INSERT CONTACT] for questions on how to
get Git Access, which is where the Baxter cookbooks live.
2. Setup Dev Environment - Setup a virtual machine for developing and testing recipes
locally. Assumptions are that Chef developers will have either:
VirtualBox (what does Baxter currently use?) installed locally with a VM equivalent to the
target deployment environment
SSH to a virtual server in AWS (Jenkins job exists to create Chef dev environment in
AWS via a CloudFormation template; see the DevOps team for access)
Appropriate text editor (Notepad ++, Sublime Text, etc.) and ChefDK installed locally
without virtual machine, along with Test Kitchen for testing recipes in virtual environment
3. Configure Local Dev (if not using AWS) From the local environment, download from
Nexus and install:
Chef Development Kit for
i. Mac OSX
ii. Windows
iii. Ubuntu
iv. Debian
v. RHEL
vi. SUSE
Git client for communicating with the Version Control System. For Windows, TortoiseGit
or Git for Windows
A robust text editor such as Sublime Text 3 for authoring recipes.
4. Checkout Recipes From the local ecosystem, create a working directory for your
recipes then use the Git client to check them out from the appropriate repository.
5. Validate Dev Environment From the working directory, run chef verify to verify your
local environment is setup.
If you completed these 5 steps, congratulations! You now have a working Chef development
ecosystem and can begin developing and testing recipes.
Local Development
In order to run the example cookbook, the chef-client must be installed on your target system.
For ease of use and to avoid dependencies that may slow down Chef developers early on, we
are recommending the use of Chef-Zero in local environments.

Chef-Zero is a client-side application that does not require connectivity to a Chef Server; rather,
it spins up a very lightweight, in-memory Chef Server that your local cookbooks are then
transferred to. When the Chef-Client run starts, it pulls cookbooks from the in-memory Chef
Server to mimic the real client-server relationship.

Cookbooks must be locally available in order to run Chef-Zero, and it requires two files:
1. Client.rb the Chef-Client configuration file. This file specifies the location where Chef-

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com 1

 Client can locate cookbooks and roles. The default location where Chef looks for the
client.rb file is /etc/chef/client.rb, but this default can be overridden with the -c or --config
command line switch. Below is an example client.rb configuration followed by an
example with just enough configuration to enable the example cookbook to run:

Client.rb for running example_cookbook on Linux:

Client.rb for running example_cookbook on Windows:

2. node.json the file containing instructions for target nodes (run list). Without this file,
Chef Zero does not know what cookbooks to run on itself. The location of the node.json
file is configurable with the j command line switch.

Below is an example node.json file specifying the dns cookbook followed by the
example_cookbook. Notice the syntax; dns::default specifies the default recipe of the
dns cookbook. Why is there no specific recipe selected for the example_cookbook?

The default behavior of a run list is to pull the default recipe if none is specified. In the
below example, both the dns and example_cookbooks are set to run their default
recipes. It was not necessary to specify the default recipe for the dns cookbook.

In order to execute the example_cookbook, ensure that it is present in the correct directory (in
this case, /opt/chef/chef-repo/cookbooks or c:\chef-repo) and then simply run:

The z flag is telling Chef-Client to run in Zero mode, which will spin up the in-memory Chef

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com 2

 Server rather than connecting to the Chef server specified in client.rb. It is not necessary to
specify the location of the node.json file, as it is specified with the json_attribs attribute inside
the client.rb.
Development
Developing Chef cookbooks and recipes is like most other forms of software development.
Developers are encouraged to work in small, iterative steps getting feedback often (via
automated tests) on their progress. Software development best practices are outside the scope
of this document, but with respect to Chef development there are some best practices.

Ruby Gems & Dependencies

Recipes and cookbooks rely on RubyGems, a code packaging system for the Ruby community.
For local development and testing, developers can use Ruby Gems hosted at
http://RubyCentral.org. This is the fastest and quickest way to get going.

For security reasons, Baxter has an internal RubyGems server with only those recipes approved
for use within Baxter . Once new recipes are working and tested locally, any new gems that
need to be added to the internal RubyGems server should be done before the new recipe(s) can
be deployed to non-local environments.

It is the responsibility of each Chef developer to identify any gems used by internal or
community cookbooks so that the required gems (and their dependencies) can be made
available on the Baxter Ruby Gems server. Please contact the DevOps for more information on
getting gems setup in the internal RubyGems server.

Cookbook Versioning
Cookbooks should always be versioned before changes are committed to Git as the Chef
Server will freeze all uploaded cookbooks by default (meaning they cannot be overwritten
without updating the version). Cookbook versions are maintained in each cookbooks
metadata.rb file in the root cookbook directory and should follow semantic versioning as follows:
MAJOR version when a breaking change is made
MINOR version when backwards-compatible functionality is added
PATCH version when backwards-compatible bug fixes are added

Examples:
1. apache 0.0.1 would be the initial commit of the apache cookbook; this cookbook is
incomplete
2. apache 0.1.0 would be the first working version of the apache cookbook; this cookbook
is not yet used in production but should run successfully
3. apache 1.0.0 would be the first working version of apache for production use
4. apache 1.1.0 would include a non-breaking update to the production apache cookbook
5. apache 1.1.1 would fix a minor bug in the apache 1.1.0 cookbook

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com 3

 Cookbook Versioning Within Run List
Run Lists for nodes should always specify cookbook versions to use, regardless of whether the
cookbook is called independently or from within a role. If a nodes run list does not specify a
cookbook version, the newest version of that cookbook found on the Chef Server will be used
by default; this means that any cookbook update committed to Git will eventually be picked up
by nodes that do not specify a cookbook version to use.

This may result in unintended changes being made to nodes, especially if the cookbook being
updated is used by many nodes. Cookbook versions in a node run list or role run list can be
specified as follows:

{
run_list: [
recipe[javamsp::groups@1.0.0],
recipe[dns@1.1.0]
]
}

In the above example, the recipe groups within the javamsp cookbook is constrained to
version 1.0.0 of the javamsp cookbook with the
cookbook_name::recipe_name@cookbook_major.cookbook_minor.cookbookpatch syntax. The
recipe default within the dns cookbook is constrained to version 1.1.0 of the dns cookbook,
because a specific recipe is not defined for the dns cookbook, so the default is used.

Testing
All cookbooks should be thoroughly tested before being pushed to Git. All role changes should
be made only after all relevant tests have passed in pre-production regions. Any role changes
intended for production release will need to secure approval of a Baxter release manager before
the Jenkins job is triggered.

Cucumber tests should be written prior to writing a cookbook and should be written to fail. As
the cookbook is beginning to take shape, the Cucumber tests should start passing until all tests
desired to verify the state of the system are passing. If the Cucumber tests and knife cookbook
tests for a particular cookbook have all passed, the cookbook is likely ready to be committed to
Git.

Local testing of Chef cookbooks should always include at least:

1. knife cookbook test COOKBOOK_NAME necessary for basic Ruby syntax verification;
many chef-client runs fail due to syntax errors. This command can be run on any Chef
Workstation because it is a test against the local /cookbooks directory. The knife.rb file
must contain the appropriate path for cookbook_path in order to locate the
COOKBOOK_NAME parameter.

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com 4

 2. Cucumber - Cucumber is a behavior-driven development framework where the desired
state of a system is written in the Gherkin style below prior to writing any code:

This .feature file (in this case, ~/chef-repo/features/apache.feature) is backed up with a Ruby file
in (~/chef-repo/features/stepdefs/apache.rb) file containing the code necessary to test this
feature. In this case, the code would appear as follows:

The above example illustrates the Ruby code and regular expression necessary to verify that
the return values from the command which httpd correctly matched the feature files
expectation that the result equal /usr/sbin/httpd.
The following example illustrates a Cucumber testing verifying that the Apache service is
running by looking for the return value httpd (pid *random pid*) is running from the command
service httpd status:

The code looks like this:

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com 5

 Notice that the code the two examples are the same. This is due to Cucumbers utilization of
regular expressions; the only difference between the two feature lines is that the second
example includes a regular expression in the actual feature.

This is because the Process ID (PID) of the httpd service is not known at runtime and is
dynamically generated, thus necessitating the use of a regular expression to match the
unknown process ID to an expected string.

Many common bash commands can utilize the same code to verify a systems state. The
Continuous Integration section of this document discusses the use of Cucumber in relation to
the Continuous Integration flow in more detail. More information about Cucumber testing can
be found here.

FoodCritic
FoodCritic is another Chef lint tool for verifying syntax and Chef best practices prior to node
convergence. FoodCritic is automatically run on every Git commit via a Jenkins job, but should
be used in conjunction with knife cookbook test locally prior to committing changes. FoodCritic
is a Ruby gem and requires Ruby 1.9.2 or greater.

Test Kitchen
Test Kitchen is a RubyGem that provides infrastructure developers an integrated testing
platform by leveraging virtual machines to provide pristine environments that are easily spun up,
tested upon, and blown away. Test Kitchen supports test environments in AWS, VirtualBox,
and OpenStack or using Vagrant locally. The Chef Development Kit also ships with Test
Kitchen for getting up-to-speed quickly.

Getting Started
1. Download VirtualBox and Vagrant. If vagrant is already installed on the system with
version lower than 1.6.5, kindly uninstall the lower version and reinstall v1.6.5 to avoid
any version conflict issues.
2. Download Git for Windows and during installation, make sure to check the Use Git and
optional Unix tools from the Windows Command Prompt checkbox, otherwise accept
the defaults.
3. Enable Virtualization in your System BIOS. Usually, this is accomplished by:
a. Restarting the computer
b. Press the ESC key during startup
c. Select System BIOS by pressing the F10 key
d. Select System Configuration or Advanced
e. Select Device Configurations
f. Select Virtualization Technology
Save and quit
4. Download the Chef Development kit for Windows

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com 6

 5. Install the ChefDK. This will create a directory in C:\opscode containing the ChefDK
executables (test-kitchen, berkshelf, chef-client, chef-vault, ohai, and foodcritic)
6. Open the command line and run chef verify (no quotes). The output should look similar
to the following:

7. Clone the Chef repositories that are needed via TortoiseGit or Git CLI.
8. Open the ~/chef-repo/cookbooks/kitchen-docs/Berksfile and update the paths to point to
each cookbook on your local filesystem.

9. Open the command line and change directory to the kitchen-docs cookbook
10. Type kitchen create kitchen-centos-66
11. Once the creation is finished, type kitchen converge kitchen-centos-66
12. If the converge fails because the Omnibus installer cannot be found, open your
command line and run vagrant plugin install vagrant-proxyconf

Test Kitchen has used the VirtualBox platform and the Vagrant VM driver to quickly build a
CentOS 6.6 virtual machine using the Vagrantfile.erb embedded Ruby template in the kitchen-
docs root directory. This template is used so that the non-default http and https_proxy values
can be set when the Operating System starts, allowing the VM to have access to the internet
through the proxy using your AD credentials.

Once the Vagrantfile has been used to provision the VM, the VM downloads the Chef client from
the omnibus URL passed into Test Kitchen from the .kitchen.yml file. After Chef has been
installed, a custom client.rb file has been generated using default values from Test Kitchen
along with the added trusted_certs_dir specified in the Provisioner section of the .kitchen.yml

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com 7

 file. Finally, Test Kitchen creates a custom dna.json file, which contains the runlist specified in
the Suites section of the .kitchen.yml file. Test Kitchen then kicks off a run of the chef-client via
a command like the following:

The .kitchen.yml file is used to describe how Test Kitchen should provision your virtual test
environment; it is also possible to hook Test Kitchen up with other service providers including
AWS, OpenStack, and Rackspace. The driver section of the file indicates which driver to use;
in our case, we simply used a local environment powered by VirtualBox and Vagrant.

Kitchen EC2 Driver

It is also possible to spin up EC2 instances using Test Kitchen. Simply install the kitchen-ec2
RubyGem, request AWS API keys through Access Central, set your AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY environment variables, and identify your subnets, security
groups, and region that you would like to use to spin up an EC2 instance for testing.

You will also need to identify the appropriate AMI ID and create an SSH key pair that Test
Kitchen can use to connect to the instance and run the Chef suites. An example .kitchen.yml for
EC2 is below:

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com 8

 The provisioner section of the file indicates how the environment should be provisioned;
possible options include:
chef_solo (deprecated way of running chef-client locally)
chef_zero (standard method of executing local Chef runs, spins up a small in-memory
Chef Server and pulls cookbooks from local Chef Server)
ansible_playbook (when using Ansible)
puppet_apply (when using Puppet)

The provisioner section also includes overrides to the client.rb file (which is dynamically
generated by Test Kitchen on the test node in /tmp/kitchen/client.rb). The .kitchen.yml file in the
kitchen-docs cookbook includes an override for the trusted_certs_dir, which points to
/etc/chef/trusted_certs. This is used for communications with the Artifact Repository; the root
Baxter CA cert must exist in /etc/chef/trusted_certs on the target node if your recipes are using
remote_file for downloading artifacts. The root_crt.rb kitchen-docs recipe includes the code to
copy the root cert to this directory.

The platforms section of the file indicates which version of OS the tests should be run on, as
well as any specific configuration you would like to apply to the OS (including URL to find the

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com 9

 VirtualBox image). Test Kitchen within Baxter should support CentOS 5.10, CentOS 6.4, and
Ubuntu 12.04 at the time of this writing.

The suites section of the file indicates which recipes to execute on the target node. You can
also customize memory and CPU for the VM within the suites section. If you have written unit
tests for your cookbook, they will be executed when you run kitchen converge, kitchen verify, or
kitchen test.

At this point, you should be able to navigate to localhost:8080 on your workstations browser
and view a test page served by Apache. You should also be able to navigate to localhost:4567
and view the Test Kitchen documentation.

Additionally, integration and unit tests have been written for the kitchen-docs cookbook. These
tests can be viewed in the test and spec folders respectively; the spec folder contains
ChefSpec unit tests while the test folder contains ServerSpec and BATS integration tests. See
below for additional detail on each testing method, but for now you should just be able to run
kitchen verify kitchen-centos-66 to see the integration tests pass.

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com

10
 When youre satisfied that your cookbook has successfully executed, just run kitchen destroy
kitchen-centos-66 to delete the entire VM.
Berksfile and Berkshelf
The Berksfile located in the kitchen-docs cookbook root directory is used much like the Bundler
gem is for RubyGems; Berkshelf is a RubyGem that handles cookbook dependencies within a
test framework like Test Kitchen, where you run tests from within a cookbook.

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com

11
 Since the .kitchen.yml file in ~/chef-repo/cookbooks/kitchen-docs doesnt know about ~/chef-
repo/cookbooks/example_cookbook and the kitchen-docs cookbook depends on it (see kitchen-
docs/metadata.rb and kitchen-docs/recipes/root_crt.rb for dependencies), Berkshelf (and the
corresponding Berksfile and Berksfile.lock) is used to handle cookbook dependencies during
resource compilation.

This allows you to seamlessly test interdependent cookbooks with Test Kitchen. Full Berkshelf
documentation is available here.

Below is a screenshot of the Berksfile located in the kitchen-docs cookbooks root directory:

The source https://supermarket.chef.io declaration at the top of the Berksfile indicates that
Berkshelf should reach out to the Chef Supermarket site to download dependent cookbooks if a
path is not otherwise specified. In our case, weve got two of the necessary cookbooks locally
(example_cookbook and kitchen-docs are custom cookbooks that are not available in the
community, but IIS is a community cookbook and can be retrieved from the Supermarket, hence
no path: specification).

In the above example, were going to reach out to the Supermarket to download version 4.0.0
of the IIS cookbook (this also illustrates locking cookbook dependencies to specific versions so
that conflicts dont emerge if a community cookbook is upgraded to a new major version).

Login to Vagrant VM
Once the vagrant boxes are created and converged, you can also login to the VM and verify any
configuration if required. Default vagrant configuration is to use vagrant user with
insecure_private_key key [default location is
%USERPROFILE%\.vagrant.d\insecure_private_key], or use root or vagrant users with
vagrant as password.

Default IP and port for vagrant VM are 127.0.0.1 and 2222 respectively. For further details on
vagrant configurations related to multiple VMs and ports, kindly refer the vagrant
documentation.

The easiest way to login to a Test Kitchen instance is to just run kitchen login [machine name].
In the kitchen-docs cookbook, that would be accomplished by running kitchen login default-
centos-66 after the VM has been created. This command will allow you to SSH into the test

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com

12
 machine as the Vagrant user on port 2222. Test Kitchen is already aware of the location of the
vagrant private key, so no additional commands are necessary.

Unit Testing with ChefSpec

The Chef Team wrote a very useful RubyGem called ChefSpec which can be used in
conjunction with the Fauxhai gem to mock out and unit test your code prior to committing to Git.
This is extremely useful when preparing a cookbook that should work on multiple versions of
operating systems (RedHat 5.10 and 6.5 for example). The Fauxhai gem allows you to mock
out different attributes before running a test such that the test is executed against multiple fake
Ohai data sets (representing different deployment platforms).

Dependencies
ChefSpec requires the ChefDK in order to run correctly, and it also requires the Ruby DevKit.
The DevKit contains Ruby Development tools including C compilers and Make, which allows
you to build native Gem extensions locally. I extracted the Ruby DevKit to C:\DevKit and then
configured my C:\DevKit\config.yml to point to the Vagrant Ruby installation as follows:

You must have the Chef Development Kit installed and your $PATH variable must include the
following IN THIS ORDER:
1. C:\opscode\chefdk\bin
2. C:\opscode\chefdk\embedded\bin

Once these environment variables are set, you should continue the below instructions.
The kitchen-docs cookbook includes multiple example ChefSpec unit tests (note: these are
rudimentary tests and should not be used as a catch-all).

Heres an example of a ChefSpec test that verifies that your code will correctly create the
directory /root/.ssh and the file /root/.ssh/known_hosts:

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com

13
 To execute the ChefSpec tests, make sure that you have a Gemfile in the kitchen-docs root
directory with the following contents:

Then, make sure you have the Bundler gem installed and navigate to the kitchen-docs directory
in your command line (terminal on Mac or Command Prompt on Windows) and run the
command rspec. This will execute all tests specified in the default_spec.rb file and post the
verbose outputs into STDOUT.

Integration Testing with BATS/ServerSpec

Unlike unit testing, which tests local code without actually executing any tests against a
configured system, the Bash Automated Testing System (BATS) and ServerSpec frameworks
can be used to write tests that verify the end state of a target system. BATS is easier for those
familiar with Bash, while ServerSpec tests are written in Ruby and play very nicely with Chef
resources. Both BATS and ServerSpec tests can be executed as part of the Test Kitchen test,
or verify commands as long as the tests exist in the expected directory within the cookbook
under test. That directory structure is as follows:

~/chef-repo/cookbooks/kitchen-docs/test/integration/kitchen/bats/some_test.bats
~/chef-repo/cookbooks/kitchen-
docs/test/integration/kitchen/serverspec/some_test_spec.rb
BATS
BATS is a method for testing the end state of target nodes by running bash commands and
expecting certain exit codes (generally 0, for success). Here is an example BATS test file:

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com

14
 In order to verify the end state of a server with BATS, we need to run bash commands that
return exit codes of 0 when the commands are successful. The default logic for BATS assumes
an exit code of 0 is success, so the Nokogiri install test from line 18-20 is actually more succinct.
However, if your test is supposed to return a non-zero exit code, you should use the logic
above. Below is an example of a test suite executing:

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com

15
 ServerSpec
ServerSpec tests must end with _spec.rb, while BATS tests must end with .bats. There must
also be a spec_helper.rb file in ~/test/integration/kitchen/serverspec/spec_helper.rb for
ServerSpec tests to appropriately execute.

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com

16
 A good spec_helper.rb file for ServerSpec is below:

The c.path variable is setting the environment PATH variable on the remote system so that
Linux commands can be executed to verify the state of a system and its components. An
example ServerSpec test file is below:

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com

17
 This integration test verifies that your cookbook actually executed what you thought it should
execute. The test framework is very legible, but if you need to look up additional ServerSpec
resources, find them here: http://serverspec.org/resource_types.html
The output of a successful ServerSpec test run (initiated with kitchen verify kitchen-centos-
66) would appear as follows:

2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com

18
 2017 Cloud Technology Partners, Inc. / Confidential / www.cloudtp.com
19