Performance Toolkit for Microsoft Dynamics TM CRM 2011

Overview
The Performance Toolkit for Microsoft DynamicsTM CRM 2011 was created by the Microsoft DynamicsTM CRM performance team to formalize testing of Microsoft DynamicsTM CRM 2011. The Performance Toolkit can be used by the Microsoft DynamicsTM CRM partners and customers to collect data to support their CRM deployment decisions. The toolkit facilitates load testing the performance of Microsoft DynamicsTM CRM 2011 deployments. By carefully planning the required dataset that the deployment needs to support and the workload requirements, the toolkit can be used to test if the scale and performance requirements of a particular deployment can be met. This methodology can be used to help with decisions on a particular deployment solution and avoid costly downtimes at a later stage. The Performance Toolkit for Microsoft DynamicsTM CRM 2011 contains the tools (listed below) that can be used in customizing the CRM installation, populating the necessary semantic data for the deployment that is preferred and conducting the benchmarking tests against the CRM installation. The tools provided in the Performance toolkit are: 1. 2. 3. 4. 5. 6. ImportCustomization Tool UserConfigGen Tool OrgStructureGenerator Tool GoalStructureGenerator Tool DbPopulator Tool CRM_Perf_Benchmark Tool

The Performance Toolkit for Microsoft Dynamics CRM 2011 is licensed under the Microsoft Software License Terms. To read the license terms, go to the documentation in the software. The Performance Toolkit contains source code format, but does not grant any redistribution rights at all. Therefore, you may use, modify and add to the source code identified in the CRM_Perf_Toolkit directory for your internal use only. Knowledge of Microsoft Visual Studio 2010 Ultimate and Microsoft Visual C# is required to use the toolkit effectively. The following are the pre-requisites to conduct the performance tests and are not included with the Performance Toolkit. Microsoft Visual Studio 2010 Ultimate running on a dedicated client system used to drive the CRM product. A dedicated workstation with 4GB RAM is sufficient to run the benchmarking tests for smaller deployments 1-320 users, more RAM is recommended for deployments larger than that. Microsoft SQLExpress 2008 (which is included with Microsoft Visual Studio 2010 Ultimate) or a full version of Microsoft SQL Server is required for the toolkit to install correctly. Microsoft DynamicsTM CRM 2011 deployment suitable for performance testing. Ideally, the hardware configuration of your deployment should resemble how you want the actual production environment be deployed. It is not recommended to run the performance toolkit against an actual production environment.

Performance Toolkit Components

The toolkit consists of following tools:

1 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

ImportCustomization Tool
This is a tool that uses the Microsoft DynamicsTM CRM 2011 API calls to import and publish customizations that can be provided using a customization xml file.

UserConfigGen Tool
This is a tool that takes in parameters about the CRM environment as well as the users that will be used to run the benchmark, it outputs an xml file that is formatted so that DBPopulator can use it to populate users in Microsoft DynamicsTM CRM 2011.

OrgStructureGenerator Tool
This is a tool that takes in parameters about the CRM environment as well as a sector start and sector count value to build a predetermined (configurable within code) organization structure for role based deployments, it outputs an xml file that is formatted so that DBPopulator can use it to build out the organization structure in Microsoft DynamicsTM CRM 2011.

GoalStructureGenerator Tool
This is a tool that takes in parameters about the CRM environment to build out predetermined data for the goal tests within the benchmark.

DbPopulator Tool
This is an xml driven tool that uses Microsoft DynamicsTM CRM 2011 API calls to load the CRM deployment system with pre-requisite data for performance testing.

CRM_Perf_Benchmark Tool
This is a tool that can be run in the Visual Studio Test infrastructure that comprises of several scenario driven tests that can be used to test the performance of the Microsoft DynamicsTM CRM deployment under a load simulation.

Setting Up the Test Environment

Pre Requisites
Install Microsoft Visual Studio 2010 Ultimate Install Microsoft Visual Studio 2010 Ultimate according to the Visual Studio installation instructions on a dedicated client computer, this dedicated client computer will be referred to as the Client Machine (CM) going forward. The Microsoft CRM 2011 Performance Toolkit uses Microsoft Visual Studio 2010 Ultimate as the platform for its load tests. This document assumes the user is familiar with using Microsoft Visual Studio 2010 Ultimate. For more information about Microsoft Visual Studio 2010 Ultimate, including documentation, see msdn.microsoft.com/vstudio/teamsystem/default.aspx. The toolkit is configured to run on the sqlexpress database on the client machine which is installed during the Visual Studio Team System installation process. Install Microsoft CRM 2011 Test System Install a Microsoft DynamicsTM CRM 2011 Testing Environment according to your deployment needs using the Microsoft DynamicsTM CRM 2011 Implementation Guide. This guide is available on the Microsoft download site at http://www.microsoft.com/downloads/en/details.aspx?FamilyID=9886ab96-3571-420f-83ad-246899482fb4 2 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

This system will be referred to as the Test Environment (TE) going forward. We recommend that this system be separated from the production environment, that is, it should have its own domain controller, Web server, and database server. Having an isolated Microsoft CRM environment dedicated to performance testing is important for several reasons, including the need for dedicated user accounts, test data, security sensitive information, and to avoid putting unnecessary load on the production environment during testing. Creation of test users in the Test Domain You must create a pool of test users on the domain for use in the performance testing of the Microsoft CRM deployment. Each of these users will be named with a prefix and an index, for instance, in our guide we will have test users named crmusr1 to crmusrN where N is the number of test users that the CRM deployment needs to support. In the test environment it is assumed all the test users have the same password. The following command script can be run to populate domain users crmusr1 to crmusr320 with the password pass, this can be done from any of the machines in the TE, so long as you are logged into the domain as a domain administrator: C:\> for /L %i in (1,1,320) do net user crmusr%i Pass /add /domain Notes The users' passwords must comply with the password complexity policies for your environment. This is one reason we recommend that you use the toolkit in an isolated environment. This script uses default settings for things such as password expiration. For more information, see the help for the net user command. The runpopulation2.cmd (defined later) requires users in blocks of 320 users as it is a pre-defined business unit structure where users have particular CRM roles. So you could have n business units defined in CRM, so for example if I wanted to do performance testing with 2 business units I would want to have 640 users in my domain so when I build out my deployment for performance testing the users will be there for population.

Performance Toolkit for Microsoft DynamicsTM CRM 2011 installation

By completing the pre-requisite steps, the CM should be ready to download and setup the Performance Toolkit. The toolkit comes in an msi package that can be run to install and configure the toolkit to your system without going through any special post setup configurations. The toolkit is available for download at. If all prerequisites are met on the client the msi package will display the following screens.

3 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

This is the initial Welcome Screen click Next

Read and accept if terms of the End-User License Agreement are agreed upon then click Next

4 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

The default is to setup the Toolkit in C:\CRMToolkit Folder. This can be changed by selecting the browse button and selecting the desired folder. After selecting the desired folder to install the toolkit in, click on Next

The next step in the setup process is to select the Toolkits Configuration Files Location. This folder is where the configuration files that are used by the toolkit will be placed. The default is ConfigFiles folder within the install folder selected from the previous screen. This folder can be customized by selecting the browse button. Once the desired Configuration files folder is selected, click on Next. 5 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

In this step, the various servers in your Microsoft DynamicsTM CRM 2011 TE will be selected and will depend on your deployment (see below). The Run as username is the domain user name of the user who deploys the Microsoft DynamicsTM CRM 2011. This user must have the Administrator privileges in the CRM system as well as the domain. An example is provided above. After all the textboxes are filled out, click Next. Metadata Server This is the server where the metadata services reside in. Report Server This is the server where the reporting services reside in. Crm Server This is the server where the web component of the CRM server resides in. Api Server This is server that the api calls are made to. This server handles api calls to the CRM system. Usually the API server can be the same server as CRM server, but it can be separated into two different servers depending on the deployment needs. Discovery Server This server handles the discovery services requests. Sql Server This is the server where the databases reside at. 6 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

In this step, youll need to enter your test organization, user base, user password, user start and count, the percent of Outlook users you want in the system, and the local Visual Studio 2010 Client Machines SQL instance (SQL as well as SQLExpress instances can be used). After all the textboxes are filled out, click Next. OrganizationName This is the name of the organization that is deployed in CRM. The toolkit can be used to test the performance of the CRM deployment of several organizations residing in several different CRM web servers. But the msi setup process creates only one organization in the configsettings.xml file which resides in the Configuration Folder specified above. The rest of the organizations can be specified in the configsettings.xml file directly after the setup process completes. A sample of the xml that gets created by the setup process is shown below: <crmservers> <crmserver> <server value="http://TEweb"/> <discoveryserver value="http://TEweb"/> <apiserver value="http://TEweb"/> <organization name="TEorg"/> <sqlcnn value="Application Name=CRM_Perf_BenchMark;Server=TEsql;Initial Catalog=TEorg_MSCRM;Integrated Security=sspi"/> <configsqlcnn value="Application Name=CRM_Perf_BenchMark;Server=TEsql;Initial Catalog=MSCRM_CONFIG;Integrated Security=sspi"/> <AD> 7 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

<userbase value="crmusr"/> <userpassword value="Pass"/> <start value="1"/> <count value="320"/> <outlookpercentage value="30"/> </AD> </crmserver> </crmservers> If more than one organization is to be setup for the deployment, the above xml can be modified to the following: <crmservers> <crmserver> <server value="http://TEweb"/> <discoveryserver value="http://TEweb"/> <apiserver value="http://TEweb"/> <organization name="testorg"/> <sqlcnn value="Application Name=CRM_Perf_BenchMark;Server=TEsql;Initial Catalog=testorg_MSCRM;Integrated Security=sspi"/> <configsqlcnn value="Application Name=CRM_Perf_BenchMark;Server=TEsql;Initial Catalog=MSCRM_CONFIG;Integrated Security=sspi"/> <AD> <userbase value="crmusr"/> <userpassword value="pass"/> <start value="1"/> <count value="320"/> <outlookstart value="1"/> <outlookcount value="200"/> </AD> </crmserver> <crmserver> <server value="http://TEweb"/> <discoveryserver value="http://TEweb"/> <apiserver value="http://TEweb"/> <organization name="testorg2"/> <sqlcnn value="Application Name=CRM_Perf_BenchMark;Server=TEsql;Initial Catalog=TEorg2_MSCRM;Integrated Security=sspi"/> <configsqlcnn value="Application Name=CRM_Perf_BenchMark;Server=TEsql;Initial Catalog=MSCRM_CONFIG;Integrated Security=sspi"/> <AD> <userbase value="crmusr"/> <userpassword value="Pass"/> <start value="321"/> <count value="640"/> <outlookpercentage value="30"/> </AD> 8 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

</crmserver> </crmservers>

Test Users BaseName This is the base name of the test users. The way the toolkit infrastructure is setup is by having a set of test users who have the same base name with an index and who have the same password. As shown in the form above, the base name is the name created for the users in the pre-requisite step. Test Users Password It is assumed by the toolkit infrastructure that all the test users have the same password which is specified in this section. Test User Start This is the number that specifies the start index of the test users. It should be specified as a number. Test User Count This is the number that specifies the number of users from the start index. If start is 321 and count is 320 then the crmusr321 to crmusr640 will be used in the toolkit. Outlook Percentage This is the percentage of your total users that you want to run Outlook specific tests. It is assumed that outlook users have the same base name as the test users.

9 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

Click Install to install the Performance Toolkit for Microsoft DynamicsTM CRM 2011. The following are the tasks that are completed by the setup process. The source files and executable files are dropped in the folders as follows under the root folder chosen during setup. If the default CRMToolkit folder is chosen during setup and the default ConfigFiles is chosen for the configuration files under the root folder, then the following are the folders created.

The data population tool and the import customization tool binaries are dropped into the Binaries folder. Two batch files called RunPopulation.cmd and RunPopulation2.cmd are also dropped in the binaries folder. o RunPopulation.cmd (SysAdmin Based) will: customize the Microsoft DynamicsTM CRM 2011 deployment with a few sample customizations add the users specified in the setup process to the CRM organization Create a comprehensive dataset for each of the users specified in the setup o RunPopulation2.cmd (Role Based*REQUIRES 320 USERS TO BE SETUP IN AD*) will: customize the Microsoft DynamicsTM CRM 2011 deployment with a few sample customizations add a role based structure of users specified in the setup process to the CRM organization 10 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

Create a comprehensive dataset for each of the users specified in the setup Add a set of basic workflow rules to the deployment The xml files that drive the DbPopulator tool are also configured for the users specified in the setup process Create the source project of the toolkit in CRM_Perf_Toolkit folder. Configure the configsettings.xml in the ConfigFiles folder. Configure the QTAgent.exe.config file in C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE with the right parameters. Creates a registry entry called MSCRMToolkit in HKLM\Software\Wow6432NodeMicrosoft\ key which specifies the folder where the configsettings.xml exists so that the crm_perf_benchmark tool can read the file wherever it is placed. If for some reason the configsettings.xml file location is changed to some other directory then this registry needs to be changed. Creates the EntityManager database in the SQL instance specified during setup on the client.

Post Installation
Data Population After the setup for the CRM Performance Toolkit is completed, the next step before running the performance tests is to create the necessary data. There are a set of sample xml files that drive the dbpopulator tool are configured for the CRM system and placed in the Binaries\DbPopulator folder, as well as a set of sample customization xml files that are also placed in Binaries\ImportCustomization folder. If the users want to get started with these pre-configured sample xml files, which create the data that the Microsoft CRM Performance team has used in their performance testing, they can simply run the RunPopulation.cmd or RunPopulation2.cmd batch file. This batch file, as mentioned above, will: customize the Microsoft DynamicsTM CRM 2011 deployment with a few sample customizations add the users (CRM System Administrator roles if RunPopulation.cmd is used, or CRM Role Based roles if RunPopulation2.cmd is used) specified in the setup process to the CRM organization Create a comprehensive dataset for each of the users specified in the setup.

Warning: It is highly recommended to back up the empty or clean Microsoft DynamicsTM CRM 2011 databases after installation has happened as well as after any customizations have been made, so that it is always easy to revert back to the beginning state of an empty Microsoft DynamicsTM CRM 2011 system if the data populated is not suitable for your needs or as testing requirements change. The data population (when running with the RunPopulation.cmd or RunPopulation2.cmd files) might take a long time, depending on your system setup as well as the setup you are using for your deployment (number of users, role based or system admin based, etc.)(The RunPopulation2.cmd on a good set of hardware with a multiple server CRM 2011 deployment takes around 8-9 hours to complete). The pre-configured xml files create a comprehensive set of data which can grow into Gigabytes in the database. It is advised for the users to read the subsequent parts on creating their own xml files that is more appropriate for their needs. If a smaller dataset is desired an easier alternative is for the users to open these xml files and reduce the counts of entities that are created for each user. This will create a wide variety of entities but reduce the number of entities per user.

11 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

Compile the CRM_Perf_Benchmark tool Once the data has been populated, open the CRM_Perf_Toolkit solution and compile the solution. Once the solution is compiled, the list of tests (each of which is a CRM user action scenario) can be viewed by opening the Test List Editor. Populate the EntityManager Database To keep the tests realistic, it is not advisable for the test infrastructure to touch the CRM databases directly during the test runs. For this reason, all the pre-requisite data used by the performance tests is copied to a local database on the Client from the CRM database servers. This database is called EntityManager database. The EntityManager database is created by the CRM Performance Toolkit setup process. The copying of the data from the CRM databases to the EntityManager database can be done by running the EMDBLoader.exe which is a part of the CRM_Perf_Toolkit solution. Note: After Data Population is done and the EntityManager data is in sync with your CRM 2011 databases. It is highly recommended to back up the EntityManager database on the client. This process ensures that both databases on the client (the system which you will be using to run the Toolkit) as well as the CRM databases are in sync (we refer to these backups as the clean state because no tests have been run yet). For accurate test runs, as well as comparison runs you should be restoring all databases to these clean states for each benchmark that is done.

Running a WebTest With this the user should be ready to run the performance tests. Most configuration and setup problems can be diagnosed by running an individual Web test. To run an individual Web test, do the following: 1. On the Visual Studio toolbar, select Test, point to Windows, and then select Test List Editor. 2. Start dbmon.exe in a dedicated Visual Studio Command Prompt window. 3. In Visual Studio, scroll down and locate the row with CreateNewNoteForAccount in the Test Name column. Select the check box for this row, right-click, and then select Run Checked Tests. 4. Monitor the output using a debugging tool (like dbmon.exe). You should see messages being generated as the data from the EMDB is loaded into memory for use by the Web test. Depending on the size of your test dataset this can take from several seconds to several minutes. 5. Monitor the Test Results window in Visual Studio. You should see a passed result if everything has run correctly. Double-clicking the Test Name in the Test Results window will show the details of the test, and is a starting point for debugging any errors that may have occurred. If errors occurred you may have to debug in order to determine the source of the problem. You can do this by selecting Debug Checked Tests. Run a Loadtest To run a loadtest, do the following: 1. From the Test List Editor window in Visual Studio, select the row for the loadtest you want to runfor example: a. If I had built out a System Administrator based deployment I would select V5_NonRoleBasedLoadTest by putting a check in the checkbox, right-click, and then select Run Checked Tests (make sure V5_NonRoleBasedLoadTest is the only row selected). b. Consequently if I had built out a Role based deployment I would select V5_RoleBasedLoadTest.loadtest by putting a check in the checkbox, right-click, and then select Run Checked Tests (make sure V5_RoleBasedLoadTest is the only row selected). 12 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

2. If you are not automatically presented with the loadtests results window, double-click the loadtest in the Test Results window (On the Visual Studio toolbar, select Test, point to Windows, and then select Test Results). Microsoft Visual Studio 2010 Ultimate lets you monitor the progress of the loadtest while it is running. There are two windows, which you can select by clicking either Graphs or Tables at the top of the window. For detailed information about the Test Result window, see the Visual Studio Team System documentation. The Web tests provided with the toolkit contain explicit transactions around key stages in each business use case scenario. For example, the Web test "CreateNewAccount" contains the transactions "NavAccountHomepage" and "CreateAccount". The performance of these transactions is viewed by clicking Tables and selecting Transactions from the drop-down list in the Load Test Result window. The other tables available in the drop-down list provide different levels of granularity when you view the results. The reference Web tests and Entity Manager that are included with the toolkit have known issues that will cause some percentage of the executed test cases to fail. This number of failed cases can be seen in the Load Test Result window in the Summary pane under the Overall Results section listed by the Test Failed category. An error rate of 1% (defined by Failed Tests / Total Tests * 100) is considered acceptable when running the reference code. Many of these errors come from deficiencies in the Entity Manager interface when dealing with the addition and creation of complex data hierarchies.

Guidelines for configuring your own performance testing

There are essentially three ways for extending the usefulness of the reference datasets and loadtest provided with the toolkit. The first technique is to customize the reference dataset. You can use dbpopulator with custom input files to more accurately reflect a particular customer scenario. The second technique is to modify the reference loadtest to test with different user populations, rate of work, and workload composition. The third technique is to create new Web tests to cover functionality not included with the out of the box Web tests.

The results of experimenting with only the reference dataset and loadtest must be interpreted with care because they reflect the performance of a system running a purely hypothetical workload. Although care was taken to make the dataset and workload in the toolkit meaningful, the results do not guarantee any level of performance for a given customer deployment. By taking the time to gather customer workload specifications and customizing the toolkit to more carefully model those specific requirements, confidence can be established early and potential design and scalability problems can be identified and managed more effectively.

Extend the Dataset and Loadtest

Testing the reference dataset and loadtest for larger user populations can be accomplished by the following steps. The descriptions later in this article assume that you have successfully created the reference dataset and executed the V5_NonRoleBasedLoadTest and that the base name for the test users is "crmusr". (This example uses the NestedSFAEntitiesWithState_Custom.xml, but applies to each xml file, but there are some caveats to think about if modifying the role based xml files (the files with predefined values for the startidx and count nodes)) 1. Back up your Microsoft DynamicsTM CRM 2011 databases, and your Entity Manager Database. 2. Add the users to the domain and to Microsoft DynamicsTM CRM 2011. For more information, see the section "Creation of users in test domain" The steps that follow assume that test users crmusr201 through crmusr400 are being tested. 3. Modify the userGroup name line in the NestedSFAEntitiesWithState_Custom.xml file to populate data for the new users as follows: 13 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

 <userGroup name='crmusr' startidx='201' count='200' password='pass'/> 4. The startidx and count attributes are used to avoid recreating data for users crmusr1 through crmusr200. 5. Use the dbpop tool with the new NestedSFAEntitiesWithState_Custom.xml input file to create data for users crmusr201 through crmusr400. 6. Back up the Microsoft DynamicsTM CRM 2011 databases. (This makes it easier to switch between different datasets). 7. Drop and re-create the EntityManager DB. The Entity Manager has to be re-created when you change the Microsoft CRM dataset. See the section "Populate the Entity Manager Database" earlier in this document. 8. Back up the new EntityManager Database. 9. Modify the configsettings.xml file in ConfigSettings folder. The new values look as follows: <crmserver> <server value="http://TEweb"/> <discoveryserver value="http://TEweb"/> <apiserver value="http://TEweb"/> <organization name="testorg"/> <sqlcnn value="Application Name=CRM_Perf_BenchMark;Server=TEsql;Initial Catalog=testorg_MSCRM;Integrated Security=sspi"/> <configsqlcnn value="Application Name=CRM_Perf_BenchMark;Server=TEsql;Initial Catalog=MSCRM_CONFIG;Integrated Security=sspi"/> <AD> <userbase value="crmusr"/> <userpassword value="pass"/> <start value="1"/> <count value="400"/> <outlookstart value="1"/> <outlookcount value="400"/> </AD>
</crmserver>

1. Adjust the loadtest to use 400 virtual users: 2. If it is not open already, double-click the V5_NonRoleBasedLoadTest.loadtest file in Solution Explorer to open the loadtest designer. 3. Select the node at V5_NonRoleBasedLoadTest, select Scenarios, select OfficialLoad (for example), and then select Step Load Pattern. 4. If the Properties window is not visible, display the Properties by pressing F4 or select Properties Window on the View menu. 5. Change the property Maximum User Count to 400. 6. Run the new loadtest.

14 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

Create Custom Datasets with dbpop

The dbpop tool works by processing an input XML file that describes what and how many Microsoft CRM entities to create, the data that each of those entities contains, and the relationships between those entities. The simplest way of customizing a dataset is to start with the reference datasets and change the user counts and/or the individual entity counts contained in the reference input XML files. If more significant changes are required, custom input XML files can be created. Several atomic input XML files are provided in the CRM_Perf_Toolkit\dbPopulator\Sample_Xmls directory, which can be used to gain a better understanding of the syntax and format of the input XML files, as well as to serve as building blocks for more complex custom input files. An example of a dbpopulator input file is listed below, taken from CRM_Perf_Toolkit\dbPopulator\Sample_Xmls\unitTest_Account.xml: <?xml version="1.0" ?> <DbPopConfig> <exec threadCount="1" /> <userGroups domain="domainname" webserver="webservername" organization=orgName> <userGroup name="crmusr" startidx="1" count="1"password="pass"/> </userGroups> <objectDist> <account count="10" save_1="settag(accountid,account,entityid)" save_2="settype(accountidtype)"> </account> </objectDist> </DbPopConfig> The key elements and attributes are: exec This is Childless element that controls various execution aspects of dbpop. This element currently supports only the threadCount attribute that sets the number of concurrent threads that dbpop will use to populate the specified data. userGroups Container element used to specify information about the TE domain, Web server and organization name. The supported attributes are domain, webserver and organization that should be set to reflect the TE domain name and Web server name, respectively. userGroup It is Child element of the userGroups element that is used to specify a group of test users for which to create the test data. Attributes of the userGroup element are name, startidx, count, and password. The name attribute defines the base name for the test users. For example, if you have created a test user pool with users named crmusr1 through crmusr200, the name attribute should be 'crmusr'. For more information, see the previous section titled "Create Test Users". The startidx and count are optional attributes that control how many test users to create data for. If data needs to be populated only for one user without an index such as administrator then name will be administrator and startidx and count can be omitted. Some examples of various settings for name, startidx, and count are as follows: The following setting populates data for crmusr1. name='crmusr' startidx='1' count='1' password=pass The following setting populates data for crmusr1 through crmusr100, inclusive. name='crmusr' startidx='1' count='100' password=pass 15 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

The following setting populates data for crmtestuser100 through crmtestuser250, inclusive. name='crmtestuser' startidx='100' count='150' password=pass The following setting populates data for administrator name='administrator' password=pass objectDist This is the container element that describes the entities to be populated. A full description of the supported child elements and syntax for the objectDist element is outside the scope of this document and should be explored through code inspection and debugging if you want to customize dbpopulator and dbpopulator input files. Children of the objectDist element represent the Microsoft CRM entities to be created. Each entity you want to create should have a representative element under the objectDist element. For a complete list of the Microsoft CRM entities currently implemented by the dbpop tool, see the switch statement inside the dbPopulator.CrmEntityTemplate.GetCrmEntityTemplate function in dbpopulator.cs. Attributes are used to control which fields of the entity are assigned to and their values. Generally, you can add any field that exists in the entity as an attribute to the entity element in the dbpop XML. These attributes can be in addition to or can override the entity fields that dbpop creates by default. To determine which fields dbpop populates by default for a given entity, examine the m_entityXmlString member in the classes derived from CrmEntityTemplate. Internally, reflection is used to map the field name to the appropriate member of the SDK object, so that your attribute name should exactly match the field name in the SDK object (see CRMService.cs). Multipart names are also supported, for example, the regardingobjectid.value field can be set on an appointment element. The attributes on the elements that are descendents of the objectdist node, and the entity fields specified in the m_entityXmlString member variables, control which entities are created, the number of those entities created, and the entity field values themselves. Note that attributes specified in the input XML file will override any default values set in the m_entityXmlString variables. There are four categories of supported attributes/attribute values, as follows: Meta values (attribute): The only supported meta attribute is count, which controls the number of entities created for this element. <account count="10" /> Text values (attribute value): If you want a given entity field to have a fixed value, you can assign the attribute a fixed string. <account creditonhold="false" /> Replacement functions (attribute value): The dbpop tool implements a variety of replacement functions that are evaluated at runtime to allow for dynamically created data. A common use of a replacement function is to insert random strings into the entities being created to provide a distribution of values over the entity field. Replacement functions have the format: replace(replacement function, optional replacement function param, ). <account name="replace(randomstring,15,40)" /> Note that in the case of the account.name field, dbpop supplies this information by default in dbPopulator.CrmAccountTemplate.m_entityXmlString: <account> <name>replace(randomstring,10,50)</name> 16 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

</account> In this case the input file overrides the default value, and account names created by this input XML are random strings between 15 and 40 characters long. For a complete list of currently dbPopulator.ReplacementFunctions. implemented replacement functions, see the methods in

Tag functions (attribute value): In order to create realistic datasets it is frequently necessary to pass information from one created entity to another. For example, to create an Annotation entity that is the child of an Account entity, the ID of the parent Account entity is needed when you create the Annotation. Tags and their associated tag functions are the mechanism by which dbpop allows data to be passed from one entity to another. Tags can be either global or dynamic, and are either set or retrieved. Global Tags: Global tags are created by using the globalTags, tag, query, column, and entity elements in the dbpop input file, and occur as descendents of the dbPopConfig root element. The following example is taken from the CRM_Perf_Toolkit\dbPopulator\Sample_Xmls\unitTest_Appointment.xml file:

<dbPopConfig> <globalTags> <tag name="allaccountids"> <query> <column name="accountid" /> <entity name="account" /> </query> </tag> </dbPopConfig> This XML fragment creates a tag named allaccountids that contains a collection of the account.accountid values in the system that are visible to the user running the dbpop tool. Global tags are implemented internally using the standard SDK query mechanisms (such as QueryExpression and ColumnSet). For more information, see the function dbPopulator.TagManager.Retrieve in the dbPopulator.cs file. Global tag processing occurs before any entities are created, in other words, they reflect only data that exists in the system before running dbpop. Dynamic Tags: Dynamic tags are created by using save_ attributes on the elements in the objectDist hierarchy and either the settag or the settype functions. The following example, taken from the CRM_Perf_Toolkit\dbPopulator\Sample_Xmls\unitTest_Appointment.xml file, demonstrates setting dynamic tags:

<objectDist> <account count="1" save_1="settag(accountid,account,entityid)" save_2="settype(accountidtype)" </account>

17 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

This XML fragment creates two dynamic tags: accountid and accountidtype. The settag function has the following format: save_N="settag(tagname,entity,field)" where tagname is the name of the tag that must be used by the retrieve functions, entity is the name of the entity, and field is the name of field that will have its value stored in the tag. Any SDK visible field on entity is allowed, as well as the special fields all and entityid. The all field retrieves all fields for the entity, and the entityid field records the unique identifier of the newly created entity. The entityid field exists as a special case because it is the only value for field that does not require an extra round trip to the server to retrieve (its value is returned by the SDK when the create method is called). This has important performance considerations - setting dynamic tags other than entityid will slow down creation considerably. The settype function has the following format: save_N="setttype(tagname)" Frequently, when you create entities with the SDK it is necessary to explicitly specify the type for a particular field when there is ambiguity as to what the value in a particular field represents. For example, a customerid field may represent either an account or a contact entity. For an example of how the settag and settype functions are used together with a customerid attribute, see the XML code in the file located at

CrmDbPopulate\dbPopulator\TU_XML\unitTest_SalesOrder.xml. Retrieving Tags: Tags are retrieved by assigning the gettag or getdyntag functions to the desired attributes. For an example of using tags, consider the following fragment taken from CRM_Perf_Toolkit\dbPopulator\Sample_Xmls\unitTest_Appointment.xml:

<appointment count="1" requiredattendees="gettag(allsystemuserids,getrandom,systemuserid.Value)" regardingobjectid.Value="getdyntag(accountid)" regardingobjectid.type="getdyntag(accountidtype)" /> In order to create a robust appointment entity it is required to specify an attendee and a regarding object. Tags allow us to dynamically supply all this information: Requiredattendees='gettag()': Here we reference the global tag "allsystemuserids" (see the previous code sample, which created a global tag for accountids for an example of creating a global tag) and select a random userid from that collection. Regardingobjectid.Value='getdyntag(accountid)': Here we reference the entityid stored when the parent account entity is created (see the previous code sample). When only a tag name is specified for the getdyntag function it is shorthand for retrieving the fast path entityid tag. 18 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

Regardingobjectid.type='getdyntag(accountidtype)': regardingobjectid is another example of a situation where the type of the entity referred to by regardingobjectid.Value is ambiguous. Here we specify explicitly that the type referred to by regardingobjectid.Value is an account.

In summary, tag retrieval is controlled with the following functions: gettag/getdyntag(tagname,retrieval_function,retrieval_field,filter_value,filter_value_map) retrieves from the tag tagname. The gettag function references global tags and the getdyntag function references dynamic tags. The getdyntag function has a special shorthand notation of getdyntag(tagname) that retrieves the fast path saved entityid. retrieval_function specifies either getrandom or getfiltered. The getrandom function selects a random entity from the collection in tagname. The getfiltered function is the most complex form of tag retrieval and allows selecting a particular entity from the collection represented by tagname. retrieval_field specifies the precise field to retrieve from the tag collection. Use dot notation for multilevel fields, for example, systemuserid.Value. Filter_value and filter_value_map control, if retrieval_function is getfiltered, the filter value used to specify a particular entity out of the tag collection. Filter_value specifies the field on the in-flight entity to use as the filter when selecting from the tag collection, and filter_value_map handles the scenario where the field name on the inflight entity differs from the field name on the tag entity. CRM_Perf_Toolkit\dbPopulator\Sample_Xmls\unitTest_Product.xml has an example of using all the filter parameters.

The ordering and nesting of objectDist child elements is significant. Consider the following sample objectDist hierarchy: <objectDist> <account save_1="settag(accountidtag,account,entityid)" save_2="settype(accountidtypetag)" count="2"> <contact parentcustomerid.Value="getdyntag(accountidtag)" parentcustomerid.type="getdyntag(accountidtypetag)" count="3" /> </account> </objectdist> This input ultimately creates eight entities. The above XML can be interpreted as "create two accounts, and for each of those accounts, create three contacts." Ultimately, two accounts, A1 and A2, and six contacts, C1 through C6, are created in the following order: A1 is created, accountidtag receives the fast path system id of A1, and accountidtypetag contains the type account. C1, C2, and C3 are created, and getdyntag(accountidtag) returns the system id for A1. getdyntag(accountidtypetag) returns the account type. A2 is created, and accountidtag receives A2's system id (so the tag accountidtag now contains the system ids of both A1 and A2 this is significant when dealing with retrieval functions such as getrandom). Because settype overwrites, instead of appending, its tag, accountidtypetag is overwritten with the account type. C4, C5, and C6 are created, and getdyntag(accountidtag) returns the ID for A2. In the absence of a retrieval function, the get*tag functions use the last entity in the tag collection to provide the tag values. The most common state data that is passed using tags is entityid, which is typically is used to facilitate parent-child relationships. The tag functions are optimized to handle this case; both through the fast path entityid argument to settag, and through the corresponding retrieval using getdyntag(tagname), which always retrieves the ID of the last entity that was created by using settag(tagname,entity,entityid). When combined with nesting, this provides a 19 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

straightforward syntax for expressing a set of objects and their parent-child relationships. The previous XML fragment can be re-written as: <objectDist> <account save_1="settag(accountidtag,account,accountid)" save_2="settype(accountidtypetag)" count="2" /> <contact parentcustomerid.Value="getdyntag(accountidtag,getrandom,accountid.value)" parentcustomerid.type="getdyntag(accountidtypetag)" count="6" /> </objectdist> This XML also creates eight entities, two accounts (A1 and A2), and six contacts (C1 through C6), but with the following differences compared to the previous example: The order is different; in this example the order of creation is A1, A2, C1, C2, C3, C4, C5, C6 Processing of accountidtag is significantly different. Even though both this example and the one before it record the IDs of the accounts that are being created, this example runs a comparatively expensive round trip to the server to retrieve the account.accountid field. The getdyntag function must also be changed to use a retrieval function, and even then you may end up with an uneven distribution of C1 through C6 across A1 and A2. If you used the same format for the tag processing as the previous example, that is using entityid in the settag function and only the tagname with getdyntag, C1 through C6 would all have A2 as their parent, because getdyntag(tagname) only retrieves the last entity in a given tag collection.

Customize the Loadtest

Developing a custom loadtest starts with gathering information about the number of users, the data that each user will own, the business use cases that each user will run, and the frequency at which those use cases occur. The section "Creating Custom Datasets with dbpopulator" earlier in this document serves as a starting point for developing a customized dataset. Ultimately, the specifications gathered from the customer must be able to be normalized down to the entities that a given user should own in the system and the relationships between those entities. If there are distinct personas in the customer specification with different data requirements, create a dbpop input XML file for each persona. Partition the test user pool into the different personas (for example, Persona1 = crmusr1 through crmusr100, Persona2 = crmusr101 through crmusr120, and so on), make sure that the userGroup element in each dbpop input file specifies the correct test users for that persona, and run dbpop one time for each input file/persona. Next, determine the set of business use cases that apply to the customer scenario. The Web tests supplied with the toolkit cover a broad range of business use cases, and can be reused or modified as required. If you must create a new Web test, start by running the business use case in Microsoft CRM while recording the http dialog that occurs between the client (browser) and the server. Microsoft Visual Studio 2010 Ultimate can be used to monitor the http traffic between the server running Microsoft CRM and the client. See the Visual Studio documentation for more detailed information. If you use do the recording facilities integrated with Microsoft Visual Studio 2010 Ultimate, you should first change the following element in the Web.config file on the CRM TE from its default value of "On" to Off: <configuration> <appSettings> 20 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

<add key="AppMode" value="Off"/> Add your new Web test to Microsoft Visual Studio Team System 2005 Solution Explorer by right-clicking CRM_Perf_Benchmark, and then select Add Web Test. One limitation of using the Visual Studio infrastructure to record new Web tests for Microsoft CRM is that web service calls are not handled properly by the current version of the Visual Studio recording tools. Because of this, other HTTP debugging proxies may be necessary. A popular and freely available HTTP debugging tool is Fiddler, available at www.fiddler2.com. Regardless of what method you use to determine the stream of requests that constitute a business use case scenario, the following topics should be considered: Dependent requests Think times Web service calls Selecting a Microsoft CRM user for the test Using the CrmRequest class Adjusting loadtest parameters

Dependent Requests A common concept in any Web application is that of dependent requests. A request is issued to the server for some page, such as an .aspx or .html page, and the response contains nested requests that must be processed in order to render the page. The dependent requests can be for static content, such as images, or they can be for dynamic pages, such as frame targets. The HTTP protocol allows for static content to be cached so that the browser can retrieve the file without additional trips to the server. There are two settings in Microsoft Visual Studio 2010 Ultimate that affect dependent request processing: The property Percentage of New Users for a scenario in the load test designer (that is, select V5_NonRoleBasedLoadTest, select Scenarios, select OfficialLoad, select Step Load Pattern and then select Properties). The property ParseDependentRequests on the WebTestRequest object.

All the requests generated by the Web tests in the toolkit derive from a base class named CrmRequest in Util.cs. CrmRequest itself derives from WebTestRequest, and, in the CrmRequest constructor, the value of WebTestRequest.ParseDependentRequests is set to false. Therefore, the Web tests that are shipped as part of the toolkit intentionally prevent Microsoft Visual Studio 2010 Ultimate from automatically issuing the dependent requests, thus simulating an environment where all users have their static content cached. If you want to follow the example set by the Web tests in the toolkit when you develop your own Web tests, you should issue your server requests by using a CrmRequest object, and issue an explicit request for all the dynamic (.aspx or .asmx) requests recorded during the execution of your business use case. If you choose instead to record your business use case scenario with the integrated Microsoft Visual Studio 2010 Ultimate facilities, you should use the integrated dependent request processing. Furthermore, if you use the integrated dependent request processing, you should experiment with different settings for the Percentage of New Users property discussed previously. This property controls whether dependent requests for static content are sent to the server; in other words, whether or not the simulated user has seen this page before, and would therefore have static content cached.

21 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

Think Times Think times are fundamental to load testing and represent pauses between actions that occur when a user is interacting with the system. Think times can occur with almost any granularity depending on what is being tested. For example, if you were load testing a keyboard driver, you would be concerned with the think time that occurs between keystrokes. In the toolkit there are two logical places where think time can occur: Between the individual requests that consist of a business use case scenario; that is, between the individual requests in a Web test. Between successive executions of a business use case.

The Web tests implemented by the toolkit only contain think time between successive executions of a business use case. This is also used to control the frequency at which business use cases are run. If you use the integrated recording functionality to record a business use case scenario, pauses between requests in your recording will be included as think times on the WebTestRequest object generated by the recorder. This will slow down the request stream and present a lighter load to the TE compared to a business use case scenario without think times set on the WebTestRequest object. The section below "Adjusting Loadtest Parameters" has more information about adjusting think times. Web Service Calls Microsoft CRM 2011 makes substantial use of Web service calls during routine processing. This is a problem working with Microsoft Visual Studio 2010 Ultimate, which does not natively support load testing with Web service calls. For example, if you use the integrated recording functionality in Microsoft Visual Studio 2010 Ultimate, the resulting request stream will not contain any Web service calls even though Web service calls most likely occurred during the recording. Therefore, if you decide to use the integrated recording tools, we strongly recommend that you cross check the recorded request stream with the request stream recorded by another application, such as Fiddler (www.fiddler2.com), for the same sequence of actions. If you find that you must insert Web service calls into the request stream generated by the integrated recording tools, you can use some facilities provided by the toolkit. To insert Web service calls, you will have to convert the recorded Web test to a coded Web test. Do this by right-clicking the root node of your Web test in the Web test designer (for example, WebTest1), and selecting Generate Code. Give the coded Web test a name and then open the new .cs file in Visual Studio. Selecting a Microsoft CRM User The requests generated by the integrated tools do not perform impersonation; in other words, the requests will present themselves to the server using the credentials of the logged on user. It is important that the Web tests impersonate one of the designated test users (that is, crmusrN). To do this, add the following code to the new class, replacing ClassName with the name of the new class: Public void ClassName_PreWebTest (object sender, PreWebTestEventArgs e) { user = EntityManager.Instance.GetNextUser(); e.WebTest.UserName = user["domainname"]; e.WebTest.Password = user["userpassword"]; }

Using the CrmRequest Class The toolkit implements a CrmRequest class and several subclasses, including a subclass for most of the URLs used by the loadtest. For a complete list of the URLs, see the file CRM_Perf_Toolkit\CRM_Perf_BenchMark\Utilities\URLs.cs. These classes make it easier to generate HTTP requests with the proper query string and form parameters.

22 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

The classes implemented have the advantage of having member variables for the generally used QueryString parameters that simplifies their usage. Using one of the classes derived from CrmRequest also has the advantage of providing basic response validation. Note that using one of the CrmRequest derived classes will set ParseDependentRequests to false for the request. The toolkit is included with WSDL.exe generated proxy code for the various Web services it uses in the CRM_Perf_Toolkit\CRM_Perf_BenchMark\WSDL directory. In order to call one of the Web services in your custom coded Web test, first enable the toolkit's hooks into the .NET Framework's SOAP infrastructure in the constructor of your class. This step, together with the edits to vstesthost.exe.config called out earlier in the document, allows the toolkit to take advantage of as much of the .NET Framework's SOAP infrastructure as possible: public ClassName () { WebRequest.RegisterPrefix("crm", new crmRequestFactory()); } Next, instantiate the appropriate proxy object and create a WebTestRequest variable to hold the generated request, as shown in this example taken from CreateNewAccount.cs: AppGridWebService.AppGridWebService agws = new AppGridWebService.AppGridWebService(); WebTestRequest wtrRefresh = null; Finally, call the desired Web service method, for example, Refresh, inside a try{}catch{} block to "hijack" the serialized SOAP request from the .NET Framework, and assign it to the WebTestRequest variable you created earlier: try { agws.Refresh(GridXmlHelper.AccountPage()); } catch (crmException x) {
wtrRefresh = x.wtr; wtrRefresh.Url = Utils.ReplaceServerName(wtrRefresh.Url, user); string urlPath = wtrRefresh.Url.Remove(0, crmServer.Length); urlPath = urlPath.Remove(0, user["organizationName"].Length + 1);

} yield return (wtrRefresh); With the new infrastructure in CRM 2011 where the organization name is included in the URL path, the toolkit provides infrastructure to add the organization name to the URL before a request is made as shown above. You can review the code in Util.cs for the crmRequestFactory, CrmRequest, and crmrequest (CrmRequest and crmrequest are separate classes) for insight into how this process is implemented. Note also the parameter passed to the Refresh method. Many of the Web service methods take complex parameters, and several helper classes are implemented by the toolkit to help call the various Web service methods. Review the Web tests shipped with the toolkit for more examples. Adjusting Loadtest Parameters The final step in creating a customized load test is to determine the frequency at which, and the relative weighting of, the Web tests that implement your business use case scenarios. Microsoft Visual Studio 2010 Ultimate needs two pieces of information to run the load test according to your specifications: 23 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT

The average time between successive Web test executions. The relative weightings of the Web tests to be executed.

It is common to express these specifications in terms of executions per user per hour. This format also lends itself to a straightforward translation into the loadtest parameters. Consider the following hypothetical workload specification: Use Case CreateNewAccount Lookup Account Lookup Appointments Executions per user per hour 1 4 5

To use this format with a Microsoft Visual Studio 2010 Ultimate loadtest, do the following: 1. Calculate the total number of operations carried out in the base time unit, in this case one hour: 1 + 4 + 5 = 10. 2. Assume that the test cases themselves take zero time, and then determine the time between successive operations in seconds so that 10 complete in one hour: 3600 seconds / 10 operations = 360 seconds per operation. 3. Determine the relative weighting of each of the operations: CreateNewAccount = 10% (1/10), LookupAccount = 40% (4/10), and Lookup Appointment = 50% (5/10). The number of seconds calculated in step 2, together with the relative weights calculated in step 3 are the variables needed to create the loadtest. The assumption made in step 2 that the test cases themselves take zero time will ultimately mean that slightly less than 10 operations will complete in an hour. But this deviation should be negligible and not have a noticeable effect on the overall loadtest. The time calculated in step 2 is entered into the property Think Time Between Test Iterations in the Properties dialog for the scenario in the load test designer window. You should also verify that the value for the property Think Profile is set to Normal Distribution on the same property page. This will introduce some variance into the think times between successive Web test executions to prevent the users from all working in lockstep. If you see sawtooth or square wave utilizations on the TE, check to ensure think times are being distributed and that there is also sufficient time between adding users in the step load profile. The weights calculated in step 3 are entered into the loadtest through the designer by right-clicking Test Mix node under the scenario in the designer main window, and then selecting Add Tests or Edit Test Mix as appropriate.

Known Issues and Resolutions

24 MICROSOFT CRM 2011 PERFORMANCE TOOLKIT