At Salesforce, performance is a huge part of application development.

From the
beginning, we knew that in order to be successful and still maintain aggressive
release cycles, we had to make performance analysis a core part of our
development lifecycle.

As a senior performance engineer here, I am deeply familiar with the scale and
performance considerations we make at every step in the feature development
process. So, how does Salesforce maintain seamless performance while
delivering multiple releases a year packed with new functionality? Read on.

1. We collaborate early and often

One way we ensure performance is considered throughout is by assigning a

dedicated performance team to every app feature scrum team. During each
release, a performance engineer works with their designated scrum team to
capture and prioritize all the performance related tasks that need to be
accomplished during the release. They provide early and actionable feedback to
the developers on any performance issues such as reviewing architecture design,
proof-of-concept tests on different implementation options, etc. This helps to
catch and fix any bugs or regressions before the feature ever hits production.

By educating teams on performance best practices, and providing self-service

tools to easily measure performance data, we have managed to greatly streamline
the performance analysis process. With the daily transactions running into
billions, our performance engineers worked diligently to optimize software code
release over release. Along with our infrastructure/capacity increase, we have
managed to keep our perf trust within the 300ms range.

To empower developers to find front-end performance issues as part of their

development cycle, we developed powerful visualization (e.g. waterfall charts
that shows dependencies, duration of an action, how much time is spent waiting
to process JS) and integrated those visualizations with the chrome developer
tool. This allows the developer to correlate the network timeline in the chrome
developer tool with the aura action characteristics of a page. By enabling
developers to easily measure the performance of the features that they are
currently building, performance engineers get to solve for the more complex
performance issues that require a deeper level of attention and analysis.
 2. Automate our regression analysis using workloads

Workloads help simulate the functionality to be tested by generating and loading

synthetic data shapes and volumes in a production-like test environment based
on expected production traffic. Workloads are huge here at Salesforce. We use
them for testing any regressions between the releases as well as measuring
stability over time.

3. Stress and stability test for peak loads

At Salesforce, our customersparticularly

our biggest enterprise customers
are constantly pushing our platform in new ways. It becomes absolutely critical
for us to be able to scale and perform well at the highest levels of expected
volumes. My teamEnterprise
Scaleworks
with some of our top strategic
customers to come up with a workload that simulates a day in the life of the
customer. It takes into account their data shape, data volumes, peak traffic
volumes, and skews. We then analyze the workload results for performance and
stability, and work with individual feature teams to file any bugs to deliver any
improvements that were uncovered. We also work with our product teams to
come up with roadmap items related to any gaps in functionality with respect to
scale or performance.

4. Monitor, analyze and react

Regardless of how much testing or analysis we do up front, there is almost

always something that goes unnoticed. Thats where some of our monitoring
tools come in handy. These uncover trends in our page load times and various
other performance metrics, which we can analyze and use to predict performance
bottlenecks. We also loop back things weve learned from customer production
cases to enhance our performance and scalability test coverage.

For many major industries like finance, healthcare, or retail, every second
matters. An increase in speed can directly help increase the revenuefor

example, by increasing the number of healthcare claims processed or increasing
the number of items sold and shipped. On the other hand, a frustrated customer
experiencing bad performance will result in poor adoption. We know that
performance is a core feature that our customers have come to expect out of our
platform. Hence performance engineering at Salesforce involves collaboration,
architecture design, test plans, analysis, visualization, workloads, monitoring and
much more.