Tuesday, February 28, 2012

3 Key Load Testing Tips

1. Test against your production environment.
I told this to an "expert" at the Association of Software Testing conference this year. He response was something like, "Are you kidding me?!! You'd have to be stupid to load test your production setup."

What occurred to me later in the conversation was that this guy works for a Fortune 100 company and has a six-figure annual budget (not counting his salary) just for performance engineering.

So what?! I still say you should test against production. And so would some others in our industry that have much more experience than I. For example, Dan Bartow taught a couple of sessions in April at STPCon in San Mateo. He mentioned several times that hammering the real production servers is the only way.

This is controversial, but I believe it. It is rarely possible to replicate your production setup - there are too many components, configurations, and network settings that can affect the results of your test. One small variable can potentially affect performance measurements by 100% or more.

Skeptical? Just try turning on/off caching. Or try running tests against the same environment except that the version of Apache or IIS is different. You will be surprised at the performance deltas. Isn't it easy to overlook the size of MySQL buffers or perhaps PHP options or other configuration selections? All of those variables in the application engineering equation can and will affect your outcomes.

So, if you load test against anything other than your production, you are always going to have doubt about the true performance of your system. There is only one way to know for sure.

2. Accurately simulate user types.
It is easy to hit your site with random heavy traffic, but that may not give you precise load test results.

Application performance can vary significantly depending on the actions of the users. For example, most CMS apps get great optimization from caching pages for anonymous users and practically none from authenticated users. Therefore, you must set up your test plans to include scenarios from both types of users.

Let's say your site normally gets 70% of its traffic from prospects just browsing your product catalog, and another 25% of customers are putting items in their shopping cart, while 5% are posting reviews. Your load testing should include three scenarios to reflect each type of user.

Additionally, you should adjust the weighting of each scenario to match the 70/25/5 percent split between the user types. That way, you will be applying load in the correct ways and massaging the code/servers/database in realistic proportions. How else can you get a true picture of performance for analysis and tuning?

3. Focus on the right metrics.
It is possible to get bogged down in all the data available to you. The exact metrics on which you concentrate may vary depending on the aspect of your web application that is being testing.

However, we recommend you always direct your attention to:

  • Response times
  • Error rates
  • Throughput
  • Active users
  • Requests per second
You may also need to measure business metrics for your managers. For example, your VP of Marketing may want to know the number of orders processed per minute. It's a bad idea to leave out the project stakeholders when deciding which metrics you need to track. We recommend that you ask them to make a list of questions for which they need answers and set the benchmark for each metric necessary to provide a useful measurement to satisfy the questions. Be sure to avoid vague questions you won't be able to test.

Ask the stakeholders if they can provide historical data or expectations regarding the metrics - this can be a big help in establishing the acceptable levels of performance from everyone's perspective. If you have historical load testing data, all the better. Share it with them.

Open communication with your business leaders goes a long way to knowing the correct metrics to track. Focusing on the right metrics is critical because you can't control what you don't measure.

Bookmark/Search this post with:

Monday, February 27, 2012

Top 10 Favorite Performance Counters in Web Site Load Testing Using VSTS 2008

Microsoft Visual Studio Team System/Test Edition provides an excellent tool to perform web site load testing. Using this load testing tool, you can monitor and measure the site performance along with system status with respect to a given load/stress.

Fortunately VSTS provides a support for wide range of performance counters, from web page request per second to condition of physical disk, memories. Unfortunately, they are too huge that, initially testers/designers get overwhelmed with all of those, to find out a clear idea about the performance of the site they built.

The number of counter parameters to be considered by the load tester/designers is greatly varies based on the type and size of the web application to be tested. Here is my favorite top 10 performance counters that I use on my each load tests, regardless of project size. These counters are based two primary categories: Web Site end and Hardware end.

Web Site Related Performance Counters

Web site related performance counters are the counters that provide valuable information about the health of web site that is under test. These parameters are categorized as Requests, Pages, Tests and Errors.

1. Request - Avg Req/Sec

Desired value range: High

This is the average number of requests per second, which includes failed and passed requests, but not cached requests, because they are not issued on web server. Please note that, all http requests, such as image, java-script, aspx, html files generates separate/individual/single request .

2. Request - Avg Req Passed/Sec

Desired value range: High

While “Request - Avg Req/Sec” provides an average with respect to all passed and failed request, “Request - Avg Req Passed/Sec” provided the average of passed requests. This info also helps to determine the average number of failed requests/sec.

3. Page - Avg Page Time (Sec)

Desired value range: Low

While a single request refers to request to a single http elements (such as css, java-script files, images, aspx, html etc), a page is the container of all of the corresponding requests generated when a web page is requested (for instance via the browser address bar). “Page - Avg Page Time (Sec)” counter refers to the average of total time taken to load a page with all of its http elements.

4. Test - Total Test

Desired value range: High

For instance, we have created a web test, that contains two web pages, pushing on a button on the first page will re-direct the user to the second page, although there will be multiple entries will be involved for Requests and Pages counters, but the whole process will be considered as a single Test.

This counter considers the total number of tests (which includes passed and failed tests) during the test period.

5. Scenario - User Load

Desired value range: High

This counter considers the maximum user load that has been provided during the test run. Please note that, for Step Load pattern, where more user volume is added on step by step basis, the maximum user load will be counted through this counter parameter.

6. Errors - Errors/Sec

Desired value range: Low

Includes average number of errors occurred per second, which includes all types of errors.


Hardware Related Performance Counters

7. Processor - % Processor Time

Desired value range: Low

This is the number of processor time being utilized in percentage.

8. Memory - Available MBytes

Desired value range: High

This the amount of Memory available in Mega byte.

9. Physical Disk - Current Disk Queue Length

Desired value range: Low

It shows how many read or write requests are waiting to execute to the disk. For a single disk, it should idle at 2-3 or lower.

10. Network Interface - Output Queue Length

Desired value range: Low

This is the number of packets in queue waiting to be sent. A bottleneck needs to be resolved if there is a sustained average of more than two packets in a queue.