1. Test against your production environment.
I told this to an "expert" at the Association of Software Testing conference this year. He response was something like, "Are you kidding me?!! You'd have to be stupid to load test your production setup."
What occurred to me later in the conversation was that this guy works for a Fortune 100 company and has a six-figure annual budget (not counting his salary) just for performance engineering.
So what?! I still say you should test against production. And so would some others in our industry that have much more experience than I. For example, Dan Bartow taught a couple of sessions in April at STPCon in San Mateo. He mentioned several times that hammering the real production servers is the only way.
This is controversial, but I believe it. It is rarely possible to replicate your production setup - there are too many components, configurations, and network settings that can affect the results of your test. One small variable can potentially affect performance measurements by 100% or more.
Skeptical? Just try turning on/off caching. Or try running tests against the same environment except that the version of Apache or IIS is different. You will be surprised at the performance deltas. Isn't it easy to overlook the size of MySQL buffers or perhaps PHP options or other configuration selections? All of those variables in the application engineering equation can and will affect your outcomes.
So, if you load test against anything other than your production, you are always going to have doubt about the true performance of your system. There is only one way to know for sure.
2. Accurately simulate user types.
It is easy to hit your site with random heavy traffic, but that may not give you precise load test results.
Application performance can vary significantly depending on the actions of the users. For example, most CMS apps get great optimization from caching pages for anonymous users and practically none from authenticated users. Therefore, you must set up your test plans to include scenarios from both types of users.
Let's say your site normally gets 70% of its traffic from prospects just browsing your product catalog, and another 25% of customers are putting items in their shopping cart, while 5% are posting reviews. Your load testing should include three scenarios to reflect each type of user.
Additionally, you should adjust the weighting of each scenario to match the 70/25/5 percent split between the user types. That way, you will be applying load in the correct ways and massaging the code/servers/database in realistic proportions. How else can you get a true picture of performance for analysis and tuning?
3. Focus on the right metrics.
It is possible to get bogged down in all the data available to you. The exact metrics on which you concentrate may vary depending on the aspect of your web application that is being testing.
However, we recommend you always direct your attention to:
- Response times
- Error rates
- Throughput
- Active users
- Requests per second
You may also need to measure business metrics for your managers. For example, your VP of Marketing may want to know the number of orders processed per minute. It's a bad idea to leave out the project stakeholders when deciding which metrics you need to track. We recommend that you ask them to make a list of questions for which they need answers and set the benchmark for each metric necessary to provide a useful measurement to satisfy the questions. Be sure to avoid vague questions you won't be able to test.
Ask the stakeholders if they can provide historical data or expectations regarding the metrics - this can be a big help in establishing the acceptable levels of performance from everyone's perspective. If you have historical load testing data, all the better. Share it with them.
Open communication with your business leaders goes a long way to knowing the correct metrics to track. Focusing on the right metrics is critical because you can't control what you don't measure.
Bookmark/Search this post with:
