How to do Performance Testing?

The exact procedure to do a performance test varies between projects and organizations. It depends what parameters are most important to the business or application under test. However, the general goals largely fall under the same broad category. Below are the steps which are followed to conduct a performance test-

1. Identify Test Environment

First step before conducting a performance test (or any test) is to identify the environment. Performance test requires an environment which is scaled to match production to get a good assessment of performance issues which might occur. We should document the architecture and understand the underlying infrastructure. It helps to create our test suite.

Please note that sometimes, we may have to do a performance test in a production environment, but this activity should be done ensuring that proper checks are in place and real customers are not disrupted.

2. Define the transactions to performance test

Think about the transactions that are widely used by customers. For example- Ordering in an ecommerce website. There are plenty of use cases possible in each application under test. However, not all of them result in a load to servers that can cause performance issues. Hence, as a performance tester, you need to analyze the transactions that generate a good amount of load on servers. We can use data analytics tools to get the specifications to design our tests. Based on data available, we need to design our scripts which can mimic the user actions and generate the required load.

3. Define the acceptance criteria

Next step is to determine the goals, constraints and thresholds that determine if a transaction is a pass or not. These come from project specifications in terms of non-functional requirements. An example of acceptance criteria is as below

4. Prepare the test data

When we know the transactions, which can result in load, next step is to create the data to simulate such kind of load in a non-production environment which has been scaled to match production (in terms of infrastructure). For example, preparing data for search, preparing data for ordering etc. If the application has different kind of user privileges, we need to create different kind of users with different access and generate load proportionate to user base in production. If we are performance testing a migration project, we can create a mix of users which have been migrated and which are still non-migrated.

5. Execute and Monitor the test

Next step is to execute the performance test using the appropriate tool. Common performance testing tools can help mock user transactions to generate the load on a given environment. Also, as a performance tester, you should monitor the test. In a well-designed performance test, the load is generated gradually before the number of virtual users reach a steady state. Once the test is in steady state, it means that application is being tested for “Peak load”, expectation is that all transactions should work as they did without the full load.

Another thing to keep in mind while monitoring the performance test is the health of underlying infrastructure. For example, if one of the JVMs is not in a healthy state, there is no point to continue the test till the end. It is better to terminate and restart the test.

Also, it is also advisable to avoid any kind of batch transactions to be run during a Peak load test since it will hamper debugging/isolating of any real performance issues.

6. Resolve the issues and Retest

Once we identify a performance issue, it can be tracked as a bug. However, one thing to note there is that PT defects should not be treated with same defect resolution SLAs as functional defects. Resolving a performance defect may take involvement of number of teams- developers, DBAs, platform architects etc. Once the issue is resolved, we should run at least 2 performance tests to see if the transaction under test is meeting the SLAs consistently to consider the bug as fixed.

Another thing to keep in mind while doing a performance test is to mimic the SOPs (Standard Operating Procedures) from production. For example, if your production application stack gets restarted once a week, same should be followed with your performance test environment. This will simulate exact scenarios as occurring in production.

7. Reporting

Performance Test Report comprises of:

• A detailed description of performance test results- Expected SLAs and achieved SLAs per transaction, how many transactions passed, how many transactions failed etc.
• Observations and findings
• Recommendations
• Identified Defects (Detected/Closed/Open)- if any.

There are few points which should be taken care for reporting.

• Before starting the performance test result analysis
  • The test should run for the defined duration
  • Separate the ramp-up, steady state and ramp-down durations
  • Separate out ‘Think Time’ from the graphs.
  • Ensure that no tool specific error should have occurred like the failure of load generator, memory issue, etc.

• During performance test result analysis-

Start the analysis with some basic metrics like,

• Number of Users: The actual load during steady state should meet the user load NFR
• Response Time: The actual response during steady state should meet the response time NFR. The response time should be measured at two levels - Individual transaction response time and end-to-end response time. If SLAs are available for both the levels, then they should meet
• Transactions per second / Iterations per hour: If any of these metrics is defined then the actual result should match with the defined figure.
• Throughput: Throughput is the rate at which the requests can be serviced by the system. It should meet the defined %.
• Error: The error count should be less than the defined error tolerance limit.
• Passed Transaction counts: Identify the number of transactions which passed and what failed. If failure rate is too high due to a possible functional issue, test should be abandoned.

Analyze the graphs:

• Set the proper granularity for all the graphs
• Read the graph carefully and note down the points
• Check the spikes and lows in the graph
• Integrated performance testing tool and monitoring tool should give details of spikes and lows on various architecture layers.
• Analyze heap dumps created during the test to give a good measure of possible issues

Performance Testing “Day in a life” Scenarios

Day in a life scenario means that we are simulating the kind of transactions which a normal customer on production will do, how will those be processed end-to-end while interacting with various backend systems (example, fulfillment systems like SAP). These scenarios become crucial when multiple front-end systems like Website, App connect to one back-end system and performance of back-end system derives the output on frontend. In such scenarios, using correct workload which is a mix of multiple frontend systems is very crucial and making tests more realistic.

Conclusion: Performing a performance test with a definite strategy is a game changer. It sets-up a smooth and channelized process to achieve the performance testing goal. Performance Testing Life Cycle helps the application development companies to develop a performance- tuned application, improvise the performance of the application and meet the customer’s expectations.