Performance Engineering Lab is a dedicated environment where software is systematically tested, analyzed, and optimized for performance metrics such as speed, scalability, stability, and resource efficiency. The lab provides the tools, infrastructure, and processes necessary to conduct thorough performance evaluations, ensuring that software applications meet required performance benchmarks before being released into production for customers to use.
For this paper, we will use the following entities to architect a performance engineering lab. Please note that the design of this PE Lab is agnostic of the application under test and the various tools/technologies that build it. The only reason names of tools and technologies are mentioned here is for the readers’ ease so that they can understand the ecosystem.
First, determine the purpose of this Performance Engineering (PE) Lab.
Some of the common reasons organizations need a SPE Lab are
A Software Performance Engineering (SPE) Laboratory focuses on analyzing, optimizing, and ensuring the performance of software systems. The technology ecosystem within an SPE lab includes tools, methodologies, and frameworks that support performance evaluation, monitoring, and tuning of software applications.
“Reference
A typical cloud-based software application that undergoes performance testing and engineering has a specific architecture. Microsoft, Amazon, and Google are some of the cloud services providers that help organizations build modern-day enterprise-level software by offering services for hosting software, data streaming, databases, caching, application security, and, last but not least, a secure network. An example of such an application using Microsoft Azure’s services is shown below
There are few important guidelines that should be followed while creating your performance engineering environment:
Think of a “requirement” as simply something your project must deliver. This could be a specific feature in a software product or a broader goal like increasing customer satisfaction. Before starting any project, it’s crucial to clearly define these requirements. This ensures everyone understands the goals and expectations. Usually, these requirements are managed in a central repository that’s easily accessible to everyone on the team. JIRA is such a tool from Atlassian. However, our SPE Lab design is system agnostic, and all it cares about is that the performance requirements are clearly articulated and have the following details:
Once the performance requirements have been baselined, a performance engineer or a developer should start translating them into source code by writing executable test scripts. There are several tools, both open-source and licensed, available in the market that help you with this step. JMeter is an extremely popular open-source tool that helps developers and engineers create performance scripts using its record & playback feature along with its standalone API call feature. Gatling is another popular tool amongst the developer community that lets you create your performance test scripts using popular programming languages like Java and Scala. On the other hand, there are licensed tools like LoadRunner, Blaze Meter, and K6. Although they come at a price, they do provide additional capabilities like:
I always recommend using a licensed tool because:
The next set of tools that you need in your SPE Lab ecosystem is a source code management system like GitHub, GitLab, or Bitbucket and a test orchestrator like Jenkins, Circle CI, or Azure DevOps. A test orchestrator is a tool that downloads source code containing performance tests from the source control systems onto an engine or a virtual machine and then executes them. Although an orchestrator is an optional tool because most of the licensed tools have built-in execution engines, if you’re relying on open-source tools like Gatling, it is always advisable to have an orchestrator. An orchestrator also facilitates continuous performance testing by plugging your performance tests into existing CI/CD pipeline/s, thus creating a fully automated/autonomous system.
My recommendation here would be to go for tools that are being used in your organization, as all we need is a means to manage your test code and plug them into existing CI/CD pipelines.
An Application Performance Monitoring (APM) tool is essential for performance testing, offering real-time insights into an application’s performance across its entire stack. It helps teams identify bottlenecks, diagnose the root causes of performance issues, and resolve them proactively, ensuring optimized application performance and a seamless user experience, especially during load testing.
APM is essentially the brain of the SPE lab. APM tools offer deep-dive capabilities to drill down into performance issues, allowing teams to identify the root cause of problems by analyzing logs, traces, and other related data, facilitating faster troubleshooting. Datadog and Dynatrace are two such tools that are leaders in this space and are must-have tools for the SPE lab.
A science lab is incomplete without a fire extinguisher, and we very well know what could happen when a science experiment goes wrong, as it could bring the entire lab down. In the same way, a performance engineering experiment gone wrong can destabilize the entire lab, making it difficult to ask the engineering teams to analyze what caused the break and what it would take to fix those issues and bring the lab back up. However, engineering teams can come up with in-house tools or scripts that can be run to restore the lab back to its pristine form. Some examples of such tools/scripts are:
It is desirable to have a centralized hub where high-level test results can be shared and communicated to the desired teams so that they can collaborate on the results. That’s where Slack and Microsoft Teams come in with their out-of-the-box integration capabilities, using their APIs to post results in a group or channel dedicated to the cause. Threaded conversations keep discussions organized and context intact.
Now that we’ve figured out all the building blocks and tools required to build the SPE Lab let’s connect the dots.
The point to be noted here is we just accomplished our objective of conducting end-to-end performance testing & engineering in a SPE Lab by utilizing the tools and processes that it provided to its users.
As I mentioned earlier, the aim of this paper is to help create a tool/technology-agnostic SPE. However, if I were to create a pictorial representation of this SPE Lab using some of the tools discussed in sections 2 and 3, then this would look like the following (
Performance metrics and Key Performance Indicators (KPIs) are critical for the success of cloud applications. They provide actionable insights to ensure that applications perform reliably, meet user expectations, and align with business objectives. Examples of Key Cloud Application Metrics:
By systematically monitoring and analyzing these metrics and KPIs, organizations can ensure the scalability, reliability, and efficiency of their cloud applications while aligning with strategic objectives.
Continuing with the example of Microsoft Azure as the cloud services provider for an application deployed on a Kubernetes setup, some of the real-life metrics and industry benchmarks are given below (
Component |
Metrics & KPIs |
||
Metric |
Description |
Benchmark |
|
Kafka/AMQ |
Messages queued/de-queued |
The number of messages that are currently in the queue and how quickly they are being processed |
Lag should not build up. All messages should get processed after the tests are complete |
Kubernetes |
CPU utilization |
CPU utilized by individual nodes/pods |
Must be within the limits specified in the resource definition of individual nodes/pods |
Kubernetes |
Memory |
Memory utilized by individual nodes/pods |
Must be within the limits specified in the resource definition of individual nodes/pods |
Throughput |
Transactions per second (TPS) or Transactions per minute (TPM) |
The number of requests handled successfully by the service per unit of time |
At least 1.5 times x requests/min x-value in production |
Database |
DTU % |
A unit of measure representing a blended measure of CPU, memory, reads, and writes. |
Average must be <= 75% |
Cache |
Server load |
The Server Load metric represents the load on the Redis Server alone |
Must be <= 80% |
Cache |
Memory |
All the data in Redis is stored in memory and is often used for caching web pages and reducing the load on the server |
Must be <= 80% of the total allocated memory |
Backend |
Response time |
Average or 95th Percentile of the observed response times are equal to or less than the specified time |
Must be <= 3000 milliseconds |
Frontend |
Response time |
Average or 95th Percentile of the observed response times are equal to or less than the specified time |
Must be <= 5000 milliseconds |
Failures |
Error % |
Total number of errors observed during the test |
Must be <= 0.05% |
Any deviation/s from these benchmarks during testing should be investigated immediately. APMs provide in-depth analysis and help determine bottlenecks in the system. For example, if the response time of a service is on the higher side, it could lead us to the root cause, which, for example, could be due to a SQL query either taking more time to execute and return data or could be using more resources (DTUs) thus slowing down the system’s performance.
To ensure accurate, reliable results and high-quality performance testing within the lab environment, it is very important that we follow some best practices.
The establishment of a Software Performance Engineering (SPE) Lab is vital for ensuring that applications fulfill performance standards across diverse scenarios. A strategically designed lab allows teams to proactively detect bottlenecks, enhance resource efficiency, and uphold software dependability.
Historically, performance engineering has been a reactive endeavor; however, contemporary methodologies advocate for its early incorporation into the software development lifecycle. The integration of cloud-based platforms, automation tools, and AI-enhanced analytics has markedly increased the efficiency of performance testing. Nonetheless, challenges such as infrastructure expenses, tool compatibility, and the intricacies of workload characterization continue to exist.
Effective collaboration among development, QA, and operations teams is crucial to fully leverage the advantages of a performance lab. By cultivating a performance-oriented culture, organizations can improve software quality, mitigate deployment risks, and elevate the overall user experience.
Establishing a Software Performance Engineering Lab necessitates thorough attention to hardware, software, and methodologies. This paper has highlighted essential elements for creating a performance lab, covering everything from requirement definition to the implementation of monitoring techniques.
An effectively organized performance lab guarantees that applications can scale effectively, manage peak demands, and provide a smooth user experience. The incorporation of automated performance testing, ongoing monitoring, and predictive analytics will continue to enhance software performance engineering.
Future developments in AI, cloud technology, and serverless architectures will further influence the transformation of performance engineering labs, making them increasingly adaptive and intelligent. Organizations that view performance engineering as a fundamental discipline will secure a competitive edge in delivering superior software solutions.