Mobile Performance TestNSW Government Agency
Case Study – NSW Government Agency
A NSW government agency had a requirement to performance test a new version of their emergency fires/hazards mobile in readiness for the fires season. The main performance requirement was to simulate the performance of the backend infrastructure for a worse case scenario and also for resiliency against 25 million watch–zones within all the states in Australia and to create 25 million notifications in under 30 minutes or as near to that target. The backend infrastructure was on AWS using Lambda for serverless, SQS queues and SNS for notifications. S3 buckets were there to store the feeds using the CAP-AU standard format from the various Jurisdictions. Spatial data was created via R to simulate the creation of watch–zone based on Geolocation data which form the input into Golang script that used to create the 25 million watch–zones for faster processing within the infrastructure. Jmeter was then used to create the load scripts and then to execute the performance test. Several bottlenecks were found in the early cycles and recommendations were given to the vendor to improve on performance to reach the acceptable NFR. Some design constraints were found as part of the performance testing. Other tests that were performed were resilient tests to ensure redundancy to ensure notifications don’t get lost and can be resubmitted if they go to a specific retransmission queue.
Due to issues with current mobile emergency during black summer (2019) there were number of inefficiencies found with performance and notification management thus a new app was created. A key requirement was to ensure that performance and the inefficiencies were thoroughly tested
The client’s documentation was not sufficient and a new knowledgebase was created around functional and non functional testing
No baselines for performance testing existed and performance scripts did not exist
After multiple NFT tests the NFRs were met and signed off for production release candidate
Through the NFT performance testing it also confirmed that components within the system could scale to handle even bigger loads
TL Consultants managed the whole performance delivery from start to the end from Discovery, test planning, environment configuration, test preparation and scripting, execution, defect triaging and reporting
Multiple workshops were required with various teams such as architecture, development, and business were held to agree on expected NFRs
Updated documentation & scripting around performance.
Performance scripts for real-time workloads were developed in JMeter and Golang scripts were maintained in GIT following best practices.
AWS X-RAY and Cloudwatch were used for monitoring
Tuning recommendations were provided on real-time workloads
On initial run of 1 million notifications, it took 3 days. With monitoring and performance scripting logs we were able to determine design issues and operational issues and were able to identify where the bottlenecks were residing in the system and also configuration recommendations on Lambda settings. This then brought the run down to 15 minutes
Further recommendations through SQS optimizations then brought the 25 million test to 40 minutes which was accepted
Scripts for spatial and geolocation data can be reused easily for future performance tests
Uplift your DevSecOps capability today
TALK TO AN EXPERT
© 2024 TL Consulting Group. All rights reserved.