Improving Productivity By 42% For Major Australian Bank With GitHub Copilot
What We Achieved
I wouldn't hesitate to consider partnering with TL Consulting again, given their exceptional expertise and contributions.
The Challenge
Sponsored by Group Technology, a major Australian bank sponsored an initiative to collect relevant information and statistics that will guide the large-scale adoption of AI pair programming technologies. As part of this initiative, TL Consulting led a GitHub proof of value as an experiment to measure the improvement gains in different use cases and perform the cost benefit analysis of using the technology.
The Methodology
Three hundred engineers that specialised in the programming language Python volunteered to participate in this PoV. The engineers were then divided into two groups of varying levels and asked to complete the same coding challenges.
- The two groups were the control group which used traditional methods to solve the challenges. While the other was the Copilot Group which used GitHub Copilot to solve the challenges.
- All participants were presented with the same coding challenges. The challenges were rated in their difficult level.
- The time each participant takes to solve each of the challenges is recorded while an average is taken for the whole group.
Key Findings
Overall, the following benefits uncovered from our assessment of GitHub Copilot were:
- Faster Code Completion: Copilot auto-suggests entire lines or blocks of code as you type, streamlining the coding process.
- The average engineer saw a +42% increase in productivity while beginners saw even more (+52%)
- Potential For Significant Cost Savings: Extrapolated value from over 3,000 engineers using Copilot was $95.8m in the first year and $138.6m in the second year.
- Improved Code Documentation: It provides inline suggestions for comments, making it easier for developers to document their code.
- Reduced Development Time: With its suggestions and code completions, developers can save time, particularly on routine or commonly used code snippets.
- Developers saw a 3x improvement in solving difficult problems.
- Fewer Code Errors: With auto-suggestions, developers might make fewer coding errors, especially when they are unsure of syntax or function usage.
- Learning Tool: Beginners can use Copilot as a learning tool. It can help them understand coding practices and introduce them to functions and libraries they might not be aware of.
- Support for Multiple Languages: Copilot helps across many languages and frameworks, making it versatile for various projects.
| Python Proficiency | Mean Total Time Spent Per Problem By - Control Group (Mins) | Mean Total Time Spent Per Problem By - Copilot Group (Mins) | Productivity Improvement | ||
|---|---|---|---|---|---|
|
Beginner |
20.07 |
9.58 |
52.27% |
||
|
Intermediate |
28.60 |
16.70 |
41.60% |
||
|
Advanced |
39.82 |
23.70 |
40.48% |
||
Potential Limitations
- Legal Alignment: Coordination with the legal team was challenging, particularly around obtaining indemnity from Microsoft. Daily follow-ups were necessary to resolve outstanding issues.
- Security Approval: The process of securing design and environment scans proved difficult, requiring numerous iterations to meet the necessary standards.
- GitHub Copilot License Agreement: The commercial and contractual processes for obtaining the required license were complex and time-consuming.
- Risk and Compliance Approvals: The AI initiative faced heightened scrutiny from various teams (risk, data, ethics), requiring extensive evidence and support from CTO and CIO to gain the necessary approvals (SEAL, ORA, ADF).
Other Case Studies
- Cloud-Native
- Data & AI
- DevSecOps
- News
- Uncategorised
Streamlining a big four bank's data pipelines and CI/CD processes to improve onboarding, strategic decision making and operational efficiency.
See how we used Azure Databricks to transform a global retailer's data & analytics to unify data sources, improve reporting and enable AI/ML capabilities.