Data & AI
Databricks
Cognos BI Migration To Azure Databricks For 12x Faster Reporting
See how we replaced a legacy reporting platform with a governed cloud data platform, enabling same-day insights, trusted data and validated disaster recovery.
A large ASX-200 engineering and industrial services company was running its entire reporting operation on a legacy IBM Cognos platform that could no longer keep pace with the business. TL Consulting replaced it end-to-end with a modern, cloud-native Data Lakehouse on Microsoft Azure in 16 weeks.
The Challenge
Cognos had become a blocker. Reports took more than 12 hours to produce, stakeholders had stopped trusting the numbers, and the platform could not scale. Making it harder: the client’s data lived across both on-premises and cloud systems, their security posture allowed zero public internet exposure, and the engagement required three fully isolated environments including a live, tested disaster recovery capability.
- Reports took 12+ hours, with data accuracy issues that had eroded trust in the numbers across the business
- Source data was split across an on-premises data warehouse and a cloud SaaS platform, each requiring a completely different ingestion approach
- Enterprise security policy mandated zero public internet exposure, requiring a private connectivity architecture across every layer of the platform
- Three fully isolated environments were required, including a disaster recovery environment that had to be tested and validated under real conditions, not just designed
Legacy platform blocking reporting
- 12+ hour reporting cycles for core operational data
- Data accuracy issues eroding stakeholder confidence
- Cognos unable to scale with growing data volumes
Complex hybrid data sources
- On-premises SQL DWH requiring bulk + incremental loads
- Avature SaaS API with rate limit and payload constraints
- Distinct ingestion patterns needed for each source
Zero public internet tolerance
- SHIR required on hardened on-premises hosts
- All connectivity via ExpressRoute only
- Private Endpoints across every platform service
- No public IPs on any compute node
Three-environment isolation with live DR
- Fully isolated DEV, PROD, and DR environments required
- DR simulation with RTO <4hrs and RPO <12hrs
- Evidence-backed failover/failback procedures
The Solution
TL Consulting delivered an end-to-end Azure Databricks Lakehouse underpinned by a Medallion architecture (Bronze, Silver, Gold) and hardened for enterprise use. Key components included:
- Ingestion: Azure Data Factory orchestrating data from an Avature SaaS API and an on-premises SQL Data Warehouse, with a Self-Hosted Integration Runtime (SHIR) brokering secure connectivity via ExpressRoute
- Storage and Processing: Delta Lake on ADLS Gen2 with Databricks Workflows handling CDC/merge logic and data quality checks at each layer
- Governance: Unity Catalog enforcing centralised lineage, access control, and fine-grained permissions across Data Admin, Data Engineer, and Data Analyst personas via Microsoft Entra ID and SCIM
- Consumption: Power BI dashboards consuming the Gold layer via Unity Catalog semantic models, promoted to production following formal UAT sign-off
- Security Hardening: No public IPs on compute, Private Endpoints for ADLS Gen2, Key Vault and the Databricks control plane, and full DEV / PROD / DR environment separation
- Delivery: Infrastructure-as-Code and CI/CD pipelines via Azure DevOps for both Databricks assets and ADF platform infrastructure
The Outcomes
Same-Day Operational Visibility
Reporting latency cut from 12+ hours to under one hour, giving teams the data they need to act on the same day it is generated.
A Trusted Single Source of Truth
Unified data across all systems ended recurring disputes over accuracy, restoring confidence in reporting across the business.
Enterprise Grade Security
Every enterprise security requirement met with zero public internet exposure, satisfying IT governance requirements without slowing delivery.
Validated Disaster Recovery
DR tested through a live simulation, with recovery confirmed within four hours and data loss contained to under 12 hours.
Scalable Platform for Future Growth
New data domains can be onboarded without re-engineering the platform, lowering the cost and effort of future reporting initiatives.
Clear Exit from Legacy Cognos
Production-ready replacements validated through formal UAT give the business a credible, low-risk path off the legacy estate.
A Trusted Single Source of Truth
Enterprise Grade Security
Technology Stack
Azure Databricks
Delta Lake
Azure Data Factory ADLS Gen2
Unity Catalog
Microsoft Fabric
Power BI
Microsoft Entra ID
ExpressRoute
16 Week Project
Mining & Engineering
Accelerate Your Business Workflows
As specialised Microsoft Solutions Partners, we can access subsidised Azure funding to accelerate your PoC, MVP, or migration.