As part of a wider engagement with one of the Global leaders in innovative air traffic solutions and airport performance, risual were approached to help solve a problem they were having with an ageing data platform. Our client was struggling with an ageing, on-premises data warehouse solution that was running out of space, incurring a huge amount of cost and not providing the performance or service needed to help the business continue to grow and develop on the modern work. In particular, the ability to be able to process key environmental performance data was causing an issue and was causing delays in providing metrics, not only to the business but also to regulatory bodies and partner organisations. Daily processing was taking over 23 hours, causing delays in providing the most up to date information available. If an error occurred in producing performance data, processing would need to be restarted which then cascaded into delays in the following day’s processing which was inefficient and costly. As part of the requirements with their regulatory authority, environmental data and progress reports needed to be submitted regularly to show their performance in driving down carbon emissions and performance against their environmental policies. Delays in reporting these metrics would incur monetary fines which was costly to the business.
Seeing an opportunity to improve both process and performance in the platform, the business engaged with risual at the conceptual early stage to discuss both the issues, business goals and technological possibilities. Engaging in workshops with both their IS Teams and Business Owners alongside risual Architectural and Data Science leads we were able to draft a solution using some of Microsoft’s new and emerging technology on both Azure and Data. The organisation had already decided on a “Cloud First” strategy so the option of migrating Data Warehouse workloads to the Cloud was explored, not just as a lift and shift, but as a strategic move to capitalise on the cloud technology and services available.
The initial stage one was to establish a data architecture on the Azure platform to prove the business case. risual worked on designing and deploying a modern Data Analytics Platform incorporating Data Factory, Data Lake, Databricks and Power BI, supporting Data Engineering,Data Science, and delivering a business data analytics capability to end users . The initial requirements delivered a data catalogue to complement the data lake, providing a solution to the discovery challenge that data consumers previously experienced. The data catalogue is easily searchable, allowing for data democracy , breaking down silos and enabling anyone in the organisation to be able to find and request access to the data assets they need. The data catalogue was key to enabling self-service analytics. Data sets should be always up to date, often in real time , allowing end users to make decisions based on the latest information. Access to real time and accurate data is enabling users to confidently plan with partnering organisations to maximise efficiencies and improve safety in both air travel and airports globally.
Once the initial stage was completed the subsequent stages were to roll out the platform further and provide an analysis and prioritisation of on-premises ETL workloads for migration to the new Azure data platform. As risual were already contracted to deliver optimisation of cloud services and had a proven track record of migrating data services into Azure with recognised cost savings, the decision was taken to optimise and re-engineer on-premises data warehouse workloads for the cloud. This was achieved by developing data pipelines in Databricks.
Step 3, which has recently been completed has seen the further re-engineering of this data. Data sets are now available to users within the business on a wider scale, controlled by user access and using Azure security services to track what data has been accessed and by who. Users are already finding their decision making is more accurate and more pro-active enabling quicker decisions to be made to make operations more efficient. The time is takes to produce regulatory reports has reduced from 23 hours to just under 1 hour and the accuracy of the reports has also improved exponentially.
The Client and risual are now committed to deploying the platform further. Moving into the support phase, risual are now working to ensure the platform always delivers what the business needs as well as ensuring optimisation of services without compromising on its original objective of improving the environment and enhancing aviation safety in both air travel and in and around the airports here in the UK and globally.