To surface key COVID-19 KPIs to Local Resilience Forum dashboards for a Central Government organisation, risual were tasked with delivering multiple data pipelines. The client performed a raw data transformation from multiple data sources and formats, using Python.
The stage data was ingested into the Consolidated Data Store (CDS) and cleansed then outputted into published dashboards.
This would seem to be the end of the task for the housing association; however, a problem arose.
The Python pipelines introduced an additional framework that required monitoring from the client’s staff. Furthermore, the use of Python, which only one of the staff was familiar with resulted in a singular point of failure. The implementation was also tied to a single user’s environment and thus caused issues if maintenance/support was required and that single individual was unavailable.
risual processed the embedding and integration of the Python functionality and code into the existing data lifecycle and management of the CDS system.
Now the client is seeing multiple benefits:
- The client has a unified framework for data management, governance and monitoring
- The pipelines can be monitored and maintained by multiple resources, instead of one individual
- The pipelines are monitored by a unified monitoring and alert system within CDS
- The single point of failure has been removed
- Computer and storage are managed within the mature data management capabilities of SQL Server rather than a single workspace
Contact risual today to ensure that your organisation challenges are overcome or follow us on Twitter for the latest risual updates.
