This is the first in a series of blogs which outline a vision of a Modern Data Platform, its components, and the benefits that can be realised from taking a holistic view of your data assets.
What is a Modern Data Platform?
A Modern Data Platform is a set of services and technologies that provide an organisation with a complete way of controlling and getting the most from their data assets. The Modern Data Platform provides;
- Data Governance
- Data Storage
- Data Security
Data Lifecycle - Data Orchestration
- Data Operation
- Data Documentation
- Data Discovery
Get the Vision
This is a technology agnostic view of a Modern Data Platform. It doesn’t matter whether your data is on-premises, co-located, in the cloud, or a hybrid of these. It is a set of functional, loosely coupled components aligned to organisational requirements. Because of this, its components may vary from organisation to organisation, depending what data assets they have, or what they do with them.
The Modern Data Platform also evolves over time as an organisation changes to cope with different varieties of data, more complex analytics and new data technologies. For example, organisations that deal with Personally Identifiable Information (PII) or are required to conform to PCI-DSS standards may need to vary the content of their Modern Data Platform vision to suit their data.
The Modern Data Platform captures metadata about itself and the assets it contains as events happen within the platform and these are stored in a metadata store. This provides a fully instrumented platform to provide management reporting at different levels of aggregation, suitable for team, department and senior management levels.
Existing Line of Business (LOB) applications can coexist with an organisations bespoke systems within the Modern Data Platform, taking advantages of its components for data processing.
The Modern Data Platform provides;
- Governance. Describing data in a standard way, using a standard taxonomy, and applying documented data standards and policies. The Modern Data Platform tracks which policies and standards apply to which data assets. This also covers data licensing, where an organisation has purchased external data assets under license, and you need to control its usage. Management dashboards and exception reporting track the contents of the Modern Data Platform.
- Storage. A Consolidated Data Store is a virtual entity that comprises relational databases, not only relational databases, big data stores, file shares, document stores and even paper files. Each data asset is stored in a suitable physical store depending on its type and content but done so using a common approach using the organisations taxonomy. Database, schema and object naming is consistent across physical data store types, so that it is known what the data is just by its location.
- Security. The Modern Data Platform incorporates a Common Security Model across all its components. The permissions given to each object are then consolidated into a security data mart, so that permissions can be analysed. A Common Audit Model is also deployed for each technology to captures changes and usage of the data assets. This is consolidated into an audit data mart for further analysis.
- Lifecycle. A Modern Data Platform manages its data assets against an organisation specific lifecycle. As data travels through an organisation it is developed and enriched, going through specific states. Keeping a copy of that data at each state is useful as it documents the data lineage; where data assets are combined to form new data assets, these relationships are documented so a picture is built of each data asset and how it was created.
- Operation. The Modern Data Platform contains an orchestration service to facilitate the movement of data within and in and out of the Consolidated Data Store. It does this using standard functions as building blocks, constructing processes and workflows to automate as much of the data processing as possible. A notification component is used to report status or key events that happen on the platform. The final component is Analytical and Presentation Tooling for the user community to enrich the data, gain insight into their organisation and surface this data within the organsiation.
- Documentation. The Modern Data Platform is deployed wherever possible as Infrastructure as Code (IaC). In addition, the standard functions and processes are captured using a common source control component. This documents how the platform is built, as well as how data is manipulated using standard functions, processes and workflow. It also providing a history of its development.
- Discovery. The components of the Modern Data Platform all capture metadata about the data assets they touch. This is surfaced to the end user community through the Data Catalogue, providing an interactive interface to discover data assets, understand how they were made, how uses them, who has access and who owns them within the organisation.
- Management. A Management layer of tooling and monitoring components, feeding in to the metadata store.
How does a Modern Data Platform help me?
Put simply, it gives you control. It is designed to help resolve the following types of questions;
- You know what data assets you have?
- You know the types of data they contain?
- You know the lineage of each data asset?
- You know the standards to which they were created?
- You know how to find the data you have?
- You know who is responsible for the data?
- You know that data is stored in a consistent way?
- You can be assured that you comply with the General Data Protection Regulation (GDPR)?
Would you like to know more?
Would you like to know more, or how a Modern Data Platform can be applied within your own organisation to bring back control of your data? Contact us on the link below.
