Data & AI

In today’s data-driven world with accelerating advancements in Artificial Intelligence (AI) and advanced analytics, organisations play an important role in ensuring that the data they collect, store, and analyse is underpinned by a strong data governance framework. Embedding the right data governance framework is the enablement of an organisation’s data strategy which requires dedicated planning and strategic direction from various business & technical stakeholders and should be driven from the “top-down” rather than “bottom-up”. To achieve this, organisations should focus on defining their information and data lifecycle management, data relationships and classification, data privacy, data quality, and data integrity to become more competitive and resilient.  The key fundamental challenge for organisations is to embed data standardisation, data security & compliance horizontally across the enterprise, thereby eliminating silos with their own disparate ways of working. In addition, it’s important for organisations to align their data governance framework with their data lifecycle, business strategy and goals, enabling a more agile approach to accommodate the organisation’s current & future needs.   Data Governance Framework Best Practices As organisations collect more and more data points, it’s important to define the right standards, policies, controls, accountability, and ownership (roles & responsibilities). A data governance framework will ensure the organisation abides by these standards while ensuring data that is collected and stored is secure, with a focus on maintaining data integrity and data quality. Ultimately, the data that is consumed by end-users should enable informative, data-driven decisions to be made. A constant re-evaluation is recommended to ensure the organisation’s data governance program is modernised and caters to the latest advancements in data and technology. Prior to defining a data governance framework, a comprehensive data discovery should be performed across the business landscape to create a unified view. This would aid in establishing data governance across the following areas: Data cataloging of data relationships, data quality, and data lineage Data classification and sourcing Metadata definition (Technical and Enterprise metadata) Data compliance, security, and privacy Data analytics & engineering Data storage & sharing The following diagram is a high-level example of a data governance framework. This model should be aligned with the organisation’s data and information management lifecycle. The framework definition should be evaluated from a People, Processes & Technology/Tooling perspective considering data stewardship, efficiencies, data security & access controls, alongside standardised processes governing the technology and tools that facilitate the production, consumption, and processing of the organisation’s data. The following sections highlight a few key areas which the data governance framework should address: Alignment to the Organisation’s Cloud Strategy When uplifting the data governance program, another important consideration for organisation’s that are building technology solutions on Cloud is to define an integrated data governance architecture across their environments, whether it be hybrid or multi-cloud. Alignment to their cloud strategy can help in the following areas: Improve data quality with better management & tooling available around data cleansing and enrichment Build a holistic, unified view of the organisation’s data through discovery and benchmarking Gain higher visibility into data lineage and track data end-to-end from source to target Build more effective data catalogs to ensure it benefits organisational needs to search and access the right data when needed Proactively review, monitor, and measure the data to ensure data consistency and data integrity is preserved For example, Microsoft offers an Azure Governance service as a management and governance cloud solution that features advanced capabilities to help manage data throughout its entire IT lifecycle and track data flows end-to-end, ensuring the right people have access to reliable, accurate data they need, whenever they need it. Data Privacy & Compliance As organisations continue building insights and implementing advanced analytics to learn more about their customers and create more tailored experiences, protecting sensitive data attributes including Personal Information (PI) should be at the heart of the organisation’s data security & data privacy practices, as part of their data governance framework. With the rise of cyber-attacks & data breaches, organisations should consider implementing data obfuscation techniques to “mask” or “encrypt” their PI source data, especially across non-production environments where the access controls are considered weaker than production environments, and the “internal” threat can be considered just as high as the external cyber threats. Applying data obfuscation techniques would ensure the PI data attributes are de-sensitized prior to their use in development, testing and data analytics. In addition, organisations should ensure data controls & access policies are reviewed more frequently than ever. Understanding who has access to the underlying data sources and platforms will help organisations maintain a good risk posture and should be assessed against their data governance framework, across their environments whether on-premise or on Cloud. Augmented Analytics & Machine Learning Without advanced analytics, data loses a lot of its usability and power. Advanced analytics combines the power of machine learning and artificial intelligence to help teams make data-driven decisions based on in-depth insights. Advanced analytics tools greatly streamline the data analysis process and help to provide a competitive edge, uncovering patterns and insights that manual data analysis may overlook. With the introduction of open-source machine learning models such as Open AI’s ChatGPT, how do organisations ensure the data that is collected, analysed, and presented is highly accurate and high quality? Depending on the data models & training algorithms used, these insights can be deeply flawed and it’s important for organisations to embed the right data governance policies around the use of open-source data models, including the collection, use, and analysis of the data points collected. A few roles that data governance plays in the world of augmented analytics, machine learning, and AI include: Providing guidance on what data is collected and how it’s used to train and validate data models for machine learning models to generate advanced analytics Providing standardization on the data science lifecycle and algorithms applied for generating insights, along with data cleansing & enrichment exercises Defining the best practices and policies when introducing new data models, along with measures to fine-tune and train models to increase data accuracy