V.ISC.2036 - Analytical Data Modelling

Summary

The purpose of this project is to support the priorities of Pillar two of the ISC data strategy, by reviewing the data structures of the ISC program databases held in the MLA Data Platform, providing recommendations for improved approaches that will enable linking of ISC program data into linked, master data sets enabling analysis to support the objectives of the ISC 2025 Strategic plan.

This project would review the current as-is environment for the ISC data model sitting in the data platform and develop a recommendation for the to-be state. When the decision from the to-be recommendation is accepted, the solution will be implemented as per the recommendation.

Objectives

• The creation of a document which provides the recommended to-be state for transforming the existing data model in the ISC Amazon S3 environment.
• The implementation of the recommended to-be state if it is approved to proceed.
• Analysis of data sources.
• Completion of data catalogue files for each data source, including database description and table descriptions.

Key findings

• We were able to align with an existing SA sheep report used previously for the SA government. This original query took over an hour to run, while running over the data model returned a result in under one minute with accurate results.
• There are a number of data issues within the existing data sources. The creation of the data checker in this solution allows these data issues to be identified and resolved.
• There are opportunities to expand the current data model across additional data sources. For example, once the eNVD data source is widely used it could be incorporated into the data model to provide additional analysis over the use of the electronic NVD process compared to the paper-based approach.

Benefits to industry

• The creation of a reporting platform for the data model enabling a broader usage of
consistent data throughout the meat industry and relevant government agencies
• Identify further data sources to include into the data model to expand its scope and value
• Provision of a centralised and consistent source of data for industry

MLA action

Support and drive the expansion of the Data Catalogue and expanding the data model to include other datasets.

Future research

Further expansion and extension of Data Catalogue to improve data discoverability and reachability.
Extending data model to include other datasets to unlock other data use cases and continue to support data driven decision making for a wider industry.

Future R&D:
▪ Expanding scope of the data model and building a reporting platform to expose the data model through to a broader audience