AusSeabed Data Hub Processing Pipeline Business User Stories

Scenario #1

Description

Profile:

1. Modern, high resolution data.

1. Multiple survey’s delivered at once.

1. Machine interface between data and processing

The AHO HIPP contractors commence data acquisition and begin submitting data to the AHO. Raw data files are provided to us in bulk (multiple surveys at a time). The raw data has been collected to the AHO’s specifications and standards, with modern instrumentation appropriate for processing through the ASB processing pipeline.

Raw files are delivered via the AHO clearing house to the ASB data warehouse, multi-survey incoming data storage area. The processing pipeline is triggered to commence processing each survey, with the option of increasing resources to process multiple survey’s in parallel.

Processing products are written to the appropriate level holding areas within the ASB data warehouse.

Point database is written to the appropriate location in the data warehouse.

Processing logs written to the appropriate location in the data warehouse.  

Coverage of the data processed is displayed on the ASB portal at the completion of processing.

Scenario #2

Description

Profile:

1. Modern, High resolution data

1. Single Survey delivered at a time

1. Human interface to the processing pipeline

PARKS Australia commissions a survey of one of their marine park area, and submits the data to the ASB data hub. Raw files are provided via file transfer directly to the ASB data warehouse single survey incoming data storage area for processing, along with a copy of the processed output that was commissioned by Parks.

An ASB data manager logs into the datahub tools and runs the raw data and gridded output through the quality assurance tools and determines that reprocessing is required to achieve an ASB compliant, and interoperable gridded output is required.

The data manager commences the processing pipeline for the single survey, the processing pipeline commences processing the survey.

Processing products are written to the appropriate level holding areas within the ASB data warehouse.

Point database is written to the appropriate location in the data warehouse.

Processing logs written to the appropriate location in the data warehouse.  

Coverage of the data processed is displayed on the ASB portal at the completion of processing.

Scenario #3

Description

1. Variable data (historical)

1. Multiple survey’s assigned for processing

1. Machine interface between data and  processing

GA, CSIRO and AAD decide to lift their historical data archive to the new data standard to support interoperability and use within the ASB data hub.

Archive data holdings are connected to the ASB data warehouse for assessment by the quality assurance tools.

1. Data is programmatically assessed as being able to be processed are moved into the multi-survey incoming data storage area. The processing pipeline is triggered to commence processing each survey, with the option of increasing resources to process multiple survey’s in parallel.

Processing products are written to the appropriate level holding areas within the ASB data warehouse.

Point database is written to the appropriate location in the data warehouse.

Processing logs written to the appropriate location in the data warehouse.  

Coverage of the data processed is displayed on the ASB portal at the completion of processing.

1. Data that is programmatically assessed as being able to be partially processed is also moved into the multi-survey incoming data storage area. The processing pipeline is triggered to commence processing each survey to the point that is possible based on the incoming data.

Processing products are written to the appropriate level holding areas within the ASB data warehouse.

Point database is written to the appropriate location in the data warehouse.

Processing logs written to the appropriate location in the data warehouse.  

1. Data that is programmatically assessed as being unable to be processed in part or in full is written to the area of the data warehouse for this data type and notifications are provided to the ASB data hub data managers to take action on.