The AusSeabed program outlines 3 program themes, one of which is the development of a “data hub”“Data Hub”.
Geoscience Australia (GA) is the lead organisation of the data hub program theme, and this is a primarily infrastructure/systems focused theme.
The data hub AusSeabed Data Hub is being developed as a multi-component suite of services that are underpinned by a federated data storage model, with each connected data store being referred to as a “local “contributing hub”. At the time of writing, there are two local contributing hubs under development, one at GA and one at CSIRO.
Outlined below is the <proposed> storage model for the AusSeabed GA Local Hubcontributing hub.
...
Details of the objects in the model above are outlined below:
Object Model Name | Description | Storage Class | Cost basis | |
AusSeabed-Public-Bathymetry | The storage container that holds all the working spaces for the GA Local Hubcontributing hub. The contents of this container is editable by the AusSeabed program team. The environment is currently set up as a publicly accessible S3 environment and will likely be constrained in the future. | Variable; controlled at object level | N/A | |
Clearing house | Is the landing zone for all data accepted into the ASB GA Local Hubcontributing hub. | Variable; controlled at an object level. | N/A | |
General 60day incoming store | Is the initial landing zone for all incoming data that has been accepted into the AusSeabed GA Local Hubcontributing hub. | S3 - Intelligent Tiering | ~$0.024 per GB per month. | |
Source Data Store | Provides a copy and full record of the data as it is provided to us without any sorting or processing applied. The Source Data Store will hold all data provided the the GA contributing hub, regardless of level. | S3 Standard - Infrequent Access (Future - Glacial) | ~$0.019 per GB per month. ~$0.005 per GB per month. | |
L0 Temporary Incoming Store | Provides a landing place for an organised collection of the L0 data. From this location, L0 data can be processed to L2 or L3 and a copy of the L0 data will be made into the distribution environment for publication. | S3 - Intelligent Tiering | ~$0.024 per GB per month. | |
L0 Archive | Provides a full copy of the data that has been identified as Level 0 or Raw data for a survey, along with the equivalent auxiliary data required for its processing. As L0 data is unchanged data, this represents the archive for all specific L0 data held in the GA Local Hubcontributing hub. | S3 Standard - Infrequent Access (Future - Glacial) | ~$0.019 per GB per month. ~$0.005 per GB per month. | |
L2 Temporary Incoming Store | Provides a landing place for an organisation collection of the L2 data. From this location, L2 data can be processed to L3 and a copy of the L2 will be added to the distribution environment. | S3 - Intelligent Tiering | ~$0.024 per GB per month. | |
L2 Archive#1 | Provides a full copy of the L2 data that has been identified as L2, along with the equivalent auxiliary data required for its processing to L3 grid. The L2 archive #1 will hold all data that is:
As L2 data is processed data, this archive represents the first only archive of all L2 products that are at the L2 level, regardless of its their publication status . (ie older versions, derived and provided, moratorium) | S3 Standard - Infrequent Access (Future - Glacial) | ~$0.019 per GB per month. ~$0.005 per GB per month. | |
L3 Temporary incoming store | Provides a landing place for an organisation collection of the L3 data. From this location, L3 grids will be copied out into our distribution environment and potentially derived to an L4 product in the future. | S3 - Intelligent Tiering | ~$0.024 per GB per month. | |
L3 Archive #1 | Provides a full copy of the data that has been identified as L3. The L3 Archive #1 will hold all data that is:
As L3 is processed data, this archive represents the only archive of all L3 products, regardless of their publication status (ie older versions, derived and provided, moratorium) | S3 Standard - Infrequent Access (Future - Glacial) | ~$0.019 per GB per month. ~$0.005 per GB per month. | |
AusSeabed-Public-Warehouse-Bathymetry | The storage container that holds all the distributable objects. Directly connects to the AusSeabed Marine Data Portal and eCat. The contents of this container is not readily editable. | Variable; controlled at object level | N/A | |
L0 Product | The L0 product folder is a static, non-editable copy of the data that ASB GA Local Hub contributing hub has elected to distribute. Distribution patterns of both the formats to be distributed and the access methods will be determined as the GA contributing hub matures. | Future - S3 Standard - Infrequent Access | ~$0.019 per GB per month. ~$0.005 per GB per month. | |
L2 Product | The L2 product folder is a static, non-editable copy of the data that ASB GA Local Hub contributing hub has elected to distribute. Distribution patterns of both the formats to be distributed and the access methods will be determined as the hub matures. | Future - S3 Standard - Infrequent Access | ~$0.019 per GB per month. ~$0.005 per GB per month. | |
L2 Distributed Archive #2 | Provides a complete archive of all items that have previously been distributed. This archive will only hold those items that have been published, past and current. | Future - S3 Glacial | ~$0.005 per GB per month. | |
L3 Product | The L2 product folder is a static, non-editable copy of the data that ASB GA Local Hub contributing hub has elected to distribute. Distribution patterns of both the formats to be distributed and the access methods will be determined as the GA contributing hub matures. | S3 - Intelligent Tiering | ~$0.024 per GB per month. | |
L3 Archive #2 | Provides a complete archive of all items that have previously been distributed. This archive will only hold those items that have been published, past and current. | Future - S3 Glacial | ~$0.005 per GB per month. | |