GA Contributing Hub Storage Model

The AusSeabed program outlines 3 program themes, one of which is the development of a “Data Hub”.

Geoscience Australia (GA) is the lead organisation of the data hub program theme, and this is a primarily infrastructure/systems focused theme.

The AusSeabed Data Hub is being developed as a multi-component suite of services that are underpinned by a federated data storage model, with each connected data store being referred to as a “contributing hub”. At the time of writing, there are two contributing hubs under development, one at GA and one at CSIRO.

Outlined below is the <proposed> storage model for the GA contributing hub.

Details of the objects in the model above are outlined below:

Object Model Name		Description	Storage Class	Cost basis
AusSeabed-Public-Bathymetry		The storage container that holds all the working spaces for the GA contributing hub. The contents of this container is editable by the AusSeabed program team. The environment is currently set up as a publicly accessible S3 environment and will likely be constrained in the future.	Variable; controlled at object level	N/A
	Clearing house	Is the landing zone for all data accepted into the GA contributing hub.	Variable; controlled at an object level.	N/A
	General 60day incoming store	Is the initial landing zone for all incoming data that has been accepted into the GA contributing hub.	S3 - Intelligent Tiering	~$0.024 per GB per month.
	Source Data Store	Provides a copy and full record of the data as it is provided to us without any sorting or processing applied. The Source Data Store will hold all data provided the the GA contributing hub, regardless of level.	S3 Standard - Infrequent Access (Future - Glacial)	~$0.019 per GB per month. ~$0.005 per GB per month.
	L0 Temporary Incoming Store	Provides a landing place for an organised collection of the L0 data. From this location, L0 data can be processed to L2 or L3 and a copy of the L0 data will be made into the distribution environment for publication.	S3 - Intelligent Tiering	~$0.024 per GB per month.
	L0 Archive	Provides a full copy of the data that has been identified as Level 0 or Raw data for a survey, along with the equivalent auxiliary data required for its processing. As L0 data is unchanged data, this represents the archive for all specific L0 data held in the GA contributing hub.	S3 Standard - Infrequent Access (Future - Glacial)	~$0.019 per GB per month. ~$0.005 per GB per month.
	L2 Temporary Incoming Store	Provides a landing place for an organisation collection of the L2 data. From this location, L2 data can be processed to L3 and a copy of the L2 will be added to the distribution environment.	S3 - Intelligent Tiering	~$0.024 per GB per month.
	L2 Archive#1	Provides a full copy of the L2 data that has been identified as L2, along with the equivalent auxiliary data required for its processing to L3 grid. The L2 archive #1 will hold all data that is: Provided directly to GA as L2 or Derived from L0 to an L2 within GA. As L2 is processed data, this archive represents the only archive of all L2 products, regardless of their publication status (ie older versions, derived and provided, moratorium)	S3 Standard - Infrequent Access (Future - Glacial)	~$0.019 per GB per month. ~$0.005 per GB per month.
	L3 Temporary incoming store	Provides a landing place for an organisation collection of the L3 data. From this location, L3 grids will be copied out into our distribution environment and potentially derived to an L4 product in the future.	S3 - Intelligent Tiering	~$0.024 per GB per month.
	L3 Archive #1	Provides a full copy of the data that has been identified as L3. The L3 Archive #1 will hold all data that is: Provided to GA as an L3 or Derived to L3 from a lower level by GA. As L3 is processed data, this archive represents the only archive of all L3 products, regardless of their publication status (ie older versions, derived and provided, moratorium)	S3 Standard - Infrequent Access (Future - Glacial)	~$0.019 per GB per month. ~$0.005 per GB per month.
AusSeabed-Public-Warehouse-Bathymetry		The storage container that holds all the distributable objects. Directly connects to the AusSeabed Marine Data Portal and eCat. The contents of this container is not readily editable.	Variable; controlled at object level	N/A
	L0 Product	The L0 product folder is a static, non-editable copy of the data that GA contributing hub has elected to distribute. Distribution patterns of both the formats to be distributed and the access methods will be determined as the GA contributing hub matures.	Future - S3 Standard - Infrequent Access	~$0.019 per GB per month. ~$0.005 per GB per month.
	L2 Product	The L2 product folder is a static, non-editable copy of the data that GA contributing hub has elected to distribute. Distribution patterns of both the formats to be distributed and the access methods will be determined as the hub matures.	Future - S3 Standard - Infrequent Access	~$0.019 per GB per month. ~$0.005 per GB per month.
	L2 Distributed Archive #2	Provides a complete archive of all items that have previously been distributed. This archive will only hold those items that have been published, past and current.	Future - S3 Glacial	~$0.005 per GB per month.
	L3 Product	The L2 product folder is a static, non-editable copy of the data that GA contributing hub has elected to distribute. Distribution patterns of both the formats to be distributed and the access methods will be determined as the GA contributing hub matures.	S3 - Intelligent Tiering	~$0.024 per GB per month.
	L3 Archive #2	Provides a complete archive of all items that have previously been distributed. This archive will only hold those items that have been published, past and current.	Future - S3 Glacial	~$0.005 per GB per month.