CDMS Requirements Analysis
The content presented on this page is a work in progress.
It may change at any stage until finalised.
NB: ET-CDMS Team Members
If you want to edit the table below, please contact Bruce first.
If not, direct edits to the table below will be lost at the next update.
- tie the CDMS to essential service needs to support a range of industries, eg drought monitoring, seasonal forecasting, monitoring of extremes, enquiries from Ministers - appropriate user stories required
- detailed analysis of all components
- explore specific climate data management user stories in depth
- rationalise user stories to those that best illustrate requirements
- high level architecture for each key user story, showing use of components
1. Use mind maps for user stories (Tools : open source Freemind version 1.01 recommended), Download available here.
2. If required, the current version of these docs may be found here:
3. During the analysis work, we expect that we will develop many user stories with more than one available per component. In time these User Stories will be rationalised to a set that best reflects the expected requirements of a Modern CDMS.
Status Classification used:
- Preliminary Analysis
- Detailed Analysis
|CDMS Component||Status||Stage (%)||Mind Map Analysis||User Story||Conceptual Diagram||Date last revised||Comments|
|3. CDMS Governance||Preliminary analysis||100||.mm ||2016-08|
|4. Time-series climate data||Detailed Analysis||50||.mm ||2016-11|
|4.1 Observations data||Detailed Analysis||50||User story: 4.1 Observations data||An Observation||2016-11|
|4.2 Logical data models||Preliminary analysis||100||.mm ||User story: Logical Data Model - Neutral CDMS Archive||2017-01||The Logical Data Model - Neutral CDMS Archive data format will probably need considerable engagement with CBS - IPET-DD, once the data model for a CDMS Database is known|
The 5.2 analysis and user stories needs considerable work. We need to cater for at least the 'Community agreed standard vocabularies' as illustrated in the _Climate data services conceptual diagram_ below.
This functionality is required to underpin federated global climate data services of consistent data sematics
|4.3 Climate metadata||Preliminary analysis||100||.mm ||User story: 4.3.3 Climate Data Provenance - W3C||2017-01||In the next version of the CDMS Specifications:|
- the Climate Observations Metadata components will be adjusted to be consistent with the WMO WIGOS (Observations) Metadata standard
- Data Intellectual Property Metadata may be also included as a subcomponent of Climate metadata. See also info in new component section in the table below.
|4.3.1 Observations metadata||Preliminary analysis||100|
|4.3.2 Dataset discovery metadata||Preliminary analysis||100|
|4.3.3 Data provenance||Preliminary analysis||100|
|4.4 WMO standard products||Preliminary analysis||100||.mm |
|4.5 Derived climate data||Detailed Analysis||30||.mm ||Need to include from Nov 2016 mind map analysis:|
- derived statistical data (Extremes, Normals, Anomalies, averages, homogenised date etc.) and other types of derived data
- Spatial Views from derived data as discrete point coverages (perhaps by variable and observations network)
- climate monitoring grids as continuous coverages derived from the spatial views
|4.6 Ancillary data||Preliminary analysis||100||.mm |
|5. Climate data management||Preliminary analysis||100||2016-08|
|5.1 Ingest and extract||Detailed Analysis||95||.mm ||User story: 5.1 Ingest and Extract||Ingest and Extract||2016-11||Need to update User Story to match mind map analysis|
|5.2 Data rescue||Preliminary analysis||100||.mm ||User story: 5.2 Climate Data Rescue - ET-CDMS |
User story: 5.2 Climate Data Rescue - W3C
|Need to approach ET-DARE for help developing appropriate user stories|
|5.3 Observations quality control||Preliminary analysis||100||.mm ||User story: 5.3 Observations Quality Control - ET-CDMS||Observations QC/QA||2016-11||Work is required to assess to suitability of using the ISO Standard 19157, Geographic Information - Data Quality: as a data quality framework for Climate Data|
|5.4 Quality assessment||Preliminary analysis||100||.mm |
|5.5 Climate metadata||Preliminary analysis||100||.mm ||User story: 4.3.3 Climate Data Provenance - W3C|
|6. Climate data analysis||Detailed Analysis||10||.mm ||User story: 6.1 Analyse and Generate Climate Data - Generic||2016-11|
|6.1.1 Climate modelling||Preliminary analysis||100|
|6.1.2 Generate derived data from climate observations||Preliminary analysis||100||User story: 6.1.2 Generate climate monitoring grids - ET-CDMS-1 |
User story: 6.1.2 Generate climate monitoring grids - derived from remotely sensed imagery - ET-CDMS-1
User story: 6.1.2 Virtual Observations - W3C
User story: 6.1.2 Generate Standard WMO Data Product
User story: 6.1.2 Generate Observations Statistical Data
|Refer '6.1.2 Generate Observations Statistical Data' to IPET-CDMP for advice on how we determine consistent rules to use for the generation of the statistics|
NB: We may need to move the user story 6.1.2 Generate WMO Standard Data Product under 5.1.2 Ingest and Extract.
To be confirmed.
It will depend on the level of analytical capability required to generate the WMO data products
|6.1.3 Data homogenization||Preliminary analysis||100||User story: 6.1.3 Climate Data Homogenisation - ET-CDMS-1||This component needs to be split between QC and also analysis|
|7. Climate data presentation||Preliminary analysis||100||.mm ||User story: 7.1 CDMS GUI - Data Exploration - ET-CDMS-1||2016-08||Also known as the User Interface|
|7.1.1 Tables and charts||Preliminary analysis||100|
|7.1.2 Manage content||Preliminary analysis||100|
|7.1.3 Visualization||Preliminary analysis||100|
|7.1.4 Integrated search of climate data||Preliminary analysis||100||User story: 7.1.4 Integrated Search - Find and Flag anomalous values - ET-CDMS-1 |
User story: 7.1.4 Integrated Search - classify observations by data quality - ET-CDMS-1
User story: 7.1.4 Integrated Search - Determine Prevailing Winds at Location - ET-CDMS-1
User story: 7.1.4 Recover from decommissioned station using Integrated Search GUI - ET-CDMS-1
User story: 7.1.4 aggregate climate stations using Integrated Search GUI - ET-CDMS-1
|7.1.5 Data download||Preliminary analysis||100|
|8. Climate data delivery services||Detailed Analysis||75||Climate data services||2016-11|
|8.1 Open spatial standards||Detailed Analysis||75||User story: 8.1 Federated Global Data Service|
|8.2 Data discovery||Preliminary analysis||100||User story: 8.2 Discover data provenance - ET-CDMS-1 |
User story: 8.2 Data Discovery - ET-CDMS
User story: 8.2.2 Linked Data access to observations - W3C
|This needs considerable work|
|8.3 Other formats||Preliminary analysis||100|
|9. Core IT infrastructure||Preliminary analysis||100||.mm ||2016-08|
|Configuration||Detailed Analysis||25||.mm ||User story: xxx CDMS Configuration - User Management |
User story: xxx CDMS Configuration - New Data Product approval
User story: xxx CDMS Configuration - New Station
User story: xxx CDMS Configuration - New Observed Variable
User story: xxx CDMS Configuration - New Sensor
|2016-11||Need much more work on analysis. Need to create user stories from current mind map analysis and add current user stories to mind map.|
|Climate Data Management||Preliminary analysis||25||User story: xxx Climate Data Management - Manage Change in Station Location||2017-01||_Manage Change in Station Location_ user story needs careful thought and analysis. It has significant implications.|
As a general note for climate data management, assume that:
- All observation data undergoes a suite of automated QC and QA processes
- Some data will undergo a further manual QC/QA process if it is deemed appropriate
- All data will be available via the climate data services and download
- The NMHS? or WMO? policy will define what level of data quality (QA Flag) is available for download
We should perhaps really aim for a position where all data is available for download, warts and all with appropriate data description in the Discovery Metadata and Data Provenance Metadata. Then it is a case of _user beware_. The data may be useful for specific use cases.
|Data Intellectual Property||Not Started||0||2016-11||Functionality is required to store and manage the Intellectual Property of data. This includes:|
- Who ‘owns' the source data?
- Has the person or organisation capturing the historical data assigned their IP rights in the data (this is particularly important with crowd-sourced data)?
- Are there any constraints on the use of the data by others?
- Copy of Data License Agreement.
This component will need to cater for datasets and for data of mixed IP where source data comes from a number of sources.
This is particularly important as NMHS move to embrace Citizen Volunteered data (e.g. 3rd Party data).
ISO standards. These documents are provided to WMO under a licence that only allows their use for developing WMO standards.