Met Office Hadley Centre observations datasets
Home > HadIOD > 

HadIOD in NEMO NetCDF feedback file format


Announcements:

None at this time

Introduction

At present feedback format data are ony available to Met Office users. These webpages provide basic information for those interested in this dataset. Please contact us if you are interested in using these data.

NEMO NetCDF feedback format (hereafter ‘feedback format’) is a format used for model data assimilation (see NEMO-OCE pdf). A tool to extract data from the HadIOD database into feedback format was created as part of the ERA-CLIM2 project. Feedback format data from HadIOD.1.1.0.0 and HadIOD.1.2.0.0 were made available to ERA-CLIM2 participants and as a pre-release to Met Office colleagues. The HadIOD feedback format files have been through a few minor iterations following user feedback, each having its own version number (see 'Dataset versions' below).

File description

Data are provided in daily files, the current version is v1.2 (see below). The v1.2 files presently cover 1978-present but could be extracted to cover other periods (HadIOD.1.2.0.0 covers 1850-present). For each day, two files are provided, one containing ocean profile temperature (in situ and potential temperature) and salinity data and the other containing sea surface temperature data. Each file contains the observed data along with metadata, bias corrections, measurement uncertainties and quality control decisions for each observation. The files contain several different bias correction and uncertainty variables; it is not possible to specify values for all of these variables for every observation and unspecified values are set to missing data. At a minimum we try to provide a quality flag and some estimate of uncertainty in each measurement (in some cases providing an uncertainty may not be possible). File contents and code tables used by certain variables are described in 'File contents' below. Sample file headers can be seen here:

Observed data are provided in 2D arrays (with stations on one axis and vertical levels on the other) and their station metadata are provided in 1D arrays. For profile data the vertical level dimension is 400 (i.e. a maximum of 400 vertical levels per profile) and for surface data the vertical level dimension is set to 1 (as these sample a single vertical level).

User guide and supplementary files

At present there is no user guide for the feedback format files. Basic information is provided on these webpages. We also strongly recommend reading the product user guide available for the HadIOD files which is of relevance here, in particular Section 2, which gives useful general information about HadIOD data and, importantly, describes the HadIOD observation error model and how to work with the bias corrections and uncertainties provided by HadIOD. Because the names used for the bias correction and uncertainty variables in the feedback format files differ from the variable names used in the product user guide, the 'File Contents' section below includes a table showing which variable names are equivalent. The 'File Contents' section also has some further remarks comparing the two file types.

The HadIOD dataset includes supplementary files that provide additional sets of bias corrections to help users explore the bias correction uncertainty. The component of uncertainty explored by these corrections is increasingly important on longer space and time scales (e.g. if concerned with decadal trends). The use of these additional corrections is recommended and is described in the product user guide. The supplementary files could be used with the feedback format files and are linkable via the DATABASE_IDENTIFIER variable.

Please contact us if you need help.

Dataset versions

The feedback format file versions listed below are for data extracted from HadIOD.1.2.0.0. During development some preliminary feedback format data were extracted from HadIOD.1.1.0.0 with processing similar (but not identical) to v1.0 below.

sst files

profile files

Data sources (HadIOD.1.2.0.0)

Observations from several different sources have been ingested into the HadIOD.1.2.0.0 database. The following sources are used in the feedback format files (CMEMS was included in the sst files from v1.1 onwards):

Terms & Conditions / Citing the data

HadIOD is subject to Crown copyright protection and is provided under a Non-Commercial Government Licence. Please do read this. Any onward licencing of the information must be for Non-Commercial purposes only.

When publishing work using the data, please quote the HadIOD version number (found in the files' 'source' attribute, see 'File contents' below), the feedback file version number (found in the files' 'file_version' attribute, see 'File contents' below), and cite the following:

Atkinson, C. P., N. A. Rayner, J. J. Kennedy and S. A. Good, 2014. An Integrated Database of Ocean Temperature and Salinity Observations, Journal of Geophysical Research: Oceans, 119, 7139-7163, doi:10.1002/2014JC010053

For example:

Users must also acknowledge the following:

If applicable, for traceability, please describe which HadIOD bias corrections were used. This information can be found in the file global attribute ‘source’. This is particularly important if any additional corrections from the supplementary files were used. See also 'how to cite the data' in the HadIOD product user guide for more information and references for bias corrections. For example:

Contact us

Please contact us if you need any assistance.

File contents

File contents are summarised in tables below.

A station (located horizontally and in time) may make one or more vertical profiles, with each profile comprising one or more observed levels (depths) of some geophysical variable. Variables listed below as 'station variables' contain station metadata (e.g. time or observation type) which apply to every observed level at a station. Variables listed below as 'data variables' contain the profile observations (temperature or salinity) and observation metadata (i.e. metadata which are potentially unique to each observation e.g. depth or a bias correction). The 'station variables' are 1D arrays with dimension N_OBS and the 'data variables' are 2D arrays with dimensions N_OBS x N_LEVELS. In the SST files N_LEVELS=1 and in the profile files N_LEVELS=400. A platform (e.g. a ship, buoy or profiling float) may make multiple stations; stations made by the same platform should share the same STATION_IDENTIFIER and STATION_TYPE (though in practice may not e.g. due to ID transcription errors).

Some general remarks on HadIOD Feedback files c.f. HadIOD files:

See also the 'User Guide' section above.

SST files


NetCDF variable Description
Dimensions
N_OBS Number of stations in the file.
N_LEVELS Maximum number of vertical levels for each station.
N_VARS Number of observed geophysical variables in file.
STRINGNAM Maximum length in characters of the VARIABLES listing the geophysical variables in the file.
STRINGWMO Maximum length in characters of the STATION_IDENTIFIER used to identify a platform.
STRINGTYP Maximum length in characters of the STATION_TYPE used to identify the platform/instrument type.
STRINGJULD Length in characters of the JULD_REFERENCE (YYYYMMDDHHMMSS) which is the date of reference for JULD.
File Variables
VARIABLES List of geophysical variables in the file.
JULD_REFERENCE Date of reference for JULD.
Station Variables
STATION_IDENTIFIER Platform identifier (For ICOADS obs platform IDs are truncated to 8-characters to fit).
STATION_TYPE Code identifying the platform/instrument type.
LONGITUDE Longitude of station.
LATITUDE Latitude of station.
JULD Date/Time of station.
OBSERVATION_QC Observation quality flag (this flags near-surface temperature observations taken from ocean profiles that are non-preferred likely duplicates of data in the profile files. If using both the SST and profile files this flag should be applied).
POSITION QC Station position quality flag.
JULD_QC Station date/time quality flag.
SST_QC Sea surface temperature quality flag.
Data Variables
DEPTH Depth of an observation.
DEPTH_CORR Depth bias correction. Added to depth to help remove biases in reported depth.
DEPTH_QC Depth quality flag.
SST_OBS Observed sea surface temperature.
SST_LEVEL_QC Sea surface temperature quality flag.
SST_OBS_TYPE_CORR Temperature correction for observations made by the given platform type (e.g. ship), see STATION_TYPE. Added to observed temperature to help remove biases in the observed quantity.
SST_OBS_TYPE_CORR_UNC The uncertainty in the value of SST_OBS_TYPE_CORR (std. dev.). Once the correction is applied, this describes the spread of a probability density function from which the (unknown) residual bias is drawn. The error (residual bias) associated with this uncertainty can be correlated with errors in other observations of the same platform type.
SST_OBS_STAT_CORR Temperature correction for observations made by an individual platform (e.g. an individual ship), see STATION_IDENTIFIER. Added to observed temperature to help remove biases in the observed quantity.
SST_OBS_STAT_CORR_UNC The uncertainty in the value of SST_OBS_STAT_CORR (std. dev.). Once the correction is applied, this describes the spread of a probability density function from which the (unknown) residual bias is drawn. The error (residual bias) associated with this uncertainty can be correlated with errors in other observations from the same individual platform.
SST_OBS_RAND_UNC The random measurement error uncertainty (std. dev.) associated with the observed temperature. The error associated with this uncertainty is not correlated with errors in other observations.
SST_OBS_COMB_UNC The uncertainty obtained by combining all other temperature uncertainty terms in quadrature (std. dev.). Valid when corrections have been applied.
DATABASE_IDENTIFIER HadIOD database identifier for an observation. This is an integer that uniquely identifies an observation within its year and month. When combined with year and month the identifier is unique within the version of HadIOD used (e.g. HadIOD.1.2.0.0).
Global Attributes
description Describes the file (e.g. 'surface feedback file').
file_version States the file version (e.g. '1.2').
source Gives more details about the source of the data in the file, e.g. HadIOD version number and which bias correction schemes were used.
usage Gives any rules that must be followed when using the data in this file (e.g. how bias corrections should be applied).
history States when the file was created, it may also record any subsequent modifications to the file.
title 'NEMO observation operator output'.

Profile files


NetCDF variable Description
Dimensions
N_OBS Number of stations in the file.
N_LEVELS Maximum number of vertical levels for each station.
N_VARS Number of observed geophysical variables in file.
N_EXTRA Number of extra observed geophysical variables in file.
STRINGNAM Maximum length in characters of the VARIABLES listing the geophysical variables in the file.
STRINGWMO Maximum length in characters of the STATION_IDENTIFIER used to identify a platform.
STRINGTYP Maximum length in characters of the STATION_TYPE used to identify the platform/instrument type.
STRINGJULD Length in characters of the JULD_REFERENCE (YYYYMMDDHHMMSS) which is the date of reference for JULD.
File Variables
VARIABLES List of geophysical variables in the file.
EXTRA List of extra geophysical variables in the file.
JULD_REFERENCE Date of reference for JULD.
Station Variables
STATION_IDENTIFIER Platform identifier.
STATION_TYPE Code identifying the platform/instrument type.
LONGITUDE Longitude of station.
LATITUDE Latitude of station.
JULD Date/Time of station.
OBSERVATION_QC Station quality flag.
POSITION QC Station position quality flag.
JULD_QC Station date/time quality flag.
POTM_QC Temperature profile quality flag.
PSAL_QC Salinity profile quality flag.
Data Variables
DEPTH Depth of an observation.
DEPTH_CORR Depth bias correction. Added to depth to help remove biases in reported depth.
DEPTH_QC Depth quality flag.
POTM_OBS Observation potential temperature.
POTM_LEVEL_QC Temperature level quality flag.
POTM_OBS_TYPE_CORR Temperature correction for observations made by a given platform type (e.g. XBT), see STATION_TYPE. Added to temperature to help remove biases.
POTM_OBS_RAND_UNC The random measurement error uncertainty (std. dev.) associated with the observed temperature. The error associated with this uncertainty is not correlated with errors in other observations.
PSAL_OBS Observed practical salinity.
PSAL_LEVEL_QC Practical salinity level quality flag.
PSAL_OBS_RAND_UNC The random measurement error uncertainty (std. dev.) associated with the observed salinity. The error associated with this uncertainty is not correlated with errors in other observations.
TEMP Observed in situ temperature
POTM_DATABASE_IDENTIFIER HadIOD database identifier for a temperature observation. This is an integer that uniquely identifies an observation within its year and month. When combined with year and month the identifier is unique within the version of HadIOD used (e.g. HadIOD.1.2.0.0).
PSAL_DATABASE_IDENTIFIER HadIOD database identifier for a salinity observation. This is an integer that uniquely identifies an observation within its year and month. When combined with year and month the identifier is unique within the version of HadIOD used (e.g. HadIOD.1.2.0.0).
Global Attributes
description Describes the file (e.g. 'subsurface feedback file').
file_version States the file version (e.g. '1.2').
source Gives more details about the source of the data in the file, e.g. HadIOD version number and which bias correction schemes were used.
usage Gives any rules that must be followed when using the data in this file (e.g. how bias corrections should be applied).
history States when the file was created, it may also record any subsequent modifications to the file.
title 'NEMO observation operator output'.

Variables explained

Further useful information about some of the variables is given below.


'STATION_TYPE' variable values

This is the NEMO station type. For EN4 observations we use the EN4 WMO_INST_TYPE variable (see EN4 profile file format) which is a simplified subset of WMO code table 1770. For other source data we map to the categories below. These are only broad categories and some observations may not be in the correct category.

Value Platform/Instrument Type
50 non-automatic ship
55 moored buoy in buoy or ship code
53 drifting buoy in ship or buoy code
58 platform/rigg
401 XBT/MBT
741 TESAC
831 profiling floats

820 BUOYS

SST file QC variables

The Met Office Hadley Centre QC suite used for the ICOADS and CMEMS observations comprises basic sanity checks, position track check, climatology check and buddy check. Flags from the individual checks have been combined and mapped to the feedback format QC variables as listed below. If any of the individual checks fail, the corresponding feedback format QC flag is set to fail.

An updated version of the Met Office Hadley Centre QC suite (MDS4) has been used for the CMEMS observations. This performs a similar suite of checks to the previous suite (MDS3) used for the ICOADS observations but with improvements to the performance of checks such as the track, climatology and buddy checks. A repeated value check was also added.

The duplicate check used for ICOADS observations is created within the HadIOD suite. This flags near-surface temperature observations taken from ocean profiles that are non-preferred likely duplicates of data in the profile files. If using both the SST and profile feedback format files this flag should be applied. For CMEMS there is minimal expected duplication with the profile data and so a duplicate check is not applied.

The SST track check used for ICOADS.2.5.1 (drifting buoy observations only) is an updated version of the drifting buoy QC checks described in Atkinson et al. [2013]. SSTs from individual drifters are tracked against a satellite-based reference and persistently poor quality observations are flagged. Position reports are used to flag buoys that have run aground or been picked-up. The coverage of the check is 1986-2015.

Feedback format flags use the following codes: 1=pass, 4=fail.

QC variable Notes

OBSERVATION_QC

ICOADS: non-preferred likely duplicate of ocean profile data.

CMEMS: not used (from v1.2 onwards this is set to '1' (pass), in v1.1 it is set to missing value).

POSITION_QC

ICOADS: implausible position, over land, track check, blacklisted.

CMEMS: implausible position, track check, blacklisted (note that there is no 'over land' check, this is handled by the sst climatology check).

JULD_QC

ICOADS: bad time, bad date.

CMEMS: bad time, bad date.

SST_QC

ICOADS: sst buddy check, sst climatology check, no sst normal, sst below-freezing, sst blacklist, sst track check (drifting buoys), sst missing.

CMEMS: sst bayesian buddy check, sst climatology check, no sst normal, sst below-freezing, repeated value, sst missing.

SST_LEVEL_QC

As for SST_QC.

DEPTH_QC

Not used.


Profile file QC variables

The quality controlled profile observations in HadIOD are taken from EN4. The EN4 profile QC has many individual checks (see EN4 profile file format). In the EN4 data the overall outcomes of groups of individual checks are summarised in the POSITION_QC, PROFILE_POTM_QC, PROFILE_PSAL_QC, POTM_CORRECTED_QC and PSAL_CORRECTED_QC variables. These are mapped to equivalent variables in the profile feedback format files as shown below. The OBSERVATION_QC, POTM_QC and PSAL_QC variables apply to all levels in a profile. The POTM_LEVEL_QC and PSAL_LEVEL_QC variables are for individual levels in a profile.

Feedback format flags use the following codes: 1=pass, 4=fail.

QC variable Notes
OBSERVATION_QC EN4: POSITION_QC (reject all station data).
POTM_QC EN4: PROFILE_POTM_QC (reject temperature profile).
PSAL_QC EN4: PROFILE_PSAL_QC (reject salinity profile).
POTM_LEVEL_QC EN4: POTM_CORRECTED_QC (reject temperature level).
PSAL_LEVEL_QC EN4: PSAL_CORRECTED_QC (reject salinity level).
POSITION_QC Not used.
JULD_QC Not used.
DEPTH_QC Not used.

Bias correction and uncertainty variables

Bias corrections should be added to the corresponding observations, e.g.:

  • SST_CORRECTED = SST_OBS + SST_OBS_TYPE_CORR + SST_OBS_STAT_CORR
  • DEPTH_CORRECTED = DEPTH + DEPTH_CORR

Further rules on applying the corrections may be given in the files' 'usage' global attribute. Do read this. For example, the 'usage' may state that "DEPTH_CORR and SST_OBS_TYPE_CORR must be applied together (and not independently)" i.e. if applying SST_OBS_TYPE_CORR to SST_OBS, then DEPTH_CORR must also be applied to DEPTH (this might be because, for example, SST corrections are made relative to some reference level).

The feedback format files use the same HadIOD observation error model as the HadIOD files. The HadIOD product user guide explains the HadIOD error model and how to work with the bias corrections and uncertainties provided by HadIOD. Reading Section 2 of the user guide is strongly recommended. Because the names used for the bias correction and uncertainty variables in the feedback format files differ from the variable names used in the product user guide, the table below lists which variable names are equivalent.

Note that the profile feedback format files do not include bias correction and uncertainty variables for quantities that are not currently specified for profile data (shown as N/A).

SST feedback file variable Profile feedback file variable HadIOD file variable
SST_OBS_TYPE_CORR POTM_OBS_TYPE_CORR temp_type_corr
SST_OBS_TYPE_CORR_UNC N/A temp_type_corr_unc
SST_OBS_STAT_CORR N/A temp_plat_corr
SST_OBS_STAT_CORR_UNC N/A temp_plat_corr_unc

SST_OBS_RAND_UNC

POTM_OBS_RAND_UNC

PSAL_OBS_RAND_UNC

temp_rand_unc

sal_rand_unc

SST_OBS_COMB_UNC N/A temp_comb_unc


Commercial and media enquiries

You can access the Met Office Customer Centre, any time of the day or night by phone, fax or e-mail. Trained staff will help you find the information or products that are right for you.
Contact the Met Office Customer Centre