Summary of the archive process

The PSA provides expert consultancy to all of the data producers throughout the archiving process. As soon as an instrument is selected, PSA begin working with the instrument team to define a set of data products and data set structures that will be suitable for ingestion into the long-term archive.

A Data Archive Working Group (DAWG) is established for each mission and meets regularly to discuss archiving matters at the top level. In these meetings the data types, data structures and the data set organisation are discussed and defined for each instrument. Many parameters used within the archive are applicable to all instruments on a mission, and these 'mission-wide' issues are agreed upon during the DAWG meetings as well.

All data providers are strongly encouraged to use the same data formats and data sets internally as those that will be delivered to the PSA. This way, only one data pipeline is required, converting from raw instrument telemetry through to archivable products. The development of the data pipeline is closely monitored by the PSA through the DAWG meetings and individual meetings and teleconferences. An 'Archive Scientist' is appointed for every mission to support the development of the archives and provide immediate consultancy to all teams, ensuring that the data sets being constructed are compliant to the PDS Standards required by the PSA, and that all necessary data/information is provided for an end-user to manipulate the data.

Consultancy is also provided to support the instrument teams with their documentation. All teams are required to include a number of documents in their data sets. Chief among these is the Experiment to Archive Interface Control Document (EAICD), which can be found in the DOCUMENT directory for all ESA planetary mission data sets. The EAICD provides a full description of the data set(s) being archived by an instrument, including a summary of the data pipeline, the structure of the data products, data sets and any calibration or geometry information provided. This document should be the starting point for anybody when trying to understand a data set.

The aim is to have data pipelines and supporting documentation prepared before launch so that a team can immediately begin delivering data without the need for considerable development during the active mission phase. During the mission, data are delivered to the PSA at regular pre-determined intervals. Typically deliveries are made every 3 to 6 months, although this depends upon how active a mission is in collecting data (e.g. cruise phase deliveries may be more sporadic if a spacecraft is not actively collecting science data for longer periods).

All data from ESA's planetary missions have a proprietary period, usually between 6 and 12 months. During this time, the data are only available to instrument team members. At the close of the proprietary period, the data are released and made available to any user via the PSA.

At the close of a mission's active operations, the archive will undergo a period of maintenance during which updates from the Principal Investigator teams are accepted and supplementary data sets can be added to the PSA for long-term archiving. This should be the closing phase of an archive, preparing the data for long-term storage and ensuring that all data within the archive will be useful and understandable in years to come without further significant updates.

Each phase of the archive process is controlled by a corresponding external peer review, during which external experts are asked to validate the data and documentation for their suitability for long-term archiving. Typically, the review process follows the three major steps in archive development:

  • first review phase concentrating on the EAICD and the initial pipeline
  • second review phase concentrating on the first data delivery
  • third review phase reviewing the full archive at the end of a mission