Image of the Week

the Gaia ESA Archive: a first step towards GAia Data release 4

Figure 1. The new appearance of the DataLink wizard as well as an excerpt of the content of the new Gaia DR3 epoch photometry serialized in the INDIVIDUAL data structure. Needless to say, the Gaia ESA Archive release 3.6 will be accompanied by a major update of the DataLink-related content in the Archive Help pages to reflect and document all these changes.

At first sight, the Gaia ESA Archive may look static these days, but in fact, nothing could be further from the truth. The team of expert engineers and scientists at the ESAC Science Data Centre (ESDC) supporting the Archive is busy with various background preparations necessary to serve the many products that will be published in Gaia Data Release 4. This vast dataset will represent a giant leap in terms of product diversity and data volume compared to previous Gaia releases. This applies both to catalogues accessible via the TAP protocol (listed in the tables tree area of the ADQL “Advanced” form of the web interface) and to non-tabular products, such as spectra and light curves, that are served via the DataLink protocol.

In preparation of Gaia Data Release 4 (Gaia DR4), the next imminent update of the Gaia ESA Archive (version 3.6 – updates will be posted to the Archive release notes here) introduces several changes intended to enhance the user experience when retrieving DataLink products. To begin with, the file-writing optional functionality of the Astroquery.Gaia.load_data method, which retrieves DataLink products, has been optimized (this FAQ explains how to install the latest Astroquery version to benefit from this update).

In addition, the DataLink wizard in the Archive web interface (for more information, see this link) has been updated (see the above Figure 1). First, the names of the DataLink products will appear in alphabetical order - a minor change that will be particularly helpful for Gaia DR4 when many products will be listed in the wizard. Second, the “Data release” drop-down menu will be removed, as the Gaia data release tag name will be automatically retrieved from the job metadata (i.e., the output of an ADQL query; see the Writing queries section of the Archive Help pages for examples).

Another important change concerns the data structure of the DataLink products (the serialization format is described in the DataLink products serialization tutorial). The COMBINED data structure, which stores a given DataLink product for multiple sources into a single file, has been deprecated because it does not scale efficiently. In other words, serving the largest Gaia DR4 DataLink products in this structure would be unfeasible.

But perhaps the most relevant update – and breaking change – is the new serialization of the Gaia DR3 epoch photometry (light curves) in the INDIVIDUAL data structure. The new serialization contains the source_id (and solution_id) information in the file’s metadata, and it is now compliant with the data model proposed in the Time Series: Annotation of light curves in VOTable document. This will facilitate the inspection of this product using IVOA-compliant software (for example, by simplifying the creation of light curves in the three Gaia photometric passbands with TOPCAT). Also important, the new serialization will also be adopted for the Gaia DR4 epoch photometry and therefore this change will allow users to get prepared well in advance for Gaia’s next data release. For reference, this .zip file contains the Gaia DR3 epoch photometry for one random source serialized in the new INDIVIDUAL data structure in all the file output formats served by the Archive.

The changes described above will be implemented on 10 December when the Gaia Archive is upgraded to ESA Gaia Archive version 3.6.

The Gaia Archive will be down on 10 December between 08:00 and 10:00 CET.

At that day, also the tutorials as mentioned above will be updated.

Story written by Héctor Cánovas and Jos de Bruijne

Credits: ESA/Gaia/DPAC

[Published: 03/12/2024]