Gaia Data Processing and Analysis Consortium (DPAC)

Gaia DPAC at its 3rd consortium meeting in Abano Terme, Italy - October 2018. The consortium meeting that was planned in March 2020 was cancelled due to the pandemic. Credit: ESA/Gaia/DPAC

Gaia Data processing and analysis consortium (DPAC)

Gaia DPAC is responsible for the processing of the Gaia data from its raw format (as it comes down from the spacecraft) into scientific data (as served from the Gaia Archive). The processing consists of many different steps and iterations and takes place at several data processing centres. The full cycle of processing is explained in detail in the Gaia Data Release Documentation and in the Gaia mission paper.

The data comes down from the spacecraft and is gathered in the main database located at ESAC, Spain. One of the first steps in the processing consists of checking the validity of the data when it just came down from the spacecraft. The FirstLook team comes in play here. They perform a first 'health check' on the data to see if it is in the shape as expected. After this quality control, the data goes into the initial data treatment (IDT) pre-processing where, among other things, the many observations are assigned to sources. The systems responsible for alerts to the community make use of this daily IDT output and when needed, Gaia Photometric Science alerts as well as Solar System alerts are published. The pre-processed data is subsequently passed along to the different Gaia data processing centres.

The astrometric processing is in the hands of Coordination Unit 3. The intermediate data update (IDU) updates calibrations, performs a run to improve the links between observations and sources and computes first estimates for the astrometric parameters. The AGIS pipeline (AGIS stands for Astrometric Global Iterative Solution) run produces the first data in scientific format. Its output are the positions, parallaxes, and proper motions of all objects, together with the spacecraft attitude. The Global Sphere Reconstruction (AVU-GSR) provides an independent verification method for solving the astrometric parameters.

More or less in parallel, photometric processing takes place, in which the brightnesses and blue and red photometer magnitudes are determined for all sources. The photometric processing is undertaken by Coordination Unit 5 and uses the PhotPipe processing pipeline.

The data then continues its route and the remaining processing activities start to become parallel. This processing is currently ongoing and the results from these processing pipelines will be published in the full Gaia data release 3, expected in 2022. This entails the radial velocities as produced by Coordination Unit 6, the classifications of variable stars as performed by Coordination Unit 7, the astrophysical parameters computed by Coordination Unit 8, and the solar system objects, non-single stars, and extended objects as processed by Coordination Unit 4.

To ensure that the data is of good quality and can be accessed by the community, Coordination Unit 9 is in charge of "Catalogue Access". This Unit performs a validation of the data and provides valuable feedback to the different Coordination Units on the quality of "their" data to allow further improvements in future processing rounds. Coordination Unit 9 also prepares the data to make it available to the world. The processing of the data needs documenting, to allow users to understand the different steps and the choices that have been made and to help users access the data.

The Coordination Units are not only working when the actual processing is ongoing in the data processing centres. In advance of processing, time is spent on calibration and further development of the software as well as on preparations for upcoming, larger data volumes. After each processing run, the teams are busy checking the validity of the data and discussing required adjustments and filtering. Added complexity comes from the scientific interfaces between the various teams which need to be aligned carefully to allow for the Gaia catalogue to be consistent across all data products.

​​​​​​​​​​​​​​

Left: This map gives an overview of the countries involved in the creation of the Gaia EDR3 catalogue (countries as shown in red). The institutes involved in Gaia DPAC are extracted from the affiliations of the authors on the Gaia EDR3 Collaboration papers. No difference is made between the activity of one person at an institute in a country as compared to the activity of a large group of people active in multiple institutes in another country. Image credit: ESA/Gaia/DPAC, CC BY-SA 3.0 IGO

​​​​​​​Right: Participants of the 2nd DPAC Consortium Meeting in Sitges, Spain. Image credit: ESA/Gaia/DPAC

DPAC statistics

As of December 2020, Gaia DPAC consists of 438 scientists and engineers from many different cultural backgrounds and speaking many different languages. This number of members does not reflect the people available fulltime for Gaia. Many of our Gaia DPAC members have other duties next to Gaia data processing, like research and teaching at universities or helping out with preparations or the processing for other space missions. The number changes over time, as new members join Gaia DPAC and others pursue their career outside the Gaia mission. The total amount has been for several years now between 400 and 450.

Below are some statistics of DPAC and its Coordination Units. The statistics show that there is an overlap between the people active in the Gaia Data Processing Centres and the people working in the different Coordination Units. One and the same member can be active in multiple teams. We have not corrected for this: such persons are counted as full member for each group in such cases. Finally, the numbers on gender are very crude assumptions based on the name of our DPAC members (we do not store data on people's gender).

​​​​​​​

DPAC statistics (November 2020)

DPAC Gender statistics (November 2020)

DPAC during the pandemic

Anthony Brown - Gaia DPAC chair - about the impact of the pandemic on Gaia DPAC. Credit: ESA/Gaia/DPAC

Published: 3 December 2020 - Credit: ESA/Gaia/DPAC

​

We would like to thank all Gaia DPAC members for their dedication and the enthusiam they put in their work for the Gaia Mission. Without them, we could not produce these Gaia catalogues.  A big thanks to all Gaia DPAC teams across Europe and beyond.

​​​​​​​The Gaia Collaboration