DataLink Service

Authors: Héctor Cánovas and Jos de Bruijne

In each Gaia data release, the key parameters of all sources are stored in the gaia_source table that contains the (mean) astrometric, photometric, and radial-velocity data as well as astrophysical parameters. This table, along with the complementary tables for, for instance, variable stars or solar system objects, is accessible by means of the IVOA compliant TAP+ table access protocol that allows to explore astronomical datasets stored in relational data bases using the ADQL query language. In addition to the gaia_source table, Gaia DR3 includes vast amounts of non-tabular data such as mean spectra, epoch photometry, and Monte Carlo Markov Chain samples for millions of sources (while Gaia DR4 will include epoch astrometry and epoch photometry for the whole sample plus billions of mean and epoch spectra). Storing these non-tabular datasets as plain tables in a monolithic, relational data base is impractical. Instead, these products are hosted by a dedicated service designed to handle massive data requests that is accessible via the DataLink protocol. DataLink is a data access protocol compliant with the IVOA architecture that provides a linking mechanism between datasets offered by different services. In practice, it can be seen and used as a web service providing the list of additional data products available for each object outside the main catalogue(s).

Since the Archive upgrade to version 2.14 the VOTables generated by the Archive contain a new resource that facilitates the access to the DataLink products to IVOA compliant clients (like TOPCAT):

</RESOURCE> <RESOURCE type="meta" utype="adhoc:service" name="ancillary"> <DESCRIPTION>Retrieve DataLink file containing ancillary data for source</DESCRIPTION> <PARAM name="standardID" datatype="char" arraysize="*" value="ivo://ivoa.net/std/DataLink#links-1.0"/> <PARAM name="accessURL" datatype="char" arraysize="*" value="https://gea.esac.esa.int/data-server/datalink/links"/> <PARAM name="contentType" datatype="char" arraysize="*" value="application/x-votable+xml;content=datalink"/> <GROUP name="inputParams"> <PARAM name="ID"  datatype="char" arraysize="*" value="" ref="DESIGNATION"/> </GROUP> </RESOURCE>

The entry point to the DataLink server is indicated by the "accessURL" parameter. To invoke the service and find out the resources associated to a given source it is necessary to combine the entry point with the target ID as:  

https://gea.esac.esa.int/data-server/datalink/links?ID=Gaia+DR3+30343944744320

The output is a xml file not intended for humans but for IVOA-compliant clients. Opening this file with TOPCAT reveals the following content:

Figure 1: Content of the xml file generated by the DataLink service when invoked as explained above.

This file contains the URLs that give access to the DataLink products associated to the target source. For information about how to access to these products from the Gaia Archive web interface and programmatically, please see the DataLink: Access from the Archive GUI and the Command line access: DataLink tutorials, respectively. The structure and content of the DataLink products are described in the Datamodel Chapter in the Gaia DR3 documentation and DataLink products serialisation tutorial.

DataLink: Access from the Archive web interface

Authors: Héctor Cánovas, Jos de Bruijne, and Alcione Mora

Last update: 2024-12-10

The Gaia DR3 data release includes vast amounts of non-tabular data such as high- and low-resolution (mean) spectra, epoch photometry, and Monte Carlo Markov Chain samples for millions of sources. These products are made available through a dedicated service designed to handle massive data requests using the DataLink protocol. This intermediate-level tutorial introduces the concepts needed to retrieve these non-tabular products using the Gaia ESA Archive web interface via its Advanced (ADQL) form. The complementary DataLink: command line access and DataLink: Python access tutorials describe the programmatic access to these products using the Unix curl command-line utility and the Python package Astroquery.Gaia, respectively, while the DataLink products serialisation tutorial describes the structure of these products. In case of difficulties following this tutorial, please consult the DataLink service and Advanced (ADQL) tab tutorials.

Tutorial content:

1. How it works
2. Basic use case
3. FPR tables
4. User tables
   1. List of sources
   2. List of coordinates

1. How it works

As explained in the DataLink Service tutorial above, the DataLink protocol allows users to search for and retrieve DataLink products associated with a list of Gaia designations, or alternatively, with a combination of Gaia source IDs and Gaia data releases. Direct access to this service through the web interface is not possible. Instead, users must first execute an ADQL query that generates an output table containing the source_id and/or designation fields. Since the release of GACSv3.6, the output table metadata includes a Gaia release tag (e.g., "Gaia DR3" or "Gaia DR2"), which is automatically fetched by the Archive web interface when preparing the request to the DataLink server.

2. Basic use case

One of the simplest use cases is likely: "I want to search for the DataLink products associated with the output of this Gaia DR3 query." The ADQL query in the example below performs a cone search with a radius of 0.25 degrees (see the Query examples section for more complex queries):

SELECT DISTANCE(266.41683, -29.00781, ra, dec) AS separation, * FROM gaiadr3.gaia_source WHERE DISTANCE(266.41683, -29.00781, ra, dec) < 0.25ORDER BY separation ASC

The first step to retrieve the DataLink products associated with this sample is to click on the double chain ("paperclip") icon in the job list area of the Advanced (ADQL) form. In this case, an error message will appear, explaining that the query result contains 19,758 sources, exceeding the threshold of 5,000 sources imposed to prevent overloading the DataLink server. To resolve this, you can either reduce the cone search radius or, preferably, use the has_<datalink_product> fields available in the gaiadr3.gaia_source table as follows:

SELECT DISTANCE(266.41683, -29.00781,ra, dec) AS separation, * FROM gaiadr3.gaia_source WHERE DISTANCE(266.41683, -29.00781, ra, dec) <0.25 AND-- Retrieve only sources with associated DataLink products 
  has_epoch_photometry ='True' ANDhas_xp_sampled = 'True'
  ORDER BY separation ASC

The output of this query contains just 26 sources, all of which have epoch photometry and both XP sampled and XP continuous spectra. Note: advanced users can use the job_upload mechanism to apply these filters to their previous queries. Clicking the double chain icon in the job list area again will launch the DataLink wizard, as shown in Fig. 1. This window lists all the available products associated with the sample generated by the previous query. One can retrieve only selected products (e.g., just RVS mean spectra) or download all products at once by clicking the "Save All Data" button. Note that, depending on the volume of data, preparing the dataset for download may take several minutes. In the DataLink wizard, one can also select different combinations of data structures and download formats (see the DataLink: products serialisation tutorial for details).

Figure 1: Screenshot of the Gaia ESA Archive web interface DataLink wizard that appears when clicking on the DataLink icon (double chain link encompassed by a red circle above) in the job lists area. The horizontal and vertical arrows point to the drop-down menus that allow to select the data structure (output serialisation) and file format of the files (see the DataLink: Products serialisation tutorial), respectively. 

The phot_variable_flag field in the main Gaia DR2 catalogue (gaia_dr2.gaia_source) allows to extract only the sources that include DR2 epoch photometry. The Gaia DR2 equivalent of the previous Gaia DR3 ADQL query is therefore:

SELECT DISTANCE(266.41683, -29.00781, ra, dec) AS separation, * FROM gaiadr2.gaia_source WHERE DISTANCE(266.41683, -29.00781, ra, dec) < 0.25 AND-- Retrieve only sources with associated DataLink products 
   phot_variable_flag = 'VARIABLE'
  ORDER BY separation ASC

3. FPR tables

The Gaia Focused Product Release (FPR) contains 12 catalogues organised into five categories: "Crowded Fields," "Extra-galactic," "Solar System," "Spectroscopy," and "Variability." Among these, only the Extra-galactic (gaiafpr.lens_catalogue_name, gaiafpr.lens_observation, and gaiafpr.lens_outlier) and Variability (gaiafpr.vari_epoch_radial_velocity, gaiafpr.vari_long_period_variable, and gaiafpr.vari_rad_vel_statistics) tables include sources with associated DataLink products that were already published in Gaia Data Release 3. To access these products, it is necessary to JOIN these tables with any of the tables from Gaia Data Release 3 either using the JOB UPLOAD or the table upload mechanism in a two-step way, as shown in the example below:

SELECT TOP 10 * FROM gaiafpr.vari_epoch_radial_velocity

SELECT gaia.designation, fpr.* FROM gaiadr3.gaia_source AS gaia JOIN job_upload."job1733156356739I" AS fpr  ON gaia.source_id = fpr.source_id

Once the output result has been generated, please proceed as explained in the Basic use case section above.

4. User tables

Another popular use case is: "I want to retrieve the DataLink products associated with my list of sources (either valid Gaia source IDs/designations or coordinates)". In this case, the first step is to upload a table with the list of sources to one's user space, as detailed in the Upload a user table tutorial. The second step is to combine that table with any of the catalogues included in Gaia Data Release 3 (or Gaia DR2 if your list of sources belong to this release). Depending on the uploaded data (either source IDs/designations or coordinates), the ADQL query used to combine the two tables will involve a JOIN or a cross-match operation, as detailed below.

4.1 List of sources

If the uploaded list of sources contains valid Gaia source IDs and/or designations, a JOIN operation (as also explained in the Joining Catalogues section of the Catalogue combination tutorial) can be used to combine the tables as follows:

SELECT my_table.*, gaia.source_id FROM <user_name>.<uploaded_table_name> AS my_table JOIN gaiadr3.gaia_source AS gaia  ON my_table.selected_source_ids = gaia.source_id

In this example, the field containing valid Gaia DR3 source IDs in the uploaded table is named as selected_source_ids. Once the output of this query has been generated, please proceed as explained in the Basic use case section above.

4.2 List of coordinates

In this alternative case, the uploaded table contains a list of coordinates; therefore, a cross-match is needed to identify the counterparts in the Gaia catalogue. The Cross-matching two catalogues (basic) section included in the Catalogue combination tutorial explains in detail how to do this using different approaches. For simplicity and to avoid duplication, we provide an example below that applies a basic cross-match - without extra filters or epoch propagation - to a list of equatorial coordinates uploaded to the Archive user space.

SELECT my_table.*, gaia.source_id FROM <user_name>.<uploaded_table_name> AS my_table JOIN gaiadr3.gaia_source AS gaia  ON DISTANCE(my_table.ra, my_table.dec, gaia.ra, gaia.dec) < 5. / 3600.

In this example, the cross-match radius is set to 5 arcseconds, and the fields containing the right ascension and declination coordinates (in the table uploaded by the user) are named ra and dec, respectively. Once the output of this query has been generated, please proceed as explained in the Basic use case section above.

datalink products serialisation

Authors: Héctor Cánovas, Jos de Bruijne, María Henar, Elena Racero, and Alcione Mora

Last update: 2024-12-10

The DataLink IVOA protocol implemented in the Gaia ESA Archive gives access to six different products (epoch photometry, medium- and low-resolution spectra, and probability density distributions for the different astrophysical parameters) available for a significant fraction of the sources included in the main Gaia DR3 table (gaia_dr3.gaia_source). These products are serialised according to different data models, and all of them can be retrieved in multiple file formats as well as multiple data structures. This document describes the contents (both data and metadata) of the DataLink products in the various serialisations generated by the Archive. Further information about the implementation of the DataLink protocol in the Archive is briefly described in the DataLink Service, while the Datalink: Command line access and the DataLink: Access from the web interface describe how to retrieve the DataLink products through the Archive web interface and programmatically, respectively. This other tutorial shows how to use the Astroquery.Gaia Python package to download these products.

The DataLink products served by the Archive (and the data models applied to serialise them) are listed below:

(*) The DR2 serialisation follows the IVOA Times Series Cube serialisation.

The serialisation of each product is detailed in the following sections.

Tutorial content:

1. Retrieval parameters
   1. Data structure
   2. Download file format
   3. Output file naming
2. Data models
   1. Epoch Photometry
   2. MCMCs: GSP-Phot & MSC Samples
   3. XP Continuous Spectra
   4. XP Sampled mean Spectra and RVS mean Spectra

1. retrieval parameters

1.1 Data Structure

This parameter defines the structure of the file that is being prepared for download. There are two possible options:

1. INDIVIDUAL (default): one single file per product per selected source(s), with the data serialised in tabular format (one element per table cell).
2. RAW: one single file per product, with the data for multiple sources serialised in a tabular format (one or more elements per table cell).

The latter format is the one used internally by the DPAC consortium, and it is documented in the Gaia Data Release 3 documentation (see the Datamodel description chapter).

Since GACS 3.6, the COMBINED data structure has been deprecated. In practise, this structure can be described as a collection of vertically stacked data serialised in the INDIVIDUAL structure.

1.2 Download FILE Format

Available file download formats are: 

1. VOTable (both binary and plain-text formats, .xml extension)
2. FITS
3. CSV
4. *ECSV (Enhanced Character Separated Values)

The VOTable, FITS, and ECSV file formats provide the table fields and metadata with column descriptions, UCDs, UTYPEs, and units when applicable. The CSV file format only includes the column names.

1.3. output file naming

The data structure and download format define the names of the retrieved files as follows:

By default, the ouput data is downloaded as a compressed .gzip file. However, some internet browsers like, for instance, Safari, automatically expand these files (without asking the user). Note that the file name is agnostic to the data structure selected when retrieving ancillary data associated with a single source.

2. Data Models

The data model for the products serialised in the INDIVIDUAL data structure is described in the following subsections. FOr the sake of simplicity, only the content that differs from the DPAC (RAW) datamodel is detailed below.

2.1 EPOCH PHOTOMETRY

Since GACS 3.6, the INDIVIDUAL serialisation of the epoch photometry follows the IVOA Time Series: Annotation of light curves in VOTable (Case #1). For reference, the previous serialisation is detailed in the Archive Help historical content. The "source_id" and "solution_id" information is provided in the file metadata (as PARAM elements in the VOTable format), and all the table fields are described in the DPAC (RAW) data model except for the "rejected_by_photometry" and "<band>_other_flags" fields detailed below:

Example Python code to extract the Boolean flags encoded in the "g_other_flags" bitmask:

from astropy.table import Table epoch   = Table.read('EPOCH_PHOTOMETRY-Gaia DR3 5616512450772689536.xml') bitmask = epoch['g_other_flags'].tolist() # Decode photometry_flag_sm_reject: bit_pos      = 0 ind_flag_dec = [bool((inp >> bit_pos) & 1) for inp in bitmask] # Decode photometry_flag_af2_unavailable: bit_pos      = 15 ind_flag_dec = [bool((inp >> bit_pos) & 1) for inp in bitmask]

2.2 MCMC's: GSP-PHOT & MSC samples

Both the serialisation of the MCMC GSP-Phot samples and the MCMC MSC samples follows the DPAC data model, with the only exceptions being the "nsamples" field that is not included and the array serialisation (the array columns are flattened to one value per entry). No metadata is added into the file header, and all the information is repeated through the output table so each row is self-contained. 

2.3 XP CONTINUOUS spectra

The XP Continuous mean spectra are serialised without deviations from the original DPAC data model for any data structure. The table metadata contains neither units nor UTYPEs. 

2.4 XP SAMPLED mean spectra and rvs mean spectra

The INDIVIDUAL serialisation of the XP sampled mean spectra and the RVS mean spectra follows the IVOA Spectra Data Model. In the table below, the rows in white and green background indicate the fields included in the table metadata and data, respectively. Note that, due to the 8-characters length limit imposed by the FITS format to the keyword names, the (added) table metadata parameters are re-named when serialising this product in FITS format. None of the metadata fields is included in the files generated in .csv format, which follows a particular serialisation (similar to the DPAC RAW serialisation but including a wavelength column).

(*) The combined_transits, combined_ccds, and deblended_ccd PARAM elements are only present in the RVS mean spectra.

(**) XP sampled mean spectra flux and flux_error units are [W.m**-2.nm**-1], while the RVS mean spectra is normalised and hence its flux and flux_error have no units.

(***) The spatial location contains the ra and dec fields included in the XP sampled mean spectra and RVS mean spectra.

Figure 1: excerpt from the XP Sampled Spectrum table and metadata as shown by TOPCAT.

Datalink: python access

Authors: Héctor Cánovas and Jos de Bruijne

Last update: 2024-12-10

The main goal of the Jupyter Notebook displayed below is to teach how to retrieve and inspect the DataLink products using the Astroquery.Gaia Python package. This code has been tested in Python >= 3.10, using the Astroquery.Gaia 0.4.8.dev9604. At the moment of publishing this tutorial, this version is not yet available via the popular "pip install" or "conda install" commands. Instead, it has to be retrieved from the Astroquery Github repository and then installed locally (as explained in the "Building from source" section in the main Astroquery documentation page). This Jupyter notebook and the complementary Tutorial - Programmatic download of large datasets through DataLink notebook are included in this zip file that also contains a yaml file that can be used to create a conda environment with all dependencies needed to execute it (as explained in the conda documentation page).

 

Tutorial: Retrieve (all) the DataLink products associated to a sample¶

 

Release number: v1.1 (2024-12-10)

Applicable Gaia Data Releases: Gaia EDR3, Gaia DR3

Author: Héctor Cánovas Cabrera; hector.canovas@ext.esa.int

Summary:

This code shows how to retrieve the different DataLink products from an input list of Gaia DR3 sources. These products are serialised in two different data structures:

• INDIVIDUAL (that follows the IVOA data model recommendations whenever possible), and
• RAW (the internal format used by the Gaia collaboration.

Both data structures contain virtually the same information. This notebook shows how to access to the data and metadata of the products retrieved in INDIVIDUAL data structure. For simplicity, the files are downloaded in VOTable format (to have an overview of the serialisation options of the DataLink products served by the Gaia ESA Archive please read the DataLink products serialisation tutorial).

Useful URLs:

• Questions or suggestions
• Tutorials, documentation, and more
• Known issues in the Gaia data
• Gaia data credits and acknowledgements
• GaiaXPy: GaiaXPy is a Python library to facilitate handling Gaia BP/RP spectra as distributed from the Gaia archive.
• Astropy: VOTable XML handling

from astroquery.gaia import Gaia
import matplotlib.pyplot as plt        

def plot_e_phot(inp_table, colours  = ['green', 'red', 'blue'], title = '', fontsize = 12, show_legend = True, show_grid = True, figsize = [20,5]):
              ""
              "Epoch photometry plotter. 'inp_table' MUST be an Astropy-table object."
              ""
              fig       = plt.figure(figsize=figsize)
              time_zero = 2455197.5 # See: https://gea.esac.esa.int/archive/documentation/GDR3/Gaia_archive/chap_datamodel/sec_dm_photometry/ssec_dm_epoch_photometry.html#epoch_photometry-g_transit_time
              xlabel    = f'JD date [TCB - {time_zero}]'
              ylabel    = f'magnitude'
              plt.gca().invert_yaxis()
              plt.plot(inp_table['g_transit_time'], inp_table['g_transit_mag'], 'o', label = 'G',  color = 'green')
              plt.plot(inp_table['bp_obs_time'],    inp_table['bp_mag'],        'o', label = 'BP', color = 'blue')
              plt.plot(inp_table['rp_obs_time'],    inp_table['rp_mag'],        'o', label = 'RP', color = 'red')
              make_canvas(title = title, xlabel = xlabel, ylabel = ylabel, fontsize= fontsize, show_legend=show_legend, show_grid = show_grid)
              plt.show()
def plot_sampled_spec(inp_table, color = 'blue', title = '', fontsize = 14, show_legend = True, show_grid = True, linewidth = 2, figsize = [20,5], show_plot = True):
              ""
              "RVS & XP sampled spectrum plotter. 'inp_table' MUST be an Astropy-table object."
              ""
              if show_plot:
              fig      = plt.figure(figsize=figsize)
              xlabel   = f'Wavelength [{inp_table["wavelength"].unit}]'
              ylabel   = f'Flux [{inp_table["flux"].unit}]'
              plt.plot(inp_table['wavelength'], inp_table['flux'], '-', linewidth = linewidth)
              make_canvas(title = title, xlabel = xlabel, ylabel = ylabel, fontsize= fontsize, show_legend=False, show_grid = show_grid)
              if show_plot:
                 plt.show()
def make_canvas(title = '', xlabel = '', ylabel = '', show_grid = False, show_legend = False, fontsize = 12):
              ""
              "Create generic canvas for plots"
              ""
              plt.title(title,    fontsize = fontsize)
              plt.xlabel(xlabel,  fontsize = fontsize)
              plt.ylabel(ylabel , fontsize = fontsize)
              plt.xticks(fontsize = fontsize)
              plt.yticks(fontsize = fontsize)
              if show_grid:
                 plt.grid()
              if show_legend:
                 plt.legend(fontsize = fontsize*1.25)
            

Gaia.login()

INFO: Login to gaia TAP server [astroquery.gaia.core]

INFO: OK [astroquery.utils.tap.core] INFO: Login to gaia data server [astroquery.gaia.core] INFO: OK [astroquery.utils.tap.core]

query = f"SELECT source_id, ra, dec, pmra, pmdec, parallax \
            FROM gaiadr3.gaia_source \
            WHERE has_epoch_photometry = 'True' \
            AND has_xp_sampled = 'True'\
            AND has_rvs = 'True' \
            AND has_mcmc_msc = 'True' \
            AND has_mcmc_gspphot = 'True' \
            AND random_index between 0 and 200000"
job     = Gaia.launch_job_async(query)
results = job.get_results()
print(f'Table size (rows): {len(results)}')
results          

INFO: Query finished. [astroquery.utils.tap.core] Table size (rows): 3

retrieval_type = 'ALL'          # Options are: 'EPOCH_PHOTOMETRY', 'MCMC_GSPPHOT', 'MCMC_MSC', 'XP_SAMPLED', 'XP_CONTINUOUS', 'RVS', 'ALL'
data_structure = 'INDIVIDUAL'   # Options are: 'INDIVIDUAL' and 'RAW'
data_release   = 'Gaia DR3'     # Options are: 'Gaia DR3' (default), 'Gaia DR2'
datalink = Gaia.load_data(ids=results['source_id'], data_release = data_release, retrieval_type=retrieval_type, data_structure = data_structure, verbose = False)
dl_keys  = [inp for inp in datalink.keys()]
dl_keys.sort()
print()
print(f'The following Datalink products have been downloaded:')
print('-' * 160)
for dl_key in dl_keys:
   print(f' * {dl_key}')        

The following Datalink products have been downloaded: ---------------------------------------------------------------------------------------------------------------------------------------------------------------- * EPOCH_PHOTOMETRY-Gaia DR3 4911590910260264960.xml * EPOCH_PHOTOMETRY-Gaia DR3 5924045608237672448.xml * EPOCH_PHOTOMETRY-Gaia DR3 6196457933368101888.xml * MCMC_GSPPHOT-Gaia DR3 4911590910260264960.xml * MCMC_GSPPHOT-Gaia DR3 5924045608237672448.xml * MCMC_GSPPHOT-Gaia DR3 6196457933368101888.xml * MCMC_MSC-Gaia DR3 4911590910260264960.xml * MCMC_MSC-Gaia DR3 5924045608237672448.xml * MCMC_MSC-Gaia DR3 6196457933368101888.xml * RVS-Gaia DR3 4911590910260264960.xml * RVS-Gaia DR3 5924045608237672448.xml * RVS-Gaia DR3 6196457933368101888.xml * XP_CONTINUOUS-Gaia DR3 4911590910260264960.xml * XP_CONTINUOUS-Gaia DR3 5924045608237672448.xml * XP_CONTINUOUS-Gaia DR3 6196457933368101888.xml * XP_SAMPLED-Gaia DR3 4911590910260264960.xml * XP_SAMPLED-Gaia DR3 5924045608237672448.xml * XP_SAMPLED-Gaia DR3 6196457933368101888.xml

In [14]:

dl_key   = dl_keys[0]           # Select an element from the dl_keys() list displayed above
votable  = datalink[dl_key][0]  # Select the first (and only) element of the list

# Display VOTable metadata ================
print(f'File metadata:')
print('-' * 160)
for item in votable.iter_fields_and_params():
    print(item)

# Display VOTable data ====================
print(f'File data:')
print('-' * 160)
votable_data = votable.to_table()
display(votable_data[0:3])

source_id = votable.get_field_by_id('source_id').value

if 'EPOCH_PHOTOMETRY' in dl_key:
   title     = f'Epoch photometry Gaia DR3 {source_id}'
   plot_e_phot(votable_data, colours  = ['green', 'red', 'blue'], title = title, fontsize = 12, show_legend = True, show_grid = True, figsize = [20,5])
else:
   if 'RVS' or 'XP_SAMPLED' in dl_key:
        title     = f'Spectra: Gaia DR3 {source_id}'
        plot_sampled_spec(votable_data, title = title, fontsize = 12, show_legend = True, show_grid = True, figsize = [20,5], linewidth= 3)  

 

File metadata: ---------------------------------------------------------------------------------------------------------------------------------------------------------------- <PARAM ID="solution_id" datatype="long" name="solution_id" ucd="meta.version" value="375316653866487564"/> <PARAM ID="source_id" datatype="long" name="source_id" ucd="meta.id;meta.main" value="4911590910260264960"/> <FIELD ID="transit_id" datatype="long" name="transit_id" ucd="meta.id"/> <FIELD ID="g_transit_time" datatype="double" name="g_transit_time" ref="time_frame" ucd="time.epoch" unit="d" utype="spec:Spectrum.Data.TimeAxis.Value"/> <FIELD ID="g_transit_flux" datatype="double" name="g_transit_flux" ucd="phot.flux;em.opt" unit="electron / s" utype="spec:Spectrum.Data.SpectralAxis.Value"/> <FIELD ID="g_transit_flux_error" datatype="double" name="g_transit_flux_error" ucd="stat.error;phot.flux;em.opt" unit="electron / s"/> <FIELD ID="g_transit_flux_over_error" datatype="float" name="g_transit_flux_over_error" ucd="stat.snr;phot.flux;em.opt"/> <FIELD ID="g_transit_mag" datatype="double" name="g_transit_mag" ucd="phot.mag;em.opt" unit="mag"/> <FIELD ID="bp_obs_time" datatype="double" name="bp_obs_time" ref="time_frame" ucd="time.epoch" unit="d" utype="spec:Spectrum.Data.TimeAxis.Value"/> <FIELD ID="bp_flux" datatype="double" name="bp_flux" ucd="phot.flux;em.opt.B" unit="electron / s" utype="spec:Spectrum.Data.SpectralAxis.Value"/> <FIELD ID="bp_flux_error" datatype="double" name="bp_flux_error" ucd="stat.error;phot.flux;em.opt.B" unit="electron / s"/> <FIELD ID="bp_flux_over_error" datatype="float" name="bp_flux_over_error" ucd="stat.snr;phot.flux;em.opt.B"/> <FIELD ID="bp_mag" datatype="double" name="bp_mag" ucd="phot.mag;em.opt.B" unit="mag"/> <FIELD ID="rp_obs_time" datatype="double" name="rp_obs_time" ref="time_frame" ucd="time.epoch" unit="d" utype="spec:Spectrum.Data.TimeAxis.Value"/> <FIELD ID="rp_flux" datatype="double" name="rp_flux" ucd="phot.flux;em.opt.R" unit="electron / s" utype="spec:Spectrum.Data.SpectralAxis.Value"/> <FIELD ID="rp_flux_error" datatype="double" name="rp_flux_error" ucd="stat.error;phot.flux;em.opt.R" unit="electron / s"/> <FIELD ID="rp_flux_over_error" datatype="float" name="rp_flux_over_error" ucd="stat.snr;phot.flux;em.opt.R"/> <FIELD ID="rp_mag" datatype="double" name="rp_mag" ucd="phot.mag;em.opt.R" unit="mag"/> <FIELD ID="variability_flag_g_reject" datatype="boolean" name="variability_flag_g_reject" ucd="meta.code.status"/> <FIELD ID="variability_flag_bp_reject" datatype="boolean" name="variability_flag_bp_reject" ucd="meta.code.status"/> <FIELD ID="variability_flag_rp_reject" datatype="boolean" name="variability_flag_rp_reject" ucd="meta.code.status"/> <FIELD ID="g_other_flags" datatype="long" name="g_other_flags" ucd="meta.code.status"/> <FIELD ID="bp_other_flags" datatype="long" name="bp_other_flags" ucd="meta.code.status"/> <FIELD ID="rp_other_flags" datatype="long" name="rp_other_flags" ucd="meta.code.status"/> <FIELD ID="rejected_by_photometry" datatype="boolean" name="rejected_by_photometry" ucd="meta.code.status"/> File data: ----------------------------------------------------------------------------------------------------------------------------------------------------------------

 

Table length=3

transit_id	g_transit_time	g_transit_flux	g_transit_flux_error	g_transit_flux_over_error	g_transit_mag	bp_obs_time	bp_flux	bp_flux_error	bp_flux_over_error	bp_mag	rp_obs_time	rp_flux	rp_flux_error	rp_flux_over_error	rp_mag	variability_flag_g_reject	variability_flag_bp_reject	variability_flag_rp_reject	g_other_flags	bp_other_flags	rp_other_flags	rejected_by_photometry
	d	electron / s	electron / s		mag	d	electron / s	electron / s		mag	d	electron / s	electron / s		mag
int64	float64	float64	float64	float32	float64	float64	float64	float64	float32	float64	float64	float64	float64	float32	float64	bool	bool	bool	int64	int64	int64	bool
14922212505719965	1666.7066178487407	414993.78049738676	487.08184643327724	852.0001	11.642262897057917	1666.706924867667	199352.02416028673	243.10506798363699	820.0242	12.089490591507708	1666.7070100568133	304486.59318223904	277.650662825362	1096.6536	11.038975063618746	False	False	False	4097	0	0	False
14936045572629106	1666.9567829719524	415223.6152432905	756.6865282971511	548.73926	11.641661753520346	1666.9570978731522	198827.86089921332	234.5247765447549	847.7904	12.09234911540527	1666.9571841362197	304275.1443979558	273.45150733390045	1112.7207	11.039729308163398	False	False	False	4609	0	0	False
14945786367190460	1667.132946715427	412579.85777377436	264.43392219174564	1560.238	11.648596812514562	1667.1332616640113	409389.33978470624	339.71967650240606	1205.0798	11.308200891433074	1667.133347609519	407707.1616243589	321.11383421937455	1269.6655	10.722024650739701	False	True	True	4097	1024	2048	False

 

Tutorial - Programmatic download of large datasets through DataLink

Authors: Héctor Cánovas and Jos de Bruijne

Last update: 2024-12-10

The main goal of the Jupyter Notebook displayed below is to teach how to retrieve large amounts (data for more than 5000 sources) of DataLink products using the Astroquery.Gaia Python package. This code has been tested in Python >= 3.10, using the Astroquery.Gaia 0.4.8.dev9604. At the moment of publishing this tutorial, this version is not yet available via the popular "pip install" or "conda install" commands. Instead, it has to be retrieved from the Astroquery Github repository and then installed locally (as explained in the "Building from source" section in the main Astroquery documentation page). This Jupyter notebook and the complementary DataLink: Python access notebook are included in this zip file that also contains a yaml file that can be used to create a conda environment with all dependencies needed to execute it (as explained in the conda documentation page).

 

Tutorial: Download DataLink products for >5000 sources¶

 

Release number: v1.1 (2024-12-10)

Applicable Gaia Data Releases: Gaia EDR3, Gaia DR3

Author: Héctor Cánovas Cabrera; hector.canovas@ext.esa.int

Summary:

This Jupyter Notebook allows to overcome the Gaia Archive DataLink products download threshold (5,000 sources) by first splitting an input source list into multiple chunks of 500 sources each. Our tests show that:

• retrieving the data in chunks is at least three times faster (exact values depend on the selected product and file format) than retrieving the data individually (one call per product per source ID), and
• varying the chunk sizes between 50 and 1,000 sources has negligible impact in the total download time.

In this tutorial, we set:

• the chunk size to 500 elements (source IDs),
• the data structure to INDIVIDUAL, and
• the format to VOTable

See the DataLink: products serialisation tutorial for info about the data structures and formats served by the Gaia Archive. This complementary tutorial shows how to download and inspect all the different DataLink products via Astroquery.Gaia for a small sample of sources.

Useful URLs:

• Questions or suggestions
• Tutorials, documentation, and more
• Known issues in the Gaia data
• Gaia data credits and acknowledgements
• Astropy: VOTable XML handling

from astropy.table import Table
              from astroquery.gaia import Gaia
              import matplotlib.pyplot as plt
            

def chunks(lst, n):
              ""
              "Split an input list into multiple chunks of size =< n"
              ""
              for i in range(0, len(lst), n):
                 yield lst[i:i + n]

def plot_e_phot(inp_table, colours  = ['green', 'red', 'blue'], title = '', fontsize = 12, show_legend = True, show_grid = True, figsize = [20,5]):
              ""
              "Epoch photometry plotter. 'inp_table' MUST be an Astropy-table object."
              ""
              fig       = plt.figure(figsize=figsize)
              time_zero = 2455197.5 # See: https://gea.esac.esa.int/archive/documentation/GDR3/Gaia_archive/chap_datamodel/sec_dm_photometry/ssec_dm_epoch_photometry.html#epoch_photometry-g_transit_time
              xlabel    = f'JD date [TCB - {time_zero}]'
              ylabel    = f'magnitude'
              plt.gca().invert_yaxis()
              plt.plot(inp_table['g_transit_time'], inp_table['g_transit_mag'], 'o', label = 'G',  color = 'green')
              plt.plot(inp_table['bp_obs_time'],    inp_table['bp_mag'],        'o', label = 'BP', color = 'blue')
              plt.plot(inp_table['rp_obs_time'],    inp_table['rp_mag'],        'o', label = 'RP', color = 'red')
              make_canvas(title = title, xlabel = xlabel, ylabel = ylabel, fontsize= fontsize, show_legend=show_legend, show_grid = show_grid)
              plt.show()

def plot_sampled_spec(inp_table, color = 'blue', title = '', fontsize = 14, show_legend = True, show_grid = True, linewidth = 2, figsize = [20,5], show_plot = True):
              ""
              "RVS & XP sampled spectrum plotter. 'inp_table' MUST be an Astropy-table object."
              ""
              if show_plot:
              fig      = plt.figure(figsize=figsize)
              xlabel   = f'Wavelength [{inp_table["wavelength"].unit}]'
              ylabel   = f'Flux [{inp_table["flux"].unit}]'
              plt.plot(inp_table['wavelength'], inp_table['flux'], '-', linewidth = linewidth)
              make_canvas(title = title, xlabel = xlabel, ylabel = ylabel, fontsize= fontsize, show_legend=False, show_grid = show_grid)
              if show_plot:
                 plt.show()

def make_canvas(title = '', xlabel = '', ylabel = '', show_grid = False, show_legend = False, fontsize = 12):
              ""
              "Create generic canvas for plots"
              ""
              plt.title(title,    fontsize = fontsize)
              plt.xlabel(xlabel,  fontsize = fontsize)
              plt.ylabel(ylabel , fontsize = fontsize)
              plt.xticks(fontsize = fontsize)
              plt.yticks(fontsize = fontsize)
              if show_grid:
              plt.grid()
              if show_legend:
              plt.legend(fontsize = fontsize*1.25)

Gaia.login()      

INFO: Login to gaia TAP server [astroquery.gaia.core]

INFO: OK [astroquery.utils.tap.core] INFO: Login to gaia data server [astroquery.gaia.core] INFO: OK [astroquery.utils.tap.core]

query = "SELECT TOP 5100 source_id, ra, dec, parallax from gaiadr3.gaia_source \
            WHERE has_epoch_photometry = 'True' AND \
            has_mcmc_gspphot = 'True' AND \
            has_mcmc_msc = 'True' AND \
            has_xp_sampled = 'True' AND \
            has_rvs = 'True'"
job     = Gaia.launch_job_async(query)
results = job.get_results()
results[0:5]          

INFO: Query finished. [astroquery.utils.tap.core]

chunk_size   = 500
ids          = results['source_id']
ids_chunks   = list(chunks(ids, chunk_size))
datalink_all = []
print(f'* Input list contains {len(ids)} source_IDs')
print(f'* This list is split into {len(ids_chunks)} chunks of <= {chunk_size} elements each')          

* Input list contains 5100 source_IDs * This list is split into 11 chunks of <= 500 elements each

retrieval_type = 'RVS'        # Options are: 'EPOCH_PHOTOMETRY', 'MCMC_GSPPHOT', 'MCMC_MSC', 'XP_SAMPLED', 'XP_CONTINUOUS', 'RVS' 
data_structure = 'INDIVIDUAL' # Options are: 'INDIVIDUAL' and 'RAW'.
data_release   = 'Gaia DR3'   # Options are: 'Gaia DR3' (default), 'Gaia DR2'
datalink_all   = []
ii = 0
for chunk in ids_chunks:
   ii = ii + 1
   print(f'Downloading Chunk #{ii}; N_files = {len(chunk)}')
   datalink  = Gaia.load_data(ids=chunk, data_release = data_release, retrieval_type=retrieval_type, format = 'votable', data_structure = data_structure, dump_to_file=True)
   datalink_all.append(datalink)       

Downloading Chunk #1; N_files = 500 INFO: DataLink products will be stored in the ~/datalink_main/datalink_output_20241022T122745.zip file [astroquery.gaia.core] Downloading Chunk #2; N_files = 500 INFO: DataLink products will be stored in the ~/datalink_main/datalink_output_20241022T122806.zip file [astroquery.gaia.core] Downloading Chunk #3; N_files = 500 INFO: DataLink products will be stored in the ~/datalink_main/datalink_output_20241022T122827.zip file [astroquery.gaia.core] Downloading Chunk #4; N_files = 500 INFO: DataLink products will be stored in the ~/datalink_main/datalink_output_20241022T122848.zip file [astroquery.gaia.core] Downloading Chunk #5; N_files = 500 INFO: DataLink products will be stored in the ~/datalink_main/datalink_output_20241022T122908.zip file [astroquery.gaia.core] Downloading Chunk #6; N_files = 500 INFO: DataLink products will be stored in the ~/datalink_main/datalink_output_20241022T122929.zip file [astroquery.gaia.core] Downloading Chunk #7; N_files = 500 INFO: DataLink products will be stored in the ~/datalink_main/datalink_output_20241022T122949.zip file [astroquery.gaia.core] Downloading Chunk #8; N_files = 500 INFO: DataLink products will be stored in the ~/datalink_main/datalink_output_20241022T123010.zip file [astroquery.gaia.core] Downloading Chunk #9; N_files = 500 INFO: DataLink products will be stored in the ~/datalink_main/datalink_output_20241022T123031.zip file [astroquery.gaia.core] Downloading Chunk #10; N_files = 500 INFO: DataLink products will be stored in the ~/datalink_main/datalink_output_20241022T123051.zip file [astroquery.gaia.core] Downloading Chunk #11; N_files = 100 INFO: DataLink products will be stored in the ~/datalink_main/datalink_output_20241022T123112.zip file [astroquery.gaia.core]

datalink_out = datalink_all[0]

for inp_dict in datalink_all[1:]:
   datalink_out.update(inp_dict)

keys = list(datalink_out.keys())
print(f'* The merged dictionary contains {len(keys)} elements')          

* The merged dictionary contains 5100 elements

In [9]:

# Examine and display data for one selected element
index = 0

votable  = datalink_out[keys[0]][0]                      # Select the first (and only) element of the list

# Display VOTable metadata ================
print(f'File metadata:')
print('-' * 160)
for item in votable.iter_fields_and_params():
   print(item)

# Display VOTable data ====================
print(f'File data:')
print('-' * 160)
votable_data = votable.to_table()
display(votable_data[0:3])

source_id = votable.get_field_by_id('source_id').value

title     = f'Spectra: Gaia DR3 {source_id}'
plot_sampled_spec(votable_data, title = title, fontsize = 12, show_legend = True, show_grid = True, figsize = [20,5], linewidth= 3)          

 

File metadata: ---------------------------------------------------------------------------------------------------------------------------------------------------------------- <PARAM ID="source_id" datatype="long" name="source_id" ucd="meta.id;meta.main" value="5817850518933465216"/> <PARAM ID="solution_id" datatype="long" name="solution_id" ucd="meta.version" value="5950420259779346465"/> <PARAM ID="combined_transits" datatype="int" name="combined_transits" ucd="meta.number" value="30"/> <PARAM ID="combined_ccds" datatype="int" name="combined_ccds" ucd="meta.number" value="83"/> <PARAM ID="deblended_ccds" datatype="int" name="deblended_ccds" ucd="meta.number" value="11"/> <PARAM ID="spatialLocation" arraysize="2" datatype="double" name="spatialLocation" ucd="pos.eq" unit="deg" utype="spec:Char.SpatialAxis.Coverage.Location.Value" value="[252.35787028 -63.58134854]"/> <PARAM ID="TimeAxisCoverageLocation" datatype="double" name="TimeAxisCoverageLocation" ucd="time.epoch" unit="yr" utype="spec:Char.TimeAxis.Coverage.Location.Value" value="2016.0"/> <PARAM ID="TimeAxisCoverageBoundsExtent" datatype="double" name="TimeAxisCoverageBoundsExtent" ucd="time.duration" unit="yr" utype="spec:Char.TimeAxis.Coverage.Bounds.Extent" value="2.83"/> <PARAM ID="spectralAccuracyStatError" datatype="double" name="spectralAccuracyStatError" ucd="stat.error;em.wl" unit="nm" utype="spec:Char.SpectralAxis.Accuracy.StatError" value="0.0"/> <PARAM ID="spectralLocation" datatype="double" name="spectralLocation" ucd="em.wl;instr.bandpass" unit="nm" utype="spec:Char.SpectralAxis.Coverage.Location.Value" value="0.0"/> <PARAM ID="spectralCoverageBoundsExtent" datatype="double" name="spectralCoverageBoundsExtent" ucd="instr.bandwidth" unit="nm" utype="spec:Char.SpectralAxis.Coverage.Bounds.Extent" value="24.0"/> <PARAM ID="spectralCoverageBoundsStart" datatype="double" name="spectralCoverageBoundsStart" ucd="em.wl;stat.min" unit="nm" utype="spec:Char.SpectralAxis.Coverage.Bounds.Start" value="846.0"/> <PARAM ID="spectralCoverageBoundsStop" datatype="double" name="spectralCoverageBoundsStop" ucd="em.wl;stat.max" unit="nm" utype="spec:Char.SpectralAxis.Coverage.Bounds.Stop" value="870.0"/> <PARAM ID="SpatialExtent" datatype="double" name="SpatialExtent" ucd="instr.fov" unit="deg" utype="spec:Spectrum.Char.SpatialAxis.Coverage.Bounds.Extent" value="0.000491106"/> <PARAM ID="DataModel" arraysize="13" datatype="char" name="DataModel" utype="spec:Spectrum.DataModel" value="Spectrum 1.01"/> <PARAM ID="Publisher" arraysize="13" datatype="char" name="Publisher" ucd="meta.curation" utype="spec:Curation.Publisher" value="ESA/Gaia/DPAC"/> <PARAM ID="Title" arraysize="8" datatype="char" name="Title" ucd="meta.title;meta.dataset" utype="spec:DataID.Title" value="Spectrum"/> <PARAM ID="SpectralAxisUcd" arraysize="5" datatype="char" name="SpectralAxisUcd" ucd="meta.ucd" utype="spec:Spectrum.Char.SpectralAxis.Ucd" value="em.wl"/> <PARAM ID="SpectralAxisUnit" arraysize="2" datatype="char" name="SpectralAxisUnit" ucd="meta.unit" unit="nm" utype="spec:Spectrum.Char.SpectralAxis.Unit" value="nm"/> <PARAM ID="FluxAxisUcd" arraysize="29" datatype="char" name="FluxAxisUcd" ucd="meta.ucd" utype="spec:Spectrum.Char.FluxAxis.Ucd" value="phot.flux.density;arith.ratio"/> <PARAM ID="FluxAxisUnit" arraysize="10" datatype="char" name="FluxAxisUnit" ucd="meta.unit" utype="spec:Spectrum.Char.FluxAxis.Unit" value="Flux value"/> <FIELD ID="wavelength" datatype="double" name="wavelength" ucd="em.wl" unit="nm" utype="spec:Data.SpectralAxis.Value"/> <FIELD ID="flux" datatype="float" name="flux" ucd="phot.flux;em.opt.I"/> <FIELD ID="flux_error" datatype="float" name="flux_error" ucd="stat.error;phot.flux;em.opt.I"/> File data: ----------------------------------------------------------------------------------------------------------------------------------------------------------------

 

Table length=3

wavelength	flux	flux_error
nm
float64	float32	float32
846.0	0.9846479	0.009583678
846.01	0.9915793	0.01039188
846.02	0.9861507	0.009392991

 

Back to Gaia Help