DataLink Service

Authors: Héctor Cánovas and Jos de Bruijne

In each Gaia data release, the key parameters of all sources are stored in the gaia_source table that contains the (mean) astrometric, photometric, and radial-velocity data as well as astrophysical parameters. This table, along with the complementary tables for, for instance, variable stars or solar system objects, is accessible by means of the IVOA compliant TAP+ table access protocol that allows to explore astronomical datasets stored in relational data bases using the ADQL query language. In addition to the gaia_source table, Gaia DR3 includes vast amounts of non-tabular data such as mean spectra, epoch photometry, and Monte Carlo Markov Chain samples for millions of sources (while Gaia DR4 will include epoch astrometry and epoch photometry for the whole sample plus billions of mean and epoch spectra). Storing these non-tabular datasets as plain tables in a monolithic, relational data base is impractical. Instead, these products are hosted by a dedicated service designed to handle massive data requests that is accessible via the DataLink protocol. DataLink is a data access protocol compliant with the IVOA architecture that provides a linking mechanism between datasets offered by different services. In practice, it can be seen and used as a web service providing the list of additional data products available for each object outside the main catalogue(s).

Since the Archive upgrade to version 2.14 the VOTables generated by the Archive contain a new resource that facilitates the access to the DataLink products to IVOA compliant clients (like TOPCAT):

</RESOURCE>
<RESOURCE type="meta" utype="adhoc:service" name="ancillary">
    <DESCRIPTION>Retrieve DataLink file containing ancillary data for source</DESCRIPTION>
    <PARAM name="standardID" datatype="char" arraysize="*" value="ivo://ivoa.net/std/DataLink#links-1.0"/>
    <PARAM name="accessURL" datatype="char" arraysize="*" value="https://gea.esac.esa.int/data-server/datalink/links"/>
    <PARAM name="contentType" datatype="char" arraysize="*" value="application/x-votable+xml;content=datalink"/>
    <GROUP name="inputParams">
        <PARAM name="ID"  datatype="char" arraysize="*" value="" ref="DESIGNATION"/>
    </GROUP>
</RESOURCE>

The entry point to the DataLink server is indicated by the "accessURL" parameter. To invoke the service and find out the resources associated to a given source it is necessary to combine the entry point with the target ID as:  

https://gea.esac.esa.int/data-server/datalink/links?ID=Gaia+DR3+30343944744320

The output is a xml file not intended for humans but for IVOA-compliant clients. Opening this file with TOPCAT reveals the following content:

Figure 1: Content of the xml file generated by the DataLink service when invoked as explained above.

This file contains the URLs that give access to the DataLink products associated to the target source. For information about how to access to these products from the Gaia Archive web interface and programmatically, please see the DataLink: Access from the Archive GUI and the Command line access: DataLink tutorials, respectively. The structure and content of the DataLink products are described in the Datamodel Chapter in the Gaia DR3 documentation and DataLink products serialisation tutorial.

DataLink: Access from the Archive web interface

Authors: Héctor Cánovas, Jos de Bruijne, and Alcione Mora

Gaia DR3 includes vast amounts of non-tabular data such as high- and low-resolution (mean) spectra, epoch photometry, and Monte Carlo Markov Chain samples for millions of sources. These products are hosted by a dedicated service designed to handle massive data requests that is accessible via the DataLink protocol. This intermediate-level tutorial introduces the concepts needed to access and retrieve these products using the Gaia Archive web interface via its Advanced (ADQL) form. The complementary DataLink: command line access and DataLink: Python Access tutorials describe the programmatic access to these products using the Unix curl command-line utility and the Python package Astroquery.Gaia, respectively, while the DataLink products serialisation tutorial describes the structure of these products. In case of difficulties following this tutorial, please consult the DataLink service and Advanced (ADQL) tab tutorials.

Tutorial content:

1. How it works
2. Basic use cases
3. Advanced use cases

1. How it works

The DataLink service searches for the DataLink product(s) associated to a list of Gaia designation(s) or, alternatively, a combination of Gaia source ID(s) and Gaia data release. By default, the server searches for the products associated to Gaia DR3. Users interested in searching the DataLink products from Gaia DR2 (which only contains epoch photometry data) simply have have to select "Gaia DR2" in the "Data release" dropdown menu that is indicated by the inclined arrow in Fig. 1.

2. Basic use cases

One of the most simple use cases that can be defined is: "I want to search for the DataLink products associated to the output of this (Gaia DR3) query". In a first attempt, we may run a simple, 0.25 degrees radius ADQL cone search similar to the first example directly accessible from the ADQL query editor (see also the Query examples):

SELECT DISTANCE(POINT(266.41683, -29.00781), POINT(ra, dec)) AS separation, *
FROM gaiadr3.gaia_source
WHERE DISTANCE(POINT(266.41683, -29.00781),POINT(ra, dec)) < 0.25
ORDER BY separation ASC

The first step to retrieve the DataLink products associated to this sample consists in clicking on the double chain ("paperclip") icon available in the job list area of the Advanced (ADQL) form (see Fig. 1). In this case, we will receive an error message explaining that the result of the query above contains 19,758 sources, which exceeds the threshold (5000 sources) imposed to not overload the DataLink server. To avoid this problem, we may reduce the cone search radius or, even better, make use of the "has_<datalink_product>" fields available in the gaiadr3.gaia_source table. Filtering the previous query as follows:

SELECT DISTANCE(POINT(266.41683, -29.00781),POINT(ra, dec)) AS separation, *
FROM gaiadr3.gaia_source
WHERE DISTANCE(POINT(266.41683, -29.00781),POINT(ra, dec)) <0.25
AND
-- Retrieve only sources with associated DataLink products 
has_epoch_photometry ='True' AND
has_xp_sampled = 'True'
ORDER BY separation ASC

reduces the query output to just 26 sources, all of them having epoch photometry and XP sampled (as well as XP continuous) spectra. Note: advanced users may use the job_upload mechanism to apply these filters to their old queries without having to re-run the entire query. Clicking again in the double chain icon in the job list area will launch the DataLink wizard as shown in Fig. 1 below. This window lists all the available products associated with the sample generated by the previous query. It is possible to retrieve only selected products (e.g., just RVS mean spectra) or download all the DataLink products at once (by simply clicking on the "Save All Data" button). Note that, depending on the amount of data to be retrieved, the preparation of the dataset prior to starting the download may take up to several minutes. From the DataLink wizard, it is also possible to select different combinations of data structures and download formats (see the DataLink: products serialisation tutorial for details).

Figure 1:Gaia ESA Archive web interface DataLink wizard that appears when clicking on the DataLink icon (double chain link encompassed by a red circle above) in the job lists area.

The vertical arrows point to the drop-down menus that allow to select the data structure (output serialisation) and file format of the files (see the DataLink: Products serialisation tutorial). 

The "phot_variable_flag" field in the main Gaia DR2 catalogue (gaia_dr2.gaia_source) allows to filter out the sources containing (DR2) epoch photometry. The DR2 equivalent of the last ADQL query would then be:

SELECT DISTANCE(
  POINT(266.41683, -29.00781),
  POINT(ra, dec)) AS separation, *
FROM gaiadr2.gaia_source
WHERE DISTANCE(POINT(266.41683, -29.00781), POINT(ra, dec)) < 0.25
AND
-- Retrieve only sources with associated DataLink products 
phot_variable_flag = 'VARIABLE'
ORDER BY separation ASC

3. Advanced use caseS

From the DataLink wizard, it is also possible to 1) select the input ID columns to be used by the massive data server when searching for the DataLink products associated to a given sample, and 2) select the associated data release (see the dropdown menus highlighted by the inclined arrows in Fig. 1). These two options become relevant for users aiming to search for DataLink products in external or user-provided tables (see the the Upload a user table and Download data from an external TAP server tutorials to learn how to upload a table to the user space). The example shown in Fig. 2 illustrates the case of a table uploaded by a (registered) user that contains three different ID fields (and right ascension and declination). If the selected column contains Gaia source IDs but not designations, then the user must indicate as well the desired data release.

Figure 2: Same as Fig.1, but illustrating what happens when searching for DataLink products in catalogues containing several fields that could be used as input ID columns for the DataLink service.

datalink products serialisation

Authors: Héctor Cánovas, María Henar, Jos de Bruijne, Elena Racero, and Alcione Mora

The DataLink IVOA protocol implemented in the Gaia ESA Archive gives access to six different products (epoch photometry, medium- and low-resolution spectra, and probability density distributions for the different astrophysical parameters) available for a significant fraction of the sources included in the main Gaia DR3 table (gaia_dr3.gaia_source). These products are serialised according to different data models, and all of them can be retrieved in multiple file formats as well as multiple data structures. This document describes the contents (both data and metadata) of the DataLink products in the various serialisations generated by the Archive. Further information about the implementation of the DataLink protocol in the Archive is briefly described in the DataLink Service, while the Datalink: Command line access and the DataLink: Access from the web interface describe how to retrieve the DataLink products through the Archive web interface and programmatically, respectively. This other tutorial shows how to use the Astroquery.Gaia Python package to download these products.

The DataLink products served by the Archive (and the data models applied to serialise them) are listed below:

The serialisation of each product is detailed in the following sections.

Tutorial content:

1. Retrieval parameters
   1. Data structure
   2. Download file format
   3. Output file naming
2. Data models
   1. Epoch Photometry
   2. MCMC GSP-Phot Samples
   3. MCMC MSC Samples
   4. XP Continuous Spectra
   5. XP Sampled Spectra
   6. RVS Spectra

1. retrieval parameters

1.1 Data Structure

This parameter defines the structure of the file that is being prepared for download. There are three possible options:

1. INDIVIDUAL (default): one single file per product per selected source(s), with the data serialised in tabular format (one element per table cell).
2. COMBINED: one single file per product, with the data for multiple sources serialised in a tabular format (one element per table cell).
3. RAW: one single file per product, with the data for multiple sources serialised in a tabular format (one or more elements per table cell).

The latter format is the one used internally by the DPAC consortium, and it is documented in the Gaia Data Release 3 documentation (see the Datamodel description chapter).

1.2 Download FILE Format

Available file download formats are: 

1. VOTable (both binary and plain-text formats, .xml extension)
2. FITS
3. CSV
4. *ECSV (Enhanced Character Separated Values)

*Note: It is not possible to download XP mean sampled spectra or RVS mean spectra in a COMBINED data structure using ECSV as file format. This format fundamentally does not support storing multiple tables (with their associated metadata) in a single file.

The VOTable, FITS, and ECSV file formats provide the table fields and metadata with column descriptions, UCDs, UTYPEs, and units when applicable. The CSV file format only includes the column names.

1.3. output file naming

The data structure and download format define the names of the retrieved files as follows:

By default, the ouput data is downloaded as a compressed .gzip file. However, some internet browsers like, for instance, Safari, automatically expand these files (without asking the user).

2. Data Models

The data model for the products serialised in the INDIVIDUAL and COMBINED data structures is described in the following subsections. In all the tables below, the fields that are added by the Archive (i.e., those that are not included in the DPAC data model) when serialisaing the different products are highlighted in bold fonts.

2.1 EPOCH PHOTOMETRY

The serialisation of the epoch photometry is based on IVOA Time series cube data model. No metadata is added into the file header, and all the information is repeated through the output table so each row is self-contained. Column names therefore remain identical in the INDIVIDUAL and COMBINED data structures.

Long descriptions for the added fields:

band: Photometric band. Values: G (per-transit combined SM-AF flux), BP (blue photometer integrated flux), and RP (red photometer integrated flux).

rejected_by_photometry: Rejected by DPAC photometric processing. Unavailable or rejected by DPAC photometric processing, or negative (unphysical) flux.

other flags: Additional processing flags. This field contains extra information on the data used to compute the fluxes and their quality. It provides debugging information that may be safely ignored for many general purpose applications. The field is a collection of binary flags, whose values can be recovered by applying bit shifting and masking operations. Each band has different binary flags in different positions, as shown below. Bit numbering is as follows: least significant bit = 1 and most significant bit = 64.

• G band:
  • Bit 1: SM transit rejected by photometric processing.
  • Bit 2: AF1 transit rejected by photometric processing.
  • Bit 3: AF2 transit rejected by photometric processing.
  • Bit 4: AF3 transit rejected by photometric processing.
  • Bit 5: AF4 transit rejected by photometric processing.
  • Bit 6: AF5 transit rejected by photometric processing.
  • Bit 7: AF6 transit rejected by photometric processing.
  • Bit 8: AF7 transit rejected by photometric processing.
  • Bit 9: AF8 transit rejected by photometric processing.
  • Bit 10: AF9 transit rejected by photometric processing.
  • Bit 13: G band flux scatter larger than expected (all CCDs considered).
  • Bit 14: SM transit unavailable by photometric processing.
  • Bit 15: AF1 transit unavailable by photometric processing.
  • Bit 16: AF2 transit unavailable by photometric processing.
  • Bit 17: AF3 transit unavailable by photometric processing.
  • Bit 18: AF4 transit unavailable by photometric processing.
  • Bit 19: AF5 transit unavailable by photometric processing.
  • Bit 20: AF6 transit unavailable by photometric processing.
  • Bit 21: AF7 transit unavailable by photometric processing.
  • Bit 22: AF8 transit unavailable by photometric processing.
  • Bit 23: AF9 transit unavailable by photometric processing.
• BP band:
  • Bit 11: BP transit rejected by photometric processing.
  • Bit 24: BP transit photometry rejected by variability processing.
• RP band:
  • Bit 12: RP transit rejected by photometric processing.
  • Bit 25: RP transit photometry rejected by variability processing.

Figure 1: excerpt from the Epoch Photometry table as shown by TOPCAT.

2.2 MCMC GSP-PHOT samples

The serialisation of the MCMC GSP-Phot samples follows the DPAC data model, with the only exceptions being the "nsamples" field that is not included and the array serialisation (the array columns are flattened to one value per entry). No metadata is added into the file header, and all the information is repeated through the output table so each row is self-contained. Column names therefore remain identical in the INDIVIDUAL and COMBINED data structures. The table metadata does not contain any UTYPEs.

Figure 2: excerpt from the MCMC GSP-Phot table as shown by TOPCAT.

2.3 MCMC MSC SAMPLES

The serialisation of the MCMC MSC samples follows the DPAC data model, with the only exceptions being the "nsamples" field that is not included and the array serialisation (the array columns are flattened to one value per entry). No metadata is added into the file header, and all the information is repeated through the output table so each row is self-contained. Column names therefore remain identical in the INDIVIDUAL and COMBINED data structures. The table metadata does not contain any UTYPEs.

Figure 3: excerpt from the MCMC MSC table as shown by TOPCAT.

2.4 XP CONTINUOS spectra

The XP Continuous mean spectra are serialised without deviations from the original DPAC data model for any data structure. The table metadata contains neither units nor UTYPEs.

Figure 4: excerpt from the XP Continuous Spectrum table as shown by TOPCAT.

2.5 XP SAMPLED spectra

The serialisation of the XP sampled mean spectra follows the IVOA Spectra Data Model. Both the INDIVIDUAL and COMBINED data structures contain the same data and metadata. However, due to the 8-characters length limit imposed by the FITS format to the keyword names, the (added) table metadata parameters are re-named when serialising this product in FITS format. In the table below, the rows in white and green background indicate the fields included in the table metadata and data, respectively. None of the metadata fields is included in the files generated in .csv format, which follows a particular serialisation (similar to the DPAC RAW serialisation but including a wavelength column).

Figure 5: excerpt from the XP Sampled Spectrum table and metadata as shown by TOPCAT.

2.6 RVS SPECTRA

The serialisation of the RVS mean spectra follows the IVOA Spectra Data Model. Both the INDIVIDUAL and COMBINED data structures contain the same data and metadata. However, due to the 8-characters length limit imposed by the FITS format to the keyword names, the (added) table metadata parameters are re-named when serialising this product in FITS format. In the table below, the rows in white and green background indicate the fields included in the table metadata and data, respectively. None of the metadata fields is included in the files generated in .csv format, which follows a particular serialisation (similar to the DPAC RAW serialisation but including a wavelength column).

Figure 6: excerpt from the RVS Spectra table and metadata as shown by TOPCAT.

Datalink: python access

Authors: Héctor Cánovas and Jos de Bruijne

The main goal of the Jupyter Notebook displayed below is to teach how to retrieve and inspect the DataLink products using the Astroquery.Gaia Python package. This code has been tested in Python >= 3.8.  The Jupyter notebook is included in this .zip file that also contains complementary notebooks, supplementary files, and a "tutorials.yml" environment file that can be used to create a conda environment with all dependencies needed to execute it (as explained in the official conda documentation).

 

Tutorial: Retrieve (all) the DataLink products associated to a sample¶

 

Release number: v1.0.1 (2022-12-06)

Applicable Gaia Data Releases: Gaia EDR3, Gaia DR3

Author: Héctor Cánovas Cabrera; hector.canovas@esa.int

Summary:

This code shows how to retrieve the different DataLink products from an input list of Gaia DR3 sources. These products are serialised in three different data structures:

• INDIVIDUAL
• COMBINED, and
• RAW

Although all data structures contain virtually the same information, the RAW format - the internal format used by the Gaia collaboration - is not intended for the final users (see for details the DataLink: Products serialisation tutorial). This notebook shows the content of the INDIVIDUAL & COMBINED products, whose serialisation follows different IVOA data model recommendations and it allows to easily inspect the product content. We recommend to select the COMBINED format when downloading DataLink products for large (>1000) amounts of sources to reduce the total download time.

Useful URLs:

• Questions or suggestions
• Tutorials, documentation, and more
• Known issues in the Gaia data
• Gaia data credits and acknowledgements
• GaiaXPy: GaiaXPy is a Python library to facilitate handling Gaia BP/RP spectra as distributed from the Gaia archive.

In [1]:

from astroquery.gaia import Gaia
import matplotlib.pyplot as plt

In [2]:

def extract_dl_ind(datalink_dict, key, figsize = [15,5], fontsize = 12, linewidth = 2, show_legend = True, show_grid = True):
    ""
    "Extract individual DataLink products and export them to an Astropy Table"
    ""
    dl_out  = datalink_dict[key][0].to_table()
    if 'time' in dl_out.keys():
        plot_e_phot(dl_out, colours  = ['green', 'red', 'blue'], title = 'Epoch photometry', fontsize = fontsize, show_legend = show_legend, show_grid = show_grid, figsize = figsize)
    if 'wavelength' in dl_out.keys():
        if len(dl_out) == 343:  title = 'XP Sampled'
        if len(dl_out) == 2401: title = 'RVS'
        plot_sampled_spec(dl_out, color = 'blue', title = title, fontsize = fontsize, show_legend = False, show_grid = show_grid, linewidth = linewidth, legend = '', figsize = figsize)
    return dl_out


def plot_e_phot(inp_table, colours  = ['green', 'red', 'blue'], title = 'Epoch photometry', fontsize = 12, show_legend = True, show_grid = True, figsize = [15,5]):
    ""
    "Epoch photometry plotter. 'inp_table' MUST be an Astropy-table object."
    ""
    fig      = plt.figure(figsize=figsize)
    xlabel   = f'JD date [{inp_table["time"].unit}]'
    ylabel   = f'magnitude [{inp_table["mag"].unit}]'
    gbands   = ['G', 'RP', 'BP']
    colours  = iter(colours)

    plt.gca().invert_yaxis()
    for band in gbands:
        phot_set = inp_table[inp_table['band'] == band]
        plt.plot(phot_set['time'], phot_set['mag'], 'o', label = band, color = next(colours))
    make_canvas(title = title, xlabel = xlabel, ylabel = ylabel, fontsize= fontsize, show_legend=show_legend, show_grid = show_grid)
    plt.show()


def plot_sampled_spec(inp_table, color = 'blue', title = '', fontsize = 14, show_legend = True, show_grid = True, linewidth = 2, legend = '', figsize = [12,4], show_plot = True):
    ""
    "RVS & XP sampled spectrum plotter. 'inp_table' MUST be an Astropy-table object."
    ""
    if show_plot:
        fig      = plt.figure(figsize=figsize)
    xlabel   = f'Wavelength [{inp_table["wavelength"].unit}]'
    ylabel   = f'Flux [{inp_table["flux"].unit}]'
    plt.plot(inp_table['wavelength'], inp_table['flux'], '-', linewidth = linewidth, label = legend)
    make_canvas(title = title, xlabel = xlabel, ylabel = ylabel, fontsize= fontsize, show_legend=show_legend, show_grid = show_grid)
    if show_plot:
        plt.show()


def make_canvas(title = '', xlabel = '', ylabel = '', show_grid = False, show_legend = False, fontsize = 12):
    ""
    "Create generic canvas for plots"
    ""
    plt.title(title,    fontsize = fontsize)
    plt.xlabel(xlabel,  fontsize = fontsize)
    plt.ylabel(ylabel , fontsize = fontsize)
    plt.xticks(fontsize = fontsize)
    plt.yticks(fontsize = fontsize)
    if show_grid:
        plt.grid()
    if show_legend:
        plt.legend(fontsize = fontsize*0.75)

 

Connect to the Gaia Archive¶

The DataLink products are available to both registered & anonymous users. However, we recommend to access as a registered user due to their extra benefits when executing long queries (as explained in this FAQ).

In [3]:

Gaia.login()

 

INFO: Login to gaia TAP server [astroquery.gaia.core]
User: hcanovas
Password: ········
OK
INFO: Login to gaia data server [astroquery.gaia.core]
OK

 

Download data sample¶

The query below retrieves a random sample of Gaia (E)DR3 sources having all types of DataLink products.

In [4]:

query = f"SELECT source_id, ra, dec, pmra, pmdec, parallax \
FROM gaiadr3.gaia_source \
WHERE has_epoch_photometry = 'True' \
AND has_xp_sampled = 'True'\
AND has_rvs = 'True' \
AND has_mcmc_msc = 'True' \
AND has_mcmc_gspphot = 'True' \
AND random_index between 0 and 200000"


job     = Gaia.launch_job_async(query)
results = job.get_results()
print(f'Table size (rows): {len(results)}')
results

 

INFO: Query finished. [astroquery.utils.tap.core]
Table size (rows): 3

Out[4]:

Table length=3

source_id	ra	dec	pmra	pmdec	parallax
	deg	deg	mas / yr	mas / yr	mas
int64	float64	float64	float64	float64	float64
6196457933368101888	202.80436078238418	-21.178991138861807	80.54562044679744	-32.95247075512294	10.167137280246173
5924045608237672448	257.635024432604	-53.35065341915946	-4.404105752618793	-6.63122508730231	0.19938320884996538
4911590910260264960	24.783541498908786	-55.317468647500505	40.64757827861938	10.758104689073546	6.2453699013330555

 

Download DataLink Products: INDIVIDUAL¶

The example below retrieves ALL available DataLink products for the input sample of Gaia Source IDs. This option significantly increases the total download time, and here it is selected only for teaching purposes. If you are not interested in downloading all products we recommend you to specify the DataLink product in retrieval_type.

The downloaded files can be stored locally by specifying the output file directory via the output_file option in the load_data method below. Note that:

• The DataLink products are stored in a .gz compressed directory. To avoid errors, this shoud be considered when naming the output file, e.g., output_file = 'datalink_output.gz'
• The individual files will also be saved in the same directory from where this notebook is being launched. This is a known bug and we are working to fix it.
• Finally, the metadata of some of the products raises an Astropy units warning. This is a known issue and we are also working on it.

In [5]:

retrieval_type = 'ALL'          # Options are: 'EPOCH_PHOTOMETRY', 'MCMC_GSPPHOT', 'MCMC_MSC', 'XP_SAMPLED', 'XP_CONTINUOUS', 'RVS', 'ALL'
data_structure = 'INDIVIDUAL'   # Options are: 'INDIVIDUAL', 'COMBINED', 'RAW'
data_release   = 'Gaia DR3'     # Options are: 'Gaia DR3' (default), 'Gaia DR2'


datalink = Gaia.load_data(ids=results['source_id'], data_release = data_release, retrieval_type=retrieval_type, data_structure = data_structure, verbose = False, output_file = None)
dl_keys  = [inp for inp in datalink.keys()]
dl_keys.sort()

print()
print(f'The following Datalink products have been downloaded:')
for dl_key in dl_keys:
    print(f' * {dl_key}')

 

WARNING: UnitsWarning: Unit 'em' not supported by the VOUnit standard. Did you mean Elm or Em? [astropy.units.format.vounit]
WARNING: UnitsWarning: Unit 'wl' not supported by the VOUnit standard.  [astropy.units.format.vounit]

 

The following Datalink products have been downloaded:
 * EPOCH_PHOTOMETRY-Gaia DR3 4911590910260264960.xml
 * EPOCH_PHOTOMETRY-Gaia DR3 5924045608237672448.xml
 * EPOCH_PHOTOMETRY-Gaia DR3 6196457933368101888.xml
 * MCMC_GSPPHOT-Gaia DR3 4911590910260264960.xml
 * MCMC_GSPPHOT-Gaia DR3 5924045608237672448.xml
 * MCMC_GSPPHOT-Gaia DR3 6196457933368101888.xml
 * MCMC_MSC-Gaia DR3 4911590910260264960.xml
 * MCMC_MSC-Gaia DR3 5924045608237672448.xml
 * MCMC_MSC-Gaia DR3 6196457933368101888.xml
 * RVS-Gaia DR3 4911590910260264960.xml
 * RVS-Gaia DR3 5924045608237672448.xml
 * RVS-Gaia DR3 6196457933368101888.xml
 * XP_CONTINUOUS-Gaia DR3 4911590910260264960.xml
 * XP_CONTINUOUS-Gaia DR3 5924045608237672448.xml
 * XP_CONTINUOUS-Gaia DR3 6196457933368101888.xml
 * XP_SAMPLED-Gaia DR3 4911590910260264960.xml
 * XP_SAMPLED-Gaia DR3 5924045608237672448.xml
 * XP_SAMPLED-Gaia DR3 6196457933368101888.xml

 

Detailed content¶

The DataLink products are stored inside a Python Dictionary, where each element (key) contains a one-element list. In addition:

• The epoch photometry, MCMC's, and XP continuous products consist in a table that includes a "source_id" field.

• The XP sampled and RVS products consist in a table that is serialised following the IVOA Spectrum Data Model (see for details the DataLink: Products serialisation tutorial). As a result, a number of parameters (including the the source_id) associated to these files is stored in the table metadata. The cell below shows how to extract these parameters, and how to export the table content to an Astropy Table object.

In [12]:

dl_key   = 'RVS-Gaia DR3 6196457933368101888.xml' # Try out using other XP_Sampled or RVS products (e.g., 'XP_SAMPLED-Gaia DR3 4911590910260264960.xml')
product  = datalink[dl_key][0]
items    = [item for item in product.iter_fields_and_params()]

if 'RVS' in dl_key or 'XP_SAMPLED' in dl_key:
    for item in items:
        print(item)
    print()
    print(f'Showing data for source_id: {product.get_field_by_id("source_id").value}')

prod_tab = product.to_table()
prod_tab[0:5]

 

<PARAM ID="source_id" datatype="long" name="source_id" ucd="meta.id;meta.main" value="6196457933368101888"/>
<PARAM ID="solution_id" datatype="long" name="solution_id" ucd="meta.version" value="5950420259779346465"/>
<PARAM ID="combined_transits" datatype="int" name="combined_transits" ucd="meta.number" value="26"/>
<PARAM ID="combined_ccds" datatype="int" name="combined_ccds" ucd="meta.number" value="73"/>
<PARAM ID="deblended_ccds" datatype="int" name="deblended_ccds" ucd="meta.number" value="6"/>
<PARAM ID="spatialLocation" arraysize="2" datatype="double" name="spatialLocation" ucd="pos.eq" unit="deg" utype="spec:Char.SpatialAxis.Coverage.Location.Value" value="[202.80436078 -21.17899114]"/>
<PARAM ID="TimeAxisCoverageLocation" datatype="double" name="TimeAxisCoverageLocation" ucd="time.epoch" unit="yr" utype="spec:Char.TimeAxis.Coverage.Location.Value" value="2016.0"/>
<PARAM ID="TimeAxisCoverageBoundsExtent" datatype="double" name="TimeAxisCoverageBoundsExtent" ucd="time.duration" unit="yr" utype="spec:Char.TimeAxis.Coverage.Bounds.Extent" value="2.83"/>
<PARAM ID="spectralAccuracyStatError" datatype="double" name="spectralAccuracyStatError" ucd="stat.error;em.wl" unit="nm" utype="spec:Char.SpectralAxis.Accuracy.StatError" value="0.0"/>
<PARAM ID="spectralLocation" datatype="double" name="spectralLocation" ucd="instr.bandpass" unit="nm" utype="spec:Char.SpectralAxis.Coverage.Location.Value" value="0.0"/>
<PARAM ID="spectralCoverageBoundsExtent" datatype="double" name="spectralCoverageBoundsExtent" ucd="instr.bandwidth" unit="nm" utype="spec:Char.SpectralAxis.Coverage.Bounds.Extent" value="24.0"/>
<PARAM ID="spectralCoverageBoundsStart" datatype="double" name="spectralCoverageBoundsStart" ucd="stat.min" unit="nm" utype="spec:Char.SpectralAxis.Coverage.Bounds.Start" value="846.0"/>
<PARAM ID="spectralCoverageBoundsStop" datatype="double" name="spectralCoverageBoundsStop" ucd="stat.max" unit="nm" utype="spec:Char.SpectralAxis.Coverage.Bounds.Stop" value="870.0"/>
<PARAM ID="SpatialExtent" datatype="double" name="SpatialExtent" ucd="instr.fov" unit="deg" utype="spec:Spectrum.Char.SpatialAxis.Coverage.Bounds.Extent" value="0.000491106"/>
<PARAM ID="DataModel" arraysize="13" datatype="char" name="DataModel" utype="spec:Spectrum.DataModel" value="Spectrum 1.01"/>
<PARAM ID="Publisher" arraysize="13" datatype="char" name="Publisher" ucd="meta.curation" utype="spec:Curation.Publisher" value="ESA/Gaia/DPAC"/>
<PARAM ID="Title" arraysize="*" datatype="char" name="Title" utype="spec:DataID.Title" value=""/>
<PARAM ID="SpectralAxisUcd" arraysize="5" datatype="char" name="SpectralAxisUcd" unit="em wl" utype="spec:Spectrum.Char.SpectralAxis.Ucd" value="em.wl"/>
<PARAM ID="SpectralAxisUnit" arraysize="2" datatype="char" name="SpectralAxisUnit" unit="nm" utype="spec:Spectrum.Char.SpectralAxis.Unit" value="nm"/>
<PARAM ID="FluxAxisUcd" arraysize="29" datatype="char" name="FluxAxisUcd" utype="spec:Spectrum.Char.FluxAxis.Ucd" value="phot.flux.density;arith.ratio"/>
<PARAM ID="FluxAxisUnit" arraysize="*" datatype="char" name="FluxAxisUnit" utype="spec:Spectrum.Char.FluxAxis.Unit" value=""/>
<FIELD ID="wavelength" datatype="double" name="wavelength" ucd="em.wl" unit="nm" utype="spec:Data.SpectralAxis.Value"/>
<FIELD ID="flux" datatype="float" name="flux" ucd="phot.flux;em.opt.I"/>
<FIELD ID="flux_error" datatype="float" name="flux_error" ucd="stat.error;phot.flux;em.opt.I"/>

Showing data for source_id: 6196457933368101888

Out[12]:

Table length=5

wavelength	flux	flux_error
nm
float64	float32	float32
846.0	0.9810099	0.016027793
846.01	0.96773505	0.02387072
846.02	0.9449764	0.02818237
846.03	0.9631098	0.022609279
846.04	0.99672407	0.020400077

 

The code below creates a plot if the downloaded product contains epoch photometry or a sampled spectrum (RVS or XP). Try yourself and examine the content of the different products by commenting/uncommenting the dl_key variable below. The table displayed below only shows the first 5 elements to shorten this Notebook.

In [13]:

dl_key  = 'EPOCH_PHOTOMETRY-Gaia DR3 4911590910260264960.xml'
# dl_key  = 'MCMC_MSC-Gaia DR3 5924045608237672448.xml'
# dl_key  = 'MCMC_GSPPHOT-Gaia DR3 5924045608237672448.xml'
# dl_key  = 'XP_CONTINUOUS-Gaia DR3 4911590910260264960.xml'
# dl_key  = 'RVS-Gaia DR3 6196457933368101888.xml'
# dl_key  = 'XP_SAMPLED-Gaia DR3 6196457933368101888.xml'

dl_out  = extract_dl_ind(datalink, dl_key, figsize=[20,7])   # Change the figsize to e.g. figsize=[20,7] to increase the size of the displayed image.
dl_out[0:5]                                                  # Remove the '[0:5]' to display the entire table.

 

Out[13]:

Table length=5

source_id	transit_id	band	time	mag	flux	flux_error	flux_over_error	rejected_by_photometry	rejected_by_variability	other_flags	solution_id
			d	mag	electron / s	electron / s
int64	int64	object	float64	float64	float64	float64	float32	bool	bool	int64	int64
4911590910260264960	14922212505719965	G	1666.7066178487407	11.642262897057917	414993.78049738676	487.08184643327724	852.0001	False	False	4097	375316653866487564
4911590910260264960	14936045572629106	G	1666.9567829719524	11.641661753520346	415223.6152432905	756.6865282971511	548.73926	False	False	4609	375316653866487564
4911590910260264960	14945786367190460	G	1667.132946715427	11.648596812514562	412579.85777377436	264.43392219174564	1560.238	False	False	4097	375316653866487564
4911590910260264960	14949878655266891	G	1667.2069563968746	11.643335932713073	414583.84376818856	531.3042654941166	780.3134	False	False	4097	375316653866487564
4911590910260264960	14959619461231507	G	1667.3831203042998	11.649341777555474	412296.8680857516	249.81574135962032	1650.4039	False	False	4097	375316653866487564

 

Download COMBINED DataLink Products¶

As it happens with the INDIVIDUAL example above, the following example retrieves ALL available DataLink products for the input sample of Gaia Source IDs. If you are not interested in downloading all products we recommend you to specify the DataLink product in retrieval_type (e.g., retrieval_type = 'RVS')

In [14]:

retrieval_type = 'ALL'          # Options are: 'EPOCH_PHOTOMETRY', 'MCMC_GSPPHOT', 'MCMC_MSC', 'XP_SAMPLED', 'XP_CONTINUOUS', 'RVS', 'ALL'
data_structure = 'COMBINED'     # Options are: 'INDIVIDUAL', 'COMBINED', 'RAW'
data_release   = 'Gaia DR3'     # Options are: 'Gaia DR3' (default), 'Gaia DR2'


datalink  = Gaia.load_data(ids=results['source_id'], data_release = data_release, retrieval_type=retrieval_type, data_structure = data_structure, verbose = False, output_file = None)
dl_keys  = [inp for inp in datalink.keys()]
dl_keys.sort()

print()
print(f'The following Datalink products have been downloaded:')
for dl_key in dl_keys:
    print(f' * {dl_key}')

 

WARNING: UnitsWarning: Unit 'em' not supported by the VOUnit standard. Did you mean Elm or Em? [astropy.units.format.vounit]
WARNING: UnitsWarning: Unit 'wl' not supported by the VOUnit standard.  [astropy.units.format.vounit]

 

The following Datalink products have been downloaded:
 * EPOCH_PHOTOMETRY_COMBINED.xml
 * MCMC_GSPPHOT_COMBINED.xml
 * MCMC_MSC_COMBINED.xml
 * RVS_COMBINED.xml
 * XP_CONTINUOUS_COMBINED.xml
 * XP_SAMPLED_COMBINED.xml

 

Detailed content¶

The DataLink products are stored inside a Python Dictionary, where each element (key) contains a one- or a multi-element list, depending on the product:

• The epoch photometry, MCMC's, and XP continuous products consist in a single-element list, which is a table that includes a "source_id" field.

• The XP sampled and RVS products consist in a multi-element list, where each element is a table serialised following the IVOA Spectrum Data Model.

 

Extract data for individual sources (epoch photometry, MCMC's, and XP continuous)¶

The table displayed below only shows the first 5 elements to shorten this Notebook.

In [15]:

dl_key      = 'EPOCH_PHOTOMETRY_COMBINED.xml'     # Try also with 'XP_CONTINUOUS_COMBINED.xml', 'MCMC_MSC_COMBINED.xml', 'MCMC_GSPPHOT_COMBINED.xml'
product     = datalink[dl_key][0]
product_tb  = product.to_table()                  # Export to Astropy Table object.
source_ids  = list(set(product_tb['source_id']))  # Detect source_ids.
print(f' There is data for the following Source IDs:')
for source_id in source_ids:
    print(f'* {source_id}')


inp_source = source_ids[0]                        # Replace "1" by "0" or "2" to show the data for the individual sources.
product_tb = product_tb[product_tb['source_id'] == inp_source]

print()
print(f'Showing data for source_id {inp_source}')
product_tb[0:5]                                   # Remove the '[0:5]' to display the entire table.

 

 There is data for the following Source IDs:
* 4911590910260264960
* 6196457933368101888
* 5924045608237672448

Showing data for source_id 4911590910260264960

Out[15]:

Table length=5

source_id	transit_id	band	time	mag	flux	flux_error	flux_over_error	rejected_by_photometry	rejected_by_variability	other_flags	solution_id
			d	mag	electron / s	electron / s
int64	int64	object	float64	float64	float64	float64	float32	bool	bool	int64	int64
4911590910260264960	14922212505719965	G	1666.7066178487407	11.642262897057917	414993.78049738676	487.08184643327724	852.0001	False	False	4097	375316653866487564
4911590910260264960	14936045572629106	G	1666.9567829719524	11.641661753520346	415223.6152432905	756.6865282971511	548.73926	False	False	4609	375316653866487564
4911590910260264960	14945786367190460	G	1667.132946715427	11.648596812514562	412579.85777377436	264.43392219174564	1560.238	False	False	4097	375316653866487564
4911590910260264960	14949878655266891	G	1667.2069563968746	11.643335932713073	414583.84376818856	531.3042654941166	780.3134	False	False	4097	375316653866487564
4911590910260264960	14959619461231507	G	1667.3831203042998	11.649341777555474	412296.8680857516	249.81574135962032	1650.4039	False	False	4097	375316653866487564

 

Extract data for individual sources (XP sampled and RVS)¶

In [16]:

dl_key   = 'RVS_COMBINED.xml'    # Try also with 'XP_SAMPLED_COMBINED.xml'
product  = datalink[dl_key][0]   # Replace "1" by "0" or "2" to show the data for the individual sources.
items    = [item for item in product.iter_fields_and_params()]

for item in items:
    print(item)

print()
print(f'Showing data for source_id: {product.get_field_by_id("source_id").value}')
prod_tab = product.to_table()
prod_tab[0:5]

 

<PARAM ID="source_id" datatype="long" name="source_id" ucd="meta.id;meta.main" value="6196457933368101888"/>
<PARAM ID="solution_id" datatype="long" name="solution_id" ucd="meta.version" value="5950420259779346465"/>
<PARAM ID="combined_transits" datatype="int" name="combined_transits" ucd="meta.number" value="26"/>
<PARAM ID="combined_ccds" datatype="int" name="combined_ccds" ucd="meta.number" value="73"/>
<PARAM ID="deblended_ccds" datatype="int" name="deblended_ccds" ucd="meta.number" value="6"/>
<PARAM ID="spatialLocation" arraysize="2" datatype="double" name="spatialLocation" ucd="pos.eq" unit="deg" utype="spec:Char.SpatialAxis.Coverage.Location.Value" value="[202.80436078 -21.17899114]"/>
<PARAM ID="TimeAxisCoverageLocation" datatype="double" name="TimeAxisCoverageLocation" ucd="time.epoch" unit="yr" utype="spec:Char.TimeAxis.Coverage.Location.Value" value="2016.0"/>
<PARAM ID="TimeAxisCoverageBoundsExtent" datatype="double" name="TimeAxisCoverageBoundsExtent" ucd="time.duration" unit="yr" utype="spec:Char.TimeAxis.Coverage.Bounds.Extent" value="2.83"/>
<PARAM ID="spectralAccuracyStatError" datatype="double" name="spectralAccuracyStatError" ucd="stat.error;em.wl" unit="nm" utype="spec:Char.SpectralAxis.Accuracy.StatError" value="0.0"/>
<PARAM ID="spectralLocation" datatype="double" name="spectralLocation" ucd="instr.bandpass" unit="nm" utype="spec:Char.SpectralAxis.Coverage.Location.Value" value="0.0"/>
<PARAM ID="spectralCoverageBoundsExtent" datatype="double" name="spectralCoverageBoundsExtent" ucd="instr.bandwidth" unit="nm" utype="spec:Char.SpectralAxis.Coverage.Bounds.Extent" value="24.0"/>
<PARAM ID="spectralCoverageBoundsStart" datatype="double" name="spectralCoverageBoundsStart" ucd="stat.min" unit="nm" utype="spec:Char.SpectralAxis.Coverage.Bounds.Start" value="846.0"/>
<PARAM ID="spectralCoverageBoundsStop" datatype="double" name="spectralCoverageBoundsStop" ucd="stat.max" unit="nm" utype="spec:Char.SpectralAxis.Coverage.Bounds.Stop" value="870.0"/>
<PARAM ID="SpatialExtent" datatype="double" name="SpatialExtent" ucd="instr.fov" unit="deg" utype="spec:Spectrum.Char.SpatialAxis.Coverage.Bounds.Extent" value="0.000491106"/>
<PARAM ID="DataModel" arraysize="13" datatype="char" name="DataModel" utype="spec:Spectrum.DataModel" value="Spectrum 1.01"/>
<PARAM ID="Publisher" arraysize="13" datatype="char" name="Publisher" ucd="meta.curation" utype="spec:Curation.Publisher" value="ESA/Gaia/DPAC"/>
<PARAM ID="Title" arraysize="*" datatype="char" name="Title" utype="spec:DataID.Title" value=""/>
<PARAM ID="SpectralAxisUcd" arraysize="5" datatype="char" name="SpectralAxisUcd" unit="em wl" utype="spec:Spectrum.Char.SpectralAxis.Ucd" value="em.wl"/>
<PARAM ID="SpectralAxisUnit" arraysize="2" datatype="char" name="SpectralAxisUnit" unit="nm" utype="spec:Spectrum.Char.SpectralAxis.Unit" value="nm"/>
<PARAM ID="FluxAxisUcd" arraysize="29" datatype="char" name="FluxAxisUcd" utype="spec:Spectrum.Char.FluxAxis.Ucd" value="phot.flux.density;arith.ratio"/>
<PARAM ID="FluxAxisUnit" arraysize="*" datatype="char" name="FluxAxisUnit" utype="spec:Spectrum.Char.FluxAxis.Unit" value=""/>
<FIELD ID="wavelength" datatype="double" name="wavelength" ucd="em.wl" unit="nm" utype="spec:Data.SpectralAxis.Value"/>
<FIELD ID="flux" datatype="float" name="flux" ucd="phot.flux;em.opt.I"/>
<FIELD ID="flux_error" datatype="float" name="flux_error" ucd="stat.error;phot.flux;em.opt.I"/>

Showing data for source_id: 6196457933368101888

Out[16]:

Table length=5

wavelength	flux	flux_error
nm
float64	float32	float32
846.0	0.9810099	0.016027793
846.01	0.96773505	0.02387072
846.02	0.9449764	0.02818237
846.03	0.9631098	0.022609279
846.04	0.99672407	0.020400077

In [19]:

dl_key      = 'XP_SAMPLED_COMBINED.xml'          # Try also with 'RVS_COMBINED.xml'
source_ids  = [product.get_field_by_id("source_id").value for product in datalink[dl_key]]
tables      = [product.to_table()                         for product in datalink[dl_key]]


fig          = plt.figure(figsize=[20,7])        # Change the figsize to e.g. figsize=[30,7] to increase the size of the displayed image.
source_ids_i = iter(source_ids)
for inp_table in tables:
    plot_sampled_spec(inp_table, title=dl_key.replace('_COMBINED.xml', ''), legend = f'source ID = {next(source_ids_i)}', show_plot=False)
plt.show()

 

Tutorial - Programmatic download of large datasets through DataLink

Authors: Héctor Cánovas and Jos de Bruijne

The main goal of the Jupyter Notebook displayed below is to teach how to retrieve large amounts (data for more than 5000 sources) of DataLink products using the Astroquery.Gaia Python package. This code has been tested in Python >= 3.8. The Jupyter notebook is included in this .zip file that also contains complementary notebooks, supplementary files, and a "tutorials.yml" environment file that can be used to create a conda environment with all dependencies needed to execute it (as explained in the official conda documentation).

 

Tutorial: Download DataLink products for >5000 sources¶

 

Release number: v1.0 (2022-07-06)

Applicable Gaia Data Releases: Gaia EDR3, Gaia DR3

Author: Héctor Cánovas Cabrera; hector.canovas@esa.int

Summary:

This Jupyter Notebook allows to overcome the Gaia Archive DataLink products download threshold by first splitting an input source list into multiple chunks, each of them having $\leq$ 5000 sources. Then, a sequential download begins and the multiple outputs are finally merged. As explained in the DataLink: products serialisation tutorial, it is possible to retrieve DataLink products in various data structures and formats. We suggest to retrieve the DataLink products in COMBINED data structure (as shown in all the examples below) because our tests indicate that this is the most efficient data structure to download large amounts of products. For simplicity, all the products in the following examples are downloaded in VOTable. This allows to easily export them to several other formats using the tools available within the Astropy.table module. This complementary tutorial shows how to download and inspect all the different DataLink products via Astroquery.Gaia for an small sample of sources. Finally, while executing this notebook it is posisble to receive a few warnings about the units included in the product metadata. Those are known issues and we are working on them.

Useful URLs:

• Questions or suggestions
• Tutorials, documentation, and more
• Known issues in the Gaia data
• Gaia data credits and acknowledgements

In [1]:

from astropy.table import Table, vstack
from astroquery.gaia import Gaia
import numpy as np

In [2]:

def chunks(lst, n):
    ""
    "Split an input list into multiple chunks of size =< n"
    ""
    for i in range(0, len(lst), n):
        yield lst[i:i + n]

 

Connect to the Gaia Archive¶

The DataLink products are available to both registered & anonymous users. However, we recommend to access as a registered user due to their extra benefits when executing long queries (as explained in this FAQ).

In [3]:

Gaia.login()

 

INFO: Login to gaia TAP server [astroquery.gaia.core]
User: hcanovas
Password: ········
OK
INFO: Login to gaia data server [astroquery.gaia.core]
OK

 

Execute ADQL Query¶

The query below retrieves data for 12000 sources that have associated all the DataLink products offered in Gaia DR3.

In [4]:

query = "SELECT TOP 5100 source_id, ra, dec, parallax from gaiadr3.gaia_source \
WHERE has_epoch_photometry = 'True' AND \
has_mcmc_gspphot = 'True' AND \
has_mcmc_msc = 'True' AND \
has_xp_sampled = 'True' AND \
has_rvs = 'True'"

job     = Gaia.launch_job_async(query)
results = job.get_results()
results[0:5]

 

INFO: Query finished. [astroquery.utils.tap.core]

Out[4]:

Table length=5

source_id	ra	dec	parallax
	deg	deg	mas
int64	float64	float64	float64
2263166706630078848	295.13035167754015	70.28624696426813	17.357227526090668
2263178457660566784	294.86955515586925	70.52640371163079	5.99456673538563
2268372099615724288	285.63663592006975	75.41851051257491	23.857068308325488
5912901375001820288	263.99225124991324	-58.82661905857226	6.476061657906406
2266609140096698112	275.7457014457717	72.17444369607303	7.253739784978569

 

Download Datalink Products¶

Warning: The load_data method allows to retrieve all types of DataLink products (epoch photometry, MCMC's, and spectra) in one single call (see below). However, selecting this option when attempting to retrieve DataLink products for large (>1000) amount of sources can severely delay the dataset preparation on the server side, and even result in a download error. Therefore, we strongly recommend to select one a product at a time in this case.

 

Split the input list into several chunks containing =<5000 elements each¶

In [5]:

dl_threshold = 5000               # DataLink server threshold. It is not possible to download products for more than 5000 sources in one single call.
ids          = results['source_id']
ids_chunks   = list(chunks(ids, dl_threshold))
datalink_all = []


print(f'* Input list contains {len(ids)} source_IDs')
print(f'* This list is split into {len(ids_chunks)} chunks of <= {dl_threshold} elements each')

 

* Input list contains 5100 source_IDs
* This list is split into 2 chunks of <= 5000 elements each

In [6]:

retrieval_type = 'RVS'        # Options are: 'EPOCH_PHOTOMETRY', 'MCMC_GSPPHOT', 'MCMC_MSC', 'XP_SAMPLED', 'XP_CONTINUOUS', 'RVS' 
data_structure = 'COMBINED'   # Options are: 'INDIVIDUAL', 'COMBINED', 'RAW' - but as explained above, we strongly recommend to use COMBINED for massive downloads.
data_release   = 'Gaia DR3'   # Options are: 'Gaia DR3' (default), 'Gaia DR2'
dl_key         = f'{retrieval_type}_{data_structure}.xml'


ii = 0
for chunk in ids_chunks:
    ii = ii + 1
    print(f'Downloading Chunk #{ii}; N_files = {len(chunk)}')
    datalink  = Gaia.load_data(ids=chunk, data_release = data_release, retrieval_type=retrieval_type, format = 'votable', data_structure = data_structure)
    datalink_all.append(datalink)

 

Downloading Chunk #1; N_files = 5000

 

WARNING: UnitsWarning: Unit 'em' not supported by the VOUnit standard. Did you mean Elm or Em? [astropy.units.format.vounit]
WARNING: UnitsWarning: Unit 'wl' not supported by the VOUnit standard.  [astropy.units.format.vounit]

 

Downloading Chunk #2; N_files = 100

 

Concatenate the DataLink outputs into one single table¶

The sampled spectra (XP and RVS) are serialised following the IVOA Spectrum Data Model and as a result a number of parameters, including the associated source_id, are stored in the table metadata. This is taken into account in the cells below.

 

Epoch Photometry, MCMC, or XP Continuous¶

In this case, the merged product is one single table that includes the source_id in one of the table fields. The code below includes an example showing how to write the entire table using the Astropy.table module.

Warning: the written table can have a size >1 Gb.

In [7]:

if 'RVS' not in dl_key and 'XP_SAMPLED' not in dl_key:
    temp       = [inp[dl_key][0].to_table() for inp in datalink_all]
    merged     = vstack(temp)
    file_name  = f"{dl_key}_{data_release.replace(' ','_')}.vot"

    print(f'Writting table as: {file_name}')
    merged.write(file_name, format = 'votable', overwrite = True)

    display(merged)

 

XP sampled or RVS¶

In this case, the merged product is one Python list whose elements are all the individual products. The code below includes an example showing how to write an individual table using the Astropy.table module

In [9]:

if 'RVS' in dl_key or 'XP_SAMPLED'  in dl_key:
    product_list_tb  = [item                                    for sublist in datalink_all for item in sublist[dl_key]]
    product_list_ids = [item.get_field_by_id("source_id").value for sublist in datalink_all for item in sublist[dl_key]]


    ii          = 12     # Try different values to display the content of the individual products.
    source_id   = product_list_ids[ii]
    product_tab = product_list_tb[ii].to_table()
    file_name   = f"{dl_key.replace('_COMBINED.xml', '')}_{data_release.replace(' ','_')}_{source_id}.vot"

    print(f'Writting table as: {file_name}')
    product_tab.write(file_name, format = 'votable', overwrite = True)
    print()
    print(f'Showing {retrieval_type} for source_id = {source_id}')
    display(product_tab[:5])

 

Writting table as: RVS_Gaia_DR3_5912768368455408000.vot

Showing RVS for source_id = 5912768368455408000

 

Table length=5

wavelength	flux	flux_error
nm
float64	float32	float32
846.0	0.961344	0.03171042
846.01	0.9489333	0.020776467
846.02	0.9774552	0.017536303
846.03	0.9911668	0.014816107
846.04	0.9947043	0.013110418

Back to Gaia Help