Help support

Should you have any question, please check the Gaia FAQ section or contact the Gaia Helpdesk

Units transformation (Astropy + Astroquery)

Authors: Héctor Cánovas and Jos de Bruijne

 

The main goal of the Jupyter Notebook displayed below is to demonstrate how to exploit the capabilities of the Astropy.units module in combination with the Astroquery.Gaia module to apply astronomical unit transformations. This code has been tested in Python >= 3.8. The Jupyter notebook is included in this .zip file that also contains complementary notebooks, supplementary files, and a "tutorials.yml" environment file that can be used to create a conda environment with all dependencies needed to execute it (as explained in the official conda documentation).

 

 

Tutorial: Transform input & output units using Astropy+Astroquery





 

Release number: v1.0 (2022-13-06)

Applicable Gaia Data Releases: Gaia EDR3, Gaia DR3

Author: Héctor Cánovas Cabrera; hector.canovas@esa.int

Summary:

This code shows how to use the Astropy.units Python module to transform the units associated to different fields in several Gaia DR3 catalogues. For more information, see also the Astropy equivalencies documentation.

Useful URLs:

In [1]:
from astropy import units as u
from astroquery.gaia import Gaia
import matplotlib.pyplot as plt
 

Connect to the Gaia Archive

In [2]:
Gaia.login()
 
INFO: Login to gaia TAP server [astroquery.gaia.core]
User: hcanovas
Password: ········
OK
INFO: Login to gaia data server [astroquery.gaia.core]
OK
 

Retrieve all sources contained in a spherical volume (roughly) centred on the Solar System barycentre, and estimate their distances in light years

The query below retrieves all Gaia (E)DR3 sources having distance estimates lower than 10 light years and RUWE <1.4. Then it converts distance units (parallax) to light years, and radial velocity units to m/s. Note the use of Python f-strings (https://docs.python.org/3/tutorial/inputoutput.html) to easily include variables into the ADQL query.

In [3]:
dist_lim    = 10.0 * u.lightyear                                # Spherical radius in Light Years
dist_lim_pc = dist_lim.to(u.parsec, equivalencies=u.parallax()) # Spherical radius in Parsec

query = f"SELECT source_id, ra, dec, parallax, distance_gspphot, teff_gspphot, azero_gspphot, phot_g_mean_mag, radial_velocity \
FROM gaiadr3.gaia_source \
WHERE distance_gspphot <= {dist_lim_pc.value}\
AND ruwe <1.4"

job     = Gaia.launch_job_async(query)
results = job.get_results()
print(f'Table size (rows): {len(results)}')

results['distance_lightyear'] = results['distance_gspphot'].to(u.lightyear)
results['radial_velocity_ms'] = results['radial_velocity'].to(u.meter/u.second)
results
 
INFO: Query finished. [astroquery.utils.tap.core]
Table size (rows): 4
Out[3]:
Table length=4
source_id ra dec parallax distance_gspphot teff_gspphot azero_gspphot phot_g_mean_mag radial_velocity distance_lightyear radial_velocity_ms
  deg deg mas pc K mag mag km / s lyr m / s
int64 float64 float64 float64 float32 float32 float32 float32 float32 float32 float32
5853498713190525696 217.39232147200883 -62.67607511676666 768.0665391873573 1.3011 2829.3542 0.0 8.984749 -21.942726 4.243621 -21942.727
4472832130942575872 269.44850252543836 4.739420051112412 546.975939730948 1.8275 3099.6335 1e-04 8.1939745 -110.46822 5.960508 -110468.23
762815470562110464 165.83095967577933 35.948653032660104 392.75294543876464 2.5453 3511.045 0.0 6.551172 -85.11064 8.301659 -85110.64
4075141768785646848 282.4587890175222 -23.83709744872712 336.0266016683708 2.9762 3117.376 0.0002 9.126414 -11.11644 9.707067 -11116.439
 

Retrieve sources with estimated $\mathrm{H\alpha}$ equivalent widths above an arbitrary threshold in nm. Convert it to $\unicode{x212B}$

In [4]:
query = f"SELECT source_id, classprob_dsc_allosmod_star, teff_gspphot, logg_gspphot, mh_gspphot, distance_gspphot, azero_gspphot, radius_gspphot, ew_espels_halpha,  spectraltype_esphs \
FROM gaiadr3.astrophysical_parameters \
WHERE ew_espels_halpha BETWEEN -10 AND -5 AND \
distance_gspphot <300 \
AND spectraltype_esphs != 'unknown' \
ORDER BY ew_espels_halpha"

job     = Gaia.launch_job_async(query)
results = job.get_results()
print(f'Table size (rows): {len(results)}')

results['ew_espels_halpha_AA'] = results['ew_espels_halpha'].to(u.Angstrom)
results
 
INFO: Query finished. [astroquery.utils.tap.core]
Table size (rows): 19
Out[4]:
Table length=19
source_id classprob_dsc_allosmod_star teff_gspphot logg_gspphot mh_gspphot distance_gspphot azero_gspphot radius_gspphot ew_espels_halpha spectraltype_esphs ew_espels_halpha_AA
    K log(cm.s**-2) dex pc mag solRad nm   Angstrom
int64 float32 float32 float32 float32 float32 float32 float32 float32 object float32
6014696635395955200 1.0 5806.3936 4.3735 -4.084 152.1869 5.8891 0.9871 -7.905174 M -79.051735
548687019867141248 1.0 3469.1199 4.8496 -0.1202 234.534 0.0096 0.4027 -7.546674 M -75.466736
6237174395139447808 1.0 3220.2583 4.5578 0.4907 136.8067 0.655 0.6153 -7.336281 M -73.36281
5977507918028936576 0.9999994 3126.9832 4.8348 0.4441 183.2773 0.9097 0.3931 -7.310384 M -73.103836
... ... ... ... ... ... ... ... ... ... ...
5788126115413073920 1.0 3110.1384 3.8806 0.4897 184.8856 1.2249 1.1557 -5.578936 M -55.78936
148449845165337600 1.0 5933.19 4.5812 -3.7281 139.1848 7.3076 0.6936 -5.4947324 M -54.947323
5201160663373986176 1.0 4325.425 4.0847 0.172 181.1893 9.0735 1.4356 -5.2507443 M -52.507442
6021414101695348736 1.0 4610.3657 4.6517 -4.0795 161.0902 4.1502 0.5191 -5.1868486 O -51.868484
 

Retrieve masses from low-mass binary pairs, and convert them to Jupiter masses.

Note: Brown dwarfs upper limit ~ 0.075 $M_{\odot}$ (~ 75 $M_\mathrm{Jup}$)

In [5]:
query = f"SELECT * \
FROM gaiadr3.binary_masses \
WHERE m1 <0.125 \
ORDER BY m1"

job     = Gaia.launch_job_async(query)
results = job.get_results()
print(f'Table size (rows): {len(results)}')

results['m1_jup_mass'] = results['m1'].to(u.jupiterMass)
results['m2_jup_mass'] = results['m2'].to(u.jupiterMass)
results
 
INFO: Query finished. [astroquery.utils.tap.core]
Table size (rows): 5
Out[5]:
Table length=5
source_id m1 m1_lower m1_upper m2 m2_lower m2_upper fluxratio fluxratio_lower fluxratio_upper combination_method m1_ref flag m1_jup_mass m2_jup_mass
  solMass solMass solMass solMass solMass solMass             jupiterMass jupiterMass
int64 float32 float32 float32 float32 float32 float32 float32 float32 float32 object object object float32 float32
4580216762992082048 0.118804775 0.07159802 0.16601153 -- 0.04972383 0.07908957 -- 0.13 -- Orbital+M1 IsocLum   124.45579 nan
5557776367807067648 0.12185247 0.07464338 0.17008784 0.067699336 0.053124726 0.08328539 -- 0.0 0.0 Orbital+M1 IsocLum   127.64845 70.919495
845583819682861184 0.122110225 0.074233726 0.16946633 0.08028531 0.06197444 0.09972578 -- 0.0 0.0 Orbital+M1 IsocLum   127.918465 84.104126
4406489184157821952 0.122521155 0.07496838 0.1705908 0.07285931 0.05727609 0.089689784 -- 0.0 0.0 Orbital+M1 IsocLum   128.34894 76.324905
3230836376154368384 0.124612674 0.076774806 0.17298208 0.04927069 0.038072873 0.0609185 -- 0.0 0.0 Orbital+M1 IsocLum   130.53995 51.614277
 

Select Cepheids with Fundamental pulsation mode below 15 hours.

In [6]:
pf_lim_hd = 15 * u.hour
pf_lim_dd = pf_lim_hd.to(u.day)

query = f'SELECT source_id, pf, pf_error, ap.teff_gspphot, ap.logg_gspphot, ap.mh_gspphot, ap.distance_gspphot, ap.azero_gspphot, ap.radius_gspphot, ap.ew_espels_halpha,  ap.spectraltype_esphs \
FROM gaiadr3.vari_cepheid \
JOIN gaiadr3.astrophysical_parameters AS ap USING (source_id) \
WHERE pf <{pf_lim_dd.value}'


job     = Gaia.launch_job_async(query)
results = job.get_results()
print(f'Table size (rows): {len(results)}')

results['pf_hour']        = results['pf'].to(u.hour)
results['pf_hour'].format = '5.3f'
results
 
INFO: Query finished. [astroquery.utils.tap.core]
Table size (rows): 9
Out[6]:
Table length=9
source_id pf pf_error teff_gspphot logg_gspphot mh_gspphot distance_gspphot azero_gspphot radius_gspphot ew_espels_halpha spectraltype_esphs pf_hour
  d d K log(cm.s**-2) dex pc mag solRad nm   h
int64 float64 float32 float32 float32 float32 float32 float32 float32 float32 object float64
5188379665334866176 0.6219627800893595 8.736139e-06 6173.523 3.1676 -0.9927 2478.2385 0.1601 5.1871 0.19384572 unknown 14.927
4685852200899088512 0.5822570063616527 1.6833055e-06 5521.2 4.7306 -3.7822 3203.7131 0.0121 0.5741 --   13.974
5861380317401194752 0.5866106007286351 1.3795823e-06 9060.484 4.1245 -0.7119 1867.7528 4.696 1.8368 0.03517555 F 14.079
5861754598036185344 0.6002593393511508 2.7811316e-06 -- -- -- -- -- -- 0.2204732 G 14.406
5926569816398338304 0.5910341676058103 8.15525e-07 8406.448 4.065 -1.2239 1941.2487 3.4149 1.7847 0.16679318 B 14.185
4702941119846080128 0.5644549970314298 5.2175283e-06 6616.0835 4.5777 -2.4239 6389.3247 0.0061 0.7866 --   13.547
5422477922959825792 0.5597106717764326 1.2912872e-06 8226.137 3.5066 -0.066 1930.5466 5.4515 4.6715 0.26228103 B 13.433
5864522588553955584 0.5990258658944467 1.9770755e-07 7514.7837 3.3402 -0.296 2169.1348 4.0485 5.2538 0.23103149 F 14.377
4688941489297074944 0.6209256688645836 3.7721848e-06 7132.224 4.582 -1.9969 7770.706 0.0011 0.8403 --   14.902
 

Compute Flux density (in mJy) from XP sampled spectrum.

The code below retrieves XP sample spectra for 10 arbitrary sources in Gaia DR3. This product can be downloaded via a dedicated DataLink server (for details, see the DataLink tutorials available here). Note: the download of this product raises a warning in the Astropy.units module. This is a known issue and we are working on it.

 

Retrieve a set of sources with XP Sampled spectra

In [7]:
query = f"SELECT TOP 10 * FROM gaiadr3.gaia_source \
WHERE has_xp_sampled = 'True'"

job     = Gaia.launch_job_async(query)
results = job.get_results()
results
 
INFO: Query finished. [astroquery.utils.tap.core]
Out[7]:
Table length=10
solution_id DESIGNATION source_id random_index ref_epoch ra ra_error dec dec_error parallax parallax_error parallax_over_error pm pmra pmra_error pmdec pmdec_error ra_dec_corr ra_parallax_corr ra_pmra_corr ra_pmdec_corr dec_parallax_corr dec_pmra_corr dec_pmdec_corr parallax_pmra_corr parallax_pmdec_corr pmra_pmdec_corr astrometric_n_obs_al astrometric_n_obs_ac astrometric_n_good_obs_al astrometric_n_bad_obs_al astrometric_gof_al astrometric_chi2_al astrometric_excess_noise astrometric_excess_noise_sig astrometric_params_solved astrometric_primary_flag nu_eff_used_in_astrometry pseudocolour pseudocolour_error ra_pseudocolour_corr dec_pseudocolour_corr parallax_pseudocolour_corr pmra_pseudocolour_corr pmdec_pseudocolour_corr astrometric_matched_transits visibility_periods_used astrometric_sigma5d_max matched_transits new_matched_transits matched_transits_removed ipd_gof_harmonic_amplitude ipd_gof_harmonic_phase ipd_frac_multi_peak ipd_frac_odd_win ruwe scan_direction_strength_k1 scan_direction_strength_k2 scan_direction_strength_k3 scan_direction_strength_k4 scan_direction_mean_k1 scan_direction_mean_k2 scan_direction_mean_k3 scan_direction_mean_k4 duplicated_source phot_g_n_obs phot_g_mean_flux phot_g_mean_flux_error phot_g_mean_flux_over_error phot_g_mean_mag phot_bp_n_obs phot_bp_mean_flux phot_bp_mean_flux_error phot_bp_mean_flux_over_error phot_bp_mean_mag phot_rp_n_obs phot_rp_mean_flux phot_rp_mean_flux_error phot_rp_mean_flux_over_error phot_rp_mean_mag phot_bp_rp_excess_factor phot_bp_n_contaminated_transits phot_bp_n_blended_transits phot_rp_n_contaminated_transits phot_rp_n_blended_transits phot_proc_mode bp_rp bp_g g_rp radial_velocity radial_velocity_error rv_method_used rv_nb_transits rv_nb_deblended_transits rv_visibility_periods_used rv_expected_sig_to_noise rv_renormalised_gof rv_chisq_pvalue rv_time_duration rv_amplitude_robust rv_template_teff rv_template_logg rv_template_fe_h rv_atm_param_origin vbroad vbroad_error vbroad_nb_transits grvs_mag grvs_mag_error grvs_mag_nb_transits rvs_spec_sig_to_noise phot_variable_flag l b ecl_lon ecl_lat in_qso_candidates in_galaxy_candidates non_single_star has_xp_continuous has_xp_sampled has_rvs has_epoch_photometry has_epoch_rv has_mcmc_gspphot has_mcmc_msc in_andromeda_survey classprob_dsc_combmod_quasar classprob_dsc_combmod_galaxy classprob_dsc_combmod_star teff_gspphot teff_gspphot_lower teff_gspphot_upper logg_gspphot logg_gspphot_lower logg_gspphot_upper mh_gspphot mh_gspphot_lower mh_gspphot_upper distance_gspphot distance_gspphot_lower distance_gspphot_upper azero_gspphot azero_gspphot_lower azero_gspphot_upper ag_gspphot ag_gspphot_lower ag_gspphot_upper ebpminrp_gspphot ebpminrp_gspphot_lower ebpminrp_gspphot_upper libname_gspphot
        yr deg mas deg mas mas mas   mas / yr mas / yr mas / yr mas / yr mas / yr                                 mas       1 / um 1 / um 1 / um               mas         deg               deg deg deg deg     electron / s electron / s   mag   electron / s electron / s   mag   electron / s electron / s   mag             mag mag mag km / s km / s               d km / s K log(cm.s**-2) dex   km / s km / s   mag mag       deg deg deg deg                             K K K log(cm.s**-2) log(cm.s**-2) log(cm.s**-2) dex dex dex pc pc pc mag mag mag mag mag mag mag mag mag  
int64 object int64 int64 float64 float64 float32 float64 float32 float64 float32 float32 float32 float64 float32 float64 float32 float32 float32 float32 float32 float32 float32 float32 float32 float32 float32 int16 int16 int16 int16 float32 float32 float32 float32 int16 bool float32 float32 float32 float32 float32 float32 float32 float32 int16 int16 float32 int16 int16 int16 float32 float32 int16 int16 float32 float32 float32 float32 float32 float32 float32 float32 float32 bool int16 float64 float32 float32 float32 int16 float64 float32 float32 float32 int16 float64 float32 float32 float32 float32 int16 int16 int16 int16 int16 float32 float32 float32 float32 float32 int16 int16 int16 int16 float32 float32 float32 float32 float32 float32 float32 float32 int16 float32 float32 int16 float32 float32 int16 float32 object float64 float64 float64 float64 bool bool int16 bool bool bool bool bool bool bool bool float32 float32 float32 float32 float32 float32 float32 float32 float32 float32 float32 float32 float32 float32 float32 float32 float32 float32 float32 float32 float32 float32 float32 float32 object
1636148068921376768 Gaia DR3 5937173266047638272 5937173266047638272 314958060 2016.0 250.22690656136763 0.0239966 -51.60318993759893 0.017609632 0.6521282730801226 0.030326193 21.503796 16.673893 1.8685996067645547 0.031973384 -16.568857116365553 0.025367465 0.010337387 -0.21418677 -0.10345264 -0.4363342 -0.10887727 -0.40680984 0.06184543 0.030531619 0.018303124 0.1261247 392 0 392 0 6.1149173 592.8164 0.13811097 6.2785025 31 False 1.4527762 -- -- -- -- -- -- -- 45 20 0.045606557 48 17 0 0.017614717 26.214508 0 0 1.2267392 0.18513173 0.27097407 0.13742293 0.41828167 -153.08228 -0.24025124 6.9000425 -33.213203 False 411 26589.599916908806 7.248126 3668.4792 14.625587 47 11275.952258059631 15.611262 722.296 15.208159 47 22326.68693522532 17.63591 1265.9788 13.875834 1.2637513 0 0 0 0 0 1.332325 0.582572 0.749753 -86.44704 18.858374 2 8 6 7 3.9088132 -- -- 425.07895 -- 6250.0 1.0 -1.0 111 -- -- -- -- -- -- -- NOT_AVAILABLE 334.5316527259258 -3.398680882782161 256.083604219929 -29.114880107260994 False False 0 True True False False False True True False 1.020579e-13 5.143651e-13 0.9999964 15001.529 15000.393 15004.231 4.2155 4.2095 4.2199 -0.2306 -0.2623 -0.2103 2439.8772 2429.876 2454.8875 3.24 3.2385 3.2413 2.8408 2.8396 2.8418 1.5555 1.5548 1.5561 OB
1636148068921376768 Gaia DR3 5937144197726034816 5937144197726034816 409734603 2016.0 251.46476095641572 0.16335373 -50.77317140047803 0.13020523 0.6378314286856485 0.25412685 2.509894 1.292189 -0.4120940440736278 0.23243496 -1.2247167218950332 0.1760335 0.03803266 -0.17221262 -0.074222684 -0.3912635 -0.08951158 -0.36605006 0.13065445 0.08541156 0.029731749 0.2990954 420 0 390 30 79.74002 21668.531 1.7483033 821.74414 95 False -- 1.3202974 0.04097113 -4.5607594e-05 0.014479563 -0.17473318 -0.08003091 0.14245875 45 17 0.34106937 50 16 0 0.13933316 7.6331325 84 0 4.983722 0.22662018 0.27790675 0.2764132 0.6595103 168.91518 -13.659251 14.864428 -33.067112 False 378 19542.730498352787 25.998741 751.6799 14.959904 48 7419.4852809890235 23.214443 319.6064 15.662608 47 27307.319652656457 31.529078 866.09955 13.657198 1.7769679 0 48 0 47 0 2.0054102 0.7027044 1.3027058 -47.441692 3.7269695 2 12 7 8 5.1557302 -- -- 783.4667 -- 4500.0 1.5 -1.5 111 -- -- -- 13.278212 0.22160137 5 -- NOT_AVAILABLE 335.66898647365656 -3.4418802794696344 256.8171486425856 -28.178722254754728 False False 0 True True False False False False True False 1.0209601e-13 5.891978e-12 0.9999927 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --  
1636148068921376768 Gaia DR3 5937144197726039936 5937144197726039936 440987907 2016.0 251.45836750025484 0.015479398 -50.77132195865252 0.01100153 1.4683939054423996 0.022162465 66.25589 8.608242 -2.619945655163551 0.021888493 -8.19986026582952 0.015430358 0.043865558 -0.15064028 -0.064317875 -0.39369085 -0.11211365 -0.38227904 0.047593243 0.09975452 0.017356893 0.28807402 414 0 413 1 -0.6554093 391.31244 0.0 0.0 31 False 1.479171 -- -- -- -- -- -- -- 47 18 0.03172368 50 17 0 0.016953178 17.956432 0 0 0.97633344 0.22421812 0.33453202 0.21561244 0.5046696 165.80637 -8.486173 19.770205 -32.0365 False 413 45637.21393929329 25.960531 1757.946 14.039069 47 20717.43005789006 40.85247 507.12796 14.547703 45 35631.80963292374 57.79166 616.5563 13.3683 1.2347213 0 7 0 1 0 1.1794024 0.5086336 0.67076874 -22.728088 5.2690153 2 19 11 10 8.166246 -- -- 927.7823 -- 5750.0 1.0 0.0 111 -- -- -- 12.9804535 0.038362145 8 -- VARIABLE 335.66776888882777 -3.437603465412785 256.8123082079012 -28.17747204543682 False False 0 True True False True False True True False 1.02699967e-13 5.1298115e-13 0.99993294 5139.183 5115.769 5207.8975 4.2553 4.2251 4.3571 -0.5361 -0.5776 -0.4635 618.9207 517.5465 644.5404 0.4595 0.44 0.5095 0.3682 0.3522 0.4089 0.1983 0.1897 0.2201 MARCS
1636148068921376768 Gaia DR3 5937144197726350592 5937144197726350592 205487013 2016.0 251.47234546031828 0.019895842 -50.76727118486512 0.013826283 0.17526834572993327 0.030309262 5.782666 5.1607804 -1.1711576390867762 0.027722465 -5.026136086985977 0.019159406 0.115696535 -0.3214559 -0.16002212 -0.41005212 -0.14725529 -0.4040042 0.02630467 0.20778184 0.08896684 0.30612004 410 0 410 0 -0.27591974 385.7335 0.0 0.0 31 False 1.2827144 -- -- -- -- -- -- -- 47 17 0.04084493 50 17 0 0.008956739 161.39064 0 0 0.9895006 0.28802025 0.32991526 0.3129251 0.54236054 168.28624 -10.720112 16.289307 -33.000294 False 420 27706.444290150273 8.531579 3247.5166 14.5809145 47 5856.201268238487 47.27629 123.87184 15.919502 47 34065.93357095369 95.46398 356.84595 13.417095 1.4408971 0 0 0 1 0 2.502407 1.3385878 1.1638193 -34.97791 1.6097668 2 22 14 10 8.293457 -- -- 745.5058 -- 4250.0 1.5 -0.75 111 -- -- -- 12.794375 0.08171257 8 -- NOT_AVAILABLE 335.67660215737334 -3.4417039070621684 256.82157391305276 -28.17219510543954 False False 0 True True False False False True True False 1.0203731e-13 5.096712e-13 0.9999985 4727.4097 4608.9995 4901.4683 2.0874 1.9288 2.3259 -0.2883 -0.3037 -0.2808 4549.154 3597.4263 5327.672 3.5258 3.3969 3.6897 2.484 2.382 2.6159 1.3457 1.2868 1.4217 PHOENIX
1636148068921376768 Gaia DR3 5937173300407375616 5937173300407375616 169494443 2016.0 250.24704334410666 0.017679643 -51.59382627681073 0.012922672 0.20778617875849348 0.022678073 9.162427 5.6557727 -4.489017938427424 0.0233065 -3.440419031965129 0.018091926 0.014787478 -0.29111868 -0.006672091 -0.39784777 -0.120790765 -0.38303235 0.08639189 -0.050483014 0.010714794 0.11505898 370 370 370 0 0.7452751 966.22327 0.13048622 16.01963 31 False 1.2832072 -- -- -- -- -- -- -- 42 20 0.032927927 45 15 0 0.03093478 5.903772 0 0 1.0267888 -- -- -- -- -- -- -- -- False 387 263292.32756901917 235.24489 1119.2266 12.136271 42 54151.05636197208 211.6225 255.88516 13.504525 40 315749.28631923825 759.4236 415.77493 10.999539 1.4049037 0 0 0 0 0 2.5049858 1.3682537 1.1367321 14.852337 0.80758554 1 17 13 12 48.121964 6.8776093 1.9254254e-10 928.10736 10.115913 4250.0 1.0 -0.5 444 -- -- -- 10.492216 0.01811704 4 -- NOT_AVAILABLE 334.5469715366394 -3.4018896184545313 256.0961002935055 -29.103702474942565 False False 0 True True False False False False True False 1.02024136e-13 5.0960543e-13 0.9999998 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --  
1636148068921376768 Gaia DR3 5937081353758582016 5937081353758582016 1530038650 2016.0 251.6453157948988 0.018873343 -51.61333388705421 0.0132323485 0.19899952026593154 0.023696996 8.397669 1.5678374 -0.5856565999864463 0.026226835 -1.4543452822051757 0.019187879 0.18300082 -0.22813645 -0.16516718 -0.40139687 -0.0036018759 -0.38657758 -0.1487166 -0.017912302 0.080677986 0.37858176 433 0 429 4 1.057775 443.72272 0.0 0.0 31 False 1.3178779 -- -- -- -- -- -- -- 50 20 0.03926723 53 19 0 0.015814753 1.7933123 0 0 1.0357476 0.19043407 0.32772923 0.23566131 0.41409686 171.96785 -10.144089 17.043184 -34.528023 False 455 27294.443449297396 6.930448 3938.3374 14.597181 49 6879.836171656016 9.31669 738.4421 15.744597 48 30135.04225336285 19.934744 1511.6844 13.550216 1.3561324 0 0 0 0 0 2.1943817 1.1474161 1.0469656 -122.37873 2.0260947 2 17 10 11 7.6918716 -- -- 763.8885 -- 4250.0 1.0 -0.5 111 -- -- -- 13.023737 0.089120515 7 -- NOT_AVAILABLE 335.1007709312551 -4.072115096889941 257.08181538353045 -28.993992671997994 False False 0 True True False False False True True False 1.0202967e-13 5.096331e-13 0.9999993 4540.0796 4532.843 4547.935 2.3788 2.3651 2.4102 0.2313 0.2303 0.2339 2491.513 2411.611 2523.5083 2.4429 2.4335 2.4527 1.7478 1.7406 1.7555 0.9238 0.92 0.928 MARCS
1636148068921376768 Gaia DR3 5937081383743372032 5937081383743372032 1388896741 2016.0 251.62238319226026 0.029433807 -51.62029027189454 0.020515807 0.6924009910207137 0.036633335 18.900845 5.762399 -2.1962513998024753 0.041033864 -5.32744995436843 0.03017893 0.07881943 -0.18317024 -0.0902052 -0.41693145 -0.055258308 -0.37683496 -0.017432967 0.018904045 0.15508316 0.29985723 424 270 424 0 21.224941 2526.7124 0.28364065 67.70416 31 False 1.5341111 -- -- -- -- -- -- -- 49 20 0.060006 54 17 0 0.031195693 14.095997 29 0 1.8156136 -- -- -- -- -- -- -- -- False 352 128395.12969023235 50.17742 2558.8228 12.915996 49 67358.68812353928 57.54373 1170.5652 13.267558 50 92071.00855188111 108.90684 845.4107 12.337588 1.2417114 0 49 0 50 0 0.9299698 0.3515625 0.5784073 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- NOT_AVAILABLE 335.086222112606 -4.065747567198785 257.0668428683234 -29.002917747180796 False False 0 True True False False False True True False 1.02131946e-13 5.1014396e-13 0.99998915 9607.438 9513.894 9772.668 3.9733 3.9396 4.0248 0.3169 0.2221 0.3834 1375.4949 1294.5084 1433.022 1.9518 1.9397 1.9703 1.7455 1.7337 1.7639 0.9426 0.9359 0.9528 A
1636148068921376768 Gaia DR3 5937173334767134336 5937173334767134336 1157832125 2016.0 250.21566824117917 0.059527557 -51.595920889486656 0.04271079 0.4702682635100296 0.07166432 6.5620975 3.4092753 -1.938231765299124 0.07593253 -2.804713168282807 0.060880087 0.051077027 -0.20518416 -0.080830626 -0.43357953 -0.09915663 -0.3913832 0.03832911 0.002058486 0.05821425 0.137561 374 0 372 2 45.74267 3584.2673 0.56429577 112.12796 31 False 1.514927 -- -- -- -- -- -- -- 43 20 0.10856331 46 17 0 0.044881888 22.828472 13 0 3.0976706 0.15475534 0.29480875 0.12611878 0.39342368 -159.89705 -4.4460692 11.778325 -32.8211 False 386 31558.894737116752 21.077711 1497.2638 14.439563 41 16130.656604160753 18.109373 890.7352 14.819412 36 23815.380522198586 71.041046 335.2341 13.805752 1.2657616 3 8 0 4 0 1.0136604 0.37984943 0.633811 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- NOT_AVAILABLE 334.53250037174223 -3.3886382930011103 256.0744276709499 -29.10877333882436 False False 0 True True False False False True True False 1.0207059e-13 5.0983745e-13 0.99999523 9269.764 9201.429 9449.048 3.7875 3.7318 3.9789 0.2939 -0.0666 0.4526 3178.5076 2359.795 3469.735 2.1056 2.0933 2.1177 1.8642 1.8523 1.876 1.0085 1.0021 1.0152 A
1636148068921376768 Gaia DR3 5937173334780460032 5937173334780460032 678129793 2016.0 250.21356325663368 0.23394246 -51.59396329321622 0.19552518 -0.3508260735074602 0.30190325 -1.162048 1.6001855 1.073248649206787 0.29730964 -1.1868996821513624 0.2584137 0.06328716 -0.20351417 -0.07027874 -0.41031334 -0.17773803 -0.37811938 0.16410476 -0.0030481764 -0.0031035203 0.14021164 373 9 341 32 150.71007 132927.52 2.3321652 3471.7947 95 False -- 1.6892734 0.057007648 -0.0964132 -0.03026436 -0.07226244 0.04191873 0.24999605 40 19 0.42922014 46 46 0 0.18801957 15.406193 71 0 10.78346 -- -- -- -- -- -- -- -- False 282 64011.30718913338 48.864918 1309.9645 13.671725 41 42860.99511271846 53.568825 800.1108 13.758387 40 54560.38493285597 48.257896 1130.6001 12.905702 1.5219402 0 1 0 0 0 0.852685 0.08666134 0.76602364 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- NOT_AVAILABLE 334.5331071374701 -3.386363373671223 256.0726035824586 -29.107040650639295 False False 0 True True False False False False True False 1.6654e-09 7.449532e-10 0.9999764 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --  
1636148068921376768 Gaia DR3 5937144266449624832 5937144266449624832 1623142172 2016.0 251.43324440910578 0.014669927 -50.77413489978534 0.010906572 0.41842144724333197 0.020740218 20.174398 1.853632 0.3252629089347102 0.019978473 -1.8248714140155078 0.015091202 -0.04721357 -0.12086837 -0.003795892 -0.47315618 -0.14569509 -0.4635919 0.13550735 0.06636885 -0.08241941 0.14478737 439 0 438 1 -1.5463891 423.98978 0.0 0.0 95 False -- 1.4685915 0.0034969677 -0.074927785 0.111765906 -0.20162709 -0.12379193 0.14058335 50 17 0.028448544 52 52 0 0.016981049 11.932347 0 0 0.94709694 0.20634997 0.29949948 0.20517065 0.5272535 177.81488 -7.7354107 14.658453 -32.942963 False 436 54841.20365068016 12.59829 4353.067 13.8396 50 24517.535498153924 22.031908 1112.8195 14.36485 48 44526.903642145546 30.803144 1445.5311 13.126339 1.2589884 0 50 0 1 0 1.2385111 0.52525043 0.71326065 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- NOT_AVAILABLE 335.6552971174124 -3.427339589205473 256.7949289949452 -28.182537219651195 False False 0 True True False False False False True False 1.0206264e-13 5.956727e-11 0.999996 8596.297 8481.228 8603.199 3.452 3.4049 3.4684 -0.3281 -0.3338 -0.3236 2582.0444 2537.1003 2722.1555 2.4509 2.4126 2.4537 2.1134 2.0769 2.1157 1.1535 1.1325 1.1548 PHOENIX
 

The metadata of the XP Sampled products raises an Astropy units warning. This is a known issue and we are also working on it.

In [8]:
datalink = Gaia.load_data(results['source_id'], data_structure = 'INDIVIDUAL', retrieval_type = 'XP_SAMPLED')
 
WARNING: UnitsWarning: Unit 'em' not supported by the VOUnit standard. Did you mean Elm or Em? [astropy.units.format.vounit]
WARNING: UnitsWarning: Unit 'wl' not supported by the VOUnit standard.  [astropy.units.format.vounit]
In [9]:
# Extract XP sampled spectra for a given source =======
outputs = [datalink[key][0] for key in datalink.keys()]
xp      = outputs[5].to_table()                        #  Plot the different spectra by changing '5' to e.g., 0, 1, ..., 9

# Add flux-density columns ============================
xp['flux_jy']         = xp['flux'].to(u.Jansky, equivalencies = u.spectral_density(xp['wavelength'].value * xp['wavelength'].unit))
xp['flux_mjy']        = xp['flux_jy'].to(u.millijansky)
xp['flux_mjy'].format = '7.4f'
display(xp)


# Plotter ==============================================
def make_canvas(xlabel = '', ylabel = '', fontsize = 12):
    plt.xlabel(xlabel,  fontsize = fontsize)
    plt.ylabel(ylabel, fontsize = fontsize)
    plt.xticks(fontsize = fontsize)
    plt.yticks(fontsize = fontsize)
    plt.grid()

fig       = plt.figure(figsize=[30,7])

plt.subplot(121)
plt.plot(xp['wavelength'], xp['flux'], linewidth = 2)
make_canvas(xlabel = f"Wavelength [{xp['wavelength'].unit}]", ylabel = f"Flux [{xp['flux'].unit}]", fontsize = 22)

plt.subplot(122)
plt.plot(xp['wavelength'], xp['flux_mjy'], linewidth = 2)
make_canvas(xlabel = f"Wavelength [{xp['wavelength'].unit}]", ylabel = f"Flux density [{xp['flux_mjy'].unit}]", fontsize = 22)

plt.show()
 
Table length=343
wavelength flux flux_error flux_jy flux_mjy
nm W / (m2 nm) W / (m2 nm) Jy mJy
float64 float32 float32 float64 float64
336.0 7.9894715e-17 1.2711064e-17 0.003008679305218825 3.0087
338.0 6.811754e-17 1.1104153e-17 0.0025958026386069277 2.5958
340.0 5.87597e-17 8.086387e-18 0.0022657746024016693 2.2658
342.0 5.7587097e-17 7.838529e-18 0.0022467600885352906 2.2468
... ... ... ... ...
1014.0 7.951429e-17 3.393551e-17 0.027270959468149857 27.2710
1016.0 8.386952e-17 3.6400103e-17 0.028878252085620608 28.8783
1018.0 8.631832e-17 3.7284885e-17 0.0298385594395844 29.8386
1020.0 9.186709e-17 3.9505603e-17 0.031881563667395155 31.8816
 
In [ ]:

 

BP/RP (XP) spectra retrieval from pleiades sample

Authors: Héctor Cánovas and Jos de Bruijne

 

 

The main goal of the Jupyter notebook displayed below is to demonstrate how to retrieve BP/RP (XP) spectra from a sample of (Gaia DR3) stars whose proper motions are similar to those of the Pleiades open cluster. In addition, this notebook uses the pyESASky widget that allows to visualise and inspect the data against different sky backgrounds. 

This notebook has been tested in Python >= 3.8 and it can be downloaded from this link. The "tutorials_advanced.yml" environment file included in the compressed .zip file can be used to create a conda environment with all dependencies needed to execute it, as explained in the official conda documentation.

 

Tutorial: Retrieve the BP/RP (XP) spectra associated to a sample of candidate Pleiades stars





 

Release number: v1.0 (2022-11-02)

Applicable Gaia Data Releases: Gaia EDR3, Gaia DR3

Author: Héctor Cánovas Cabrera; hector.canovas@esa.int

Summary:

This code shows how to retrieve the different DataLink products associated to a sample of stars associated to the Pleiades open cluster.

Useful URLs:

In [1]:
from astropy.table import Table
from astroquery.gaia import Gaia
import numpy as np
import matplotlib.pyplot as plt
from pyesasky import ESASkyWidget
from pyesasky import Catalogue
from pyesasky import CatalogueDescriptor
from pyesasky import MetadataDescriptor
from pyesasky import MetadataType
In [2]:
def make_canvas(title = '', xlabel = '', ylabel = '', show_grid = False, show_legend = False, fontsize = 12):
    ""
    "Generic function to simplify the editing of plots"
    ""
    plt.title(title,    fontsize = fontsize)
    plt.xlabel(xlabel,  fontsize = fontsize)
    plt.ylabel(ylabel , fontsize = fontsize)
    plt.xticks(fontsize = fontsize)
    plt.yticks(fontsize = fontsize)
    if show_grid:
        plt.grid()
    if show_legend:
        plt.legend(fontsize = fontsize)


def esasky_oplot(inp_cat, catalogueName = 'my_cat', color = 'red', lineWidth = 2):
    ""
    "ESASky catalogue plotter"
    ""
    catalogue = Catalogue(catalogueName = catalogueName, cooframe = 'J2000', color = color, lineWidth = lineWidth)
    for i in range(len(inp_cat)):
        catalogue.addSource(inp_cat['source_id'][i], inp_cat['ra'][i], inp_cat['dec'][i], i + 1, [], [])
    esasky.overlayCatalogueWithDetails(catalogue)


def plot_sampled_spec(inp_table, color = 'blue', title = '', fontsize = 16, show_legend = True, show_grid = True, linewidth = 2, legend = '', figsize = [25,7], show_plot = True):
    ""
    "RVS & XP sampled spectrum plotter. 'inp_table' MUST be an Astropy-table object."
    ""
    if show_plot:
        fig      = plt.figure(figsize=figsize)
    xlabel   = f'Wavelength [{inp_table["wavelength"].unit}]'
    ylabel   = f'Flux [{inp_table["flux"].unit}]'
    plt.plot(inp_table['wavelength'], inp_table['flux'], '-', linewidth = linewidth, label = legend)
    make_canvas(title = title, xlabel = xlabel, ylabel = ylabel, fontsize= fontsize, show_legend=show_legend, show_grid = show_grid)
    if show_plot:
        plt.show()
 

Download data sample

The query below retrieves a 1-degree cone-search centred in Alcyone (brightest star of the Pleiades open cluster)

In [3]:
radius  = 1.0        # Degrees
inp_ra  = 56.87125   # Degrees
inp_dec = 24.10493   # Degrees
In [4]:
query = f"SELECT * FROM gaiadr3.gaia_source_lite \
WHERE DISTANCE(POINT({inp_ra}, {inp_dec}),POINT(ra, dec)) < {radius} AND \
ruwe <1.4 AND parallax_over_error >10"

job     = Gaia.launch_job_async(query)
results = job.get_results()
print(f'Table size (rows): {len(results)}')
results['source_id', 'ra', 'dec', 'pmra' ,'pmdec', 'parallax']
 
INFO: Query finished. [astroquery.utils.tap.core]
Table size (rows): 4032
Out[4]:
Table length=4032
source_id ra dec pmra pmdec parallax
  deg deg mas / yr mas / yr mas
int64 float64 float64 float64 float64 float64
69805798521250560 56.24863115999279 24.642582257680882 2.190487348950862 -13.465314592937117 0.9437596679009509
69805901600465152 56.24727995894163 24.65570538537785 0.49759409475044836 -3.8273885658895668 0.8311574217226121
69805901600465536 56.248963878828214 24.646361688721974 7.372404958972663 -6.369812663837182 1.5491388879533554
69805935960203392 56.27054240577331 24.6474754189256 6.8928967344856975 -8.05494305625505 1.025008949947923
69806039039416832 56.27740714247925 24.678409040272616 20.24411977407674 -44.80748682496666 7.5284014269211195
69806107758894976 56.24396966595952 24.665878429795082 30.429650578356245 -21.351212463789484 2.9152954059140637
69806107758895104 56.241044070606705 24.664036267040704 7.383219341257357 -1.4144879294953903 1.0237043621074358
69806142118633984 56.21224469443437 24.666776076898838 -2.299934195324371 -7.141252554177461 0.849064870039556
69806176478371456 56.228100350737336 24.683007540848468 1.3298150335447376 -0.5604760322096058 1.5462215155701013
... ... ... ... ... ...
66872438939447296 57.41927069557885 24.902859187628646 5.269959320361026 -17.235031261876014 1.8900901689665883
66872713817354496 57.39148986292233 24.91686642164138 26.64832725091828 -22.776599280393107 1.6380916416227402
66872954335515008 57.4317225476395 24.94532691334358 2.460806281233295 -5.845198117146882 0.7673624782044982
66872988695252608 57.45556569586933 24.936496091566315 -0.3013591385803767 -0.25918539719043726 0.9662221646798007
66873053115788800 57.44519493675477 24.949412751851238 -6.089689651619146 7.787121246684261 2.236350307216108
66873396713149824 57.341657924358806 24.918972204413215 -3.233880867237348 -8.526102126701794 1.6219504123800708
66873504091336832 57.34333186577304 24.942320483949963 5.781356192421169 -6.157438497006506 0.7036249265336612
66873675890029952 57.32669282077237 24.940204177040435 6.704281589243597 -34.28370298186868 4.466245382112969
66874294365308800 57.36907834160493 24.993130509031612 3.16843484519884 -1.0195910753054078 0.48783947420933466
66922088761382784 57.24753589995385 25.033553369136268 89.06361877306728 -32.905036492498375 6.943053929927199
 

Inspect results

Pleiades candidate sources are selected by filtering a circular area (radius = 3 mas/yr) around the Pleiades proper motion distribution (centre at ~ [20,-50] mas/yr in [pmra, pmdec])

In [5]:
# Identify Pleiades candidate members =======
radius_pm = 3   # Radius applied to select Pleiades sample in the PMRA-PMDEC space
pmra_c    = 20  # Approx. centre of the Pleiades Cluster pmra.
pmdec_c   = -45 # Approx. centre of the Pleiades Cluster pmdec.
els       = np.sqrt((results['pmra']-pmra_c)**2 + (results['pmdec']-pmdec_c)**2) <radius_pm # Selected Pleiades sample
pl_samp   = results[els]



# Plot & Zoom ===============================
fig       = plt.figure(figsize=[30,12])
fontsize  = 18

# Panel 1 ===============
plt.subplot(121)
plt.plot(results['pmra'], results['pmdec'], 'bo', alpha = 0.25)
make_canvas(xlabel='pmra  [mas/yr]',ylabel='pmdec [mas/yr]', fontsize = fontsize, show_grid = True)
z_fac = 140   # Zoom-in factor
plt.xlim([-z_fac,z_fac])
plt.ylim([-z_fac,z_fac])


# Panel 2 ===============
plt.subplot(122)
plt.plot(results['pmra'], results['pmdec'], 'bo', alpha = 0.20, label = 'Field stars')
plt.plot(pl_samp['pmra'], pl_samp['pmdec'], 'ro', alpha = 0.50, label = 'Pleiades')
z_fac = 50   # Zoom-in factor
plt.xlim([-z_fac,z_fac])
plt.ylim([-z_fac,z_fac])


make_canvas(title = 'Zoom-in', xlabel='pmra  [mas/yr]',ylabel='pmdec [mas/yr]', fontsize = fontsize, show_grid = True, show_legend = True)
plt.show()
 
 

Show results in ESASky

In [7]:
esasky = ESASkyWidget()
esasky
 
 
 
In [7]:
# Plot Cone Search in ESASky =================
esasky_oplot(results, catalogueName = 'Pleiades Cone Search', color = 'blue')
esasky.setFoV(radius*1.5)
esasky.setGoToRADec(results['ra'].mean(), results['dec'].mean())
esasky_oplot(pl_samp, catalogueName = 'Pleiades selected', color = 'red')
 

Show different sub-samples in ESASky

In [8]:
esasky_oplot(pl_samp[pl_samp['has_xp_sampled'] == True],    catalogueName = 'Pleiades XP Sampled',    color = 'white')
esasky_oplot(pl_samp[pl_samp['has_xp_continuous'] == True], catalogueName = 'Pleiades XP Continuous', color = 'green')

# esasky_oplot(pl_samp[pl_samp['has_epoch_photometry'] == True], catalogueName = 'Pleiades Epoch Phot', color = 'white')
# esasky_oplot(pl_samp[pl_samp['has_rvs'] == True],              catalogueName = 'Pleiades RVS',        color = 'green')
 

DataLink products are labelled as: "<retrieval_type>-<data release> <source_id>.<format>", where:

  • retrieval_type = 'EPOCH_PHOTOMETRY', 'MCMC_GSPPHOT', 'MCMC_MSC', 'XP_SAMPLED', 'XP_CONTINUOUS', 'RVS', 'ALL'
  • data_structure = 'INDIVIDUAL', 'COMBINED', 'RAW'

For more information about DataLink products please read the tutorials in this Section.

 

XP Sampled spectra

The first line in the code below selects the sources in the "Pleiades sample" that contain XP sampled products and then retrieves these products.

In [9]:
inp_samp       = pl_samp[pl_samp['has_xp_sampled'] == True]
retrieval_type = 'XP_SAMPLED'
data_structure = 'COMBINED'
dl_key         = 'XP_SAMPLED_COMBINED.xml'  # With Python f-strings: dl_key = f'{retrieval_type}_{data_structure}.xml'


datalink    = Gaia.load_data(ids=inp_samp['source_id'], retrieval_type=retrieval_type, data_structure = data_structure)

print(f"* {len(datalink['XP_SAMPLED_COMBINED.xml'])} XP sampled spectra have been downloaded")
 
 
 
* 120 XP sampled spectra have been downloaded
 

The datalink instance retrieved above consists of a 1-element Python dictionary that, in turn, contains various individual spectra. Each of these contain data (stored in a table) and metadata. The code below shows the content of the XP Spectra for one selected source, namely the one with index=100. To change the source, simply modify the index=100 variable in line #2.

In [10]:
source_ids  = [product.get_field_by_id("source_id").value for product in datalink[dl_key]] # Source IDs are stored in the table metadata.
index       = 100

source_id  = source_ids[index]
xp_samp_tb = datalink['XP_SAMPLED_COMBINED.xml'][index].to_table()

print(f'Listing XP spectra for: Gaia DR3 {source_id}')
display(xp_samp_tb)

plot_sampled_spec(xp_samp_tb, title=dl_key.replace('_COMBINED.xml', ''), legend = f'source ID = {source_id}', show_plot=True, figsize=[20,7])
 
Listing XP spectra for: Gaia DR3 66808220587759104
 
Table length=343
wavelength flux flux_error
nm W / (m2 nm) W / (m2 nm)
float64 float32 float32
336.0 2.11973e-17 7.942415e-18
338.0 1.15127155e-17 5.8046068e-18
340.0 9.963061e-18 4.7220395e-18
342.0 1.3008566e-17 4.0508565e-18
344.0 1.6232869e-17 3.870236e-18
346.0 1.6739485e-17 3.829343e-18
348.0 1.4194591e-17 3.9990886e-18
350.0 1.12620136e-17 4.279232e-18
352.0 1.2844933e-17 4.6978734e-18
... ... ...
1002.0 9.708737e-17 3.0144269e-18
1004.0 9.6774566e-17 3.070413e-18
1006.0 9.78583e-17 3.3961079e-18
1008.0 9.7349033e-17 3.9039583e-18
1010.0 9.817977e-17 4.587646e-18
1012.0 9.966355e-17 5.2730427e-18
1014.0 9.9550465e-17 5.697663e-18
1016.0 1.02366724e-16 6.0409778e-18
1018.0 1.0218127e-16 6.0034047e-18
1020.0 1.04994935e-16 6.1550385e-18
 
 

XP Continuous

The cell below shows how to retrieve XP Continuous spectra for just 3 sources. The GaiaXPy tool contains various functionalities to exploit the information contained in these files.

In [11]:
inp_samp       = pl_samp[pl_samp['has_xp_continuous'] == True][0:3]
retrieval_type = 'XP_CONTINUOUS'
data_structure = 'COMBINED'
dl_key         = 'XP_CONTINUOUS_COMBINED.xml'  # With Python f-strings: dl_key = f'{retrieval_type}_{data_structure}.xml'


datalink     = Gaia.load_data(ids=inp_samp['source_id'], retrieval_type=retrieval_type, data_structure = data_structure)
datalink_out = datalink['XP_CONTINUOUS_COMBINED.xml'][0].to_table()

print(f"* {len(datalink_out)} XP Continuous spectra have been downloaded")
 
* 3 XP Continuous spectra have been downloaded

ICRF2 sources (DR1)

Author: Alcione Mora

This tutorial was developed for Data Release 1 and is kept for legacy (DR1 old data still available).

​​​​​​​​

Gaia DR1 contains information on ICRF reference sources and variable stars, in addition to the main gaia_source table, which contains the astrometry and average photometry for 1.14 billion sources in the sky, and the selection of pulsating variables inn the Large Magellanic Cloud. The following sections provide hints on how to work with these data using the Archive.

This is an intermediate level tutorial that assumes a basic knowledge of the general interface and workflow. The introductory tutorials White dwarfs exploration and Cluster analysis are recommended in case of difficulties following this exercise.

  1. ICRF sources

    Mignard et al. 2016 A&A 595A, 5M presents the Gaia astrometric solution created to align DR1 with the ICRF. The abstract context and aims are reproduced below.

    Context. As part of the data processing for Gaia Data Release 1 (Gaia DR1) a special astrometric solution was computed, the so-called auxiliary quasar solution. This gives positions for selected extragalactic objects, including radio sources in the second realisation of the International Celestial Reference Frame (ICRF2) that have optical counterparts bright enough to be observed with Gaia. A subset of these positions was used to align the positional reference frame of Gaia DR1 with the ICRF2. Although the auxiliary quasar solution was important for internal validation and calibration purposes, the resulting positions are in general not published in Gaia DR1.

    Aims. We describe the properties of the Gaia auxiliary quasar solution for a subset of sources matched to ICRF2, and compare their optical and radio positions at the sub-mas level.

    Table gaiadr1.aux_qso_icrf2_match in the archive contains the data used in that paper. It is relatively small (2191 entries), so a full download can be carried out using the following ADQL query within the Search→ADQL Form tab.

    select * from gaiadr1.aux_qso_icrf2_match

    The table is fully functional within the archive, though. In the following, it will be shown how to reproduce Figures 5, 6 and 7 of Mignard et al. 2016 using the Archive and Topcat.

  2. Get ICRF catalogue from CDS and upload as user table 'icrf2'

    Follow steps 1-7 from ‘White dwarfs exploration’ tutorial, but using the ICRF2 catalogue (Fey et al. 2015 AJ 150, 58F) and 'icrf2' as the user table name. The CDS catalogue is J/AJ/150/58/icrf2

  3. Gaia QSO to ICRF2 cross-match on source name

    Execute the following ADQL query in the Search→ADQL Form tab, replacing <username> by the appropriate Archive user name.

    SELECT gaia.ra AS gaia_ra, gaia.dec AS gaia_dec,
  icrf2.icrf2, icrf2.source,
  icrf2.raj2000 AS icrf2_ra,
  icrf2.dej2000 AS icrf2_dec
FROM gaiadr1.aux_qso_icrf2_match AS gaia
JOIN user_<username>.icrf2 AS icrf2
  ON gaia.icrf2_match = icrf2.icrf2

    A quick look on the results ('White dwarfs exploration' step 10) shows the combined output:

  4. Gaia to ICRF2 positional differences computation

    The position difference between Gaia and ICRF2 can also be computed within the archive adding two new columns to the output: ra_diff and dec_diff (units: mas)

    SELECT gaia.ra AS gaia_ra, gaia.dec AS gaia_dec,
  icrf2.icrf2, icrf2.source,
  icrf2.raj2000 AS icrf2_ra, icrf2.dej2000 AS icrf2_dec,
  (gaia.ra - icrf2.raj2000) * cos(radians(icrf2.dej2000)) * 3600000 AS ra_diff,
  (gaia.dec - icrf2.dej2000) * 3600000 AS dec_diff
FROM gaiadr1.aux_qso_icrf2_match AS gaia
JOIN user_<username>.icrf2 AS icrf2
  ON gaia.icrf2_match = icrf2.icrf2

  5. Data export to Topcat and representation

    The table can be exported to the Topcat visualization tool via SAMP following ‘White dwarfs tutorial’ steps 15-17. The data will be divided into three subsets according to the contents of column 'source'

    ​​​​​​​

    ​​​​​​​​​​​​​​​​​​​​​

    ​​​​​​​

    We will now represent dec_diff vs ra_dif as a 2D plot

    ​​​​​​​

    Initially, only the points presenting big discrepancies are apparent. Changing the plot range to ±10 mas in both axes, a much more informative plot is obtained.

    ​​​​​​​

    Finally, each subset can be plotted using different colours to reveal the behaviour of the different subsamples. This plot is equivalent to Fig.7 top right panel in Mignard et al. 2016.

    ​​​​​​​

  6. Gaia to ICRF2 positional differences histogram

    The 2D data studied in the previous sections can also be displayed as a collection of 1D histograms. This can directly be carried out within most plotting programs, such as Topcat. However, this approach might become impractical when the number of data points reach billions in size (far from the case for ICRF sources). The following query shows how to compute a histogram directly within the Archive:

    SELECT 0.2 * index AS dec_diff, n from (
  SELECT floor(5 * (gaia.dec - icrf2.dej2000) * 3600000 + 0.1) AS index,
    count(*) AS n
  FROM gaiadr1.aux_qso_icrf2_match AS gaia
  JOIN user_<username>.icrf2 AS icrf2
    ON gaia.icrf2_match = icrf2.icrf2
  GROUP BY index
) AS subquery
ORDER BY dec_diff

    where an intermediate integral index is defined such that the difference in declination is 0.2 mas for each increment of a full init. The accumulations are carried out in a subquery. The outer query reverses the multiplying factor to recover the original mas scale. The output data can then be exported to Topcat and represented as a 1D histogram, using the weighting the 'dec_diff' column as a function of the 'n' number of objects in that bin.

    ​​​​​​​

    After adjusting the x-axis range to ±10 mas

    ​​​​​​​

    and the bin size to 0.2 mas (the value used in the Archive query), the plot is now equivalent to Fig. 6 left panel in Mignard et al. 2016.

    ​​​​​​​

​​​​​​​

Cluster Analysis GUI

Authors: Raúl Gutiérrez, Alcione Mora, José Hernández

This is a tutorial is focused on possible scientific exploration exercise using the Gaia Archive. Realistic science use cases created from users are really welcome and they could be shared in this section with the proper reference/contact point.

We are going to explore a known cluster as the Pleiades (m45) using Gaia data. First, we are going to retrieve all the available data in the region of interest:

  1. As we are going to use the private storage area, we have to log in the archive.

  2. Go to Search → Simple Form. In positional search, enter "pleiades" in the field "Name". Once it is resolved, select a 2 degrees search radius and make sure "Gaia source" is selected. Click on "Show Query" button.

    ​​​​​

  3. Enter "m45" in the field Job name . Edit the ADQL query and remove the TOP 500 restriction. The query should be:

      
    SELECT *
FROM gaiadr1.gaia_source
WHERE CONTAINS(POINT(gaiadr1.gaia_source.ra,gaiadr1.gaia_source.dec),CIRCLE(56.75,24.1167,2))=1

    Execute the query. Around 1e5 results are found.

  4. The number of results are small enough to be represented by a local application. Open Topcat and click Send to SAMP button. As the job is private, Topcat will ask for your credentials.
    Once the data have been loaded, you could show the results in the sphere, or create a proper motion plot to identify the cluster.

  5. Go to the archive and filter data by quality. For that, create a new DB table in your local environment from the job results clicking on the corresponding upload icon (see image):
    ​​​​​

    ​​​​​​​ In this upload operation, no heavy network traffic occurs between the user machine and the server. Job results are stored in the server, so data to be ingested in the user area from job results do not leave the server in the upload process.

    Enter "m45PmFilter" as job name and perform the next query (do not forget to replace <username> with your user name):

      
    SELECT * FROM user_<username>.m45
WHERE abs(pmra_error/pmra)<0.10
AND  abs(pmdec_error/pmdec)<0.10
AND pmra IS NOT NULL AND abs(pmra)>0
AND pmdec IS NOT NULL AND abs(pmdec)>0;

    Execute the query. You can send the results to Topcat via SAMP and plot the new results over the proper motion plot to see the sources with sufficient proper motion quality.

  6. Create the table m45pmfilter from m45PmFilter job results. Now we are going to take the candidate objects to be in the cluster. Based on the proper motion plot, we execute the next filter:

    SELECT * FROM user_<username>.m45pmfilter
WHERE pmra BETWEEN 15 AND 25
AND pmdec BETWEEN -55 AND -40;

    Name the job as m45cluster and execute the job. You could send the results to Topcat and plot over the previous proper motion plots.

  7. Using the ADQL interface we can perform analysis queries on the results. Create the table m45cluster from m45cluster job results. Name the job as m45clusterParallaxAvg . Execute the next query:

    SELECT avg(parallax) as avg_parallax FROM user_<username>.m45cluster

    Execute the query and show the results using the corresponding button from the job results.

    ​​​​​​​

  8. Now, we want to add information from other catalogues. To do so, we will make use of the crossmatch functionality of the archive. We need to identify which columns of our table have the geometrical information ( ra and dec in our case). Select m45cluster table by checking its check box and click on the Edit table button. Find column ra and set the flag Ra . Find the column dec and set the flag Dec . Click on Update button .

    ​​​​​This action will create a positional index on Ra and Dec, and the table will be identified as a geometrical table. This allows the use of the crossmatch functionality.

    ​​​​​​​

  9. Click on crossmatch button. Select user_<username>.m45cluster for TableA and gaiadr1.twomass_original_valid for TableB . Click on Execute .
    This will create a new job of crossmatch type and a new table called xmatch_m45cluster_tmass_original_valid . This table is a joint table between m45cluster and twomass_original_valid tables. A helper function is available in the crossmatch job to create a join query between the two tables.

     
    Positional crossmatch is performed for the time being. All counterparts falling in the search radius are considered matches. Distance is provided for further filtering.

  10. To create a table with all the information from the two tables, click in the Show join query button of the crossmatch job. A query like this is automatically loaded in the ADQL query panel:

      
    SELECT c."dist", a."dec", a."m45cluster_oid", a."ra", a."astrometric_chi2_ac", a."astrometric_chi2_al", a."astrometric_delta_q", a."astrometric_excess_noise",
		a."astrometric_excess_noise_sig", a."astrometric_go_f", a."astrometric_n_obs_ac", a."astrometric_n_obs_al", a."astrometric_n_outliers_ac",
		a."astrometric_n_outliers_al", a."astrometric_params_solved", a."astrometric_primary_flag", a."astrometric_priors_used", a."astrometric_rank_defect",
		a."astrometric_relegation_factor", a."astrometric_weight_ac", a."astrometric_weight_al", a."dec_error", a."dec_parallax_corr", a."dec_pmdec_corr",
		a."dec_pmra_corr", a."dec_pmradial_corr", a."m45_oid", a."m45pmfilter_oid", a."matched_observations", a."parallax", a."parallax_error",
		a."parallax_pmdec_corr", a."parallax_pmra_corr", a."parallax_pmradial_corr", a."phot_bp_mean_flux", a."phot_bp_mean_flux_error", a."phot_bp_mean_mag",
		a."phot_bp_n_obs", a."phot_g_mean_flux", a."phot_g_mean_flux_error", a."phot_g_mean_mag", a."phot_g_n_obs", a."phot_rp_mean_flux",
		a."phot_rp_mean_flux_error", a."phot_rp_mean_mag", a."phot_rp_n_obs", a."phot_variable_flag", a."pmdec", a."pmdec_error", a."pmdec_pmradial_corr",
		a."pmra", a."pmradial", a."pmradial_error", a."pmra_error", a."pmra_pmdec_corr", a."pmra_pmradial_corr", a."ra_dec_corr", a."radial_velocity",
		a."radial_velocity_constancy_probability", a."radial_velocity_error", a."ra_error", a."random_index", a."ra_parallax_corr", a."ra_pmdec_corr",
		a."ra_pmra_corr", a."ra_pmradial_corr", a."ref_epoch", a."scan_direction_mean_k1", a."scan_direction_mean_k2", a."scan_direction_mean_k3",
		a."scan_direction_mean_k4", a."scan_direction_strength_k1", a."scan_direction_strength_k2", a."scan_direction_strength_k3", a."scan_direction_strength_k4",
		a."solution_id", a."source_id",
		b."dec", b."ra", b."designation", b."err_ang", b."err_maj", b."err_min", b."ext_key", b."h_m", b."h_msigcom", b."j_date", b."j_m", b."j_msigcom",
		b."k_m", b."k_msigcom", b."ph_qual", b."tmass_oid"
FROM user_<username>.m45cluster AS a, public.tmass_original_valid AS b, user_<username>.xmatch_m45cluster_tmass_original_valid AS c
WHERE (c.m45cluster_m45cluster_oid = a.m45cluster_oid AND c.tmass_original_valid_tmass_oid = b.tmass_oid)
    All the columns of the joined tables are shown for convenience. This way, the user has only to remove colums to get the desired output.

    Name the job as xmatch and execute it. Show the results and verify that both Gaia and 2Mass data are present.

  11. Now, we are going to use the sharing functionality of the archive to share this results with some colleagues. First, create a new table cluster_2mass from the previous job. Go to the tab SHARE > Groups and create a group called cluster . The new group appears in the tree. Select the group and click on Edit . Use the User to include field to search for your colleague click on add. Repeat this search to add any other colleagues you want. Then click on Update . The group has been updated with the new members.
    Return to the ADQL search page, check the cluster_2mass table and click on Share button. Select group cluster , click Add and then Update . You will see that a little Share icon is added to the table icon. From this moment, the users in cluster group will be notified and they will have access to this table.

     
    ​​​​​​​ Size of shared tables is only accounted for the quota of the table owner.

Cluster Analysis Python

Authors: Deborah Baines

This tutorial has taken the Cluster analysis tutorial and adapted it to python. The tutorial uses the Gaia TAP+ (astroquery.gaia) module .

 

This tutorial is focused on a possible scientific exploration exercise for a known cluster, the Pleiades (M45), using data from the Gaia Archive.

You can import and run this tutorial in your own Jupyter Notebook using this file: Download

First, we import all the required python modules:

 

In [1]:
import astropy.units as u
from astropy.coordinates.sky_coordinate import SkyCoord
from astropy.units import Quantity
from astroquery.gaia import Gaia
 
Created TAP+ (v1.0) - Connection:
	Host: gea.esac.esa.int
	Use HTTPS: True
	Port: 80
	SSL Port: 443
In [2]:
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

# Suppress warnings. Comment this out if you wish to see the warning messages
import warnings
warnings.filterwarnings('ignore')
 

Do the following to load and look at the available Gaia table names:

In [3]:
from astroquery.gaia import Gaia
tables = Gaia.load_tables(only_names=True)
for table in (tables):
    print (table.get_qualified_name())
 
Retrieving tables...
Parsing tables...
Done.
public.dual
public.tycho2
public.igsl_source
public.hipparcos
public.hipparcos_newreduction
public.hubble_sc
public.igsl_source_catalog_ids
tap_schema.tables
tap_schema.keys
tap_schema.columns
tap_schema.schemas
tap_schema.key_columns
gaiadr1.phot_variable_time_series_gfov
gaiadr1.ppmxl_neighbourhood
gaiadr1.gsc23_neighbourhood
gaiadr1.ppmxl_best_neighbour
gaiadr1.sdss_dr9_neighbourhood
gaiadr1.rrlyrae
gaiadr1.allwise_neighbourhood
gaiadr1.gsc23_original_valid
gaiadr1.tmass_original_valid
gaiadr1.allwise_best_neighbour
gaiadr1.cepheid
gaiadr1.urat1_neighbourhood
gaiadr1.ppmxl_original_valid
gaiadr1.tmass_neighbourhood
gaiadr1.ucac4_best_neighbour
gaiadr1.ucac4_neighbourhood
gaiadr1.aux_qso_icrf2_match
gaiadr1.phot_variable_time_series_gfov_statistical_parameters
gaiadr1.sdssdr9_original_valid
gaiadr1.urat1_best_neighbour
gaiadr1.variable_summary
gaiadr1.ucac4_original_valid
gaiadr1.tmass_best_neighbour
gaiadr1.gsc23_best_neighbour
gaiadr1.gaia_source
gaiadr1.ext_phot_zero_point
gaiadr1.sdss_dr9_best_neighbour
gaiadr1.tgas_source
gaiadr1.urat1_original_valid
gaiadr1.allwise_original_valid
 

Next, we retrieve all the available data in the region of interest.

To do this we perform an asynchronous query (asynchronous rather than synchronous queries should be performed when retrieving more than 2000 rows) centred on the Pleides (coordinates: 56.75, +24.1167) with a search radius of 2 degrees and save the results to a file.

Note: The query to the archive is with ADQL (Astronomical Data Query Language). For a description of ADQL and more examples see the Gaia DR1 ADQL cookbook.​​

In [4]:
job = Gaia.launch_job_async("SELECT * \
FROM gaiadr1.gaia_source \
WHERE CONTAINS(POINT(gaiadr1.gaia_source.ra,gaiadr1.gaia_source.dec),CIRCLE(56.75,24.1167,2))=1;" \
, dump_to_file=True)

print (job)
 
Launched query: 'SELECT * FROM gaiadr1.gaia_source WHERE CONTAINS(POINT(gaiadr1.gaia_source.ra,gaiadr1.gaia_source.dec),CIRCLE(56.75,24.1167,2))=1;'
Retrieving async. results...
Jobid: 1495017686224O
Phase: None
Owner: None
Output file: async_20170517124518.vot
Results: None
 

Inspect the output table and number of rows (around 1e5 results are found):

In [5]:
r = job.get_results()
print (r['source_id'])
 
    source_id
-----------------
66926207631181184
66818318054203520
66917823855519360
66830859358837888
66809423175240448
66944761890240000
66980191076373760
66781621852927232
66827805636652928
66947545031024640
              ...
66649989694512256
65666785781176576
64014803920669568
64137880504644992
66542306274948224
64005909043397504
66689881351473664
66436615718993792
65645757620918912
66718434293042560
64103559419447296
Length = 98538 rows
 

To identify the cluster, create a proper motion plot of proper motion in RA (pmra) versus proper motion in DEC (pmdec) in the range pmra [-60,80] and pmdec [-120,30]:

In [6]:
plt.scatter(r['pmra'], r['pmdec'], color='r', alpha=0.3)
plt.xlim(-60,80)
plt.ylim(-120,30)

plt.show()
 
 
 

Perform another asynchronous query to filter the results by quality:

In [7]:
job2 = Gaia.launch_job_async("SELECT * \
FROM gaiadr1.gaia_source \
WHERE CONTAINS(POINT(gaiadr1.gaia_source.ra,gaiadr1.gaia_source.dec),CIRCLE(56.75,24.1167,2))=1 \
AND abs(pmra_error/pmra)<0.10 \
AND abs(pmdec_error/pmdec)<0.10 \
AND pmra IS NOT NULL AND abs(pmra)>0 \
AND pmdec IS NOT NULL AND abs(pmdec)>0;", dump_to_file=True)
 
Launched query: 'SELECT * FROM gaiadr1.gaia_source WHERE CONTAINS(POINT(gaiadr1.gaia_source.ra,gaiadr1.gaia_source.dec),CIRCLE(56.75,24.1167,2))=1 AND abs(pmra_error/pmra)<0.10 AND abs(pmdec_error/pmdec)<0.10 AND pmra IS NOT NULL AND abs(pmra)>0 AND pmdec IS NOT NULL AND abs(pmdec)>0;'
Retrieving async. results...
 

Again, inspect the output table and number of rows:

In [8]:
j = job2.get_results()
print (j['source_id'])
 
    source_id
-----------------
66623395256627712
66581957412169728
65614730777165824
64053561704835584
65828207832006400
66858862543722112
66863054431798272
66588520122169728
66657823714473856
66592437132341248
              ...
66912360656318208
66471215975411200
66729257611496704
64013704408496896
64013807487711360
66506331628024832
66715101399291392
66724447247218048
66610957031353856
66857281995760000
66570549979009280
Length = 218 rows
 

Plot these new filtered results on the same plot as the previous search:

In [9]:
plt.scatter(r['pmra'], r['pmdec'], color='r', alpha=0.3)
plt.scatter(j['pmra'], j['pmdec'], color='b', alpha=0.3)
plt.xlim(-60,80)
plt.ylim(-120,30)

plt.show()
 
 
 

Now we are going to take the candidate objects to be in the cluster. Based on the proper motion plot, we execute the same job with the following constraints on the proper motions in RA and DEC: pmra between 15 and 25, pmdec between -55 and -40:

In [10]:
job3 = Gaia.launch_job_async("SELECT * \
FROM gaiadr1.gaia_source \
WHERE CONTAINS(POINT(gaiadr1.gaia_source.ra,gaiadr1.gaia_source.dec),CIRCLE(56.75,24.1167,2))=1 \
AND abs(pmra_error/pmra)<0.10 \
AND abs(pmdec_error/pmdec)<0.10 \
AND pmra IS NOT NULL AND abs(pmra)>0 \
AND pmdec IS NOT NULL AND abs(pmdec)>0 \
AND pmra BETWEEN 15 AND 25 \
AND pmdec BETWEEN -55 AND -40;", dump_to_file=True)
 
Launched query: 'SELECT * FROM gaiadr1.gaia_source WHERE CONTAINS(POINT(gaiadr1.gaia_source.ra,gaiadr1.gaia_source.dec),CIRCLE(56.75,24.1167,2))=1 AND abs(pmra_error/pmra)<0.10 AND abs(pmdec_error/pmdec)<0.10 AND pmra IS NOT NULL AND abs(pmra)>0 AND pmdec IS NOT NULL AND abs(pmdec)>0 AND pmra BETWEEN 15 AND 25 AND pmdec BETWEEN -55 AND -40;'
Retrieving async. results...
 

Again, inspect the output table and number of rows, and call the job 'm45cluster':

In [11]:
m45cluster = job3.get_results()
print (m45cluster['parallax'])
 
     parallax
    Angle[mas]
------------------
7.4545648282310184
7.5239065408350996
6.9301093717323772
7.5788921365825894
 7.328286272889863
7.0233646883680159
8.1389671643305235
8.1445555234101779
7.3648468313130566
7.1694418609395765
               ...
7.2826513541337299
7.4681735932280162
 7.658985698496882
7.2023395286122085
8.2920957945484393
7.1461398221089798
7.8361028050064272
7.8762955633015945
 7.066701274479442
7.3330438493994405
Length = 106 rows
 

Plot these new filtered results on the same plot as the previous search:

In [12]:
plt.scatter(r['pmra'], r['pmdec'], color='r', alpha=0.3)
plt.scatter(j['pmra'], j['pmdec'], color='b', alpha=0.3)
plt.scatter(m45cluster['pmra'], m45cluster['pmdec'], color='g', alpha=0.3)
plt.xlim(-60,80)
plt.ylim(-120,30)

plt.show()
 
 
 

Calculate the average parallax and standard deviation of the parallax for the M45 cluster candidates:

In [13]:
avg_parallax = np.mean(m45cluster['parallax'])
stddev_parallax = np.std(m45cluster['parallax'])
print (avg_parallax, stddev_parallax)
 
7.4686695575 0.834822732559
 

Now, we want to add information from other catalogues, in this example from 2MASS. To do this we make use of the pre-computed cross-matched tables provided in the Gaia archive.

We obtain the 2MASS photometric data by using the Gaia - 2MASS cross-matched best neighbour table (gaiadr1.tmass_best_neighbour) to identify the sources and the 2MASS original table (gaiadr1.tmass_original_valid) to retrieve the photometry:

In [14]:
job4 = Gaia.launch_job_async("SELECT * \
FROM gaiadr1.gaia_source AS g, gaiadr1.tmass_best_neighbour AS tbest, gaiadr1.tmass_original_valid AS tmass \
WHERE g.source_id = tbest.source_id AND tbest.tmass_oid = tmass.tmass_oid \
AND CONTAINS(POINT(g.ra,g.dec),CIRCLE(56.75,24.1167,2))=1 \
AND abs(pmra_error/pmra)<0.10 \
AND abs(pmdec_error/pmdec)<0.10 \
AND pmra IS NOT NULL AND abs(pmra)>0 \
AND pmdec IS NOT NULL AND abs(pmdec)>0 \
AND pmra BETWEEN 15 AND 25 \
AND pmdec BETWEEN -55 AND -40;", dump_to_file=False)
 
Launched query: 'SELECT * FROM gaiadr1.gaia_source AS g, gaiadr1.tmass_best_neighbour AS tbest, gaiadr1.tmass_original_valid AS tmass WHERE g.source_id = tbest.source_id AND tbest.tmass_oid = tmass.tmass_oid AND CONTAINS(POINT(g.ra,g.dec),CIRCLE(56.75,24.1167,2))=1 AND abs(pmra_error/pmra)<0.10 AND abs(pmdec_error/pmdec)<0.10 AND pmra IS NOT NULL AND abs(pmra)>0 AND pmdec IS NOT NULL AND abs(pmdec)>0 AND pmra BETWEEN 15 AND 25 AND pmdec BETWEEN -55 AND -40;'
Retrieving async. results...
Query finished.
 

Finally, confirm the output table has Gaia and 2MASS photometry and check the number of rows in the table:

In [15]:
p = job4.get_results()
print (p['phot_g_mean_mag', 'j_m', 'h_m', 'ks_m'])
 
 phot_g_mean_mag        j_m            h_m            ks_m
  Magnitude[mag]   Magnitude[mag] Magnitude[mag] Magnitude[mag]
------------------ -------------- -------------- --------------
10.757800849005008      9.6029997      9.2019997      9.0939999
6.0733019438444877      5.9679999      6.0510001      5.9759998
10.643169127592188      9.5349998      9.2189999      9.1370001
6.8275023878511796      6.6989999      6.7329998      6.6919999
 10.15654586177056      9.1169996          8.868      8.7580004
7.5371472521505787      7.2800002           7.29          7.257
8.1328652600027205      7.5879998      7.5279999      7.4699998
8.5268119167756886      7.8920002      7.7670002      7.7379999
9.0234235063721506      8.2360001          8.033      8.0019999
7.0246160054011426          6.848      6.9200001          6.895
               ...            ...            ...            ...
10.511033521483427      9.4770002      9.1400003      9.0719995
11.597470392962894         10.261          9.835      9.7299995
10.591447284586406      9.5310001      9.1920004      9.1230001
10.325731863675678      9.3500004          9.092      8.9960003
9.9103711452615997      8.9829998      8.7110004          8.632
9.3335117945827335          8.533      8.3290005      8.2819996
8.1126755868950937      7.6729999          7.599      7.5760002
6.3275477760376688      6.2280002      6.2480001          6.257
8.1878498252772722          7.526      7.3930001      7.3520002
11.149659997470195      9.9490004      9.5640001      9.4390001
Length = 106 rows
 

All of the above has been performed as an anonymous user to the Gaia archive. To log in to the archive, keep and share your results, see the following instructions: http://astroquery.readthedocs.io/en/latest/gaia/gaia.html#authenticated-access

 

Additional information

The above query to obtain the 2MASS catalogue data can also be performed by using an 'INNER JOIN' in the ADQL query. For example:

In [16]:
job5 = Gaia.launch_job_async("SELECT * \
FROM gaiadr1.gaia_source \
INNER JOIN gaiadr1.tmass_best_neighbour ON gaiadr1.gaia_source.source_id = gaiadr1.tmass_best_neighbour.source_id \
INNER JOIN gaiadr1.tmass_original_valid ON gaiadr1.tmass_original_valid.tmass_oid = gaiadr1.tmass_best_neighbour.tmass_oid \
WHERE CONTAINS(POINT(gaiadr1.gaia_source.ra,gaiadr1.gaia_source.dec),CIRCLE(56.75,24.1167,2))=1 \
AND abs(pmra_error/pmra)<0.10 \
AND abs(pmdec_error/pmdec)<0.10 \
AND pmra IS NOT NULL AND abs(pmra)>0 \
AND pmdec IS NOT NULL AND abs(pmdec)>0 \
AND pmra BETWEEN 15 AND 25 \
AND pmdec BETWEEN -55 AND -40;", dump_to_file=True)
 
Launched query: 'SELECT * FROM gaiadr1.gaia_source INNER JOIN gaiadr1.tmass_best_neighbour ON gaiadr1.gaia_source.source_id = gaiadr1.tmass_best_neighbour.source_id INNER JOIN gaiadr1.tmass_original_valid ON gaiadr1.tmass_original_valid.tmass_oid = gaiadr1.tmass_best_neighbour.tmass_oid WHERE CONTAINS(POINT(gaiadr1.gaia_source.ra,gaiadr1.gaia_source.dec),CIRCLE(56.75,24.1167,2))=1 AND abs(pmra_error/pmra)<0.10 AND abs(pmdec_error/pmdec)<0.10 AND pmra IS NOT NULL AND abs(pmra)>0 AND pmdec IS NOT NULL AND abs(pmdec)>0 AND pmra BETWEEN 15 AND 25 AND pmdec BETWEEN -55 AND -40;'
Retrieving async. results...
 

Confirm the output table has Gaia and 2MASS photometry and check the number of rows in the table is the same as above (106 rows):

In [17]:
test = job5.get_results()
print (test['phot_g_mean_mag', 'j_m', 'h_m', 'ks_m'])
 
 phot_g_mean_mag        j_m            h_m            ks_m
  Magnitude[mag]   Magnitude[mag] Magnitude[mag] Magnitude[mag]
------------------ -------------- -------------- --------------
10.757800849005008      9.6029997      9.2019997      9.0939999
6.0733019438444877      5.9679999      6.0510001      5.9759998
10.643169127592188      9.5349998      9.2189999      9.1370001
6.8275023878511796      6.6989999      6.7329998      6.6919999
 10.15654586177056      9.1169996          8.868      8.7580004
7.5371472521505787      7.2800002           7.29          7.257
8.1328652600027205      7.5879998      7.5279999      7.4699998
8.5268119167756886      7.8920002      7.7670002      7.7379999
9.0234235063721506      8.2360001          8.033      8.0019999
7.0246160054011426          6.848      6.9200001          6.895
               ...            ...            ...            ...
10.511033521483427      9.4770002      9.1400003      9.0719995
11.597470392962894         10.261          9.835      9.7299995
10.591447284586406      9.5310001      9.1920004      9.1230001
10.325731863675678      9.3500004          9.092      8.9960003
9.9103711452615997      8.9829998      8.7110004          8.632
9.3335117945827335          8.533      8.3290005      8.2819996
8.1126755868950937      7.6729999          7.599      7.5760002
6.3275477760376688      6.2280002      6.2480001          6.257
8.1878498252772722          7.526      7.3930001      7.3520002
11.149659997470195      9.9490004      9.5640001      9.4390001
Length = 106 rows
 

Visually inspect the results are the same by plotting the same as above:

In [18]:
plt.scatter(r['pmra'], r['pmdec'], color='r', alpha=0.3)
plt.scatter(j['pmra'], j['pmdec'], color='b', alpha=0.3)
plt.scatter(m45cluster['pmra'], m45cluster['pmdec'], color='g', alpha=0.3)
plt.scatter(test['pmra'], test['pmdec'], color='y', alpha=0.3)
plt.xlim(-60,80)
plt.ylim(-120,30)

plt.show()
 
 

 

White Dwarfs Exploration

Authors: Jesús Salgado, Juan-Carlos Segovia

This is a tutorial is focused on possible scientific exploration exercise using the Gaia Archive. Realistic science use cases created from users are really welcome and they could be shared in this section with the proper reference/contact point.

We are going to explore white dwarfs observed by Gaia. First, we have a typical known white dwarfs catalogue at Vizier:

  1. Go to Vizier to:
    IR photometry of 2MASS/Spitzer white dwarfs
    http://vizier.u-strasbg.fr/viz-bin/VizieR-3?-source=J/ApJ/657/1013&-out.max=50&-out.form=HTML%20Table&-out.add=_r&-out.add=_RAJ,_DEJ&-sort=_r&-oc.form=sexa

  2. Download the catalogue by:
    • Select in Preferences: max=unlimited, output format = VOTable.
    • Click on Submit.
    • A file called vizier_votable.vot will be created.

  3. Open in a different tab the Gaia Archive:
    https://archives.esac.esa.int/gaia/

  4. Log in Gaia Archive.

  5. Click on SEARCH Tab and, inside it, the Advanced ADQL Form.

  6. Click on the upload table button

    and select the vizier_votable.vot VOTable.

  7. Use as table name "dwarfs" and click on the Upload button. As some of the column names of the downloaded VOTable are not compatible with TAP/ADQL, the system will automatically rename them showing the next notice:

  8. Select the uploaded table user_<your_login_name>.dwarfs (under 'User tables') and click on the edit table button:

    ​​​​​​​​​​​​​​

  9. For column col_raj2000 select the flag Ra. And for column col_dej2000 select the flag Dec. Then click UPDATE. The table icon in the tree will change to a "Positional indexed table"

    ​​​​​​​

  10. Inspect the table content inside Gaia Archive. Type in the form at the top of the page:

    select top 100 * from user_<your_user_name>.dwarfs

    where <your_user_name> is your own username, and click 'Submit Query'. A new job will start to be exectued and, when finished, the table result could be inspected by clicking on the "Display top 2000 results" button:

    ​​​​​​​

    Now, we want to obtain metadata of Gaia catalogues for these sources. In order to do that, counterparts should be found by the execution of a crossmatch operation. A positional crossmatch (identification by position) will be executed. More complex algorithms will be offered in future Gaia Archive versions.

  11. Click on the crossmatch button:

    ​​​​​​​

  12. Select as Table A: public.igsl_source and as Table B: user_<your_user_name>.dwarfs with a radius of 1 arcsecond

    Note: IGSL is a combination catalogue from other external catalogues. It has the size of the expected future Gaia catalogue and a synthetic photometry on band G (Gaia). The calculation of this photomtetry could fail for peculiar objects. See more info on IGSL at: http://www.cosmos.esa.int/web/gaia/iow_20131008

  13. Execute the crossmatch. A new job will start. At the end of the execution, a new join table (called xmatch_igsl_source_dwarfs by default) will be created between IGSL and "dwarfs" catalogues.

  14. When finished, click on the "Show join query"

    ​​​​​​​

    This query is an example on how to contain all the metadata of the two catalogues. Reduce the content of the metadata by replacing the SELECT part of the ADQL sentence as follows

    SELECT a."source_id", b."dwarfs_oid", b."name", a."ra", a."dec", b."col_raj2000",  b."col_dej2000",
a."mag_g", b."f_hmagc", b."f_jmagc", b."f_kmagc", b."hmag2", b."hmagc", b."jmag2",
b."jmagc", b."kmagc", b."ksmag2"

    Note: FROM and WHERE conditions must be preserved
    This query will contain the ids of the source in both catalogues (to explore possible duplications), the name of the dwarf stars and magnitudes from both catalogues.

    Click on "Submit Query" to launch a new job.

  15. Open the VO application Topcat:
    http://www.star.bris.ac.uk/~mbt/topcat/topcat-lite.jnlp

  16. Download results and open them in Topcat.

    Click on Download button

    ​​​​​​​

    Open results with Topcat.

  17. on the "Plane Plotting Window" button

    ​​​​​​​

  18. Add, by clicking on the "Add a new positional control to the stack", to create three plots:

    • hmagc versus kmagc
    • jmagc versus kmagc
    • jmagc versus mag_g

    The result should be as follows

    ​​​​​​​

    Most of the points are located in a clear line except some few up and two of them clearly below it. That could imply that they emits on the G band more than the expected (as the G band is synthetic for IGSL, this is not fully clear) or that the crossmatch is not fully correct for these sources.

    Click on the "Display table cell data" button on the main Topcat window.

    ​​​​​​​

  19. Click on the Plan Plot, one at once, on the two strange sources. Once you click on one source, the two plots will be synchronized

    ​​​​​​​​​​​​​​

    ​​​​​​​

  20. Checking the name column in the Table browser, the two sources are:

    • LTT 4816
    • L745-46A

     

    The first object has been identified as a pulsating white dwarf and the second as a simple white dwarf.

    The analysis of the result could suggest a failure on the calculation of the synthetic G magnitude for these objects or some peculiarity on the emission.

    http://simbad.u-strasbg.fr/simbad/sim-id?Ident=LTT%204816

Variable sources (DR1)

Author: Alcione Mora

This tutorial was developed for Data Release 1 and is kept for legacy (DR1 old data still available). The variable stars data model has evolved substantially, and the light curves are not accessible via TAP anymore, but through the DataLink and Massive data services. Please take a look at the DataLink and light curves tutorial if interested in epoch photometry.

​​

Gaia DR1 contains information on a selection of pulsating variables, mostly in the Large Magellanic Cloud, in addition to the other deliverables, mainly the gaia_source table, which contains the astrometry and average photometry for 1.14 billion sources in the sky.

This is an intermediate level tutorial that assumes a basic knowledge of the general interface and workflow. The introductory tutorials White dwarfs exploration and Cluster analysis are recommended in case of difficulties following this exercise.

Variable sources

The main references dealing with the analysis of variable star light curves in DR1 are Eyer et al. 2017 and Clementini et al. 2016 A&A 595A, 133C. The former provides an overview of the all the variability analyses carried out, while the latter focuses on the Cepheids and RR Lyrae pipeline and the results finally published in DR1. The Clementini et al. 2017 abstract is reproduced below.

Context. The European Space Agency spacecraft Gaia is expected to observe about 10,000 Galactic Cepheids and over 100,000 Milky Way RR Lyrae stars (a large fraction of which will be new discoveries), during the five-year nominal lifetime spent scanning the whole sky to a faint limit of G = 20.7 mag, sampling their light variation on average about 70 times.

Aims. We present an overview of the Specific Objects Study (SOS) pipeline developed within the Coordination Unit 7 (CU7) of the Data Processing and Analysis Consortium (DPAC), the coordination unit charged with the processing and analysis of variable sources observed by Gaia, to validate and fully characterise Cepheids and RR Lyrae stars observed by the spacecraft. The algorithms developed to classify and extract information such as the pulsation period, mode of pulsation, mean magnitude, peak-to-peak amplitude of the light variation, subclassification in type, multiplicity, secondary periodicities, and light curve Fourier decomposition parameters, as well as physical parameters such as mass, metallicity, reddening, and age (for classical Cepheids) are briefly described.

Methods. The full chain of the CU7 pipeline was run on the time series photometry collected by Gaia during 28 days of ecliptic pole scanning law (EPSL) and over a year of nominal scanning law (NSL), starting from the general Variability Detection, general Characterization, proceeding through the global Classification and ending with the detailed checks and typecasting of the SOS for Cepheids and RR Lyrae stars (SOS Cep&RRL). We describe in more detail how the SOS Cep&RRL pipeline was specifically tailored to analyse Gaia's G-band photometric time series with a south ecliptic pole (SEP) footprint, which covers an external region of the Large Magellanic Cloud (LMC), and to produce results for confirmed RR Lyrae stars and Cepheids to be published in Gaia Data Release 1 (Gaia DR1).

Results. G-band time series photometry and characterisation by the SOS Cep&RRL pipeline (mean magnitude and pulsation characteristics) are published in Gaia DR1 for a total sample of 3194 variable stars (599 Cepheids and 2595 RR Lyrae stars), of which 386 (43 Cepheids and 343 RR Lyrae stars) are new discoveries by Gaia. All 3194 stars are distributed over an area extending 38 degrees on either side from a point offset from the centre of the LMC by about 3 degrees to the north and 4 degrees to the east. The vast majority are located within the LMC. The published sample also includes a few bright RR Lyrae stars that trace the outer halo of the Milky Way in front of the LMC.

Gaia Archive tables

The variability data are distributed among the following tables:

  • variable_summary. It contains a list of the stars classified in gaia_source as variables, together with the first fundamental frequency and best classification.
  • cepheid. Additional fit parameters for Cepheids, including best sub-class and light curve Fourier analysis (period, peak-to-peak amplitude, first to second harmonic ratio, ...)
  • rrlyrae. Similar results, but for RR Lyrae.
  • phot_variable_time_series_gfov. The G-band light curves: fluxes, errors and magnitudes as a function of time.
  • phot_variable_time_series_gfov_statistical_parameters. Basic statistical analysis of each light curve: number of points, first fourth moments, minimum, maximum, median, ...

In addition, gaia_source has the specific field phot_variable_flag set to 'VARIABLE' for all variable stars in this data release.

Getting summary data

The following queries show how to retrieve a basic summary of the first 10 Cepheids and RR Lyrae in the archive. Note the contents of four different tables is joined to provide a full overview.

Cepheids
SELECT TOP 10
  gaia.source_id, gaia.ra, gaia.dec, gaia.parallax,
  variable.classification, variable.phot_variable_fundam_freq1,
  phot_stats.mean,
  cepheid.peak_to_peak_g, cepheid.num_harmonics_for_p1, cepheid.r21_g, cepheid.phi21_g,
  cepheid.type_best_classification, cepheid.type2_best_sub_classification, cepheid.mode_best_classification
FROM gaiadr1.gaia_source AS gaia
INNER JOIN gaiadr1.variable_summary AS variable
  ON gaia.source_id = variable.source_id
INNER JOIN gaiadr1.phot_variable_time_series_gfov_statistical_parameters AS phot_stats
  ON gaia.source_id = phot_stats.source_id
INNER JOIN gaiadr1.cepheid AS cepheid
  ON gaia.source_id = cepheid.source_id
RR Lyrae
SELECT TOP 10
  gaia.source_id, gaia.ra, gaia.dec, gaia.parallax,
  variable.classification, variable.phot_variable_fundam_freq1,
  phot_stats.mean,
  rrlyrae.peak_to_peak_g, rrlyrae.num_harmonics_for_p1, rrlyrae.r21_g, rrlyrae.phi21_g,
  rrlyrae.best_classification
FROM gaiadr1.gaia_source AS gaia
INNER JOIN gaiadr1.variable_summary AS variable
  ON gaia.source_id = variable.source_id
INNER JOIN gaiadr1.phot_variable_time_series_gfov_statistical_parameters AS phot_stats
  ON gaia.source_id = phot_stats.source_id
INNER JOIN gaiadr1.rrlyrae AS rrlyrae
  ON gaia.source_id = rrlyrae.source_id

The results for the RR Lyrae query are summarised below:

 

 

 

Light curve reconstruction and folding

G-band light curves are only provided for objects classified as variable stars in DR1. They are included in table phot_variable_time_series_gfov. It contains one row per star and observing time. The main fields are:

  • source_id. The source identifier.
  • observation_time. The time scale is TCB, measured in Julian days. The zero point is 2010-01-01T00:00:00.
  • g_flux, g_error. Units are electrons per second.
  • g_magnitude. Vega scale. Converted from g_flux using the zero points in table ext_phot_zero_point.
  • rejected_by_variability_processing. Identified outliers.

Magnitude errors are not provided, because they cannot be easily quantified with a single number for low signal to noise fluxes. In the high signal regime, the following approximate relation can be used.

ΔG ≈ 2.5/log(10) * Δf/f

where G is the magnitude and f the flux. Note that using fluxes instead of magnitudes is recommended whenever precise analyses are required (e.g. photometric system cross-calibration, spectral energy distribution construction, ...).

The following query illustrates how to retrieve the light curve in magnitudes of a given variable: the RR Lyrae with source_id = 5284240582308398080 in this example.

SELECT
  curves.observation_time,
  mod(curves.observation_time - rrlyrae.epoch_g, rrlyrae.p1)/ rrlyrae.p1 as phase,
  curves.g_magnitude,
  2.5/log(10)* curves.g_flux_error/ curves.g_flux
     AS g_magnitude_error,
  rejected_by_variability_processing AS rejected
FROM gaiadr1.phot_variable_time_series_gfov AS curves
INNER JOIN gaiadr1.rrlyrae AS rrlyrae
  ON rrlyrae.source_id = curves.source_id
WHERE rrlyrae.source_id = 5284240582308398080

The output contains the time (TCB), phase, G-band magnitude and estimated error and a flag indicating whether this point has been used by the variability processing. The phase is estimated folding the time using the best fit period. The origin is taken at the epoch of maximum flux in the fitted harmonic model.

The data can then be exported for further use. Topcat plots of the unfolded and folded light curves with error bars are provided below.

 

Miscellaneous plots from Clementini et al. 2016

Additional ADQL queries are provided to reproduce some plots (using Topcat) originally included in the Clementini et al. (2016)

Fig. 28. Histogram of RR Lyrae periods (all Gaia sources)
SELECT floor(p1 * 50) / 50 AS period, count(*) AS n
FROM gaiadr1.rrlyrae
GROUP BY period
ORDER BY period

 

Fig. 30 top panel. RR Lyrae period – G-band amplitude diagram
SELECT p1, peak_to_peak_g, best_classification
FROM gaiadr1.rrlyrae

Two exclusive subsets are created based on the best_classification column.

 

Fig. 34 top panel. Cepheids period-luminosity diagram
SELECT p1, int_average_g,
type_best_classification, type2_best_sub_classification, mode_best_classification
FROM gaiadr1.cepheid

 

Topcat views of the query result shown as a table and the corresponding subset definition used in the plot above are included below.

​​​​

Back to Gaia Help