How to extract Gaia data programmatically - Gaia Users
Help supportShould you have any question, please check the Gaia FAQ section or contact the Gaia Helpdesk |
Python Access: Astroquery
ESA Gaia Archive provides a Python Astroquery (Astropy) package for easy access to the ESA Gaia Archive: astroquery.gaia. The latest Astroquery version can be instaled following the instructions given in the "How can I install the latest version of Astroquery.Gaia" FAQ.
Also, a generic package is provided for accessing any TAP and TAP+ compliant service: astroquery.utils.tap.
See a Python tutorial for Jupyter Notebook example here.
Command line access: TAP/TAP+
The entry point is a TAP (Table Access Protocol) server. You may use HTTP protocol to execute TAP requests at https://gea.esac.esa.int/tap-server/tap. TAP provides two operation modes:
- Synchronous: the response to the request will be generated as soon as the request received by the server.
- Asynchronous: the server will start a job that will execute the request. The first response to the request is the required information (a link) to obtain the job status. Once the job is finished, the results can be retrieved.
Our TAP server provides two access modes:
- Public: this is the standard TAP access. A user can execute ADQL queries and upload tables to be used in a query 'on-the-fly' (these tables will be removed once the query is executed). The results are available to any other user and they will remain in the server for a limited space of time.
- Authenticated: some functionalities are restricted to authenticated users only. The results are saved in a private user space and they will remain in the server for ever (they can be removed by the user).
- ADQL queries and results are saved in a user private area.
- Built-in cross-match: a catalogue cross-match operation can be executed. Cross-match operations results are saved in a user private area.
- Persintance of uploaded tables: a user can upload a table in a private space. These tables can be used in queries as well as in cross-matches operations.
Here you can find some examples about how to interact with a TAP server (we are using the curl tool):
1. Non authenticated access
2. Authenticated access
3. Interface
1. Non authenticated access
Content:
1.1. Getting all public tables1.2. Synchronous query
1.3. Synchronous query on an 'on-the-fly' uploaded table
1.4. Asynchronous query
1.5. Python low-level script to execute an asynchronous query and wait until the results are available.
1.1. Getting all public tables
curl "https://gea.esac.esa.int/tap-server/tap/tables"
1.2. Synchronous query
curl "https://gea.esac.esa.int/tap-server/tap/sync?REQUEST=doQuery&LANG=ADQL&FORMAT=votable&QUERY=SELECT+TOP+5+source_id,ra,dec+FROM+gaiadr1.gaia_source"
The retrieved results is a VO table by default (see '3.2. Synchronous Queries' section parameters to specify a different output format). The results can be saved in a file and inspected using any analysis tool like TOPCAT, for instance.
1.3. Synchronous query on an 'on-the-fly' uploaded table
curl --form UPLOAD="table_c,param:table1" --form table1=@test_ra_dec.vot --form LANG=ADQL --form REQUEST=doQuery --form QUERY="select top 5 * from tap_upload.table_c" https://gea.esac.esa.int/tap-server/tap/sync
Where 'test_ra_dec.vot'
is a file that contains the VOTable to be uploaded (in order to be used by the query)
The retrieved results is a VO table by default (see '3.2. Synchronous Queries' section parameters to specify a different output format). The results can be saved in a file and inspected using any analysis tool like TOPCAT, for instance.
1.4. Asynchronous query
curl -i -X POST --data "PHASE=run&LANG=ADQL&LANG=ADQL&REQUEST=doQuery&QUERY=select+top+5+*+from+gaiadr1.gaia_source" "https://gea.esac.esa.int/tap-server/tap/async"
Note that there is the possibility to use the optional parameters "JOBNAME" to assign a name to the job and "JOBDESCRIPTION" to add a description:
curl -i -X POST --data "PHASE=run&LANG=ADQL&JOBNAME=optionalJobName&JOBDESCRIPTION=optionalDescription&LANG=ADQL&REQUEST=doQuery&QUERY=select+top+5+*+from+gaiadr1.gaia_source" "https://gea.esac.esa.int/tap-server/tap/async"
The response will contain the URL of the job running at server side (see Location header):
HTTP/1.1 303 See Other Date: Mon, 30 Jun 2014 14:44:39 GMT Server: Apache-Coyote/1.1 Location: https://gea.esac.esa.int/tap-server/tap/async/1404139480755A Content-Type: application/x-www-form-urlencoded Connection: close Transfer-Encoding: chunked
To obtain the status of the running job:
curl "https://gea.esac.esa.int/tap-server/tap/async/1404139480755A"
The status response is something like:
<?xml version="1.0" encoding="UTF-8"?> <uws:job xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <uws:jobId><![CDATA[1404139480755A]]></uws:jobId> <uws:runId xsi:nil="true" /> <uws:ownerId><![CDATA[anonymous]]></uws:ownerId> <uws:phase>COMPLETED</uws:phase> <uws:quote xsi:nil="true" /> <uws:startTime>2014-06-30T16:44:40.766+0200</uws:startTime> <uws:endTime>2014-06-30T16:44:40.830+0200</uws:endTime> <uws:executionDuration>0</uws:executionDuration> <uws:destruction>2014-07-07T16:44:40.754+0200</uws:destruction> <uws:parameters> <uws:parameter id="maxRec"><![CDATA[100000]]></uws:parameter> <uws:parameter id="query"><![CDATA[select top 5 * from gaiadr1.gaia_source]]></uws:parameter> <uws:parameter id="request"><![CDATA[doQuery]]></uws:parameter> <uws:parameter id="format"><![CDATA[votable]]></uws:parameter> <uws:parameter id="keepAuthenticatedUserJobs"><![CDATA[true]]></uws:parameter> <uws:parameter id="lang"><![CDATA[ADQL]]></uws:parameter> <uws:parameter id="version"><![CDATA[1.0]]></uws:parameter> </uws:parameters> <uws:results> <uws:result id="result" xlink:type="simple" xlink:href="http%3A%2F%2Fgea.esac.esa.int%2Ftap-server%2Ftap%2Fasync%2F1404139480755A%2Fresults%2Fresult" mime="application/x-votable+xml" size="8863" rows="6" /> </uws:results> <uws:errorSummary xsi:nil="true" />
To obtain the results of the job (once the job is finished):
curl "https://gea.esac.esa.int/tap-server/tap/async/1404139480755A/results/result"
The retrieved results is a VO table by default (see '3.2 Synchronous Queries' section parameters to specify a different output format).
The results can be saved in a file and inspected using any analysis tool like TOPCAT, for instance.
1.5. Python low-level script to execute an asynchronous query and wait until the results are available.
It is also possible to access the Archive using a low-level Python script that does not require external libraries like Astroquery.
#ASYNCHRONOUS REQUEST #Python 2 #import httplib #import urllib #Python 3 import http.client as httplib import urllib.parse as urllib import time from xml.dom.minidom import parseString host = "gea.esac.esa.int" port = 443 pathinfo = "/tap-server/tap/async" #------------------------------------- #Create job params = urllib.urlencode({\ "REQUEST": "doQuery", \ "LANG": "ADQL", \ "FORMAT": "votable_plain", \ "PHASE": "RUN", \ "JOBNAME": "Any name (optional)", \ "JOBDESCRIPTION": "Any description (optional)", \ "QUERY": "SELECT DISTANCE(POINT(266.41683,-29.00781),POINT(ra,dec)) AS dist, * FROM gaiadr1.gaia_source WHERE DISTANCE(POINT(266.41683,-29.00781),POINT(ra,dec)) < 0.08333333 ORDER BY dist ASC" }) headers = {\ "Content-type": "application/x-www-form-urlencoded", \ "Accept": "text/plain" \ } connection = httplib.HTTPSConnection(host, port) connection.request("POST",pathinfo,params,headers) #Status response = connection.getresponse() print ("Status: " +str(response.status), "Reason: " + str(response.reason)) #Server job location (URL) location = response.getheader("location") print ("Location: " + location) #Jobid jobid = location[location.rfind('/')+1:] print ("Job id: " + jobid) connection.close() #------------------------------------- #Check job status, wait until finished while True: connection = httplib.HTTPSConnection(host, port) connection.request("GET",pathinfo+"/"+jobid) response = connection.getresponse() data = response.read() #XML response: parse it to obtain the current status #(you may use pathinfo/jobid/phase entry point to avoid XML parsing) dom = parseString(data) phaseElement = dom.getElementsByTagName('uws:phase')[0] phaseValueElement = phaseElement.firstChild phase = phaseValueElement.toxml() print ("Status: " + phase) #Check finished if phase == 'COMPLETED': break #wait and repeat time.sleep(0.2) connection.close() #------------------------------------- #Get results connection = httplib.HTTPSConnection(host, port) connection.request("GET",pathinfo+"/"+jobid+"/results/result") response = connection.getresponse() data = response.read().decode('iso-8859-1') #print(type(data)) #print(data) outputFileName = "example_votable_output.vot" outputFile = open(outputFileName, "w") outputFile.write(data) outputFile.close() connection.close() print ("Data saved in: " + outputFileName)
The saved file is a VO table (by default, see '3.2 Synchronous Queries' section parameters to specify a different output format). The file can be inspected using any analysis tool like TOPCAT, for instance.
2. Authenticated access
Content:
2.1. Login2.2. Logout
2.3. Getting public and user tables
2.4. Upload user table
2.5. Delete user table
2.6. Asynchronous query
2.7. Shared tables
2.8. Share capability
2.9. Events capability
2.10. Notifications capability
2.11. Table flags editing capability
2.12. Listing jobs
2.13. Deleting jobs
2.14. Queries on jobs
2.15. External TAP queries
2.16. Global Tap Schema (GloTS) search
2.1. Login
curl -k -c cookies.txt -X POST -d username=USERNAME -d password=PASSWORD -L "https://gea.esac.esa.int/tap-server/login"
2.2. Logout
curl -k -b cookies.txt -X POST -d -L "https://gea.esac.esa.int/tap-server/logout"
2.3. Getting public and user tables
curl -k -b cookies.txt -X POST -L "https://gea.esac.esa.int/tap-server/tap/tables"
2.4. Upload user table
curl -k -b cookies.txt -X POST -F FILE=@file.name -F TABLE_NAME=table_name "https://gea.esac.esa.int/tap-server/Upload"
Where 'file.name'
is a file that contains the VOTable to be uploaded. TABLE_DESC
parameter can be used to specify a table description. FORMAT
parameter can be used to specify the table data format. Available formats: 'VOTable'
, 'CSV'
and 'ASCII'
: see section 3.7 for more details.
2.5. Delete user table
curl -k -b cookies.txt -X POST -F TABLE_NAME=table_name -F DELETE=TRUE "https://gea.esac.esa.int/tap-server/Upload"
Where 'table.name'
is the argument used when the table was uploaded (argument TABLE_NAME=table_name). To force a table removal, use FORCE_REMOVAL=TRUE flag:
curl -k -b cookies.txt -X POST -F TABLE_NAME=table_name -F DELETE=TRUE -F FORCE_REMOVAL=TRUE "https://gea.esac.esa.int/tap-server/Upload"
2.6. Asynchronous query
curl -k -b cookies.txt -i -X POST --data "PHASE=run&LANG=ADQL&REQUEST=doQuery&QUERY=select+top+5+*+from+gaiadr1.gaia_source" "https://gea.esac.esa.int/tap-server/tap/async"
curl -k -b cookies.txt -i -X POST --data "PHASE=run&LANG=ADQL&JOBNAME=optionalJobName&JOBDESCRIPTION=optionalDescription&REQUEST=doQuery&QUERY=select+top+5+*+from+gaiadr1.gaia_source" "https://gea.esac.esa.int/tap-server/tap/async"
The response will contain the URL of the job running at server side:
HTTP/1.1 303 See Other Date: Mon, 30 Jun 2014 15:02:00 GMT Server: Apache-Coyote/1.1 Location: https://gea.esac.esa.int/tap-server/tap/async/1404140520859A Content-Type: application/x-www-form-urlencoded Connection: close Transfer-Encoding: chunked
To obtain the status of the running job:
curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/async/1404140520859A"
The status response is something like:
<?xml version="1.0" encoding="UTF-8"?> <uws:job xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <uws:jobId><![CDATA[1404141177261A]]></uws:jobId> <uws:runId xsi:nil="true" /> <uws:ownerId><![CDATA[USERNAME]]></uws:ownerId> <uws:phase>COMPLETED</uws:phase> <uws:quote xsi:nil="true" /> <uws:startTime>2014-06-30T17:12:57.273+0200</uws:startTime> <uws:endTime>2014-06-30T17:12:57.864+0200</uws:endTime> <uws:executionDuration>0</uws:executionDuration> <uws:destruction>2014-07-07T17:23:31.362+0200</uws:destruction> <uws:parameters> <uws:parameter id="jobdescription"><![CDATA[]]></uws:parameter> <uws:parameter id="jobname"><![CDATA[]]></uws:parameter> <uws:parameter id="session"><![CDATA[1404141103551]]></uws:parameter> <uws:parameter id="maxRec"><![CDATA[100000]]></uws:parameter> <uws:parameter id="query"><![CDATA[SELECT DISTANCE(POINT(ra,dec), POINT(266.41683,-29.00781)) AS dist, * FROM gaiadr1.gaia_source WHERE DISTANCE(POINT(266.41683,-29.00781),POINT(ra,dec)) < 0.08333333 ORDER BY dist ASC]]></uws:parameter> <uws:parameter id="request"><![CDATA[doQuery]]></uws:parameter> <uws:parameter id="keepAuthenticatedUserJobs"><![CDATA[true]]></uws:parameter> <uws:parameter id="format"><![CDATA[votable]]></uws:parameter> <uws:parameter id="lang"><![CDATA[ADQL]]></uws:parameter> <uws:parameter id="version"><![CDATA[1.0]]></uws:parameter> </uws:parameters> <uws:results> <uws:result id="result" xlink:type="simple" xlink:href="http%3A%2F%2Fgea.esac.esa.int%2Ftap-server%2Ftap%2Fasync%2F1404141177261A%2Fresults%2Fresult" mime="application/x-votable+xml" size="2468741" rows="4734" /> </uws:results> <uws:errorSummary xsi:nil="true" />
To obtain the results of the job (once the job is finished) and store them locally in a file:
curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/async/1404140520859A/results/result" >query_out.vot.gzip
By default, the downloaded product is a compressed (.gzip) file in VOTable format. Other output formats are also possible (see '3.2 Synchronous Queries' section parameters to specify a different output format). The following example shows how to retrieve the results in .fits format:
curl -k -b cookies.txt -i -X POST --data "PHASE=run&LANG=ADQL&REQUEST=doQuery&QUERY=select+top+5+*+from+gaiadr1.gaia_source&FORMAT=fits" "https://gea.esac.esa.int/tap-server/tap/async"
curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/async/<job_id>/results/result" >query_out.fits.gzip
2.7. Shared tables
Shared tables are accessible once the user is authenticated. You can use the tables in the same way you use your own tables. For instance, if you want to get the first 5 rows of the table 'shared_table' from user 'user_joe' (i.e. 'user_joe._shared_table') in a synchronous mode, you can type:
curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/sync?REQUEST=doQuery&LANG=ADQL&FORMAT=votable&QUERY=SELECT+TOP+5+*+FROM+user_joe.shared_table"
in asynchronous mode:
curl -k -b cookies.txt -i -X POST --data "PHASE=run&LANG=ADQL&REQUEST=doQuery&QUERY=select+top+5+*+from+user_joe.shared_table" "https://gea.esac.esa.int/tap-server/tap/async"
If you want to obtain a list of available tables and columns, you can type:
curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/tables?share_info=true&share_accessible=true"
If you want to obtain a list of available tables, you can type:
curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/tables?share_info=true&share_accessible=true&only_tables=true"
2.8. Share capability
You have to create a shared group in order to share a table.
The following example creates a group named 'new_group' and adds two users ('user_id_1' and 'user_id_2') to it:
curl -k -b cookies.txt -X POST --data "action=CreateOrUpdateGroup&title=my_new_group&description=descripion&users_list=user_id1,user_id2" "https://gea.esac.esa.int/tap-server/tap/share"
The response will contain the group identifier.
OK Group '<user_id>_1425480932331JC' successfully created.
Then, you can update a group (to add/remove users to a group, see 'Adding/Removing a user' below):
curl -k -b cookies.txt -X POST --data "action=CreateOrUpdateGroup&group_id=<group_identifier>&title=my_new_group&description=descripion&users_list=<new_users_list>" "https://gea.esac.esa.int/tap-server/tap/share"
To obtain a list of groups associated to the user (either the user is the owner or the user belongs to):
curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/share?action=GetGroups"
The response:
<?xml version="1.0" encoding="UTF-8"?> <sharedGroups> <sharedGroup id="user_id_1425482038968JC" owner="user_id"> <title><![CDATA[my_new_group]]></title> <description><![CDATA[descripion]]></description> </sharedGroup> <sharedGroup id="other_user_1417615009193I" owner="other_user"> <title><![CDATA[gacs]]></title> <description><![CDATA[group of gacs developers]]></description> </sharedGroup> </sharedGroups>
In this case, the user 'user_id' is the owner of the group 'user_id_1425482038968JC', while the user 'other_user' owns 'other_user_1417615009193I' group.
To remove a group:
curl -k -b cookies.txt -X POST --data "action=RemoveGroup&group_id=<group_identifier>" "https://gea.esac.esa.int/tap-server/tap/share"
Adding a user:
curl -k -b cookies.txt -X POST --data "action=CreateUserGroup&group_id=<group_identifier>&user_id=<user_id>" "https://gea.esac.esa.int/tap-server/tap/share"
Removing a user:
curl -k -b cookies.txt -X POST --data "action=RemoveUserGroup&group_id=<group_identifier>&user_id=<user_id>" "https://gea.esac.esa.int/tap-server/tap/share"
To share a table named 'user_schema.table1' (full qualified table name) you need to create a shared item. In this example, a new item is created and shared to 'my_new_group' group:
curl -k -b cookies.txt -X POST --data "action=CreateOrUpdateItem&resource_type=0&title=user_schema.table1&description=description&items_list=group_id|Group|Read" "https://gea.esac.esa.int/tap-server/tap/share"
The response will contain the shared resource identifier.
OK Resource '<user_id>_1425542442398JC' (type '0') successfully created.
Then, you can update a shared table:
curl -k -b cookies.txt -X POST --data "action=CreateOrUpdateItem&resource_id=<resource_identifier>&resource_type=0&title=user_schema.table1&description=description&items_list=group_id|Group|Read" "https://gea.esac.esa.int/tap-server/tap/share"
To obtain a list of shared tables:
curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/share?action=GetSharedItems"
The response
<?xml version="1.0" encoding="UTF-8"?> <sharedItems owner="user_id"> <sharedItem id="user_id_1425485873983JC" type="0"> <title><![CDATA[user_schema.table]]></title> <description><![CDATA[description]]></description> <sharedToItems> <sharedToItem shareTo="group_id_1425482038968JC" shareType="Group" shareMode="Read"/> </sharedToItems> </sharedItem> </sharedItems>
To remove a sharing (it does not remove the actual table):
curl -k -b cookies.txt -X POST --data "action=RemoveItem&resource_id=<resource_identifier>&resource_type=0" "https://gea.esac.esa.int/tap-server/tap/share"
2.9. Events capability
Each action (table creation/removal, sharing creation/removal, etc.) generates an event. Events are associated to a session and they are kept while the session (connecton) is open.
Events are grouped by types and each time has a 'latest modification time' (latest time that this type of event was generated).
The following example retrieves the events for a user:
curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/event"
The response will contain the events and times associated.
event_type1=timestamp1 event_type2=timestamp2 ...
2.10. Notifications capability
Notifications provides information about some activities. Notifications are available for authenticated users only and are kept for a month.
The following example retrieves the notifications for a user:
curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/notification?action=GetNotifications"
A response example:
notification_id1[type: notification_type1,notification_subtype1]=explanation1 notification_id2[type: notification_type2,notification_subtype2]=explanation2 ...
2.11. Table flags editing capability
Tables flags can be modified.
The following example explains how to set the flags of a table:
curl -k -b cookies.txt -X POST --data "ACTION=radec&TABLE_NAME=schema.table_name&RA=ra_column_name&DEC=dec_column_name" "https://gea.esac.esa.int/tap-server/TableTool"
Where schema.table_name is a full qualified table name (e.g. user_userid.table_name). ra_column specifies the ra column name. dec_column specifies the dec column name.
2.12. Listing jobs
Jobs can be listed using the following request:
curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/jobs/list?offset=index&limit=jobs_number&order=order"
Where list specifies the jobs list (e.g. 'sync' or 'async'). It is mandatory.
index specifies the number of jobs to skip before beginning to return the first job. By default, it is '0'.
limit specifies the number of jobs to be returned. No limit by default is set.
order specifies the order of the results.
Example:
curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/jobs/async?offset=0&limit=20&order=CREATION_TIME:DESC"
2.13. Deleting jobs
Jobs can be deleted using their identifiers:
curl -k -b cookies.txt -X POST --data "JOB_IDS=job_id1,job_id2..." "https://gea.esac.esa.int/tap-server/tap/deletejobs"
2.14. Queries on jobs
A job can be used as a table in a query. Launch an asynchronous job:
curl -k -b cookies.txt -i -X POST --data "PHASE=run&LANG=ADQL&REQUEST=doQuery&QUERY=select+top+5+*+from+gaiadr1.gaia_source" "https://gea.esac.esa.int/tap-server/tap/async"
Response:
HTTP/1.1 303 See Other Date: Mon, 30 Jun 2014 14:44:39 GMT Server: Apache-Coyote/1.1 Location: https://gea.esac.esa.int/tap-server/tap/async/1404139480755A Content-Type: application/x-www-form-urlencoded Connection: close Transfer-Encoding: chunked
Use the job identifier (1404139480755A in this case) as a table:
curl -k -b cookies.txt --output my_job_results.vot.gz "https://gea.esac.esa.int/tap-server/tap/sync?REQUEST=doQuery&LANG=ADQL&FORMAT=votable&QUERY=SELECT+TOP+5+source_id,ra,dec+FROM+JOB_UPLOAD.\"job1404139480755A\""
NOTE always add 'job' to your job identifier, e.g. job1404139480755A
2.15. External TAP queries
A query can be executed in an external TAP server. When an external query is executed, an asynchronous job is launched to the remote TAP server. (See '3.15. TAP+ external TAP access capability' section parameters)
In synchronous mode, the results are served on the fly (in this example, we are accessing Herschel TAP service).
curl "https://gea.esac.esa.int/tap-server/tap/sync?REQUEST=external_tap&EXT_TAP_URL=http://archives.esac.esa.int/hsa/whsa-tap-server/tap&LANG=ADQL&FORMAT=votable_plain&QUERY=SELECT+TOP+5+observation_id,ra,dec+FROM+hsa.v_active_observation"
In asynchronous mode, the results are saved locally (in this example, we access to Vizier TAP service)
curl -i -X POST --data "REQUEST=external_tap&EXT_TAP_URL=http://tapvizier.u-strasbg.fr/TAPVizieR/tap&PHASE=run&LANG=ADQL&LANG=ADQL&QUERY=select+top+5+recno,lambda,flux+from+%22J/A%2BA/605/L8/quasar_a%22" "https://gea.esac.esa.int/tap-server/tap/async"
Response:
HTTP/1.1 303 See Other Date: Mon, 30 Jun 2014 14:44:39 GMT Server: Apache-Coyote/1.1 Location: https://gea.esac.esa.int/tap-server/tap/async/1404139480755A Content-Type: application/x-www-form-urlencoded Connection: close Transfer-Encoding: chunked
Now, you have a local job (1404139480755A in this case) with the results of your external TAP query.
2.16. Global Tap Schema (GloTS) search
A global tap schema search service is available.
Search for services that contains a table name or description.
curl "https://gea.esac.esa.int/tap-server/tap/glots?ACTION=search_tables&keyword=quasars+new&MODE=all&FORMAT=csv"
In this case, we are searching for the keywords 'quasars new'
Response:
accessurl,ivoid,table_name,table_desc,utype http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/A+AS/134/483/table3,"New QSOs from the Hamburg Quasar Survey Object ( Hagen H.-J., Engels D., Reimers D.)", http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/AJ/113/2000/table1,"New quasar radio detections ( Bischof O.B., Becker R.H.)", http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/AJ/122/518/table1,"New LBQS quasars ( Hewett P.C., Foltz C.B., Chaffee F.H.)", http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/AJ/142/78/table1,"Parameters of 14 new 2.2<z<3.0 quasars in S82 ( Wu X.-B., Wang R., Schmidt K.B., Bian F., Jiang L., Fan X.)", http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/AJ/145/159/table1,"Catalog of new quasars in the vicinity of M31 and M33 discovered by LAMOST in fields observed in 2010 and 2011 ( Huo Z.-Y., Liu X.-W., Xiang M.-S., et al.)", http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/AJ/151/24/table1,"Parameters of the 1180 New Quasars Discovered in LAMOST DR1 ( Ai Y.L., Wu X.-B., Yang J., Yang Q., Wang F., Guo R., Zuo W., Dong X., Zhang Y.-X., Yuan H.-L., Song Y.-H., Wang J., Dong X., Yang M., Wu H., Shen S.-Y., Shi J.-R., He B.-L., Lei Y.-J., Li Y.-B., Luo A.-L., Zhao Y.-H., Zhang H.-T.)", http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/ApJS/194/22/table1,"29 new, 12 plausible, and 3 previously known quasars behind the SMC ( Kozlowski S., Kochanek C. S., Udalski A.)", http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/ApJS/227/11/table5,"Spectroscopic observations of the new PS1 quasars (77 unique sources) ( Banados E., Venemans B.P., Decarli R., et al.)", http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/other/RAA/15.1438/table1,"Catalog of new quasars in fields in the vicinity of M31 and M33 discovered by LAMOST in the 2013 observational season ( Huo Z.-Y., Liu X.-W., Xiang M.-S., Shi J.-R., Yuan H.-B., Huang Y., Zhang Y., Hou Y.-H., Wang Y.-F., Yang M.)", http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,VII/182/table6,"new UV excess quasars (paper II) ( Warren S.J., Hewett P.C., Irwin M.J., Osmer P.S.)",
In order to retrieve all columns of a table, we have to provide the 'ivoid' service identifier and the table name:
curl "https://gea.esac.esa.int/tap-server/tap/glots?ACTION=get_columns&IVOID=ivo%3A%2F%2Fcds.vizier%2Ftap&TABLE_NAME=J%2FA%2BAS%2F134%2F483%2Ftable3&FORMAT=csv"
Response:
ivoid,table_name,column_name,description,unit,ucd,utype,datatype,ndhnpmiighpa,principal,indexed,std ivo://cds.vizier/tap,J/A+AS/134/483/table3,Bmag,B magnitude,mag,phot.mag;em.opt.B, ,DOUBLE,mt.size,0,0,1 ivo://cds.vizier/tap,J/A+AS/134/483/table3,DE2000,Declination (2000.0),deg,pos.eq.dec;meta.main, ,DOUBLE,mt.size,1,0,1 ivo://cds.vizier/tap,J/A+AS/134/483/table3,Name,Object name, ,meta.id;meta.main, ,VARCHAR,mt.size,0,0,1 ivo://cds.vizier/tap,J/A+AS/134/483/table3,RA2000,Right ascension (2000.0),deg,pos.eq.ra;meta.main, ,DOUBLE,mt.size,1,0,1 ivo://cds.vizier/tap,J/A+AS/134/483/table3,Run,Run number, ,meta.id;obs, ,SMALLINT,mt.size,0,0,1 ivo://cds.vizier/tap,J/A+AS/134/483/table3,l_Bmag,[~] Approximate flag on Bmag, ,meta.code.error, ,CHAR(1),mt.size,0,0,1 ivo://cds.vizier/tap,J/A+AS/134/483/table3,recno,Record number assigned by the VizieR team. Should Not be used for identification., ,meta.record, ,INTEGER,mt.size,0,0,1 ivo://cds.vizier/tap,J/A+AS/134/483/table3,z,Redshift, ,src.redshift, ,DOUBLE,mt.size,0,0,1
3. Interface
See the following specifications:
- TAP (Table Access Protocol) v1.0
- UWS (Universal Worker Service) v1.0
- ADQL (Astronomical Data Query Language) v2.0
Content:
3.1. TAP resources3.2. Synchronous Queries
3.3. Asynchronous Queries
3.4. Query on an 'on-the-fly' uploaded table
3.5. TAP+ login
3.6. TAP+ logout
3.7. TAP+ upload user table
3.8. TAP+ delete user table
3.9. TAP+ sharing capability
3.10. TAP+ tables capabilities
3.11. TAP+ events capabilities
3.12. TAP+ notifications
3.13. Table flags editing
3.14. TAP+ jobs capabilities
3.15. TAP+ deletejobs capability
3.16. TAP+ external TAP access capability
3.17. Global Tap Schema (GloTS) search capability
3.1. TAP resources
https://gea.esac.esa.int/tap-server/tap/
Tables |
https://gea.esac.esa.int/tap-server/tap/tables |
|
Synchronous access |
https://gea.esac.esa.int/tap-server/tap/sync |
|
Asynchronous access |
https://gea.esac.esa.int/tap-server/tap/async |
|
Service availability |
https://gea.esac.esa.int/tap-server/tap/availability |
|
Sharing capability |
https://gea.esac.esa.int/tap-server/tap/share |
TAP+ |
Events capability |
https://gea.esac.esa.int/tap-server/tap/event |
TAP+ |
Notifications capability |
https://gea.esac.esa.int/tap-server/tap/notification |
TAP+ |
Jobs listing capability |
https://gea.esac.esa.int/tap-server/tap/jobs |
TAP+ |
Jobs removal capability |
https://gea.esac.esa.int/tap-server/tap/deletejobs |
TAP+ |
3.2. Synchronous Queries
Parameter |
Value |
Comments |
REQUEST |
doQuery |
Requests to execute the provided query |
LANG |
ADQL |
Query language |
FORMAT |
- votable |
Results output format |
QUERY |
ADQL query |
query |
3.3. Asynchronous Queries
Parameter |
Value |
Comments |
Same parameters as defined in 3.2 Synchronous Queries and |
||
PHASE |
run |
Query job initial phase |
The response header will contain the location of the job.
3.4. Query on an 'on-the-fly' uploaded table
Use a multipart/form-data (see IETF RFC 2388) HTTP POST
Parameter |
Value |
Comments |
Same parameters as defined in 3.2 Synchronous Queries and |
||
UPLOAD |
query_table,param:parameter_table_name |
query_table: the name of the table used in the query |
parameter_table_name |
file |
file name that contains the table to be uploaded |
For instance, in the following request:
curl --form UPLOAD="table_c,param:table1" --form table1=@test_ra_dec.vot --form LANG=ADQL --form REQUEST=doQuery --form QUERY="select top 5 * from tap_upload.table_c" https://gea.esac.esa.int/tap-server/tap/sync
table_c
is the name of the table used in the query: QUERY="select top 5 * from tap_upload.table_c"
,
table1
is the name of the HTTP parameter that provides the file: table1=@test_ra_dec.vot
and test_ra_dec.vot
is the file name that contains the table to be uploaded.
3.5. TAP+ login
Parameter |
Value |
Comments |
username |
user_name |
User name |
password |
user_password |
User password |
The response header will contain the session identifier.
3.6. TAP+ logout
Parameter |
Value |
Comments |
session identifier |
session identifier |
Session identifier provided by a login request |
3.7. TAP+ upload user table
Use a multipart/form-data (see IETF RFC 2388) HTTP POST
Parameter |
Value |
Comments |
session identifier |
session identifier |
Session identifier provided by a login request |
FILE |
file |
file name that contains the table to be uploaded |
TABLE_NAME |
table name |
table name associated to the uploaded table |
TABLE_DESC |
table description |
table description associated to the uploaded table |
FORMAT |
-VOTable (see file format) |
table data format |
3.8. TAP+ delete user table
Parameter |
Value |
Comments |
session identifier |
session identifier |
Session identifier provided by a login request |
TABLE_NAME |
table name |
table name associated to the uploaded table |
DELETE |
TRUE |
requested action |
FORCE_REMOVAL |
TRUE |
To force a table removal |
3.9. TAP+ sharing capability
3.9.1. Tables list
'table' TAP capability allows the user to retrieve shared items information. The following parameters can be used:
Parameter |
Value |
Comments |
session identifier |
session identifier |
Session identifier provided by a login request |
share_info |
TRUE |
provides information about shared items |
share_accessible |
TRUE |
includes tables that are accessible to the user |
3.9.2. Shared groups
Create shared group TAP capability is required in order to allow user to share tables. The following parameters can be used:
Parameter |
Value |
Comments |
session identifier |
session identifier |
Session identifier provided by a login request |
action |
CreateOrUpdateGroup |
Request to create or update a group |
group_id |
Group identifier |
If not provided: a new group is created and its identifier is returned as part of the response. |
title |
Group title |
Mandatory. |
description |
Group description |
Optional. |
user_list |
List of user identifiers |
Optional. |
To remove a shared group the following parameters must be used:
Parameter |
Value |
Comments |
session identifier |
session identifier |
Session identifier provided by a login request |
action |
RemoveGroup |
Request to remove a group |
group_id |
Group identifier |
Mandatory. |
3.9.3. Shared items
Create shared item TAP capability allows the user to share a table to one or more groups (user groups). The following parameters can be used:
Parameter |
Value |
Comments |
session identifier |
session identifier |
Session identifier provided by a login request |
action |
CreateOrUpdateItem |
Request to create or update a group |
resource_id |
Shared resource identifier |
If not provided: a new shared resource is created and its identifier is returned as part of the response. |
resource_type |
0 |
Mandatory. |
title |
Full qualified table name |
Mandatory. |
description |
Shared table description |
Optional. |
items_list |
List of groups identifiers that share this item |
Mandatory. |
To remove a shared table the following parameters must be used:
Parameter |
Value |
Comments |
session identifier |
session identifier |
Session identifier provided by a login request |
action |
RemoveItem |
Request to remove a shared item |
resource_id |
Shared item identifier |
Mandatory. |
resource_type |
0 |
Mandatory. |
3.9.4. Sharing: other actions
Other Shared TAP+ capabilities are:
Parameter |
Value |
Comments |
session identifier |
session identifier |
Session identifier provided by a login request |
action |
GetGroups |
Returns a list of groups where the user belongs to or created by the user |
GetGroupsDescriptionsOnly |
Returns a list of groups where the user belongs to or created by the user (only descriptions). |
|
GetSharedItems |
Returns a list of items shared by the user. |
|
GetSharedItemsDescriptionsOnly |
Returns a list of items shared by the user (only descriptions). |
|
include_users |
'true' or 'false' |
Valid for 'GetGroups' action only. |
groups_list |
list of groups ids |
Valid for 'GetGroups' action only. |
3.10. TAP+ tables capabilities
In addition to the standar 'tables' TAP capability, the following parameters can be used too:
Parameter |
Value |
Comments |
tables |
comma separated full qualified table names |
A lis of the specified tables will be returned |
schemas |
comma separated schema names |
A list of the specified schemas will be returned |
only_tables |
TRUE / FALSE (default: FALSE) |
TRUE: no columns information will be returned |
only_schemas |
TRUE / FALSE (default: FALSE) |
TRUE: no tables nor columns information will be returned |
These parameters are handled based on the following priorities
Priority |
Parameter |
Comments |
1. |
tables != null |
No more checks are performed (share_info and share_accessible are handled if present) |
2. |
tables == null (default) |
More checks are performed |
2.1. |
only_schemas = TRUE |
No more checks are performed |
2.2. |
only_schemas = FALSE (default) |
The following checks are performed |
2.2.1. |
schema_names != null |
The following parameters are applied to the specified schemas only |
2.2.2 |
only_tables = TRUE |
No columns data are generated |
2.2.3 |
only_tables = FALSE (default) |
Columns data are generated |
3.11. TAP+ events capabilities
Tables creation/removal/sharing actions generate events. Events are grouped by types and each type has a 'last modification time' associated. Those events and times can be retrieved using:
Parameter |
Value |
Comments |
id |
event type identifier |
Optional parameter. If not provided, all events are returned. If provided, only the requested type is returned. |
The current event types are:
Value |
Comments |
100 |
Job created |
101 |
Job updated |
102 |
Job removed |
210 |
Shared item created |
211 |
Shared item updated |
212 |
Shared item removed |
220 |
Shared group created |
221 |
Shared group updated |
222 |
Shared group removed |
230 |
Shared user created |
231 |
Shared user updated |
232 |
Shared user removed |
300 |
Log in |
301 |
Log out |
401 |
Database quota updated |
402 |
File quota updated |
500 |
Notification created |
501 |
Notification removed |
4000 |
Table created |
4001 |
Table updated |
4002 |
Table removed |
3.12. TAP+ notifications
Notifications are kept for one month. Each notification provides a message that explains the notification. Notifications are available for authenticated users only.
Parameter |
Value |
Comments |
session identifier |
session identifier |
Session identifier provided by a login request |
action |
GetNotifications |
Notificaions associated to the user |
3.13. Table flags editing
Parameter |
Value |
Comments |
session identifier |
session identifier |
Session identifier provided by a login request |
ACTION |
radec |
Requests a Ra/Dec update |
TABLE_NAME |
Full qualified table name |
|
RA |
ra column name |
|
DEC |
dec column name |
|
3.14. TAP+ jobs capabilities
Parameter |
Value |
Comments |
session identifier |
session identifier |
Session identifier provided by a login request |
list |
job list identifier |
Mandatory |
offset |
Number of jobs to skip |
Optional. '0' by default. |
limit |
Number of jobs to return |
Optinal. No limit by default. |
order |
List of column order records. |
Optional. |
3.15. TAP+ deletejobs capability
Parameter |
Value |
Comments |
session identifier |
session identifier |
Session identifier provided by a login request |
JOB_IDS |
comma separated job identifiers |
Mandatory |
3.16. TAP+ external TAP access capability
Synchronous mode
Parameter |
Value |
Comments |
Same parameters as defined in 3.2 Synchronous Queries and: |
||
REQUEST |
external_tap |
Request an external TAP access |
EXT_TAP_URL |
URL |
URL of the external TAP service |
Asynchronous mode
Parameter |
Value |
Comments |
Same parameters as defined in 3.3 Asynchronous Queries and: |
||
REQUEST |
external_tap |
Request an external TAP access |
EXT_TAP_URL |
URL |
URL of the external TAP service |
The response header will contain the location of the job.
3.17. Global Tap Schema (GloTS) search capability
Search by table or description
Parameter |
Value |
Comments |
ACTION |
search_table |
|
KEYWORD |
text |
Descriptons or table names |
MODE |
-all (default) |
-Search in both: table names and descriptions |
FORMAT |
-votable (default) |
Results output formt |
Retrieve table columns
Parameter |
Value |
Comments |
ACTION |
get_columns |
|
IVOID |
text |
Service descriptor identifier |
TABLE_NAME |
table name |
|
FORMAT |
-votable (default) |
Results output formt |
Command line access: Datalink
Authors: Héctor Cánovas, Jos de Bruijne, and José Osinde
Las update: 2024-12-10
Introduction
The DataLink server (see also the DataLink service tutorial) provides a service listing all the available resources per source linked at the entry point:
gea.esac.esa.int/data-server/datalink/links?
To invoke the service, the Gaia source ID and its associated catalogue release (either RELEASE=Gaia+DR2 or RELEASE=Gaia+DR3) should be passed as follows:
curl "https://gea.esac.esa.int/data-server/datalink/links?ID=6680733225618222592&RELEASE=Gaia+DR2"
Alternatively to source ID, the Gaia source designation can be used:
curl "https://gea.esac.esa.int/data-server/datalink/links?ID=Gaia+DR3+30343944744320"
It is also possible to connect as a registered user by logging into the Gaia Archive (follow this link to create a Gaia Archive user account) and storing the credentials as follows:
# Login command:
curl -k -c cookies-datalink.txt -X POST -d username=<USERNAME> -d password=<PASSWORD> -L "https://gea.esac.esa.int/data-server/login"
curl -k -b cookies-datalink.txt "https://gea.esac.esa.int/data-server/datalink/links?ID=Gaia+DR3+30343944744320"
This option may be useful for users aiming to retrieve the DataLink products associated to the sources included in their jobs and/or user_tables (see the section explaining how to retrieve DataLink products associated to user tables in the DataLink: Access from the Archive web interface, Sect. 2 in the Command line access: TAP/TAP+ tutorial, and the Upload a User table tutorials to learn more about managing ones user space). It is recommended to wait a few seconds between the execution of the two commands listed above to allow the system to process the request.
The response is always a DataLink compliant XML file with content like:
<?xml version="1.0" encoding="UTF-8"?>
<VOTABLE version="1.4" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.ivoa.net/xml/VOTable/v1.3 http://www.ivoa.net/xml/VOTable/v1.3" xmlns="http://www.ivoa.net/xml/VOTable/v1.3">
<RESOURCE type="results">
<INFO name="standardID" value="ivo://ivoa.net/std/DataLink#links-1.0"/>
<INFO name="QUERY_STATUS" value="OK" />
<INFO name="JOBID" value="1654590222600ODL"><![CDATA[1654590222600ODL]]></INFO>
<INFO name="JOBNAME" value=""><![CDATA[Datalink job for user hcanovas]]></INFO>
<TABLE>
<FIELD name="ID" ucd="meta.id;meta.main" datatype="char" arraysize="*"/>
<FIELD name="access_url" ucd="meta.ref.url" datatype="char" arraysize="*"/>
<FIELD name="service_def" ucd="meta.ref" datatype="char" arraysize="*"/>
<FIELD name="error_message" ucd="meta.code.error" datatype="char" arraysize="*"/>
<FIELD name="semantics" ucd="meta.code" datatype="char" arraysize="*"/>
<FIELD name="description" ucd="meta.note" datatype="char" arraysize="*"/>
<FIELD name="content_type" ucd="meta.code.mime" datatype="char" arraysize="*"/>
<FIELD name="content_length" ucd="phys.size;meta.file" datatype="long" unit="byte"/>
<DATA>
<TABLEDATA>
<TR>
<TD>Gaia DR3 30343944744320</TD>
<TD><![CDATA[https://gea.esac.esa.int/data-server/data?ID=Gaia+DR3+30343944744320&RETRIEVAL_TYPE=MCMC_MSC]]></TD>
<TD></TD>
<TD></TD>
<TD>#this</TD>
<TD><![CDATA[MCMC MSC, source Gaia DR3 30343944744320]]></TD>
<TD>application/x-votable+xml</TD>
<TD></TD>
</TR>
..........
..........
<TR>
<TD>Gaia DR3 30343944744320</TD>
<TD><![CDATA[https://gea.esac.esa.int/data-server/data?ID=Gaia+DR3+30343944744320&RETRIEVAL_TYPE=RVS]]></TD>
<TD></TD>
<TD></TD>
<TD>#this</TD>
<TD><![CDATA[RVS mean spectra, source Gaia DR3 30343944744320]]></TD>
<TD>application/x-votable+xml</TD>
<TD></TD>
</TR>
</TABLEDATA>
</DATA>
</TABLE>
</RESOURCE>
</VOTABLE>
The above example file (that has been trimmed to reduce the verbosity of the tutorial) is intended for IVOA-compliant clients like TOPCAT. It contains the links to the DataLink products available for the requested source (Gaia DR3 30343944744320). The section below shows how to retrieve the different products. For details about them, please see the Datamodel description Chapter in the Gaia DR3 Documentation and the DataLink products serialisation tutorial.
data retrieval
The different DataLink products can be downloaded in a variety of output formats and data structures, as described in the DataLink products serialisation tutorial. The service entry point is:
https://gea.esac.esa.int/data-server/data?RETRIEVAL_TYPE=<retrieval_type>
The table below indicates the accepted values for all available parameters.
Parameter |
Value |
Definition |
Mandatory |
---|---|---|---|
*RETRIEVAL_TYPE |
EPOCH_PHOTOMETRY | XP_SAMPLED | XP_CONTINUOUS | MCMC_GSPPHOT | MCMC_MSC | RVS | ALL |
DataLink product(s) to be retrieved. See the DataLink products serialisation tutorial for details. |
YES |
ID |
source_id(s) | designation(s) |
Unique source identifier(s). The identifier(s) can be passed in different formats (see the input format table below). |
YES |
RELEASE |
Gaia+DR2 | Gaia+DR3 |
Gaia data release associated to a source_id. |
YES if ID is set as source_id. |
**VALID_DATA |
TRUE | FALSE |
The default value is False, which means that the epoch photometry service returns all available data, including data rows where flux is null and/or the rejected_by_photometry flag is set to True. In order to retrieve only valid data (data rows where flux is not null and/or the rejected_by_photometry flag is set to False), this request parameter should be included with VALID_DATA=True |
NO |
FORMAT |
VOTABLE | VOTABLE_PLAIN | FITS | CSV | ECSV |
Output file format. The default format is a (binary) VOTable. |
NO |
DATA_STRUCTURE |
INDIVIDUAL | RAW |
Output file data structure. The default data structure is individual (one source_id per file). |
NO |
Notes:
- (*) Strictly speaking, RETRIEVAL_TYPE defines the service entry point so it is not a parameter. It is included in the table for convenience.
- (**) The optional parameter VALID_DATA only applies to the epoch photometry product.
- When retrieving all data products (RETRIEVAL_TYPE = ALL), the output file is always a .gzip compressed file that contains the products stored in INDIVIDUAL data structure.
- The RELEASE parameter is not strictly mandatory when using a source_id as input ID (there are some combinations that successfully retrieve DataLink products without setting this parameter). However, we recommend to include it in the request for traceability. This parameter is set to Gaia+DR3 by default.
- Since Gaia Archive software release GACS 3.6 (December 2024), the optional parameter 'BAND', which allowed the retrieval of only the photometric band specified by this parameter when requesting epoch photometry products, has been deprecated. This parameter can still be used but no longer has any effect.
The table below summarises the accepted input formats to request individual or multiple ID's:
Format |
Definition |
Example |
---|---|---|
*source_id |
Individual source ID. If this format is selected, then the release parameter MUST be provided. |
1514169246023424 |
designation |
Individual source designation. |
Gaia+DR3+5912901375001820288 |
source_id1,source_id2,... |
Comma-separated source ID list. If this format is selected, then the release parameter MUST be provided. |
1514169246023424,16061459700373248,16061459700373248 |
source_id1-sourceid2 |
Source ID range. The service returns all source_id's fulfilling the condition: source_id <= source_id2 & source_id >= source_id1. |
720000000000000-1800000000000000 |
designation1,designation2,... |
Comma-separated designation list. |
Gaia+DR2+1514169246023424,Gaia+DR2+16061459700373248 |
designation1-designation2 |
Designation range. |
Gaia+DR2+720000000000000-Gaia+DR2+1800000000000000 |
job:job_id.column_name |
Result column from a job with the list of source id's/designations. |
job:1530537474992O.source_id |
user_schema.user_table.column_name |
Column from a defined user table with the list of source id's/designations. |
user_<user_name>.<table_name>.<column_name> |
Notes:
- Remember to include the RELEASE parameter when the input ID is provided as source_id (instead of a source designation).
- The retrieved dataset is a compressed .gzip file if DATA_STRUCTURE = INDIVIDUAL (default) when downloading products for multiple sources.
Examples:
Any combination of the request parameters listed in the table above is allowed. The following examples show how to retrieve selected DataLink products for a single source:
curl -k -b cookies-datalink.txt "https://gea.esac.esa.int/data-server/data?RETRIEVAL_TYPE=EPOCH_PHOTOMETRY&ID=Gaia+DR3+30343944744320" > out_epoch.xml
curl -k -b cookies-datalink.txt "https://gea.esac.esa.int/data-server/data?RETRIEVAL_TYPE=EPOCH_PHOTOMETRY&ID=30343944744320&RELEASE=Gaia+DR3&BAND=BP" > out_epoch_bp.xml
curl -k -b cookies-datalink.txt "https://gea.esac.esa.int/data-server/data?RETRIEVAL_TYPE=EPOCH_PHOTOMETRY&ID=30343944744320&RELEASE=Gaia+DR3&BAND=G&VALID_DATA=TRUE&FORMAT=ECSV" > out_epoch_bp.ecsv
curl -k -b cookies-datalink.txt "https://gea.esac.esa.int/data-server/data?RETRIEVAL_TYPE=RVS&ID=Gaia+DR3+30343944744320&format=VOTABLE_PLAIN"> out_rvs.xml
curl -k -b cookies-datalink.txt "https://gea.esac.esa.int/data-server/data?RETRIEVAL_TYPE=ALL&ID=Gaia+DR3+30343944744320&format=FITS" > datalink_all.gzip
Notes:
- The use of the ">" bash command writes the output dataset to a file instead of the standard output (terminal / screen).
- Requests that download one single file are stored uncompressed, while requests retrieving multiple files are stored as compressed .gzip files.
The examples below show how to download DataLink products for multiple sources (for the sake of simplicity, combinations of DATA_STRUCTURE and FORMAT have been avoided, but they can be included as well):
curl -k -b cookies-datalink.txt "https://gea.esac.esa.int/data-server/data?RETRIEVAL_TYPE=XP_SAMPLED&ID=5912901375001820288,5912586227479692800,5912437793408452608&RELEASE=Gaia+DR3" > xp_sampled_multi.gzip
curl -k -b cookies-datalink.txt "https://gea.esac.esa.int/data-server/data?RETRIEVAL_TYPE=MCMC_GSPPHOT&ID=Gaia+DR3+5912901375001820288,Gaia+DR3+5912586227479692800,Gaia+DR3+5912437793408452608" > mcmc_gspphot_multi.gzip
curl -k -b cookies-datalink.txt "https://gea.esac.esa.int/data-server/data?RETRIEVAL_TYPE=EPOCH_PHOTOMETRY&ID=10655814178816-30343944744320&RELEASE=Gaia+DR3" > epoch_multi.gzip
curl -k -b cookies-datalink.txt "https://gea.esac.esa.int/data-server/data?RETRIEVAL_TYPE=EPOCH_PHOTOMETRY&ID=job:<job_id>.source_id"
curl -k -b cookies-datalink.txt "https://gea.esac.esa.int/data-server/data?RETRIEVAL_TYPE=XP_CONTINUOUS&ID=<user_schema>.<user_table>.source_id&RELEASE=Gaia+DR3"
Complex combinations for multiple requests are also possible:
curl -k -b cookies-datalink.txt "https://gea.esac.esa.int/data-server/data?RETRIEVAL_TYPE=EPOCH_PHOTOMETRY&ID=10655814178816-30343944744320,46316927345664,54528905041920-56732223835904&RELEASE=Gaia+DR3" > epoch_multi_2.gzip