Implementation Guides - VOVisObs_protocols
Implementation Guides
A) Object Visibility Service
A basic web service (REST) complaint with ObjVisSAP can be downloaded as python source code from github under:
https://github.com/emiliosalazardonate/visibility-service/blob/master/README.md
It listening for requests in port 8000. It needs python 3.6 and uses two libraries django and astropy:
Django==3.0.3 astropy==3.0.4
To run the server:
/path/to/python3/bin/python manage.py runserver
Starting development server at http://127.0.0.1:8000/
B) Observation Locator Service
This page summarises how to implement an ObsLocTAP service. The full description of this protocol can be found under the IVOA (International Virtual Observatory Alliance) domain at:
http://www.ivoa.net/documents/ObsLocTAP/index.html
The Observation Locator Table Access Protocol (ObsLocTAP henceforth) specifies in a standard format the services to retrieve information about planned, scheduled and performed observations. By sharing this information, the scientific community can monitor the planning activities of the observatories, prepare coordinated observations or to allow the preparation of better proposals from the scientific point of view.
Following the approach of the newer IVOA protocols, ObsLocTAP is a Tabular Access Protocol (TAP) service. This service exposes a database table using a query language close to SQL but with astronomical operators that allow better exploration of astronomical resources. This language called Astronomical Data Query Language (ADQL)
http://www.ivoa.net/documents/REC/ADQL/ADQL-20081030.pdf
is a standard for the virtual observatory.
The implementation of a full compatible TAP service from scratch is cumbersome. However, there are a set of toolkits that allow the creation of this services without major knowledge of the underlying standards and very fast. We will describe how to implement a TAP service using a very well toolkit called taptuto but there are others. Implementors could take a look into the different alternatives and give them a try.
Other very well known toolkits are:
- DACHS from GAVO: http://soft.g-vo.org/dachs https://dachs-doc.readthedocs.io/tutorial.html
- SAADA: http://saada.unistra.fr/saada/
- TAPTuto: http://cdsportal.u-strasbg.fr/taptuto/
Points 3 and 5 are applicable for all the toolkits to create a table ObsLocTAP compliant.
1. ObsLocTAP components
Three basic components are needed to implement an ObsLocTAP service
- Database installation: In order to facilitate the geometrical queries needed for a TAP service, we recommend the use of a PostgreSQL DB instance
- Creation of ObsLocTAP complaint table (database)
- TAP service instance: Several frameworks can be used to create a TAP service. We will use, as example, the one created by Gregory Mantelet, currently working at CDS, due to its level of documentation. This package is open source
- Publication of tables through TAP server (TAP_SCHEMA)
2. Database installation
There are several pages where we can find how to install a PosgreSQL server. In most of the cases, just following the instructions of https://www.postgresql.org/download/ should be enough to have a PosgreSQL server up and running. In fact, many of the different server images have already a PosgreSQL instance that can be used. Although PosgreSQL v10 contains some interesting features, v9.6 can guarantee at this stage operational performance and compatibility with extensions. There are multiple tutorials to install PosgreSQL depending on the operating system on the web.
It is recommendable to implement periodic dumps and backups in order to restore an operational database in case of failure (see https://www.postgresql.org/docs/9.6/backup.html). Also, it is recommendable to use a special account (e.g. postgres) for the database instance.
Once you have your PosgreSQL instance up and running, there is one spatial extension that would be needed for the TAP service instance. There are other spatial indexing extensions but the TAP service instance used in point 2 is making use of pgsphere.
Installation of the extension following the instructions at http://pgsphere.github.io/download.html is more or less transparent.
3. Creation of ObsLocTAP complaint table (database)
Create a database and a table with the following structure:
CREATE SCHEMA ivoa;
CREATE TABLE ivoa.obsplan (
t_planning double precision NOT NULL,
target_name character varying,
obs_id character varying NOT NULL,
obs_collection character varying NOT NULL,
s_ra double precision,
s_dec double precision,
s_fov double precision,
s_region scircle,
s_resolution double precision,
t_min double precision NOT NULL,
t_max double precision NOT NULL,
t_exptime double precision,
t_resolution double precision,
em_min double precision,
em_max double precision,
em_res_power double precision,
o_ucd character varying NOT NULL,
pol_states character varying,
pol_xel integer,
facility_name character varying NOT NULL,
instrument_name character varying NOT NULL,
obs_release_date timestamp,
t_plan_exptime double precision NOT NULL,
category character varying,
priority integer,
execution_status character varying NOT NULL
);
CREATE INDEX i_obsplan_fov
ON ivoa.obsplan
USING gist
(s_region);
Please notice that the field of view (s_region column) is defined as a pgsphere object. Other types different than scircle could be used (see https://pgsphere.github.io/doc/types.html). For example, if the planned observation can be very well defined (including, e.g. the position angle), a typical type used for footprints would be polygon. Both, circle and polygon, can be used in the same way for ADQL searches. Unfortunately, two different types of pgsphere objects cannot be combined in the same database column (that means, you cannot have observations defined as circles and other defined as polygons into the same s_region column of the same table)
The last create index sentence defines how to create a gist index on the pgsphere object to increase the performance.
In this case, we have defined s_region as a scircle pgsphere object. Typical value would be a spoly. However, there is not an easy way to mix pgsphere circles and polygons in the same column as they do not have a generic data type. One possible solution is to store everything as spoly and convert the circles in approximated polygons.
The output of a select on s_region is typically a string representation of the Field of View. The conversion from a pgsphere output to a STC-S output when this column is used on a SELECT condition is typically done by the TAP server.
For example, polygon is defined as:
Polygon J2000 158.408362 57.781849 158.501991 57.812066 158.558573 57.762136 158.465026 57.73196
where the points of the polygon is defined after J2000 in pairs of ra dec values.
Whenever there is some uncertainty on the final pointing, a good option is to define the stc_s representation just as a circle, with the central position and the radius defined as:
Circle J200 <s_ra> <s_dec> <s_fov>
where the values of s_ra, s_dec and s_fov are the other columns in the obsplan table. Whenever the Field of View is better defined, an upgrade from the circle to something more detailed could be done.
See more info at http://ivoa.net/documents/STC-S/
It would be a good idea to populate this table with some dummy rows for testing during next steps.
If you are familiar with docker, we have created a docker instance with posgreSQL, pgsphere and the ivoa.obsplan table with some dummy fields for testing.
https://hub.docker.com/r/jsalgadodocker/postgres9.5-pgsphere-obsplan
4. TAP Service instance
There are some frameworks to be used to create a TAP instance. We recommend to use the one created by Gregory Mantelet at CDS as it is quite well documented and it has been updated to the latests ADQL (Astronomical Data Query Language) changes.
Installation instructions can be found here: http://cdsportal.u-strasbg.fr/taptuto/
Some notes:
- ObsLocTAP database could/should be named ivoa instead of the one mentioned into the example MyStarCatalogues to clarify the schema
- Table to be published will be called obsplan
The instructions provided by these pages are long but very descriptive so, in principle, you should be able to have a service running at the end of this process.
5. Publication of tables through TAP server
TAP_SCHEMA should contain the description of the published table(s) through the TAP service. Using the SQL script provided in the previous framework (or by other method), the characterization of the table in the TAP_SCHEMA should contain the following information:
| ivoa.obsplan | t_planning | adql:DOUBLE | d |
| ivoa.obsplan | target_name | adql:VARCHAR | |
| ivoa.obsplan | obs_id | adql:VARCHAR | |
| ivoa.obsplan | obs_collection | adql:VARCHAR | |
| ivoa.obsplan | s_ra | adql:DOUBLE | deg |
| ivoa.obsplan | s_dec | adql:DOUBLE | deg |
| ivoa.obsplan | s_fov | adql:DOUBLE | deg |
| ivoa.obsplan | s_resolution | adql:DOUBLE | arcsec |
| ivoa.obsplan | t_min | adql:DOUBLE | d |
| ivoa.obsplan | t_max | adql:DOUBLE | d |
| ivoa.obsplan | t_exptime | adql:DOUBLE | s |
| ivoa.obsplan | t_resolution | adql:DOUBLE | s |
| ivoa.obsplan | em_min | adql:DOUBLE | m |
| ivoa.obsplan | em_max | adql:DOUBLE | m |
| ivoa.obsplan | em_res_power | adql:DOUBLE | |
| ivoa.obsplan | o_ucd | adql:VARCHAR | |
| ivoa.obsplan | pol_states | adql:VARCHAR | |
| ivoa.obsplan | pol_xel | adql:BIGINT | |
| ivoa.obsplan | facility_name | adql:VARCHAR | |
| ivoa.obsplan | instrument_name | adql:VARCHAR | |
| ivoa.obsplan | obs_release_date | adql:TIMESTAMP | date |
| ivoa.obsplan | t_plan_exptime | adql:DOUBLE | s |
| ivoa.obsplan | category | adql:VARCHAR | |
| ivoa.obsplan | priority | adql:INTEGER | |
| ivoa.obsplan | fov | adql:VARCHAR | |
| ivoa.obsplan | stc_s | adql:VARCHAR |
The metadata that should be added to the basic/plain TAP_SCHEMA created by TAPTUTO to the TAP_SCHEMA would be the following:
INSERT INTO "TAP_SCHEMA".tables (schema_name,table_name,table_type,description,utype,dbname) VALUES ( 'TAP_SCHEMA','TAP_SCHEMA.schemas','table','List of schemas published in this TAP service.',NULL,NULL); INSERT INTO "TAP_SCHEMA".tables (schema_name,table_name,table_type,description,utype,dbname) VALUES ( 'TAP_SCHEMA','TAP_SCHEMA.tables','table','List of tables published in this TAP service.',NULL,NULL); INSERT INTO "TAP_SCHEMA".tables (schema_name,table_name,table_type,description,utype,dbname) VALUES ( 'TAP_SCHEMA','TAP_SCHEMA.coosys','table','List of coordinate systems of coordinate columns published in this TAP service.',NULL,NULL); INSERT INTO "TAP_SCHEMA".tables (schema_name,table_name,table_type,description,utype,dbname) VALUES ( 'TAP_SCHEMA','TAP_SCHEMA.columns','table','List of columns of all tables listed in TAP_SCHEMA.TABLES and published in this TAP service.',NULL,NULL); INSERT INTO "TAP_SCHEMA".tables (schema_name,table_name,table_type,description,utype,dbname) VALUES ( 'TAP_SCHEMA','TAP_SCHEMA.keys','table','List all foreign keys but provides just the tables linked by the foreign key. To know which columns of these tables are linked, see in TAP_SCHEMA.key_columns using the key_id.',NULL,NULL); INSERT INTO "TAP_SCHEMA".tables (schema_name,table_name,table_type,description,utype,dbname) VALUES ( 'TAP_SCHEMA','TAP_SCHEMA.key_columns','table','List all foreign keys but provides just the columns linked by the foreign key. To know the table of these columns, see in TAP_SCHEMA.keys using the key_id.',NULL,NULL); INSERT INTO "TAP_SCHEMA".tables (schema_name,table_name,table_type,description,utype,dbname) VALUES ( 'ivoa','obsplan','table','ObsLocTAP compatible table','',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','t_planning','','d','','','adql:DOUBLE',0,0,0,0,'t_planning',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','target_name','','','','','adql:VARCHAR',32,0,0,0,'target_name',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','obs_id','','','','','adql:VARCHAR',32,0,0,0,'obs_id',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','obs_collection','','','','','adql:VARCHAR',32,0,0,0,'obs_collection',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','s_ra','','deg','','','adql:DOUBLE',0,0,0,0,'s_ra',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','s_dec','','deg','','','adql:DOUBLE',0,0,0,0,'s_dec',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','s_fov','','deg','','','adql:DOUBLE',0,0,0,0,'s_fov',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','s_resolution','','arcsec','','','adql:DOUBLE',0,0,0,0,'s_resolution',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','t_min','','d','','','adql:DOUBLE',0,0,0,0,'t_min',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','t_max','','d','','','adql:DOUBLE',0,0,0,0,'t_max',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','t_exptime','','s','','','adql:DOUBLE',0,0,0,0,'t_exptime',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','t_resolution','','s','','','adql:DOUBLE',0,0,0,0,'t_resolution',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','em_min','','m','','','adql:DOUBLE',0,0,0,0,'em_min',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','em_max','','m','','','adql:DOUBLE',0,0,0,0,'em_max',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','em_res_power','','','','','adql:DOUBLE',0,0,0,0,'em_res_power',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','o_ucd','','','','','adql:VARCHAR',32,0,0,0,'o_ucd',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','pol_states','','','','','adql:VARCHAR',32,0,0,0,'pol_states',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','pol_xel','','','','','adql:BIGINT',0,0,0,0,'pol_xel',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','facility_name','','','','','adql::VARCHAR',32,0,0,0,'facility_name',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','instrument_name','','','','','adql::VARCHAR',32,0,0,0,'instrument_name',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','obs_release_date','date','','','','adql::TIMESTAMP',0,0,0,0,'obs_release_date',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','t_plan_exptime','s','','','','adql::DOUBLE',0,0,0,0,'t_plan_exptime',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','category','','','','','adql::VARCHAR',32,0,0,0,'category',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','priority','','','','','adql::INTEGER',0,0,0,0,'priority',''); INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES ( 'ivoa.obsplan','s_region','','','','','adql::VARCHAR',32,0,0,0,'s_region','');
6. Testing the system
Once the service is up and running, some queries can be used to test the system. Some examples are included in the specification but, as an example, we present some queries used for the ObsLocTAP Integral Observatory service. The service URL for this service is:
https://iladev.esac.esa.int/tap-dev/tap/sync?
For other services, this service URL will be different although the rest of parameters (REQUEST, PHASE, LANG, QUERY) are the typical TAP service one so these will be common for the rest of services.
6.1. Search of observations on a certain time range
ADQL Query:
SELECT o.s_ra, o.s_dec, o.target_name, o.t_min, o.t_max FROM ivoa.obsplan AS o WHERE o.t_max > 56783 ORDER BY o.t_min
Service invocation:
6.2. Search of observations on a certain astronomical area
ADQL Query:
SELECT * FROM ivoa.obsplan WHERE
1=INTERSECTS(s_region, CIRCLE('ICRS', 114.8251, 1.6179, 0.016666))
Service invocation:
6.3. Combination of time and geometrical conditions
ADQL Query:
SELECT * FROM ivoa.obsplan WHERE
t_max < 58502 AND
1=INTERSECTS(s_region, CIRCLE('ICRS', 114.8251, 1.6179, 0.016666))
Service invocation:
7. Playing with dockers
Although it is not recommended to use docker containers for an operational archive, we have defined two dockers containers for the DB and the TAP server that you can use to have a ObsLocTAP server up and running in minutes in you have the docker command line client installed.
You just need to:
docker pull jsalgadodocker/pgsphere-obsplan:latest docker pull jsalgadodocker/tapserver:latest docker network create --driver=bridge db-network docker image ls docker run -p 8080:8080 --net=db-network --name tap jsalgadodocker/tapserver:latest docker run -p 5432:5432 --net=db-network --name db jsalgadodocker/pgsphere-obsplan:latest
You will have a TAP server with an obsplan table with some Integral records as test data running on your localhost. It could be tested by
http://localhost:8080/tap/tap/
Also, you can connect to port 5432 of your localhost with a dabase client (e.g. pgadmin3) to see the records or to ingest your own ones.
In order to make public your service, you can e.g. create a proxy pass redirect from an apache web server, use port 80 and open it to internet, etc.
(please send comments on this page to Jesus.Salgado@sciops.esa.int)