OM data reduction with SAS: processing image data, step by step

Procedure
 

The PPS data package contains OM data which normally do not necessitate further processing for the purpose of calibration. However, a user may want to apply a more recent calibration, or change some default parameters to e.g. improve the source detection. It may not be necessary to run the complete chains, but just some tasks. All this can be done interactively.

The SAS_ODF environment variable shall be set to the summary file produced by odfingest, or to a directory containing the data, and access to calibration files shall be set through cifbuild (see SAS Startup Thread).

In addition to the fits files containing the OM science image data, the ODF needs to contain the OM house-keeping files:

  ${REV}_${OBSID}_OMX00000NPH.FIT - non periodic house-keeping
  ${REV}_${OBSID}_OMX00000PEH.FIT - periodic house-keeping

and some spacecraft files of the following form:

  ${REV}_${OBSID}_SCX00000SUM.SAS - ASCII observation summary file
  ${REV}_${OBSID}_SCX00000TCS.FIT - Spacecraft Time correlation file
  ${REV}_${OBSID}_SCX00000ATS.FIT - Spacecraft Attitude file

where ${REV}$ is the revolution in which the observation was taken and ${OBSID} is the ObsID. These are part of the naming convention of ODF files. We refer to the Data Files Handbook for definition of ODF file names.

Invoking omichain or omfchain will automatically start the processing of all OM data in the working directory. The duration of the process will depend on the number of exposures and windows and at the end we shall obtain the final processed files.

In the standard automatic SSC pipeline processing, the temporary image files are re-used and thus overwritten. In the task by task processing, they can be distinguished so that intermediate stage output can be looked at if desired. In the examples below we have used the naming system as used by omichain. This allows us to maintain some intermediate files which can be helpful in understanding the whole process.

Currently the chains can accept parameters to limit the processing to a given filter or a given exposure. Some of the default parameters used by individual tasks can also be tuned. (See online SAS help pages for detailed instructions for each task parameters)
 

Example of image data processing
 

A task by task procedure is presented below for processing a single exposure in image mode as

  ${REV}_${OBSID}_OMS${ENUM}00IMI.FIT,

where ${REV}$ is the revolution in which the observation was taken, ${OBSID} the ObsID, and ${ENUM}$ the exposure number to be processed. They can be defined via environment variable as shown in step0 below. The following files are necessary (they are normally part of the ODF):

  ${REV}_${OBSID}_OMS${ENUM}00WDX.FIT - Exposure priority window file
  ${REV}_${OBSID}_OMS${ENUM}00THX.FIT - Exposure tracking history file
  ${REV}_${OBSID}_OMS${ENUM}00IMI.FIT - Exposure image file

Output products of the different tasks have been given the names that omichain would have applied.

It is assumed throughout this example that the ODF data are in: ${MYDATA}/
and output products are written into a subdirectory: ${MYDATA}/image/

The steps below process a single image mode exposure and needs to be repeated for each exposure separately to process all exposures within the observation.

step 0
 

Set environment variables in order to copy-paste the commands below. Here exposure number 012 in ObsID 0650300201, taken in revolution 1868 is used as an example.

Revolution in which observation was taken:
 

   setenv REV 1874

ObsID of observation:
 

   setenv OBSID 0561580301

Exposure number to be processed:
 

   setenv ENUM 012

Directory where uncalibrated raw (=ODF) data are located
 

   setenv MYDATA .....

Then produce output directory:
 

   mkdir ${MYDATA}/image/

step 1

The OM tracking data are processed for later use, if applicable.

   omprep set=${MYDATA}/${REV}_${OBSID}_OMS${ENUM}00THX.FIT \
    nphset=${MYDATA}/${REV}_${OBSID}_OMX00000NPH.FIT \
    pehset=${MYDATA}/${REV}_${OBSID}_OMX00000PEH.FIT \
    wdxset=${MYDATA}/${REV}_${OBSID}_OMS${ENUM}00WDX.FIT \
    outset=${MYDATA}/image/I${OBSID}OMS${ENUM}TRHIS0000.FIT \
    modeset=3

In case there is no THX file, then set=DUMMYTHX.FIT. omprep will generate a dummy file needed for the rest of the chain, with zero drift in it.

step 2

A plot of the tracking data, useful to assess the S/C stability during the exposure, is generated. Note that tracking data may not be good enough, in which case no plot is produced.

   omdrifthist set=${MYDATA}/image/I${OBSID}OMS${ENUM}TRHIS0000.FIT \
    plotfile=${MYDATA}/image/P${OBSID}OMS${ENUM}TSHPLT000.ps

step 3

The count rates of tracking stars are written into a FITS file.

   omthconv thxset=${MYDATA}/image/I${OBSID}OMS${ENUM}TRHIS0000.FIT \
    nphset=${MYDATA}/${REV}_${OBSID}_OMX00000NPH.FIT \
    outset=${MYDATA}/image/P${OBSID}OMS${ENUM}TSTRTS000.FIT

step 4

Processing of the image data starts here. Some parameters are extracted for insertion into the FITS header.

   omprep set=${MYDATA}/${REV}_${OBSID}_OMS${ENUM}00IMI.FIT \
    nphset=${MYDATA}/${REV}_${OBSID}_OMX00000NPH.FIT \
    pehset=${MYDATA}/${REV}_${OBSID}_OMX00000PEH.FIT \
    wdxset=${MYDATA}/${REV}_${OBSID}_OMS${ENUM}00WDX.FIT \
    outset=${MYDATA}/image/I${OBSID}OMS${ENUM}IMAGE10000.FIT
 

step 5

Bad pixels in the image are flagged.

   omcosflag set=${MYDATA}/image/I${OBSID}OMS${ENUM}IMAGE10000.FIT \
    thxset=${MYDATA}/image/I${OBSID}OMS${ENUM}TRHIS0000.FIT

Although no real flat field correction exists for OM, nor is it necessary, the processing requires such a file which can be generated using the task omflatgen as follows. (This task can also be run at the beginning of the processing)
 

step 6

A flat field is generated.

   omflatgen outset=${MYDATA}/image/P${OBSID}OM000FLAFLD0000.FIT

The output flatfield (primary extension) will be set to unity.

step 6 A

The flat field is applied to the data.

   omflatfield set=${MYDATA}/image/I${OBSID}OMS${ENUM}IMAGE10000.FIT \
    thxset=${MYDATA}/image/I${OBSID}OMS${ENUM}TRHIS0000.FIT \
    inorbitflatset=${MYDATA}/image/P${OBSID}OM000FLAFLD0000.FIT \
    tsflatset=${MYDATA}/image/I${OBSID}OMS${ENUM}FLATF0000.FIT \
    outset=${MYDATA}/image/I${OBSID}OMS${ENUM}IMAGE20000.FIT

step 7

A very important correction is applied here: modulo-8 fixed pattern is removed.

   ommodmap set=${MYDATA}/image/I${OBSID}OMS${ENUM}IMAGE20000.FIT \
    mod8product=yes \
    mod8set=${MYDATA}/image/I${OBSID}OMS${ENUM}MOD80000.FIT \
    outset=${MYDATA}/image/I${OBSID}OMS${ENUM}IMAGE30000.FIT \
    nsig=3 \
    nbox=16
 

step 8

Image quality information is stored for later use on mosaiced images for additional source detection.

   omqualitymap srclistset=' ' \
    set=${MYDATA}/image/I${OBSID}OMS${ENUM}IMAGE30000.FIT \
    outset=${MYDATA}/image/I${OBSID}OMS${ENUM}QIMAG_0000.FIT \
    mode=setqualityimage

step 9

A searching algorithm is used to find the sources present in the image. Aperture photometry is performed on all sources to obtain corrected count rates.

   omdetect nsigma=2 minsignificance=3 detectextended=1 \
    set=${MYDATA}/image/I${OBSID}OMS${ENUM}QIMAG_0000.FIT \
    regionfile=${MYDATA}/image/I${OBSID}OMS${ENUM}REGION0000.reg \
    levelimage=${MYDATA}/image/I${OBSID}OMS${ENUM}LEVELIMAGE0000.FIT \
    backgroundimage=${MYDATA}/image/P${OBSID}OMS${ENUM}BCKIMAGE_0000.FIT \
    outset=${MYDATA}/image/P${OBSID}OMS${ENUM}SWSRLI0000.FIT

Note that all functions from this step on can be accomplished with the interactive omsource task.

step 10

Image quality information is stored for later use on mosaiced images for additional source detection.

   omqualitymap srclistset=${MYDATA}/image/P${OBSID}OMS${ENUM}SWSRLI0000.FIT \
    set=${MYDATA}/image/I${OBSID}OMS${ENUM}QIMAG_0000.FIT \
    outset=${MYDATA}/image/P${OBSID}OMS${ENUM}IMAGE_0000.FIT \
    mode=setqualityimage
 

step 11

The count rates of all sources found with omdetect are converted to magnitudes in the OM instrumental system.

   ommag set=${MYDATA}/image/P${OBSID}OMS${ENUM}SWSRLI0000.FIT
 

step 12

Astrometry corrections are performed here to derive the coordinates of all detected sources. A sky image (North on top) is produced. This is just cosmetical, no photometric measurements should be made on this image (SIMAGE file)

   omatt set=${MYDATA}/image/P${OBSID}OMS${ENUM}IMAGE_0000.FIT \
    sourcelistset=${MYDATA}/image/P${OBSID}OMS${ENUM}SWSRLI0000.FIT \
    ppsoswset=${MYDATA}/image/P${OBSID}OMS${ENUM}SIMAGE0000.FIT \
    usecat=yes rotateimage=yes \
    catfile=I${OBSID}OM_USNO.FIT

Note that if a subset of USNO catalogue, named I${OBSID}OM_USNO.FIT  is provided, then source cross correlation will be performed and the computed offsets will be added to the derived coordinates of the sources. If the catalogue file is not present, then omichain will attempt to run scat to produce the catalogue file. This requires that this utility is installed into the system, otherwise source cross correlation will not be performed. Alternatively, e.g., if the scat tool is not available or fails to run, the PPSfile of the form,

   P${OBSID}OBX000REFCAT0000.FTZ

can be used instead.

The detected sources can be overlaid on the OSW as follows:
 

step 12 a

The success of the detection can be checked.

   implot set=${MYDATA}/image/P${OBSID}OMS${ENUM}SIMAGE0000.FIT \
    withsrclisttab=yes \
    srclisttab=${MYDATA}/image/P${OBSID}OMS${ENUM}SWSRLI0000.FIT \
    device='/XW'

A similar, perhaps easier result can be obtained by displaying the image with ds9 and loading the corresponding region produced by omdetect:

   ds9 ${MYDATA}/image/P${OBSID}OMS${ENUM}IMAGE_0000.FIT \
     -region ${MYDATA}/image/I${OBSID}OMS${ENUM}REGION0000.reg \
     -zoom to fit -scale log

using the image produced in step 10 and the region file produced in step 8.

step 13

After all images for a given filter have been processed, they can be combined to form a mosaic. If the images are not suitable then the task is skipped.

   ommosaic imagesets="${MYDATA}/image/list of sky images" \
     mosaicedset=${MYDATA}/image/myfield_in_myfilter.fit

For example, if exposures 012 and 017 have been processed, the command is:

   ommosaic imagesets="${MYDATA}/image/P${OBSID}OMS012SIMAGE0000.FIT \
     ${MYDATA}/image/P${OBSID}OMS017SIMAGE0000.FIT" \
     mosaicedset=${MYDATA}/image/myfield_in_myfilter.fit

and the combined image can be viewed with,

    ds9 ${MYDATA}/image/myfield_in_myfilter.fit -zoom to fit -scale log

step 14

After processing all exposures (with one or several filters), we can combine their source lists. Color corrections will be applied to derive standard UBV magnitudes.

For example, if exposures 012 and 017 have been processed, the command is,

    omsrclistcomb sourcelistsets="${MYDATA}/image/P${OBSID}OMS012SWSRLI0000.FIT\
     ${MYDATA}/image/P${OBSID}OMS017SWSRLI0000.FIT" \
      nsigma=3 \
      outset=${MYDATA}/image/P${OBSID}OMCOMBOBSMLI0000.FIT

Note the list, surrounded by quotes, of SWSRLI files.

Here, again as in step 12, we can use a catalogue files to cross correlate the coordinates of the sources in the combined list. This is important because OM images obtained with the UV filters may have too few sources for a successful match in step 12. The coordinates of these UV detected sources can be corrected now in step 14.

step 15

The same checks of resulting products need to be performed as in the omichain thread.

Further processing

Starting with SAS 9, an additional detection can be done on mosaiced images. This will allow us to find fainter sources. The optional parameter processmosaicedimages(=yes) in omichain has to be used.

If the observation contains multiple exposures with the same filter then the detected sources can be checked for variability using, in SAS 10, the task omvariability. Please refer to the on-line documentation for more details.