OM data reduction with SAS: step by step fast mode data processing

Procedure
 

The PPS data package contains OM data which normally do not necessitate further processing for the purpose of calibration. However, a user may want to apply more recent calibrations, or change some default parameters to e.g. improve the source detection. It may not be necessary to run the complete chains, but just some tasks. All this can be done interactively.

The SAS_ODF environment variable shall be set to the summary file produced by odfingest, or to a directory containing the data, and access to calibration files shall be set through cifbuild (see SAS Startup Thread).

In addition to the fits files containing the science data, the ODF needs to contain the OM house-keeping files:

   ${REV}_${OBSID}_OMX00000NPH.FIT - non periodic house-keeping
   ${REV}_${OBSID}_OMX00000PEH.FIT - periodic house-keeping

and spacecraft files of the following form:

   ${REV}_${OBSID}_SCX00000SUM.SAS - ASCII observation summary file
   ${REV}_${OBSID}_SCX00000TCS.FIT - Spacecraft Time correlation file
   ${REV}_${OBSID}_SCX00000ATS.FIT - Spacecraft Attitude file

where ${REV}$ is the revolution in which the observation was taken and ${OBSID} is the ObsID. These are part of the naming convention of ODF files. We refer to the Data Files Handbook for definition of ODF file names.

Invoking omichain or omfchain will automatically start the processing of all OM data in the working directory. The duration of the process will depend on the number of exposures and windows and at the end we shall obtain the final processed files.

In the standard automatic SSC pipeline processing, the temporary image files are re-used and thus overwritten. In the task by task processing, they can be distinguished so that intermediate stage output can be looked at if desired. In the examples below we have used the naming system as used by omfchain. This allows us to maintain some intermediate files which can be helpful in understanding the whole process.

Some of the default parameters used by individual tasks can be tuned if necessary.
 

Example of fast mode data processing
 

A task by task procedure is presented below for processing a single exposure in image mode as,

   ${REV}_${OBSID}_OMS${ENUM}01FAE.FIT

where ${REV}$ is the revolution in which the observation was taken, ${OBSID} the ObsID, and ${ENUM}$ the exposure number to be processed. They can be defined via environment variable as shown in Step 0 below.

The following files are necessary (they are normally part of the ODF):

   ${REV}_${OBSID}_OMS${ENUM}00WDX.FIT - Exposure priority window file
   ${REV}_${OBSID}_OMS${ENUM}00THX.FIT - Exposure tracking history file
   ${REV}_${OBSID}_OMS${ENUM}01FAE.FIT - Exposure fast mode data file (fast mode window)

Output products of the different tasks have been given the names that omfchain would have applied.

It is assumed throughout this example that the ODF data are in: ${MYDATA}/
and output products are written into a subdirectory: ${MYDATA}/fast/

The steps below process a single fast mode exposure and needs to be repeated for each exposure separately to process all exposures within the observation.

step 0

Set environment variables in order to copy-paste the commands below. Here exposure number 012 in ObsID 0650300201, taken in revolution 1868 is used as an example.

Revolution in which observation was taken:

  setenv REV 1874

ObsID of observation:

  setenv OBSID 0561580301

Exposure number to be processed:

  setenv ENUM 012

Directory where uncalibrated raw (=ODF) data are located

  setenv MYDATA .....

Then produce output directory:

  mkdir ${MYDATA}/fast/

step 1
 

The OM tracking data are processed for later use, if applicable.

  omprep set=${MYDATA}/${REV}_${OBSID}_OMS${ENUM}00THX.FIT \
   pehset=${MYDATA}/${REV}_${OBSID}_OMX00000PEH.FIT \
   nphset=${MYDATA}/${REV}_${OBSID}_OMX00000NPH.FIT \
   wdxset=${MYDATA}/${REV}_${OBSID}_OMS${ENUM}00WDX.FIT \
   outset=${MYDATA}/fast/tmp_tracking \
   modeset=3

As for image data, if there is no THX file, then set=DUMMYTHX.FIT. omprep will generate a dummy file needed for the rest of the chain, with zero drift in it.

step 2

A plot of the tracking data, useful to assess the S/C stability during the exposure, is generated. Note that tracking data may not be good enough, in which case no plot is produced.

  omdrifthist set=${MYDATA}/fast/tmp_tracking \
   plotfile=${MYDATA}/fast/P${OBSID}OMS${ENUM}TSHPLT0000.PS \
   trackradius=0.5 \
   hardcopy=yes

step 3

The count rates of tracking stars are written into a FITS file.

  omthconv thxset=${MYDATA}/fast/tmp_tracking \
   nphset=${MYDATA}/${REV}_${OBSID}_OMX00000NPH.FIT \
   outset=${MYDATA}/fast/P${OBSID}OMX${ENUM}TSTRTS0000.FIT \
   modeset=1

step 4

Processing of the fast mode data starts here.

  omprep set=${MYDATA}/${REV}_${OBSID}_OMS${ENUM}01FAE.FIT \
   pehset=${MYDATA}/${REV}_${OBSID}_OMX00000PEH.FIT \
   nphset=${MYDATA}/${REV}_${OBSID}_OMX00000NPH.FIT \
   wdxset=${MYDATA}/${REV}_${OBSID}_OMS${ENUM}00WDX.FIT \
   outset=${MYDATA}/fast/F${OBSID}OMS${ENUM}EVLIST2000.FIT \
   modeset=1

In its second run, omprep is invoked for fast data (modeset=1) and the FAE raw event data is transformed into a modified event list to be used as input for evselect

step 5

Starting the processing of the detected events.

  evselect table=${MYDATA}/fast/F${OBSID}OMS${ENUM}EVLIST2000.FIT \
   withimageset=yes \
   imageset=${MYDATA}/fast/F644_${OBSID}_OMS${ENUM}01FIMI.FIT \
   xcolumn=RAWX \
   ycolumn=RAWY

The pseudo-image corresponding to the fast mode OSW has been created by this first run of evselect.

step 6

Correcting for drift.

  omfastshift nphset=${MYDATA}/${REV}_${OBSID}_OMX00000NPH.FIT \
   set=${MYDATA}/fast/F${OBSID}OMS${ENUM}EVLIST2000.FIT \
   thxset=${MYDATA}/fast/tmp_tracking \
   interpolate=yes

The X- and Y- coordinates of the photon events are corrected for spacecraft drift. New columns are added to the event list with the corrected values.

Although no real flat field correction exists for OM, nor is it necessary, the processing requires such a file which can be generated using the task omflatgen as follows. (This task can also be run at the beginning of the processing).

step 7

  omflatgen outset=${MYDATA}/fast/${REV}_${OBSID}_OMX00000FFX.FIT

The output flatfield (primary extension) will be set to unity.

step 7 A

  omfastflat set=${MYDATA}/fast/F${OBSID}OMS${ENUM}EVLIST2000.FIT \
   slewflatset=${MYDATA}/fast/${REV}_${OBSID}_OMX00000FFX.FIT \
   fastimgset=${MYDATA}/fast/P${OBSID}OMS${ENUM}IMAGE_2000.FIT \
   oswflatset=${MYDATA}/fast/F${OBSID}OMS${ENUM}FLAFLD2000.FIT

Here again, the system is prepared to apply a subset of the omflatgen generated flat field to the fast mode window data, taking into account the spacecraft drift. The flat field is set to one, and therefore this correction has no real effect. The task generates the tracking shifted F${OBSID}OMS${ENUM}FLAFLD2000.FIT (only for the fast mode window) and the corrected pseudo image image.fit.

step 8

  omdetect nsigma=6 \
   set=${MYDATA}/fast/P${OBSID}OMS${ENUM}IMAGE_2000.FIT \
   regionfile=${MYDATA}/fast/F${OBSID}OMS${ENUM}REGION2000.ASC \
   outset=${MYDATA}/fast/P${OBSID}OMS${ENUM}SWSRLI2000.FIT

The output region will allow the user to check the proper detection of the source in the small fast window, e.g., with the command,

  ds9 ${MYDATA}/fast/P${OBSID}OMS${ENUM}IMAGE_2000.FIT \
   -regions ${MYDATA}/fast/F${OBSID}OMS${ENUM}REGION2000.ASC -zoom to fit -scale log

The PSF information is used to parameterize the detected source.

step 9

  omatt set=${MYDATA}/fast/P${OBSID}OMS${ENUM}IMAGE_2000.FIT \
   sourcelistset=${MYDATA}/fast/P${OBSID}OMS${ENUM}SWSRLI2000.FIT \
   ppsoswset=${MYDATA}/fast/P${OBSID}OMS${ENUM}SIMAGE2000.FIT \
   usecat=no rotateimage=yes tolerance=3

Astrometry is applied as for image data. The pseudo-image is also north aligned.

step 10

  omregion set=${MYDATA}/fast/P${OBSID}OMS${ENUM}SWSRLI2000.FIT \
   srcnumber=1 srcradius=-6 nfwhm=3 bkginner=1.2 bkgouter=2.5 \
   bkgfile=${MYDATA}/fast/F${OBSID}OMS${ENUM}BGDREG2000.FIT \
   srcfile=${MYDATA}/fast/F${OBSID}OMS${ENUM}SRCREG2000.FIT

These regions will be used by evselect to filter out the event list, extracting the corresponding photon events for the source and the background. Optional parameters can be used to fine-tune the definition of the regions.

step 11

This step, and the next one, do sample the corrected fast mode event list in the bins defined by timebinsize (default is 10s) to extract the source and background rates.

Source events

  evselect table=${MYDATA}/fast/F${OBSID}OMS${ENUM}EVLIST2000.FIT \
   expression="((WIN_FLAG .eq. 0) .and. \
   (region(${MYDATA}/fast/F${OBSID}OMS${ENUM}SRCREG2000.FIT, CORR_X, CORR_Y)))" \
   xcolumn=CORR_X ycolumn=CORR_Y \
   rateset=${MYDATA}/fast/F${OBSID}OMS${ENUM}SCRATE2000.FIT \
   timecolumn=TIME timebinsize=10 maketimecolumn=yes withrateset=yes
 

step 12

Background events

  evselect table=${MYDATA}/fast/F${OBSID}OMS${ENUM}EVLIST2000.FIT \
   expression="((WIN_FLAG .eq. 0) .and. \
   (region(${MYDATA}/fast/F${OBSID}OMS${ENUM}BGDREG2000.FIT, CORR_X, CORR_Y)))" \
   xcolumn=CORR_X ycolumn=CORR_Y \
   rateset=${MYDATA}/fast/F${OBSID}OMS${ENUM}BGRATE2000.FIT \
   timecolumn=TIME timebinsize=10 maketimecolumn=yes withrateset=yes

These background rates are not used in the light curve if the chain was run with the default parameter bkgfromimage=yes

step 13

And finally the light curve can be obtained and plotted. The photometric corrections are also applied (coincidence loss, dead time, PSF, and time dependent sensitivity variation).

  omlcbuild srcregionset=${MYDATA}/fast/F${OBSID}OMS${ENUM}SRCREG2000.FIT \
   bkgregionset=${MYDATA}/fast/F${OBSID}OMS${ENUM}BGDREG2000.FIT \
   srcrateset=${MYDATA}/fast/F${OBSID}OMS${ENUM}SCRATE2000.FIT \
   bkgrateset=${MYDATA}/fast/F${OBSID}OMS${ENUM}BGRATE2000.FIT \
   sourcelistset=${MYDATA}/fast/P${OBSID}OMS${ENUM}SWSRLI2000.FIT \
   wdxset=${MYDATA}/${REV}_${OBSID}_OMS${ENUM}00WDX.FIT \
   outset=${MYDATA}/fast/P${OBSID}OMS${ENUM}TIMESR2000.FIT \
   imageset=${MYDATA}/${REV}_${OBSID}_OMS${ENUM}00IMI.FIT \
   bkgfromimage=yes


By default, omfchain measures the background in the image data that usually are taken together with the fast mode data. This image file is specified with the parameter imageset. The image file has to correspond to a detector window that embeds the fast mode window.

Should the user prefer to extract the background from the fast mode data (or in case there is no image file) then the parameter bkgfromimage should be set to 'no' and there would be no need of imageset.

step 14

  lcplot set=${MYDATA}/fast/P${OBSID}OMS${ENUM}TIMESR2000.FIT \
   binsize=1 plotdevice=/PS \
   plotfile=${MYDATA}/fast/F${OBSID}OMS${ENUM}TIMESR2000.PS \
   bkgdyscale=no

step 15

The same checks of resulting products need to be performed as in the omfchain thread.