XMM-Newton Current Calibration Files

Introduction

To analyse the data collected with the instruments on-board XMM-Newton, together with the SAS you will need some calibration files (Current Calibration Files a.k.a. CCF).

Contrary to the usual way of producing and distributing Calibration Data versions after a certain period of time, the XMM-Newton CCF are updated dynamically and continuously. Announcements of single or multiple file updates are issued via the ccf mailing list. Successive CCF releases do not have version numbers. Once announced, CCF data files are available for immediate download from our project servers (see below on how to do that) and their respective Release Notes are published online.

The CCF Index File

Before processing your data with the SAS you must build a CCF Index File (CIF) of all the CCF appropriate for your observations. This is carried out with the SAS task cifbuild. One of the parameters of the task, analysisdate, can be used by experts who want to compare data processed with past calibration knowledge. A "normal" user, will leave this parameter at its default: the current date.

Alternately, this interactive web page, allows you to run our own local version of cifbuild. As a result, you may download via ftp the resulting CIF file, compare it with the one used by you or download all the CCF required to process your data.

The indexing algorithm used to build the CIF is described in the SAS on-line documentation for the task cifbuild or by typing in within any installation of SAS, the command sashelp --doc=cifbuild.

The CCF Repositories

All the XMM-Newton CCF ever made public, currently ~1550 files occupying ~4.3 GB, are available for download from our project servers. Although you may copy all these files, this would only be necessary if you want to process your data with the calibration knowledge at any given time in the XMM-Newton history, since many old CCF have been superseded by more accurate calibration. If this is not your purpose, we recommend you to download only a subset of all CCF, known as the Valid CCF Set, assembled and made available for download daily. The components of this set are all the CCF you need to process any XMM-Newton ODF at the current date. Currently, this set has ~515 files with a size of ~1.18 GB.

Mirroring The CCF Repositories

Depending on your needs, you can either mirror the historical repository or the Valid CCF Set. For both we recommend to use rsync and only in case you can not use this tool for any reason, use plain ftp, that is always available.

rsync ensures that any CCF files deleted for any reason from our repositories will also be deleted from your local copy, therefore reducing the possibility that you use wrong CCF.

Using rsync

To store the CCF, please choose any directory where you have write permission. Let it be for example <dir>.

To keep it in sync with our repositories, please do as follows:

1. To synchronise with the whole repository of CCF:

   rsync -v -a --delete --delete-after --force --include='*.CCF' --exclude='*/' sasdev-xmm.esac.esa.int::XMM_CCF <dir>/

2. To synchronise with the Valid CCF Set:

   rsync -v -a --delete --delete-after --force --include='*.CCF' --exclude='*/' sasdev-xmm.esac.esa.int::XMM_VALID_CCF <dir>/

Before running these commands for the first time you may want to use the option --dry-run to see what the command would do without actually executing it.

Using plain ftp

You can always access our ftp server via anonymous user connection, as follows:

ftp sasdev-xmm.esac.esa.int Connected to sasdev-xmm.esac.esa.int. 220-+-----------------------------------------------------------------------------+ 220-| European Space Agency - Agence Spatiale Europeenne                          | 220-|                                                                             | 220-| System operated by the Sciops technical IT Unit (SITU)                      | 220-+-----------------------------------------------------------------------------+ 220-|                                                                             | 220-| This system is for the exclusive use of authorized users only.              | 220-|                                                                             | 220-| Unauthorized or improper use of this system may result in administrative    | 220-| disciplinary action and civil and criminal penalties. By continuing to use  | 220-| this system you indicate your awareness of and consent to these terms and   | 220-| conditions of use. LOG OFF IMMEDIATELY if you do not agree to the           | 220-| conditions stated in this warning.                                          | 220-|                                                                             | 220-+-----------------------------------------------------------------------------+ 220- 220-                  Welcome to the XMM-Newton SOC ftp server 220- 220-Information about your login and any transfers you do are logged on this host. 220-If you don't like that, please disconnect now. 220- 220-Please contact xmmhelp@sciops.esa.int with any technical problems you may have 220-accessing this archive. 220- 220 Name (sasdev-xmm.esac.esa.int:user): anonymous 331 Please specify the password. Password: your_user_name@valid_mail_address (not echoed)
  

You must enter a valid your_user_name@valid_mail_address mail address, otherwise our ftp server will reject the connection.

Once logged in, you should receive a welcome message similar to the following:

    230-XMM-Newton SOC - European Space Agency (ESA) 230-European Space Astronomy Centre (ESAC), Madrid, Spain 230------------------------------------------------------------ 230- 230-Last update: Nov 28, 2019 230------------------------------------------------------------ 230- 230 Login successful. Remote system type is UNIX. Using binary mode to transfer files. ftp>
  

Then, change directory to /pub/ccf and make a list of its contents as follows:

ftp> cd /pub/ccf 250 CWD command successful. ftp> ls 229 Entering Extended Passive Mode (|||31636|) 150 Here comes the directory listing. -rw-------    1 503      503          2115 Feb 18  2015 AAA_README.txt lrwxrwxrwx    1 503      503            30 Feb 18  2015 background -> constituents/extras/background drwxrwxrwx    5 503      503        122880 Dec 02 11:02 constituents drwxr-xr-x    2 503      503          4096 Aug 24  2002 doc drwxrwxrwt    2 503      503          4096 Dec 01 04:50 download lrwxrwxrwx    1 503      503            30 Jun 09  2014 esas_caldb -> constituents/extras/esas_caldb lrwxrwxrwx    1 503      503            34 Feb 18  2015 esas_caldb_old -> constituents/extras/esas_caldb_old lrwxrwxrwx    1 503      503            33 Feb 18  2015 filter_closed -> constituents/extras/filter_closed drwxrwxrwx    8 503      503          4096 Feb 20  2017 private drwxr-xr-x    4 503      503         77824 Apr 01  2016 red_constituents lrwxrwxrwx    1 503      503            29 Feb 18  2015 responses -> constituents/extras/responses drwxr-xr-x    3 503      503          4096 Mar 10  2016 to_be_deleted drwxrwxr-x    2 503      503         40960 Dec 02 04:52 valid_constituents 226 Directory send OK.
  e>

The directory constituents contains the historical repository of CCF. To get all these files, you may do it as follows:

ftp> prompt off Interactive mode off ftp> mget *.CCF
  

Similarly, to get only the Valid CCF Set, you can change directory to valid_constituents and invoke similar ftp commands.

Note that using these commands will download all specified CCF to the directory from where you started the ftp session.

Reporting Problems

Questions and problems regarding the CCF and how to download them should be addressed to the XMM-Newton Helpdesk.