Herschel Data Processing

please note that there is an issue with hipe download.

we are aware of the problem and are trying to restore hipe services, but this may take some time as hipe no longer has dedicated support.

Please be patient!

 

Introduction to Herschel Data Processing

All data obtained with Herschel will be systematically pipeline-processed with the Herschel Data Processing system, at the Herschel Science Centre (HSC), to generate a range of scientific and auxiliary data products.

All raw and processed data are ingested in the Herschel Science Archive immediately after the pipeline processing is completed, typically 1-2 days after execution. Notification e-mails are sent to the proposal PIs when basic quality control checks have been completed, something which may take from a few additional days to weeks, depending on the circumstances.

The Herschel Data Processing (DP) system is a joint development effort by the HSC, the Herschel Principal Investigator Instrument consortia, and the NASA Herschel Science Center. The software is coded in Java and Jython, and runs on Linux, Windows, and Macintosh platforms.

The software and the associated documentation are available to the community for download from the HSC (just like HSpot) and incorporate the state of the art in calibration and pipelines.

For any queries about the system, please contact the HSC Helpdesk.

Overview of the Herschel Data Processing system

The Data Processing system consists of a pipeline processing software to be run 'hands-off' and an Interactive Analysis environment where the system can be used 'hands-on' by individual astronomers.

Pipeline processing software

The Herschel Data Processing system is used by the HSC to systematically process Herschel data in order to generate products associated with each observation.

The systematic pipeline processing consists of:

  • Standard Product Generation (SPG) software, run systematically for all data at HSC to generate a set of standard products associated with each observation. It consists of routines available in the Interactive Processing environment
  • Quality Control (QC) processing. It generates quality control information on all observational data. An automated part will run systematically for all data, but the data will also be visually inspected by HSC personnel at least once per observation.

 

On-demand reprocessing

Users can request at any time the reprocessing of (a limited set of) their observations with the latest operational software release using the 'On-Demand Reprocessing' option available from the 'Shopping Basket' window of the Herschel Science Archive.

The requested observations will be placed in a queue for reprocessing at the HSC with the latest operational version of the pipeline. The reprocessing will also make use of the most up-to-date calibration files and auxiliary products, without you needing to wait until the next bulk-reprocessing exercise that the HSC runs every 6 months. Once reprocessed, you will be notified and the data will be placed in a password-protected ftp area for your retrieval for a period of one week.

Herschel Interactive Processing Environment (HIPE)

In addition to the standard product pipeline processing the Herschel Data Processing system also provides the tools to interactively reduce the data using the same routines.

The Herschel Interactive Processing Environment (HIPE) enables the user to:

  • access data from the Herschel Science Archive
  • perform interactive data reduction from raw data to publishable products, using Herschel-provided and user-developed routines, both in GUI form or in console-batch mode
  • visualise and manipulate data
  • perform science analysis
  • get access to context-sensitive documentation and help

 

How to access HIPE

Linux, Windows, or Mac installers for the latest user version of HIPE can be retrieved here. In the early phases of the mission, up to 4 user releases per year were made available to the users community to accommodate the fast evolution of the pipelines. This has evolved with time into a standard cycle of 2 user releases per year.

How to learn more, customize and contribute to HIPE

HIPE is open to external contributions; the Key Programme consortia and the astronomical community in general are encouraged to feed back their data products and share the tools and algorithms developed to produce them with the HSC for possible inclusion in the Data Processing system. HIPE pipelines are organised in modules (called tasks), easily interchangeable by user-customized tasks.

Herschel Data Processing documentation and tutorials

For a quick introduction to HIPE, please read the "Quick Start Guide" document. For detailed information on reducing and analysing your Herschel data, see the Instrument Data Reduction Guides listed below.

  • PACS Data Reduction Guide: Spectroscopy (PDF)
  • PACS Data Reduction Guide: Photometry (PDF)
  • SPIRE Data Reduction Guide (PDF)
  • HIFI Data Reduction Guide (PDF)

In addition, instructional video tutorials about data analysis, data access and data structures are available on the 'Learn HIPE' YouTube and Twitter channels.

HIPE 'documentation sprints' are regularly organised by the Data Processing Editorial Board. A documentation sprint is an occasion for users to contribute to and comment on a piece of HIPE documentation. You decide your level of involvement, starting from just a few minutes of your time.

Data Processing User Telecons

A series of topic based telecon/webex Data Processing User Telecons are organised by the HSC regularly on various data processing aspects that involve instrument experts from the HSC, ICCs and NHSC, attended by people that joined our DP interest lists some time ago.

Details on the topics addressed in each one of these DP telecons, including the current schedule of upcoming sessions, draft agendas and minutes for those telecons that already took place in the past are available at our new 'HIPE community' web site.

This is a place for open discussion and knowledge exchange about Herschel data reduction and the HIPE software, from which you can join the various on-going discussions and access other HIPE related news, including the 'Learn HIPE' YouTube and Twitter channels.

We invite you to join these telecons following the instructions provided in the above link as a good opportunity to catch up on the latest news and plans for Herschel data reduction. Also to ask any questions to the instrument experts and share information and experiences with other people processing similar data.

Herschel Data Processing Questionnaire

In order to improve our knowledge about your experience with the processing of Herschel data, we would strongly appreciate if everyone of you with recently accepted refereed publications based on Herschel data could find some time to complete the Herschel Data Processing Questionnaire.

This questionnaire should take you no more than 10 minutes to fill in and is intended to serve as a way to help us improving HIPE. If you were not directly involved in the processing of the Herschel data used for your publication, please ask the appropriate person in your research group to do that. We thank you very much in advance for your collaboration.