Table of Contents -- Index -- PS reprint -- PDF reprint

Astronomical Data Analysis Software and Systems VII
ASP Conference Series, Vol. 145, 1998
Editors: R. Albrecht, R. N. Hook and H. A. Bushouse

On-The-Fly Re-Calibration of HST Observations

B. Pirenne and A. Micol
Space Telescope - European Coordinating Facility, ESO, Garching, D-85748

D. Durand and S. Gaudet
Canadian Astronomy Data Centre, DAO, Victoria, BC

 

Abstract:

On-the-fly re-calibration of HST data (OTF) has now been available at the CADC and ST-ECF for about 18 months. With the installation of STIS and NICMOS on the spacecraft, a new breed of data types and data organisations was challenging our existing data retrieval and processing service. We briefly describe the OTF method below.

         

1. What is On-the-fly re-calibration?

HST observations are normally calibrated and stored in the archive immediately after reception by the ground system. The calibration can only be executed using the calibration reference files which are available at that time. Some may be missing or may not be the most appropriate ones as new calibration observations might be obtained later on. Moreover, the calibration software of those instruments hardly ever stabilizes. In other words, the longer one waits before calibrating an observation, the better the results should be.

This is the concept that we decided to implement. The recipe is simple in principle: recalibrate only at the time of the data delivery to the user. This is ``Just-in-time'' HST data!

2. Implementation

The implementation is best explained by considering the data flow model presented in Figure 1.

  

Figure 1: On-the-fly re-calibration data flow: from request submission to data delivery.

In this model, users perform the traditional catalogue browsing activities: selection of data according to search criteria, examination of the results, refinement of the selection etc., until the proper set of observations has been identified from the science database, perhaps helped by looking at quick-look samples of the data. Subsequently, users will mark those records for retrieval and will typically select the on-the-fly reprocessing of the datasets.

After proper identification, selection of the output media etc, the request is registered in the archive database. Then, a first automatic process reads the files required from the data repository, while a second process starts the actual re-calibration.

Currently, this processing step involves the selection of the best calibration reference files using a specialized database. This database relates the current best calibration reference files to any given HST observation. Using this information, the actual calibration reference files (flat, bias, dark, etc.) are retrieved from magnetic disks and applied to the science data being re-calibrated.

As soon as all exposures belonging to a request have been reprocessed, the data is made available to the user.

On-the-fly re-calibration has another advantage for our archive sites: we only need to keep the raw data and calibration reference files on-line for the process to be automatic. As a matter of fact, original calibration, engineering data and other auxiliary data need not be available to the re-calibration process and can remain on secondary storage.

3. Recent Improvements

The ST-ECF and CADC are currently working towards improving this system through two major new facilities.

• First, the integration of the calibration pipeline for the two new HST instruments (NICMOS and STIS) complements our existing service.
• The science catalogue is now showing associations of WFPC2 exposures that can be retrieved and re-calibrated as one single entity, enabling further processing steps such as cosmic ray rejection and image co-addition. These latest improvements are described at length in Alberto Micol's article in this volume.

4. Conclusions

The mere fact that the service is used a lot is for us both proof of the usefulness of the OTF concept and an encouragement to develop it further: The integration of more advanced processing steps (cosmic-ray removal, co-addition of frames) pushes OTF even further by allowing users to concentrate more on data analysis and leave the reduction tasks to automated procedures.

References:

Micol A., Bristow P., & Pirenne B. 1998, this volume

Crabtree, D., Durand, D., Gaudet, S., & Pirenne, B., 1996,``The CADC/ST-ECF Archives of HST data: Less is More", in Astronomical Data Analysis Software and Systems V, ASP Conf. Ser., Vol. 101, eds. G. H. Jacoby and J. Barnes (San Francisco, ASP), 505

© Copyright 1998 Astronomical Society of the Pacific, 390 Ashton Avenue, San Francisco, California 94112, USA