The STScI NICMOS Calibration Pipeline

H. Bushouse, C. Skinner, J. MacKenty, and D. Axon Space Telescope Science Institute, 3700 San Martin Drive, Baltimore, MD 21218, Email: bushouse@stsci.edu E. Stobie U. of Arizona, Steward Observatory, Tucson, AZ 85721

         

 

Abstract:

The Space Telescope Science Institute (STScI) data reduction and calibration pipeline for the Near-Infrared Camera and Multi-Object Spectrometer (NICMOS) is composed of two major tasks, CALNICA and CALNICB. CALNICA applies instrumental calibration to all images, while CALNICB applies specialized processing to sets of images associated with a single astronomical target. This paper describes some of the technical aspects of the NICMOS data file formats, the CALNICA and CALNICB software, as well as details and examples of the processing that these tasks perform.

NICMOS, calibration, pipeline, software

Introduction

The STScI NICMOS calibration pipeline is composed of two major tasks, CALNICA and CALNICB. CALNICA applies instrumental calibration to all images and includes basic corrections such as dark current subtraction, detector non-linearity correction, and flat-fielding. CALNICA operates on one image at a time. It takes as input the raw science data files produced by the Generic Conversion process in the STScI OPUS environment. Its outputs are fully calibrated files and, for MultiAccum observations only, an intermediate calibrated file.

CALNICB is only applied to associated NICMOS observations and is executed after all images in an association have first been calibrated with CALNICA. CALNICB combines images into mosaics and also removes background signal from the images. Its inputs are an association table, which contains information about the images in the association, and the calibrated images produced by CALNICA. Its outputs are an updated version of the association table and background-subtracted mosaic images, produced by combining the associated input images.

CALNICA/CALNICB Common Features

Both CALNICA and CALNICB are built as host-level tasks and can therefore be executed directly from the host operating system level (e.g. Unix or VMS), or they can be executed from within the IRAF environment in the STSDAS hst_calib.nicmos package. The tasks are completely data driven in that they derive all information that is necessary to guide the processing from values of the input image header keywords and, in the case of CALNICB, the association table. Some user-setable parameters are available for tailoring reprocessing efforts. These parameters use default values when operating in the OPUS pipeline.

The run-time file format for all data I/O is FITS files with IMAGE and BINTABLE extensions. All NICMOS images are represented by five data arrays, stored as five IMAGE extensions in a single FITS file. The group of five data arrays is known as an image set, or ``imset". MultiAccum-mode exposures, which produce multiple intermediate detector readouts during the course of an exposure, have multiple imsets stored in the FITS files. Each imset is comprised of the science image (extension name ``SCI''), a statistical error image (``ERR''), a data quality flag array (``DQ''), an array giving the number of data samples at each pixel (``SAMP''), and an exposure time per pixel array (``TIME'').

All input and output files conform to the same file structure and organization, which allows output files to be reused as input if desired. All data processing steps involving reference data (e.g. dark and flat-field images) propagate statistical uncertainties from the ERR arrays and data quality flags from the DQ arrays of the reference data into the science data being processed. Furthermore, all steps involving image combination propagate the number of samples (SAMP) and their total exposure time (TIME) used to compute each resulting science (SCI) image value.

CALNICA Processing

The overall flow of CALNICA processing is shown in Figure 1.

  

Figure 1: CALNICA Processing Flow.

Included in this figure are the names of the image header keywords that are read by CALNICA to determine the names of reference files (left column) and the names of the keyword ``switches" used to turn each step on or off.

CALNICA applies basic instrumental calibrations to individual NICMOS observations. For MultiAccum-mode observations, the processing steps are repeated for each imset in the input file. The most significant calibration steps are the subtraction of dark current, correction for non-linear photometric response, and correction for pixel-to-pixel gain variations (flat-fielding). MultiAccum-mode observations receive some specialized processing, including correction for source signal present in the initial or ``zeroth" detector readout, subtraction of the zeroth readout from all subsequent readouts, and combination of the multiple readouts into a single image, which includes cosmic-ray and bad pixel rejection. Details of some of these steps are discussed below. For details regarding all of the individual processing steps see Bushouse (1997a, 1997b).

Dark Current Subtraction

The subtraction of dark current is a critical step in processing NICMOS data because it actually includes corrections for several instrumental effects (see Skinner, Bergeron, & Daou 1997; Bergeron & Skinner 1997). The actual detector dark current is quite small for all three NICMOS cameras, being at most 0.05 e^-/sec. There are two other much larger instrumental signatures superimposed on the darks. The first is a bias level or DC offset which exhibits a signal gradient across each quadrant of an image. This is the so-called ``shading", which is a noiseless signal than can reach levels of tens or even hundreds of DN (data numbers). The other effect is known as amp glow and appears as an elevated signal in the corners of NICMOS images. This signal is due to a small amount of IR radiation emitted by the readout amplifiers, which are situated near the detector corners.

Fortunately, both the shading and the amp glow signal levels are very repeatable, and depend variously upon the number of readouts and the time since the last detector reset. Hence a proper subtraction can be achieved by applying dark current reference images having the exact same readout sequence and exposure times. CALNICA automatically selects the appropriate imset from the dark current reference file that has an exposure time matching that of the science data being processed.

Detector Non-Linearity

This step corrects for the non-linear response of the detectors. The response can be conveniently divided into three regimes. In the low signal regime, the detector response is linear, hence no correction is applied. In the mid-level regime, the response deviates from true linearity in a way that is correctable using a first-order polynomial of the form

SCI(corrected) = (c₁ + c₂*SCI) * SCI (7.1)

where c₁ and c₂ are the polynomial coefficients. The coefficients are stored, on a pixel-by-pixel basis, in the ``COEF" and ``ERR" image extensions in the NLINFILE reference file. In the high signal regime, where pixels begin to saturate, no correction is applied to the data, but a saturation flag value is set in the DQ image. The DN levels defining the boundaries between the regimes are also stored on a pixel-by-pixel basis in two ``NODE" image extensions in the NLINFILE. The linear regime typically extends up to $\sim$14500 DN, and the non-linear regime to $\sim$27500 DN, above which saturation begins to occur.

MultiAccum Zeroth Read Signal

Because NICMOS does not have a shutter, signal from bright sources can accumulate during the $\sim$0.2 second long period that elapses between resetting the detector and taking the initial, or zeroth, readout that begins an exposure. Any signal present in the zeroth read will be removed when the zeroth read image is subtracted from all subsequent readouts by CALNICA, but this can lead to an error in the linearity correction that is applied to a given pixel, since the linearity correction depends on the total amount of charge that accumulated in the pixel.

The ZSIGCORR step of CALNICA computes the count rate for each pixel by differencing the zeroth and first readouts from one another and then estimates the amount of signal that would have accumulated in the zeroth read during its 0.203 second effective exposure time. This zeroth read signal is carried through to the NLINCORR step, where it is added back into the signal level for each pixel before determining its correction. After applying the linearity correction, the zeroth read signal is again removed.

Combining MultiAccum Readouts

Once all the individual readouts of a MultiAccum observation have received all the necessary instrumental calibration steps, they are combined into a single image, identifying and rejecting both cosmic-ray hits and bad pixels in the process. The accumulated counts as a function of increasing exposure time for each pixel are fitted with a first-order polynomial, using a standard weighted least-squares algorithm. Readouts already flagged as bad in the DQ array (e.g. due to saturation) are not used in the fit. The distance of each readout value from the fit is computed and cosmic-rays are identified by their characteristic negative-to-positive residual transition from one readout to the next. Readouts containing a hit are rejected and the fit is recomputed. This process is repeated until no new hits are identified. The slope of the fit gives the final count rate for the pixel. The total number of non-rejected readouts and their exposure time are recorded in the SAMP and TIME arrays of the final output image.

Figure 2 shows some examples of the MultiAccum fitting process.

  

Figure 2: MultiAccum Ramp Fitting Examples.

The top panel shows the results for a pixel that had a cosmic-ray hit. The open circles are the original data values, while the filled circles are the result of removing the signal from the readout containing the CR hit. The bottom panel shows a pixel that has the last two readouts flagged as saturated and therefore not used in the fit.

CALNICB Processing

The overall flow of CALNICB processing is shown in Figure 3.

  

Figure 3: CALNICB Processing Flow.

CALNICB is used exclusively for processing associated sets of images, all of which must have already been calibrated using CALNICA. A NICMOS image association is defined to be a logical grouping of multiple observations of a single astronomical target. Associations are typically used for:

• making multiple exposures at a single sky position so that cosmic-ray rejection can be performed by later combining the images
• dithering the telescope field of view to remove the effects of bad pixels or flat field residuals
• building image mosaics of large angular-size targets
• chopping the field of view off of the target to measure the telescope thermal background signal

To make it easier for observers to plan dithered and chopped sets of observations, STScI created several predefined observing patterns which automatically dither or chop the telescope field of view through a grid of observing positions on the sky (see MacKenty et al. 1997, chapter 10).

The major processing steps of CALNICB include the combination of images taken at individual sky positions, the measurement and subtraction of the thermal background signal, and the mosaicing of images within an observing pattern. The following sections describe some of these operations in a bit more detail. For more information, see Bushouse (1997a, 1997c).

Input Data

CALNICB, as run in the STScI pipeline, is completely data driven, getting all the information it needs to guide the processing from the association table that accompanies the individual images, and from header keywords in the images themselves. Some user-setable task parameters are available in CALNICB versions 2.2 and later, for use when running CALNICB interactively (see below). The association table (denoted ``ASN") is a FITS file with a BINTABLE extension containing the list of file names that make up the association to be processed by CALNICB, as well as the file names of all output mosaic image files (denoted ``MOS") to be produced. CALNICB determines which names in the list are input files and which are output files based on the MEMTYPE column in the ASN table. Input files have a MEMTYPE prefix of ``EXP" (exposure), while output files have a prefix of ``PROD" (products). Furthermore, association members that are observations of the target of interest have a MEMTYPE suffix of ``TARG" (target), while observations of sky background locations (which are only produced by using chop patterns) have a suffix of ``BCKn", where n is the background region number in the pattern (see Table 1). One MOS image will always be produced for the target, and chop patterns will result in one MOS image being produced for each background region observed.

Once CALNICB has determined the list of input images from the ASN table, it reads various keywords from the image headers to determine exactly what processing needs to be performed. Certain keywords, such as PATTERN (pattern type), NUMPOS (number of pattern positions), and NUMITER (number of exposure iterations at each position) have the same value in all images and are therefore only read from the first image in the association. Others are image specific, such as PATT_POS (position number of each image in the pattern) and World Coordinate System (WCS) keywords, have unique values in each image and are therefore read from each image. From this information CALNICB determines what type of pattern was used, how many positions are in the pattern, how many images there are at each pattern position, the relative positions of each image in the pattern, how many output MOS images need to be produced, and which input images go into which MOS image.

Combining NUMITER Images

If there is more than one image at each pattern position, CALNICB will combine the images at each position into a single image. The offsets between each image at a given position are computed from their WCS information, using the first image at each pattern position as a reference. These offsets are then refined using a cross-correlation technique. The images are then aligned with their reference image, using bilinear interpolation, and then combined. The combining process computes the ERR-weighted mean for each SCI image value, with rejection of DQ-flagged pixels and iterative sigma clipping. The number of non-rejected samples, and their total exposure time, are saved in the SAMP and TIME images.

Identifying Sources

This step flags (in the DQ images) pixels suspected of containing flux from a source. This is accomplished using a 3-stage process. First, pixels that are greater than 5 sigma above the median signal level in each image are identified as candidate source pixels. Second, in order to filter out spurious identifications, only those candidates having two or more neighboring pixels also identified as containing a source are retained. Finally, the surviving pixels have their source DQ flags ``grown" to their four immediate neighbors.

Scalar Background Subtraction

This step computes and subtracts a scalar (or ``DC") background signal from all images in the association. This is accomplished as follows. First, the median signal level in each BCK image in the association is computed. The computation excludes bad and source-flagged pixels and uses iterative sigma clipping to avoid any possible contamination. If there are not any BCK images in the association (e.g. the observations are from a pure dither pattern), the same computation is performed on the target images. The resulting background levels for each image are then averaged to compute the overall mean background level for the whole association. This overall mean is then subtracted from all images. The mean background level is recorded in the MEAN_BKG keyword in the output association (denoted ``ASC") table.

2-D Background Subtraction

This step subtracts the ILLMFILE reference image from the associated images. It is intended to remove any spatial variations in the telescope background thermal pattern, but to date, little if any such spatial variations have been seen. Thus the ILLMFILEs currently in use at STScI contain dummy data (constant zero values). CALNICB recognizes that the files are dummy and skips the subtraction.

Creating Mosaics

This step performs the final combination of images to produce the output mosaics. The process is very similar to the step that combines the NUMITER images at each pattern position. The relative offsets for each image in a given mosaic are first computed from their WCS information, using the first image in each mosaic as a reference. The offsets are refined using cross-correlation, and then the images are aligned using bilinear interpolation and combined. The combining process computes the ERR-weighted mean of overlapping pixels, rejecting those with non-zero DQ values and applying an iterative sigma clipping. The number of non-rejected samples and their total exposure time are saved in the output SAMP and TIME images of the MOS files.

Updating the Association Table

The final step of CALNICB creates a new association (ASC) table. The ASC table is a copy of the input ASN table, with four new columns of information appended. An example of an ASC table is shown in Table 1. The new columns are BCKIMAGE, MEANBCK, XOFFSET, and YOFFSET.
 
The BCKIMAGE column indicates whether or not a given image was used in the scalar background computation. The example shown in Table 1 is for a TWO-CHOP pattern, with NUMITER=2, and NUMPOS=5. So there are two images at each of the five pattern positions and two of the positions are background (BCK) regions. Thus, in this example, only the images at the BCK positions were used to compute the scalar background level. The MEANBCK column records the background signal computed for each individual image. The XOFFSET and YOFFSET columns record the computed offsets for each image relative to its reference image.

CALNICB User Parameters

As mentioned above, beginning with CALNICB v2.2, there are a few parameters available to interactive users for tailoring the CALNICB processing. The subbkg parameter allows you to turn the scalar background subtraction step on or off. The meanbkg parameter allows you to use the mean scalar background value of all images, or subtract the value derived for each image from itself. The readbkg and readoffsets parameters allow you to use an ASC table as input to CALNICB and have it read either the image background or offset (or both) values from the table, rather than computing them. The crthresh parameter sets the cosmic-ray rejection threshold used during image combining, and the xcwin parameter sets the size of the cross-correlation search window box.