v1_1 (April 1, 2003)


Major changes:


- Drew Phillip's optical model is now used for initial wavelength
  guesses.  Large tweaks are sometimes still required (e.g. due to
  differences in FCS setup from the model), but nevertheless this has
  proved to be quite successful.

- Arc handling has been totally revamped, allowing good solutions for
  600-line data; among other changes, a new lamp list,
  lamp_NIST_blue.dat, should work better than the default
  lamp_NIST.dat for low-resolution/non-DEEP2 data.  We can now deal
  with multiple arcs (including combining partially saturated with
  unsaturated arcs; see COMBINE_ARCS), and can use separate arcs for
  red side vs. blue.

- Slits overlapping edges of chips are now reduced, not ignored.  This
  could add as many as 8 objects/mask (~2-6 should be typical).
  Regions with no data have inverse variance 0 and the 2^2 bit of MASK
  set to 1.

- The derivative tails at the spatial ends of slits (due to something
  like sub-pixel shifts between the slitfunction derived from flats
  and the science data, though that cannot be the actual cause) are
  now corrected when possible, giving us ~2 more rows per slit for sky.

- Non-local sky subtraction has been greatly improved, and is now used
  throughout the pipeline (including extractions & spec1d, with
  non-local results in separate HDUs).

- B-spline breakpoint handling and sky region determination has
  changed, yielding much better (though still noisy) results for short slits.

- Extractions totally revamped.  Standard extractions are no longer a
  boxcar with interpolation and a non-inverse variance weighted
  optimal extraction, but instead a boxcar with bad pixels compensated
  for based on the fraction of light from the object missing (using
  the object's profile) and an optimal extraction with inverse
  variance weighting.  A major bug (which caused the
  optimal-extraction kernel to be the same for all objects) has been fixed.

- Sky spectra are now stored in the spec1d structures, and the inverse
  variance in the extracted spectra due to covariant sky subtraction
  is properly included.

- Positions of serendipitous sources on the sky (RA/dec) are now
  determined relative to object position using slit PA.  Serendips now
  have naming convention 'sXXXXXXX[b/c/d/...]' where XXXXXXXX is the label of
  a source on the same slit, and [b/c/d/...] is a letter, b for the
  first serendip on a slit, c for the second, etc.

- Flat-fielding, and particularly slit function determination, has
  been made much more robust; spectral ends of chips are also much 
  better than before. Additionally, we have greatly improved the behavior of
  the b-spline at the spectral ends of the array.

- To deal with the time variations in the FCS scattered light, the pipeline
  now chooses amongst multiple superdarks rather than always using an
  August one.

- Bad pixel mask has been greatly improved (using superdark data).

- Old QA routines have been updated and replaced with QA_CHECK .


More minor changes and fixes:


- More keywords saved in FITS headers (including Gaussian sigma of
  skylines and PAs for rotation curve analysis).

- The SKYIND tag in spslit files has been removed to prevent
  occasional problems combining spslit structures.

- spec1d structures have been changed (e.g. to include the sky
  spectra); see documentation.  Extraction width definitions have been
  unified between boxcar and optimal; the IVARFUDGE parameter is an
  estimate of how much one might want to multiply the stated IVAR by
  to match the actual fluctuations in the extracted spectrum (this was
  added mostly as a diagnostic; it varies significantly from one only
  for sky slitlets [due to using the same rows for the sky bspline and
  extraction], bright objects [which violate the assumptions made],
  and short slits with concomitantly noisy sky subtraction (for which
  IVARFUDGE reflects the reality).

- Arc lines are now identified as blends based on what lines are
  actually in a mask's arcs, not based on the complete linelist.

- New .plan file keywords: REDARC, BLUEARC, and LINELIST .

- SPEC1D_ASCII routine to convert spec1d output to ASCII files.

- FLAG_CR now finds CR's iteratively (only used for bsplines)


To be done:

- When we have few identified arc lines, if we fit, say, 2 terms in
  the wavelength solution, fix the 3rd-6th from the optical model and
  fit the first two, rather than only including 2 terms in the
  solution total.

- Develop the ability to combine results from different arc frames,
  rather than combining all frames and then fitting.

- Handle the varying resolution along a slit that comes from big
  notches in the slit function.  The interim fix might be to only use
  sky rows with the same slitfn value as the object; longer-term, we
  might convolve all rows to matching resolution.

- Incorporate atmospheric dispersion and wavelength-dependent seeing
  into the extractions.  This will probably require a rewrite &
  generalization of extract1d.

- Flux calibration.

- Re-implement quicklook.

- Frame-by-frame QA diagnostics.

- Time-varying pixflat files, not just superdarks

------------------------------------------------------------------------------