Top/Data_Reduction/Imaging_Data_Reduction

Data Reduction Pipeline for Imaging Data

Overview

SWIMS Imaging Data Reduction Pipeline, written in Python, processes SWSB*/SWSR* FITS data to produce a final stacked image in a standard manner.

Reduction Flow Chart
flowchart_20210718.png

Download

Core scripts

FileContents
fileswsred_20220406.tgzPython core scripts (swsred/*py) and configuration/utility files

BPM (Bad-Pixel Mask)

Apr 6, 2022: BPM for S21B/S22A available.

Put files that you need on swsred/calib/bpm/ directory.
Note that BPMs for S22A are just symbolic links to S21B. If you have data on both S21B and S22A, you only need to download either BPMs and create symbolic links to them in swsred/calib/bpm/ directory, like "ln -s bpm_s21b_b1.fits bpm_s22a_b1.fits" to save your disk space.

Imaging Flat

Apr 6, 2022: Flat for S21B/S22A available.

Put files that you need on swsred/calib/flat/ directory.
Note that Flat data for S22A are just symbolic links to S21B. If you have data on both S21B and S22A, you only need to download either Flats and create symbolic links to them in swsred/calib/flat/ directory, like "ln -s flat_s21b_y_b1.fits flat_s22a_y_b1.fits" to save your disk space.

FilterS22A(=S21B)S21BS21A (May)S21A (April)S18B
Yflat_s22a_y_b1.fits
flat_s22a_y_b2.fits
flat_s21b_y_b1.fits
flat_s21b_y_b2.fits
flat_s21a2_y_b1.fits
flat_s21a2_y_b2.fits
flat_s21a1_y_b1.fits
flat_s21a1_y_b2.fits
flat_s18b_y_b1.fits
flat_s18b_y_b2.fits
Jflat_s22a_j_b1.fits
flat_s22a_j_b2.fits
flat_s21b_j_b1.fits
flat_s21b_j_b2.fits
flat_s21a2_j_b1.fits
flat_s21a2_j_b2.fits
flat_s21a1_j_b1.fits
flat_s21a1_j_b2.fits
--
--
J1flat_s22a_j1_b1.fits
flat_s22a_j1_b2.fits
flat_s21b_j1_b1.fits
flat_s21b_j1_b2.fits
flat_s21a2_j1_b1.fits
flat_s21a2_j1_b2.fits
flat_s21a1_j1_b1.fits
flat_s21a1_j1_b2.fits
flat_s18b_j1_b1.fits
flat_s18b_j1_b2.fits
J2flat_s22a_j2_b1.fits
flat_s22a_j2_b2.fits
flat_s21b_j2_b1.fits
flat_s21b_j2_b2.fits
flat_s21a2_j2_b1.fits
flat_s21a2_j2_b2.fits
flat_s21a1_j2_b1.fits
flat_s21a1_j2_b2.fits
flat_s18b_j2_b1.fits
flat_s18b_j2_b2.fits
NB1244flat_s22a_nb1244_b1.fits
flat_s22a_nb1244_b2.fits
flat_s21b_nb1244_b1.fits
flat_s21b_nb1244_b2.fits
--
--
--
--
flat_s18b_nb1244_b1.fits
flat_s18b_nb1244_b2.fits
NB1326flat_s22a_nb1326_b1.fits
flat_s22a_nb1326_b2.fits
flat_s21b_nb1326_b1.fits
flat_s21b_nb1326_b2.fits
--
--
--
--
flat_s18b_nb1326_b1.fits
flat_s18b_nb1326_b2.fits
Hflat_s22a_h_r1.fits
flat_s22a_h_r2.fits
flat_s21b_h_r1.fits
flat_s21b_h_r2.fits
flat_s21a2_h_r1.fits
flat_s21a2_h_r2.fits
flat_s21a1_h_r1.fits
flat_s21a1_h_r2.fits
--
--
KSflat_s22a_ks_r1.fits
flat_s22a_ks_r2.fits
flat_s21b_ks_r1.fits
flat_s21b_ks_r2.fits
flat_s21a2_ks_r1.fits
flat_s21a2_ks_r2.fits
flat_s21a1_ks_r1.fits
flat_s21a1_ks_r2.fits
--
--
H1flat_s22a_h1_r1.fits
flat_s22a_h1_r2.fits
flat_s21b_h1_r1.fits
flat_s21b_h1_r2.fits
flat_s21a2_h1_r1.fits
flat_s21a2_h1_r2.fits
--
--
flat_s18b_h1_r1.fits
flat_s18b_h1_r2.fits
H2flat_s22a_h2_r1.fits
flat_s22a_h2_r2.fits
flat_s21b_h2_r1.fits
flat_s21b_h2_r2.fits
flat_s21a2_h2_r1.fits
flat_s21a2_h2_r2.fits
--
--
flat_s18b_h2_r1.fits
flat_s18b_h2_r2.fits
H3flat_s22a_h3_r1.fits
flat_s22a_h3_r2.fits
flat_s21b_h3_r1.fits
flat_s21b_h3_r2.fits
flat_s21a2_h3_r1.fits
flat_s21a2_h3_r2.fits
--
--
flat_s18b_h3_r1.fits
flat_s18b_h3_r2.fits
K1flat_s22a_k1_r1.fits
flat_s22a_k1_r2.fits
flat_s21b_k1_r1.fits
flat_s21b_k1_r2.fits
flat_s21a2_k1_r1.fits
flat_s21a2_k1_r2.fits
flat_s21a1_k1_r1.fits
flat_s21a1_k1_r2.fits
flat_s18b_k1_r1.fits
flat_s18b_k1_r2.fits
K2flat_s22a_k2_r1.fits
flat_s22a_k2_r2.fits
flat_s21b_k2_r1.fits
flat_s21b_k2_r2.fits
flat_s21a2_k2_r1.fits
flat_s21a2_k2_r2.fits
flat_s21a1_k2_r1.fits
flat_s21a1_k2_r2.fits
flat_s18b_k2_r1.fits
flat_s18b_k2_r2.fits
K3flat_s22a_k3_r1.fits
flat_s22a_k3_r2.fits
flat_s21b_k3_r1.fits
flat_s21b_k3_r2.fits
flat_s21a2_k3_r1.fits
flat_s21a2_k3_r2.fits
flat_s21a1_k3_r1.fits
flat_s21a1_k3_r2.fits
flat_s18b_k3_r1.fits
flat_s18b_k3_r2.fits
NB1630flat_s22a_nb1630_r1.fits
flat_s22a_nb1630_r2.fits
flat_s21b_nb1630_r1.fits
flat_s21b_nb1630_r2.fits
--
--
--
--
flat_s18b_nb1630_r1.fits
flat_s18b_nb1630_r2.fits
NB1653flat_s22a_nb1653_r1.fits
flat_s22a_nb1653_r2.fits
flat_s21b_nb1653_r1.fits
flat_s21b_nb1653_r2.fits
--
--
--
--
flat_s18b_nb1653_r1.fits
flat_s18b_nb1653_r2.fits
NB1875flat_s22a_nb1875_r1.fits
flat_s22a_nb1875_r2.fits
flat_s21b_nb1875_r1.fits
flat_s21b_nb1875_r2.fits
--
--
--
--
flat_s18b_nb1875_r1.fits
flat_s18b_nb1875_r2.fits
NB1945flat_s22a_nb1945_r1.fits
flat_s22a_nb1945_r2.fits
flat_s21b_nb1945_r1.fits
flat_s21b_nb1945_r2.fits
--
--
--
--
flat_s18b_nb1945_r1.fits
flat_s18b_nb1945_r2.fits
NB2137flat_s22a_nb2137_r1.fits
flat_s22a_nb2137_r2.fits
flat_s21b_nb2137_r1.fits
flat_s21b_nb2137_r2.fits
--
--
--
--
flat_s18b_nb2137_r1.fits
flat_s18b_nb2137_r2.fits
NB2167flat_s22a_nb2167_r1.fits
flat_s22a_nb2167_r2.fits
flat_s21b_nb2167_r1.fits
flat_s21b_nb2167_r2.fits
--
--
--
--
flat_s18b_nb2167_r1.fits
flat_s18b_nb2167_r2.fits

Mosaic

Usually, mosaicking is performed (using swarp) based on WCS information.
These files are needed only when your data do not have enough stars for astrometry.

FileContents
fileswsred_calib_mosaic_s18a_20210131.tgzMosaic parameter files for S18A data
fileswsred_calib_mosaic_s18b_20210131.tgzMosaic parameter files for S18B data
fileswsred_calib_mosaic_s21a_20210413.tgzTentative use only!! Mosaic parameter files for S21A data

Sample data

Use these test data (taken in S18B) to check if your swsred works.

FileContents
fileswsred_sample_b.tgzJ1-band data (4 frames x 2 arrays)
fileswsred_sample_r.tgzH1-band data (4 frames x 2 arrays)

Here is a code to reduce the data. --> sample recipe

The reduced images produced with two iterations (n_iter=2) using swsred version 20210901 and the execution log (swsred.log) should be like these.

FileContents
filefs12_j1_b1_stack1.fitsStacked image (after two iterations) on the blue array #1.
filefs12_j1_b2_stack1.fitsStacked image (after two iterations) on the blue array #2.
filefs12_j1_b1_stack1_exp.fitsThe exposure map for fs12_j1_b1_stack1.fits.
filefs12_j1_b2_stack1_exp.fitsThe exposure map for fs12_j1_b2_stack1.fits.
filefs12_j1_b1_stack1_wht.fitsThe weight map for fs12_j1_b1_stack1.fits.
filefs12_j1_b2_stack1_wht.fitsThe weight map for fs12_j1_b2_stack1.fits.
filefs12_j1.fitsMosaicked image (after two iterations) on the blue arrays.
filefs12_j1_exp.fitsThe exposure map for fs12_j1.fits.
filefs12_j1_wht.fitsThe weight map for fs12_j1.fits.
fileswsred_j1.logExecution logs on reducing J1 data.
filefs12_h1_r1_stack1.fitsStacked image (after two iterations) on the red array #1.
filefs12_h1_r2_stack1.fitsStacked image (after two iterations) on the red array #2.
filefs12_h1_r1_stack1_exp.fitsThe exposure map for fs12_h1_r1_stack1.fits.
filefs12_h1_r2_stack1_exp.fitsThe exposure map for fs12_h1_r2_stack1.fits.
filefs12_h1_r1_stack1_wht.fitsThe weight map for fs12_h1_r1_stack1.fits.
filefs12_h1_r2_stack1_wht.fitsThe weight map for fs12_h1_r2_stack1.fits.
filefs12_h1.fitsMosaicked image (after two iterations) on the red arrays.
filefs12_h1_exp.fitsThe exposure map for fs12_h1.fits.
filefs12_h1_wht.fitsThe weight map for fs12_h1.fits.
fileswsred_h1.logExecution logs on reducing H1 data.

Installation

prerequisites

SWSRED requires the following python packages and astrOmatic softwares.

PackageTested version
Python2.7.173.7.10
astropy2.0.94.2
numpy1.16.61.19.2
scipy1.2.11.6.2
matplotlib2.2.33.3.4
astroquery0.40.4.1
sextractor2.19.52.19.5
scamp2.7.82.10.0
swarp2.38.02.38.0

The astrOmatic softwares can be installed as follows if you have a anaconda environment.

% conda config --append channels conda-forge
% conda install sextractor
% conda install -c conda-forge astromatic-scamp
% conda install -c conda-forge astromatic-swarp

Also, Internet connection is required during reducing data to access Pan-STARRS/2MASS catalogs.

Install SWSRED

  1. Download SWSRED files you need.
  2. Extract all the tar balls on a directory you want.
    % cd SWSRED_DIRECTORY
    % tar xvf swsred_20210527.tgz
    % tar xvf swsred_calib_mosaic_s21a_20210413.tgz
    % mkdir -p swsred/calib/bpm swsred/calib/flat
    
  3. set PYTHONPATH to SWSRED_DIRECTORY.
    % export PYTHONPATH="SWSRED_DIRECTORY:${PYTHONPATH}
    

Usage

SWSRED has no documentations yet, but you can use a dirty script 'swsred/reduce_all.py' as a quick tutorial to learn how to use the software.

Be sure to use SWSRED within python interpreter (e.g., in ipython or jupyter, etc.), not via the command line as no command-line interfaces prepared.

Here is a quick introduction to use SWSRED.

  1. First of all, prepare a list of input FITS files including absolute paths as a python 'array-like' (list, tuple, or numpy array) object, for example as follows.
    > import glob
    > in_arr1 = glob.glob("DATA/PATH/SWSB*1.fits")  # blue chip1 data 
    > in_arr1.sort()  # must be sorted before using.
    
    • Note that as shown in the flow chart above, data of chip1 and chip2 should be reduced separately (before stacking together).
    • Data taken within the same dithering set are grouped by '2_DSETID' header keyword. Using this, you can make a input list like follows,
      import glob
      import astropy.io.fits
      
      dct1 = {}
      dct2 = {}
      
      infiles = glob.glob("SWS*fits")
      infiles.sort()
      
      for infile in infiles:
          hdr = astropy.io.fits.getheader(infile)
      
          ch = hdr["2_DETCH"]
          if ch == 1:
              dct = dct1
          else:
              dct = dct2
      
          if hdr["2_DSETID"] not in dct.keys():
              dct[hdr["2_DSETID"]] = []
      
          dct[hdr["2_DSETID"]].append(infile)
      
  2. Make filenames of intermediate files.
    > import swsred.make_file_lst
    > file_lst1 = swsred.make_file_lst.make_file_lst(in_arr1)
    
    • 'file_lst1' is a dictionary which defines various filenames such as flat-fielded FITS filename. You can edit any values as you want.
      • For example, you get the following result with an input ["data/path/SWSB00000001.fits"] as in_arr1,
        KeyValueDescription
        indata/path/SWSB00000001.fitsinput filename
        ffffSWSB00000001.fitsfilename of flat-fielded data
        msmsSWSB00000001.fitsfilename of self-sky data
        ssssSWSB00000001.fitsfilename of sky-subtracted data
        wcwcSWSB00000001.fitsfilename of WCS-corrected data
        bpm/SWSRED_DIRECTORY/calib/bpm/bpm_sXXY_b1.fitsfilename of Bad-pixel mask (XXY denotes a semester)
        whtSWSB00000001_wht.fitsfilename of weight map (created by make_weight_map module)
        omskSWSB00000001_omsk.fitsfilename of object mask
        ldacssSWSB00000001.ldacfilename of SExtractor LDAC catalog
        asciissSWSB00000001.asciifilename of SExtractor ASCII catalog
        headssSWSB00000001.headfilename of Scamp HEAD file
  1. Then, start your data reduction. For example, to perform flat-fielding,
    > import swsred.flat_field
    > swsred.flat_field.flat_field(file_lst1["in"], file_lst1["ff"])
    
    • By default, the values 'file_lst1["ff"]' do not have a path which means that 'file_lst1["ff"]' files will be created on the current directory. If you want to specify any directory to put intermediate files, edit and add your path in 'file_lst1'.

Refer to swsred/reduce_all.py for all the procedures.

Questions & Answers

  1. Astrometry (performed by scamp) is failed or is not good. How do I work for it?
    • Try changing the following parameters,
      • detect_thresh_wcs: detection threshold to make a source catalog used in scamp.
      • mag_max_wcs: maximum magnitude of Pan-STARRS sources to be used in scamp.
    • You may want to set "use_scamp=False" not to use Scamp; in that case (still in beta), the swsred try to find stars in the image around reference catalog positions and calculate WCS parameters.
  2. How is the semester determined?
    • FRAMEID is used to determine when the data has been taken. You can find the definition of which data was taken in which semester in swsred/config/config.ini.
    • If swsred would assign the wrong semester, check if you are using the latest swsred (i.e., latest definition).

Known bugs

Changes

Version 2021.01.31

Contact

Please send your bug reports or questions to konishi__at__ioa.s.u-tokyo.ac.jp.


トップ   編集 凍結 差分 履歴 添付 複製 名前変更 リロード   新規 一覧 検索 最終更新   ヘルプ   最終更新のRSS
Last-modified: 2022-04-16 (土) 21:52:38