Imaging_Data_Reduction

Data Reduction Pipeline for Imaging Data †

Overview †

SWIMS Imaging Data Reduction Pipeline, written in Python, processes SWSB*/SWSR* FITS data to produce a final stacked image in a standard manner.

Download †

Core scripts †

BPM (Bad-Pixel Mask) †

Apr 6, 2022: BPM for S21B/S22A available.

Put files that you need on swsred/calib/bpm/ directory.
Note that BPMs for S22A are just symbolic links to S21B. If you have data on both S21B and S22A, you only need to download either BPMs and create symbolic links to them in swsred/calib/bpm/ directory, like "ln -s bpm_s21b_b1.fits bpm_s22a_b1.fits" to save your disk space.

Imaging Flat †

Apr 6, 2022: Flat for S21B/S22A available.

Put files that you need on swsred/calib/flat/ directory.
Note that Flat data for S22A are just symbolic links to S21B. If you have data on both S21B and S22A, you only need to download either Flats and create symbolic links to them in swsred/calib/flat/ directory, like "ln -s flat_s21b_y_b1.fits flat_s22a_y_b1.fits" to save your disk space.

Mosaic †

Usually, mosaicking is performed (using swarp) based on WCS information.
These files are needed only when your data do not have enough stars for astrometry.

Sample data †

Use these test data (taken in S18B) to check if your swsred works.

• Object: FS12
• Dithering: 4-point
• Filter: J1-band and H1-band

Here is a code to reduce the data. --> sample recipe

The reduced images produced with two iterations (n_iter=2) using swsred version 20210901 and the execution log (swsred.log) should be like these.

Installation †

prerequisites †

SWSRED requires the following python packages and astrOmatic softwares.

The astrOmatic softwares can be installed as follows if you have a anaconda environment.

% conda config --append channels conda-forge
% conda install sextractor
% conda install -c conda-forge astromatic-scamp
% conda install -c conda-forge astromatic-swarp

Also, Internet connection is required during reducing data to access Pan-STARRS/2MASS catalogs.

Install SWSRED †

1. Download SWSRED files you need.
2. Extract all the tar balls on a directory you want.

   % cd SWSRED_DIRECTORY
   % tar xvf swsred_20210527.tgz
   % tar xvf swsred_calib_mosaic_s21a_20210413.tgz
   % mkdir -p swsred/calib/bpm swsred/calib/flat
   

3. set PYTHONPATH to SWSRED_DIRECTORY.

   % export PYTHONPATH="SWSRED_DIRECTORY:${PYTHONPATH}
   

Usage †

SWSRED has no documentations yet, but you can use a dirty script 'swsred/reduce_all.py' as a quick tutorial to learn how to use the software.

Be sure to use SWSRED within python interpreter (e.g., in ipython or jupyter, etc.), not via the command line as no command-line interfaces prepared.

Here is a quick introduction to use SWSRED.

1. First of all, prepare a list of input FITS files including absolute paths as a python 'array-like' (list, tuple, or numpy array) object, for example as follows.

   > import glob
   > in_arr1 = glob.glob("DATA/PATH/SWSB*1.fits")  # blue chip1 data 
   > in_arr1.sort()  # must be sorted before using.
   

   • Note that as shown in the flow chart above, data of chip1 and chip2 should be reduced separately (before stacking together).
   • Data taken within the same dithering set are grouped by '2_DSETID' header keyword. Using this, you can make a input list like follows,

     import glob
     import astropy.io.fits
     
     dct1 = {}
     dct2 = {}
     
     infiles = glob.glob("SWS*fits")
     infiles.sort()
     
     for infile in infiles:
         hdr = astropy.io.fits.getheader(infile)
     
         ch = hdr["2_DETCH"]
         if ch == 1:
             dct = dct1
         else:
             dct = dct2
     
         if hdr["2_DSETID"] not in dct.keys():
             dct[hdr["2_DSETID"]] = []
     
         dct[hdr["2_DSETID"]].append(infile)
     

2. Make filenames of intermediate files.

   > import swsred.make_file_lst
   > file_lst1 = swsred.make_file_lst.make_file_lst(in_arr1)
   

   • 'file_lst1' is a dictionary which defines various filenames such as flat-fielded FITS filename. You can edit any values as you want.
     • For example, you get the following result with an input ["data/path/SWSB00000001.fits"] as in_arr1,

       Key	Value	Description
       in	data/path/SWSB00000001.fits	input filename
       ff	ffSWSB00000001.fits	filename of flat-fielded data
       ms	msSWSB00000001.fits	filename of self-sky data
       ss	ssSWSB00000001.fits	filename of sky-subtracted data
       wc	wcSWSB00000001.fits	filename of WCS-corrected data
       bpm	/SWSRED_DIRECTORY/calib/bpm/bpm_sXXY_b1.fits	filename of Bad-pixel mask (XXY denotes a semester)
       wht	SWSB00000001_wht.fits	filename of weight map (created by make_weight_map module)
       omsk	SWSB00000001_omsk.fits	filename of object mask
       ldac	ssSWSB00000001.ldac	filename of SExtractor LDAC catalog
       ascii	ssSWSB00000001.ascii	filename of SExtractor ASCII catalog
       head	ssSWSB00000001.head	filename of Scamp HEAD file

1. Then, start your data reduction. For example, to perform flat-fielding,

   > import swsred.flat_field
   > swsred.flat_field.flat_field(file_lst1["in"], file_lst1["ff"])
   

   • By default, the values 'file_lst1["ff"]' do not have a path which means that 'file_lst1["ff"]' files will be created on the current directory. If you want to specify any directory to put intermediate files, edit and add your path in 'file_lst1'.

Refer to swsred/reduce_all.py for all the procedures.

Questions & Answers †

1. Astrometry (performed by scamp) is failed or is not good. How do I work for it?
   • Try changing the following parameters,
     • detect_thresh_wcs: detection threshold to make a source catalog used in scamp.
     • mag_max_wcs: maximum magnitude of Pan-STARRS sources to be used in scamp.
   • You may want to set "use_scamp=False" not to use Scamp; in that case (still in beta), the swsred try to find stars in the image around reference catalog positions and calculate WCS parameters.
2. How is the semester determined?
   • FRAMEID is used to determine when the data has been taken. You can find the definition of which data was taken in which semester in swsred/config/config.ini.
   • If swsred would assign the wrong semester, check if you are using the latest swsred (i.e., latest definition).

Known bugs †

Changes †

Version 2021.01.31 †

• First release

Contact †

Please send your bug reports or questions to konishi__at__ioa.s.u-tokyo.ac.jp.