MIRAGE Quick Start

Table of Contents:

• Getting Started

• Create ``mirage` input yaml files from an APT file <#make_yaml>`__

• Single image simulation

  • Create simulation with one command

  • Running simulator steps independently

• Running multiple simulations

  • Running in series

  • Running in parallel

• Example observation list file

• Example yaml file

---

Getting Started

Important: Before proceeding, ensure you have set the MIRAGE_DATA environment variable to point to the directory that contains the reference files associated with MIRAGE.

import os

# For examining outputs
from glob import glob
from scipy.stats import sigmaclip
import numpy as np
from astropy.io import fits
from astropy.visualization import simple_norm
import matplotlib.pyplot as plt
%matplotlib inline

# Import top level functions
from mirage import imaging_simulator
from mirage.apt import apt_inputs
from mirage.yaml import yaml_generator

# Import the three steps of the simulator.
from mirage.seed_image import catalog_seed_image
from mirage.dark import dark_prep
from mirage.ramp_generator import obs_generator

---

### Create mirage input yaml files from an APT file

For convenience, observing programs with multiple pointings and detectors can be simulated starting with the program’s APT file. The xml and pointings files must be exported from APT, and are then used as input into a tool that will generate a series of yaml input files.

apt_xml_file = 'my_apt_file.xml'
apt_pointing_file = 'my_apt_file.pointing'

catalogs = {'nircam': {'sw': ['nrc_ptsrc_1.cat', 'nrc_ptsrc_2.cat'],
                       'lw': ['nrc_ptsrc_lw_1.cat', 'nrc_ptsrc_lw_2.cat']},
            'niriss': ['niriss_ptsrc_1.cat', 'niriss_ptsrc_2.cat']}
backgrounds = ['low', 'medium']

# Create a series of data simulator input yaml files
# from APT files
yam = yaml_generator.SimInput()

# Point to the xml and pointing files from your APT proposal
yam.input_xml = apt_xml_file
yam.pointing_file = apt_pointing_file

# Output directory for the collection of yaml files
yam.output_dir = './yaml_files'

# Output directory for the simulated observations
yam.simdata_output_dir = './imaging_data/'

# Observation table that lists the source catalogs to use with each filter/observation
yam.observation_table = observation_list_file

# Output data type. "raw", "linear", or "linear,raw" for both
yam.datatype = 'raw'

# Optional parameters
yam.use_JWST_pipeline = True
yam.use_linearized_darks = True

# Create the yaml files
yam.reffile_setup()
yam.create_inputs()

# Look at the list of files generated
yaml_files = glob('./yaml_files/*yaml')

---

## Create simulation with one command

This will take several minutes to run. For a better idea of what Mirage is doing, skip down several cells to the Running simulator steps independently section

The imaging_simulator function will run all three steps of the simulator. This convenience function is useful when creating simulated imaging mode data. WFSS data will need to be run in a slightly different way.

test_yaml_file = yaml_files[0]

# imaging_simulator is a wrapper around all 3 steps of Mirage
img_sim = imaging_simulator.ImgSim()
img_sim.paramfile = test_yaml_file
img_sim.create()

---

## More detail on what’s going on: Running simulation steps independently

First generate the “seed image”

This is generally a 2D noiseless countrate image that contains only simulated astronomical sources.

A seed image is generated based on a .yaml file that contains all the necessary parameters for simulating data. An example .yaml file is shown at the bottom of this notebook.

# Generate the seed image
cat = catalog_seed_image.Catalog_seed()
cat.paramfile = test_yaml_file
cat.make_seed()

Look at the seed image

# Need to flip the image vertically in order to match what ds9 would show
def show(array, title ,min=None, max=None):
    plt.figure(figsize=(12,12))
    if min is None and max is None:
            norm = simple_norm(array, 'log', percent=99)
    else:
        if min is None:
            min = np.min(array)
        if max is None:
            max = np.max(array)
        norm = simple_norm(array, 'log', min_cut=min, max_cut=max)
    plt.imshow(array[::-1,:], norm=norm)
    plt.title(title)
    plt.colorbar().set_label('DN$^{-}$/s')

show(cat.seedimage,'Seed Image', max=100, min=0)

Prepare the dark current exposure

This will serve as the base of the simulated data. This step will linearize the dark current (if it is not already), and reorganize it into the requested readout pattern and number of groups.

d = dark_prep.DarkPrep()
d.paramfile = test_yaml_file
d.prepare()

Look at the dark current

# For this, we will look at an image of the final group minus the first group
exptime = d.linDark.header['NGROUPS'] * cat.frametime
diff = (d.linDark.data[0,-1,:,:] - d.linDark.data[0,0,:,:]) / exptime
show(diff,'Dark Current Countrate', max=0.1, min=0)

Create the final exposure

Turn the seed image into a exposure of the proper readout pattern, and combine it with the dark current exposure. Cosmic rays and other detector effects are added.

The output can be either this linearized exposure, or a ‘raw’ exposure where the linearized exposure is “unlinearized” and the superbias and reference pixel signals are added, or the user can request both outputs. This is controlled from within the yaml parameter file.

obs = obs_generator.Observation()
obs.linDark = d.prepDark
obs.seed = cat.seedimage
obs.segmap = cat.seed_segmap
obs.seedheader = cat.seedinfo
obs.paramfile = test_yaml_file
obs.create()

Examine the final output image

Again, we will look at the last group minus the first group

with fits.open(obs.raw_output) as h:
    lindata = h[1].data
    header = h[0].header

exptime = header['EFFINTTM']
diffdata = (lindata[0,-1,:,:] - lindata[0,0,:,:]) / exptime
show(diffdata,'Simulated Data',min=0,max=20)

If you have multiple exposures that will use the same dark current image (with the same readout pattern, subarray size, and number of groups), you can feed the output from the initial run of dark_prep into future runs of the obs_generator, to save time. This can be accomplished with the imaging_simulator.py code, as shown below. (Note that time savings are minimal in this case, where the readout pattern is RAPID and there are only a handful of groups. This means that no averaging/skipping of frames has to be done within dark_prep.py)

# Now that the linearized dark product has been created, if you want to use it
# when running the simulator with a different yaml file (or repeating the run
# with the same yaml file) you can provide the filename of the dark product, and the
# dark_prep step will be skipped.
# NOTE: if you use the same dark product for multiple exposures, those exposures
# will contain exactly the same dark signal. This may or may not be advisable, depending
# on your goals for the simulated data.
img_sim_same_dark = imaging_simulator.ImgSim()
img_sim_same_dark.paramfile = second_yaml_file
img_sim_same_dark.override_dark = 'jw44444001001_01101_00001_nrcb1_uncal_linear_dark_prep_object.fits'
img_sim_same_dark.create()

---

## Running Multiple Simulations

Each yaml file will simulate an exposure for a single pointing using a single detector.

### Function to simulate multiple detectors/pointings in series

def make_sim(paramlist):
    '''Function to run many simulations in series
    '''
    for file in paramlist:
        m = imaging_simulator.ImgSim()
        m.paramfile = file
        m.create()

# THIS WILL RUN ALL SIMULATIONS AND CAN TAKE A LONG TIME DEPENDING ON HOW MANY YAML FILES YOU HAVE!!
for yfile in yaml_files:
    make_sim(yfile)

### Function to simulate multiple detectors/pointings in parallel

from multiprocessing import Pool

n_procs = 3 # number of cores available

with Pool(n_procs) as pool:
    pool.map(make_sim, yaml_files)

---

## Example yaml input file

Entries listed as ‘config’ have default files that are present in the config directory of the repository. The scripts are set up to automatically find and use these files. The user can replace ‘config’ with a filename if they wish to override the default.

In general, if ‘None’ is placed in a field, then the step that uses that particular file will be skipped.

Note that the linearized_darkfile entry overrides the dark entry, unless linearized_darkfile is set to None, in which case the dark entry will be used.

Inst:
  instrument: NIRCam          #Instrument name
  mode: imaging               #Observation mode (e.g. imaging, WFSS, moving_target)
  use_JWST_pipeline: False   #Use pipeline in data transformations

Readout:
  readpatt: RAPID  #Readout pattern (RAPID, BRIGHT2, etc) overrides nframe,nskip unless it is not recognized
  ngroup: 3        #Number of groups in integration
  nint: 1          #Number of integrations per exposure
  array_name: NRCB5_FULL    #Name of array (FULL, SUB160, SUB64P, etc)
  filter: F250M       #Filter of simulated data (F090W, F322W2, etc)
  pupil: CLEAR        #Pupil element for simulated data (CLEAR, GRISMC, etc)

Reffiles:                                 #Set to None or leave blank if you wish to skip that step
  dark: None   #Dark current integration used as the base
  linearized_darkfile: $MIRAGE_DATA/nircam/darks/linearized/B5/Linearized_Dark_and_SBRefpix_NRCNRCBLONG-DARK-60090141241_1_490_SE_2016-01-09T02h46m50_uncal.fits # Linearized dark ramp to use as input. Supercedes dark above
  badpixmask: $MIRAGE_DATA/nircam/reference_files/badpix/NRCB5_17161_BPM_ISIMCV3_2016-01-21_ssbspmask_DMSorient.fits # If linearized dark is used, populate output DQ extensions using this file
  superbias: $MIRAGE_DATA/nircam/reference_files/superbias/NRCB5_superbias_from_list_of_biasfiles.list.fits  #Superbias file. Set to None or leave blank if not using
  linearity: $MIRAGE_DATA/nircam/reference_files/linearity/NRCBLONG_17161_LinearityCoeff_ADU0_2016-05-22_ssblinearity_v2_DMSorient.fits    #linearity correction coefficients
  saturation: $MIRAGE_DATA/nircam/reference_files/saturation/NRCB5_17161_WellDepthADU_2016-03-10_ssbsaturation_wfact_DMSorient.fits    #well depth reference files
  gain: $MIRAGE_DATA/nircam/reference_files/gain/NRCB5_17161_Gain_ISIMCV3_2016-02-25_ssbgain_DMSorient.fits #Gain map
  pixelflat: None
  illumflat: None                               #Illumination flat field file
  astrometric: $MIRAGE_DATA/nircam/reference_files/distortion/NRCB5_FULL_distortion.asdf  #Astrometric distortion file (asdf)
  ipc: $MIRAGE_DATA/nircam/reference_files/ipc/NRCB5_17161_IPCDeconvolutionKernel_2016-03-18_ssbipc_DMSorient.fits #File containing IPC kernel to apply
  invertIPC: True       #Invert the IPC kernel before the convolution. True or False. Use True if the kernel is designed for the removal of IPC effects, like the JWST reference files are.
  occult: None                                    #Occulting spots correction image
  pixelAreaMap: $MIRAGE_DATA/nircam/reference_files/pam/NIRCam_B5_PAM_imaging.fits #Pixel area map for the detector. Used to introduce distortion into the output ramp.
  subarray_defs:   config   #File that contains a list of all possible subarray names and coordinates
  readpattdefs:    config   #File that contains a list of all possible readout pattern names and associated NFRAME/NSKIP values
  crosstalk:       config   #File containing crosstalk coefficients
  filtpupilcombo:  config   #File that lists the filter wheel element / pupil wheel element combinations. Used only in writing output file
  flux_cal:        config   #File that lists flux conversion factor and pivot wavelength for each filter. Only used when making direct image outputs to be fed into the grism disperser code.

nonlin:
  limit: 60000.0                           #Upper singal limit to which nonlinearity is applied (ADU)
  accuracy: 0.000001                        #Non-linearity accuracy threshold
  maxiter: 10                              #Maximum number of iterations to use when applying non-linearity
  robberto:  False                         #Use Massimo Robberto type non-linearity coefficients

cosmicRay:
  path: $MIRAGE_DATA/nircam/cosmic_ray_library/         #Path to CR library
  library: SUNMIN    #Type of cosmic rayenvironment (SUNMAX, SUNMIN, FLARE)
  scale: 1.5     #Cosmic ray scaling factor
  suffix: IPC_NIRCam_B5    #Suffix of library file names
  seed: 2956411739      #Seed for random number generator

simSignals:
  pointsource: my_point_sources.cat #File containing a list of point sources to add (x,y locations and magnitudes)
  psfpath: $MIRAGE_DATA/nircam/webbpsf_library/        #Path to PSF library
  psfbasename: nircam                        #Basename of the files in the psf library
  psfpixfrac: 0.25                           #Fraction of a pixel between entries in PSF library (e.g. 0.25 = files for PSF centered at 0.25 pixel intervals within pixel)
  psfwfe: predicted                               #PSF WFE value ("predicted" or "requirements")
  psfwfegroup: 0                             #WFE realization group (0 to 4)
  galaxyListFile: my_galaxies_catalog.list
  extended: None          #Extended emission count rate image file name
  extendedscale: 1.0         #Scaling factor for extended emission image
  extendedCenter: 1024,1024  #x,y pixel location at which to place the extended image if it is smaller than the output array size
  PSFConvolveExtended: True #Convolve the extended image with the PSF before adding to the output image (True or False)
  movingTargetList: None   #Name of file containing a list of point source moving targets (e.g. KBOs, asteroids) to add.
  movingTargetSersic: None  #ascii file containing a list of 2D sersic profiles to have moving through the field
  movingTargetExtended: None      #ascii file containing a list of stamp images to add as moving targets (planets, moons, etc)
  movingTargetConvolveExtended: True       #convolve the extended moving targets with PSF before adding.
  movingTargetToTrack: None #File containing a single moving target which JWST will track during observation (e.g. a planet, moon, KBO, asteroid)   This file will only be used if mode is set to "moving_target"
  zodiacal:  None                          #Zodiacal light count rate image file
  zodiscale:  1.0                            #Zodi scaling factor
  scattered:  None                          #Scattered light count rate image file
  scatteredscale: 1.0                        #Scattered light scaling factor
  bkgdrate: 0.0                         #Constant background count rate (electrons/sec/pixel)
  poissonseed: 2012872553                  #Random number generator seed for Poisson simulation)
  photonyield: True                         #Apply photon yield in simulation
  pymethod: True                            #Use double Poisson simulation for photon yield

Telescope:
  ra: 53.1                     #RA of simulated pointing
  dec: -27.8                   #Dec of simulated pointing
  rotation: 0.0                #y axis rotation (degrees E of N)

newRamp:
  dq_configfile: config          #config file used by JWST pipeline
  sat_configfile: config         #config file used by JWST pipeline
  superbias_configfile: config   #config file used by JWST pipeline
  refpix_configfile: config      #config file used by JWST pipeline
  linear_configfile: config      #config file used by JWST pipeline

Output:
  file: jw44444024002_01101_00001_nrcb1_uncal.fits   #Output filename
  directory: ./   # Directory in which to place output files
  datatype: linear,raw # Type of data to save. 'linear' for linearized ramp. 'raw' for raw ramp. 'linear,raw' for both
  format: DMS          #Output file format Options: DMS, SSR(not yet implemented)
  save_intermediates: False   #Save intermediate products separately (point source image, etc)
  grism_source_image: False   # Create an image to be dispersed?
  unsigned: True   #Output unsigned integers? (0-65535 if true. -32768 to 32768 if false)
  dmsOrient: True    #Output in DMS orientation (vs. fitswriter orientation).
  program_number: 44444    #Program Number
  title: Supernovae and Black Holes Near Hyperspatial Bypasses   #Program title
  PI_Name: Doug Adams  #Proposal PI Name
  Proposal_category: GO  #Proposal category
  Science_category: Cosmology  #Science category
  observation_number: '024'    #Observation Number
  observation_label: Obs2    #User-generated observation Label
  visit_number: '002'    #Visit Number
  visit_group: '01'    #Visit Group
  visit_id: '42424024002'    #Visit ID
  sequence_id: '1'    #Sequence ID
  activity_id: '01'    #Activity ID. Increment with each exposure.
  exposure_number: '00001'    #Exposure Number
  obs_id: 'V44444024002P0000000001101'   #Observation ID number
  date_obs: '2019-10-15'  #Date of observation
  time_obs: '06:29:11.852'  #Time of observation
  obs_template: 'NIRCam Imaging'  #Observation template
  primary_dither_type: NONE  #Primary dither pattern name
  total_primary_dither_positions: 1  #Total number of primary dither positions
  primary_dither_position: 1  #Primary dither position number
  subpix_dither_type: 2-POINT-MEDIUM-WITH-NIRISS  #Subpixel dither pattern name
  total_subpix_dither_positions: 2  #Total number of subpixel dither positions
  subpix_dither_position: 2  #Subpixel dither position number
  xoffset: 344.284  #Dither pointing offset in x (arcsec)
  yoffset: 466.768  #Dither pointing offset in y (arcsec)