MIRAGE: Multi-Instrument RAmp GEnerator¶
Overview¶
Mirage is a Python package that creates realistic simulated data products for JWST’s NIRCam, NIRISS, and FGS instruments. The output files are in a format that is as close as possible to that which real JWST data will have, which allows Mirage data to be used for data analysis software development and testing, including the official JWST calibration pipeline.
Mirage creates simulated exposures for NIRCam’s imaging and wide field slitless spectroscopy (WFSS) modes, NIRISS’s imaging, WFSS, and aperture masking interferometery (AMI) modes, and FGS’s imaging mode. It supports sidereal as well as non-sidereal tracking (i.e. sources can be made to move across the field of view within an observation).
Astronomical sources to add to the scene are specified through the use of source catalogs.
The software is designed such that a single yaml file can be used as input to create a single simulated exposure from one detector. This file contains various instrument and readout parameters, as well as lists reference files necessary for the production of the simulated data. Details on the contents of the yaml file are given on the Example Yaml page. One yaml file contains all the information necessary to produce a simulated exposure from a single detector.
For software developement and testing, users may wish to simulate all data from a given JWST proposal. In order to do this, Mirage requires one yaml file for each exposure/detector combination. To facilitate this, the Mirage package includes ancillary convenience functions that allow the user to translate an Astronomer’s Proposal Tool (APT) proposal into a series of input yaml files. These tools are described in detail on the Simulating Observations from an APT File page.
Basic Workflow¶
The basic order of steps necessary to produce simulated data with Mirage is:
1. Create the appropriate Source Catalogs. Mirage contains tools to aid in the creation of source catalogs.
2. Create the needed yaml files to descirbe the observations. An easy way to do this is to start with an APT proposal and use Mirage’s yaml file generator tool.
3. Create the simluated data. Run the appropriate module. Most often this will be the imaging simulator or the wfss simulator. To produce only noiseless, background-free countrate images of the scene, the seed image generator module can be used.
See the notebooks in the Mirage repository for basic examples of each of these situations. These include the:
Catalog generation notebook Imaging simulator example notebook WFSS simulator example notebook Moving Target notebook
and for use with APT programs:
APT program data generation notebook
Under the Hood¶
Broadly speaking, Mirage creates simulated data using three main steps. These include the creation of a noiseless “seed image” of the scene, which contains signal only from astronomical sources. Secondly, a dark current exposure is modified to fit the requested readout pattern. And finally, the seed image and dark current exposures are combined, and noise/cosmic rays are added. See the Three Stages page for details.
Limitations¶
Mirage currently does not simulate coronagraphic observations directly. However the user can input an image of an occulted PSF using the extended source catalog.
When adding PSFs to simulated data, MIRAGE uses the PSFs from a pre-computed library. The libraries currently provided in the Mirage reference files collection contain fits files that are several hundred pixels on a side. This limited size keeps the file sizes small, but cuts off the wings of the PSFs. For very bright sources, the truncated wings will be visible in the data. In addition, when MIRAGE normalizes a PSF to the requested brightness, signal that should be in the truncated wings will instead be in the pixels of the truncated PSF. The total brightness of the source will be as expected, but the signal will be slightly more concentrated in the center of the PSF than it should be.
When creating data for a target that is moving through the field of view, (e.g. a non-sidereal target in a sidereal observation, or vice versa) the velocity of the target is constant. Speed and direction cannot change within an exposure.
Source brightnesses are constant. There is currently no way to simulate a variable brightness source other than by creating multiple exposures of the same source and varying the source brightness from one exposure to the next.
The yaml_generator function currently is only able to parse a subset of all available APT templates. See the yaml generator page for details. If your proposal contains observations that use unsupported APT templates, the easiest work-around at the moment is to make a copy of your APT file and strip out the unsupported observations.
Getting Help
For help installing or running Mirage, or if something is not clear in this documentation, please open an issue on the Mirage github page.
Installing MIRAGE¶
There are two aspects to Mirage installation. First, the software itself must be installed. Once this is complete, there is a set of reference files which must be downloaded. The preferred installation method is via Pypi, as this is the latest stable version of the software.
Tip
Mirage currently supports python 3.8 and 3.9. Support for 3.6 and 3.7 has been removed, as the jwst package, which contains the JWST calibration pipeline, no longer supports python < 3.8.
Attention
For those running Mac OSX 10.14:
Some users have reported errors when installing Mirage on their machines running Mac OSX 10.14. If you see installation failures for the synphot or batman packages, they are most likely related to the OpenMP library. See the section below on Troubleshooting for Mac OSX 10.14 installtion.
Install from Pypi¶
Mirage is hosted on Pypi. To install the latest stable version of Mirage, use the commands below. In this example, we create a conda environment called “mirage” and then install the software into that environment.
conda create -n mirage python=3.9 -y
conda activate mirage
pip install mirage
Tip
Some of Mirage’s dependencies rely on Healpy,. Healpy has released different wheels for different versions of Mac OSX. For example, healpy version 1.12.5 works for MacOSX 10.13 (High Sierra) and 1.14.0 works for MacOSX 10.15 (Catalina). If the version of healpy above does not work for your system, you may need to install a different version.
Tip
This method installs webbpsf via pip. In this case, you must also manually download the collection of webbpsf data files If you install webbpsf via conda, the data files are downloaded and installed for you, however conda installation is currrently only supported for python 3.7 and below.
Install the Development Version¶
For those wishing to use the most up-to-date version of Mirage, you can install directly from github. In general, the version in the master branch should work, although that is not guaranteed. By installing this version, you will have access to updates made since the latest release on Pypi.
The installation procedure is nearly identical to the case of installing from Pypi above. The only difference is that the install command points to github.
conda create -n mirage python=3.9 -y
conda activate mirage
pip install git+https://github.com/spacetelescope/mirage
Tip
Some of Mirage’s dependencies rely on Healpy,. Healpy has released different wheels for different versions of Mac OSX. For example, healpy version 1.12.5 works for MacOSX 10.13 (High Sierra) and 1.14.0 works for MacOSX 10.15 (Catalina). If the version of healpy above does not work for your system, you may need to install a different version.
Tip
This method installs webbpsf via pip. In this case, you must also manually download the collection of webbpsf data files If you install webbpsf via conda, the data files are downloaded and installed for you, however conda installation is currrently only supported for python 3.7 and below.
Installation for Developers¶
For those wishing to contribute to the code base, you can install Mirage by cloning and installing the repository. This is only recommended for those looking to help with development. In general, those wishing only to use Mirage should install the latest stable version from Pypi.
Clone the Mirage repository:
git clone https://github.com/spacetelescope/mirage.git
Installation can then be done via pip, which uses setup.py, or using the conda environment file that is included in the package.
To install using pip and setup.py: Create and activate a new environment. In this example we call the environment “mirage”. Then move into the mirage directory, and install Mirage into the new environment:
conda create -n mirage python=3.9 -y
conda activate mirage
cd mirage
pip install .
Tip
Some of Mirage’s dependencies rely on Healpy,. Healpy has released different wheels for different versions of Mac OSX. For example, healpy version 1.12.5 works for MacOSX 10.13 (High Sierra) and 1.14.0 works for MacOSX 10.15 (Catalina). If the version of healpy above does not work for your system, you may need to install a different version.
Tip
This method installs webbpsf via pip. In this case, you must also manually download the collection of webbpsf data files If you install webbpsf via conda, the data files are downloaded and installed for you, however conda installation is currrently only supported for python 3.7 and below.
Install via Environment File¶
The Mirage repository also contains environment files, which can be used to create an environment with proper versions of all of Mirage’s dependencies. After cloning the Mirage repository, the environment file (located within the top-level directory) can be used via the following commands. The name keyword is used to specify that the name of the environment. You can name the environment anything you like.
Create a python 3.8 environment using the environment file, activate the environment, and install mirage:
cd mirage
conda env create -f environment_python_3.8.yml
conda activate mirage_py3.8
pip install .
There is also an environment file that can be used to create python 3.9 environment:
cd mirage
conda env create -f environment_python_3.9.yml
conda activate mirage_py3.9
pip install .
Tip
For the python 3.8 and 3.9 cases most packages, including webbpsf, are installed via pip (astroconda does not yet support python 3.8 and beyond). In this case you must manually download the collection of webbpsf data files.
Troubleshooting for Mac OSX 10.14 installtion¶
If you have installation errors on your machine running 10.14 (Mojave), try these solutions.
Synphot¶
If the synphot package fails to build, try installing via conda using the conda-forge channel. Do this before installing Mirage, using the command:
conda install synphot -c conda-forge
Batman¶
If the Batman package fails to build, the work-around is more complex. Mirage uses the Batman package when simulating imaging and grism Time Series Observations (TSO).
The installation errors are related to supporting Batman’s ability to run calculations in parallel. There are two options for modifying the installation, which are described in this Batman issue on github
If you do want to make use of parallel processing (or simply want to try the less invasive installation fix), you must install LLVM and OpenMP on your machine prior to installing Mirage. See this StackOverflow issue for details. If you successfully install these, then you should be able to install Mirage following the instructions in the sections above.
If you do not wish to use parallel processing within Batman, or the option above fails, then you can modify Batman such that it does not use parallel processing. This involves modifying the Batman and Mirage setup.py files and install using those. Clone the Batman package, open its setup.py file, and remove “-fopenmp”. Then you must clone Mirage and remove Batman from Mirage’s environment.yml and setup.py files. Then create the environment using environment.yml, pip install the local copy of Batman, and pip install the local copy of Mirage.
cd mirage conda env create -f environment_python_3.9.yml conda activate mirage pip install . cd ../batman pip install .
If you are having installtion problems and will not be creating TSO simulations, you could skip Batman installation altogether. In this case you will still need to clone Mirage and remove Batman from the environment.yml and setup.py files. Then install Mirage via the environment file.
Reference Files and MIRAGE_DATA Environment Variable¶
In addition to the code itself, there is a set of reference files that accompany Mirage, and are necessary for Mirage to function. These files include dark current ramps and cosmic ray and PSF libraries.
Instructions for downloading the reference files are provided on the reference files page.
Reference Files¶
NOTE to users internal to STScI: a collection of Mirage reference files is already present on the STScI network, so there is no need to download them. Please email Bryan Hilbert for the path to use for your MIRAGE_DATA environment variable.
In order to produce data that is as realistic as possible, Mirage is accompanied by a set of reference files that are used to construct the simulated data.
After installing Mirage, these reference files can be downloaded using the downloader module. As the collection of reference files is quite large, users have the option of downloading only certain subsets of the files. For example, if you will only be simulating data from one instrument, you can download only the reference files needed for that instrument. You can also choose to download only certain types of reference files in a given call to the downloader. In this way, you can break up the download into multiple, smaller calls to the downloader if desired. The basic commands to download reference files are shown below. The downloader first creates a list of files to download given the input parameters. Before attempting to download a particular file, the script first checks if the file is already present in the requested local directory. If the file is present, the download of that file is skipped. Details on calling the downloader are provided below.
from mirage.reference_files import downloader
download_path = '/path/into/which/files/are/downlaoded/'
downloader.download_reffiles(download_path, instrument='all', dark_type='linearized', skip_darks=False, single_dark=False, skip_cosmic_rays=False, skip_psfs=False)
The instrument keyword controls which subset of reference files are downloaded. You can give it the name of a single instrument, a string containing a comma-separated list of instruments, or the string all, which will download reference files for NIRCam, NIRISS, and FGS.
The dark_type keyword controls which dark current exposures are downloaded. Mirage requires linearized dark current exposures when creating simulated data. A user may provide raw dark current files, which Mirage will linearize on the fly, or linearized dark current files, which will save processing time. Set this keyword to linearized to download only the linearized versions of the darks, raw to download only the raw versions, or both for both. If omitted by the user, the script will default to downloading only the linearized darks. Note that the darks are by far the largest reference files in the collection (3GB per file, with 5 files per NIRCam detector, 20 files for NIRISS, and 8 files for FGS) and will take the most time to download.
If True, the skip_dark parameter will cause the script not to download the dark current files for the given instrument. Similarly, the skip_cosmic_rays and skip_psfs parameters, if True, will cause the script to skip downloading the cosmic ray library and PSF library, respectively, for the indicated instruments. The default for all three of these parameters is False.
If True, the single_dark parameter will cause the script to download only a single dark current file for each detector of the selected instruments. Downloading only a single dark rather than the full collection of darks will save significant time and disk space. This is intended as a quick-start of sorts, for those interested in testing Mirage. However, we recommend against creating a large amount of simulated data all based on only a single dark for each detector.
When called, the function will download the appropriate files from the STScI Box repository, unzip the files, and create the directory structure Mirage expects. It will then remind you to point your MIRAGE_DATA environment variable to the top-level location of these files, so that Mirage knows where to find them. You may wish to add this definition to your .bashrc or .cshrc file.
For example:
export MIRAGE_DATA="/my_files/jwst/simulations/mirage_data"
CRDS Environment Variables¶
In addition to the MIRAGE_DATA environment variable, there are two environment variables required by the Calibration References Data System (CRDS) that should be set. These will be used when Mirage queries CRDS for the appropriate JWST calibration reference files for the observations being simulated. While setting these environment variables before running Mirage is not strictly required, it is recommended in order to avoid confusion when CRDS is queried. The two environment variables are CRDS_PATH and CRDS_SERVER_URL. CRDS_PATH should be set to the directory where you would like JWST calibration reference files to be stored. If not set, Mirage will set this to the CRDS default location of $HOME/crds_cache. CRDS_SERVER_URL must be set to the value below.
export CRDS_PATH=$HOME/crds_cache
export CRDS_SERVER_URL=https://jwst-crds.stsci.edu
Reference File Contents¶
There are three main groups of reference files for each instrument: dark current exposures, a PSF library, a cosmic ray library. Note that users can run Mirage with alternate reference files. For example, if a specific science case requires larger PSFs, the user can create a new PSF library and replace the downloaded PSF library, or simply update the psf_path entry in their Mirage input yaml files to point to the new library. Similarly, to use a non-standard JWST calibration reference file, the appropriate entry in the yaml input file can be changed to point to the new reference file.
Source Catalog File Formats¶
Mirage accepts 7 different types of ascii source catalogs, all of which can be generated using Mirage’s catalog generation functionality. Each of these are discussed in the sections below.
Tip
For information on the optional hdf5 catalog that can be used when simulating WFSS data, see the WFSS simulation page.
The type of source catalog(s) you need depends on the scene you are simulating. The table below shows the possible catalogs for sidereal and non-sidereal observations. For sidereal observations, the point source, galaxy, and extended source catalogs can be used to create static sources in the scene. The three types of moving target catalogs can be used to create sources that trail across the field of view during the observation. For non-sidereal observations, the non-sidereal source catalog is used to create a static source in the scene. This is the non-sidereal target which JWST is tracking. The other types of catalogs can then all be used to create trailed sources. In this case, the inverse of the non-sidereal source’s velocity is applied to all targets in the other catalogs. In this way, Mirage will create trailed background stars and galaxies, as well as trailed non-sidereal targets that have velocities other than that of the main target.
Type of observation |
Static source catalogs |
Trailed (moving) source catalogs |
|---|---|---|
Sidereal |
|
|
Non-sidereal |
|
|
Grism TSO |
|
|
Imaging TSO |
|
|
Common formatting details¶
Position and Velocity Units¶
Mirage scans the top 4 lines of each catalog for certain phrases that can be used to characterize the inputs. These phrases are used to specify the units of source locations or velocities, as well as the magnitude system to use. Multiple phrases can be used in a single catalog, but only one phrase per line is allowed. Examples are shown in the catalogs below.
The locations of sources can be specified in RA, Dec or in (x,y) pixel locations on the detector. If you wish to provide positions in units of (x, y) detector pixels, then the string ‘position_pixels’ must be added after the # in one of the top 4 lines of the file.
Tip
RA and Dec values can be given in decimal degrees, colon-separated values (HH:MM:SS and DD:MM:SS), or in more conventional string formats, but all sources in a given catalog must have the same format:
53.08864 -27.83999
03:32:21.273 -27:50:23.983
03h32m21.273s -27d50m23.983s
Mirage uses AB magnitudes as the default for input sources. However, you can change the magnitude system by specifying an alternative in one of the top 4 lines. The three acceptible options are vegamag, stmag, and abmag. All sources in a given catalog must be in the same magnitude system.
For moving targets (both those that are moving across the field of view, as well as non-sidereal targets), the default unit for velocity is arcseconds per hour. If you wish to instead use pixels per hour, then velocity_pixels must be added to one of the 4 top lines of the catalog.
Source Index Numbers¶
When using Mirage’s catalog_generator.py to create source catalogs, the left-most column of the catalogs is an “index” column, which labels each source with a unique number. This can be useful when comparing source catalogs with segmentation images. By default, Mirage will begin counting at 1 and increase values from there. However, if you are planning to produce a simulation using multiple catalogs (e.g. a point source catalog and a galaxy catalog), then you must be sure that the source index numbers do not overlap between the two catalogs. To prevent having repeated source indexes, you can set the minimum index value for each catalog when calling the catalog generator by using the ‘starting_index’ keyword. In the example below, the point source catalog index column starts at 1 (the default), while the galaxy catalog index column starts at 4, since the values of 1, 2, and 3 are present in the point source catalog.
from mirage.catalogs.catalog_generator import PointSourceCatalog, GalaxyCatalog
ra = [1.1, 1.12, 1.13]
dec = [45.2, 45.25, 45.3]
mags = [18.1, 17.3, 19.3]
ptsrc = PointSourceCatalog(ra=ra, dec=dec)
ptsrc.add_magnitude_column(mags, magnitude_system='abmag', instrument='nircam',
filter_name='f200w')
ptsrc.save('my_point_sources.cat')
gal_ra = [1.13, 1.15, 1.17]
gal_dec = [45.25, 45.26, 45.32]
ellipticity = [0.1, 0.05, 0.4]
radius = [0.02, 0.1, 0.1]
sersic_index = [1.0, 3.4, 1.5]
position_angle = [5.2, 10.5, 127.8]
gal = GalaxyCatalog(ra=gal_ra, dec=gal_dec, ellipticity=ellipticity, radius=radius,
sersic_index=sersic_index, position_angle=position_angle,
starting_index=4)
gal.add_magnitude_column(mags, magnitude_system='abmag', instrument='nircam',
filter_name='f200w')
gal.save('my_galaxies.cat')
Stamp Images for Optical Ghosts¶
Users may also optionally provide a list of fits files containing images to be used when creating optical ghosts. See the Addition of Ghosts to Mirage Simulations page for details on adding this column to a source catalog and how Mirage’s behavior will change if this column is present.
Point Sources¶
Point sources are specified using a catalog that includes the locations of the sources in RA and Dec (or x,y pixel locations on the detector) and the corresponding magnitudes through the filter or filter/pupil combination specified by the user. Currently the simulator supports the use of ABMAG [Oke, 1983]_, STMAG [Stone, 1996]_ , and VEGAMAG () systems, with ABMAG as the default.
An example point source catalog is shown below with the positions given in RA and Dec.
#
#
#
# abmag
#
# Magnitudes are converted from input flux densities.
index x_or_RA y_or_Dec nircam_f200w_clear_magnitude
1 53.0886395 -27.8399952 20.0
2 53.0985009 -27.8398137 19.2
Mirage looks for the exact column names shown above when reading point source catalogs. Changing these column names will cause the simulator to fail when attempting to read in the file.
Galaxies (aka 2D Sersic profiles)¶
Below is an example of a galaxy source catalog. In this case, galaxy positions are given in RA, Dec decimal degrees, and the half light radii are in units of arcseconds. The half light radius can also be specified in units of pixels. In that case, you must add radius_pixels after the # in one of the top four lines.
Position angle values are defined as degrees east of north.
The simulator software looks for the exact column names shown below when reading these catalogs. Changing these column names will cause the simulator to fail when attempting to read in the file.
#
#
#
# abmag
#
# Magnitudes are converted from input flux densities.
index x_or_RA y_or_Dec radius ellipticity pos_angle sersic_index niriss_f200w_clear_magnitude
1 53.05 -27.83 0.17 0.46 104.35 3.3 18.06
2 53.10 -27.83 0.73 0.01 195.50 2.7 16.86
Extended Objects¶
The extended object catalog lists files containing stamp images to be added to the seed image. For example, a source such as a nebula or spiral galaxy that cannot be simulated via a 2-dimensional Sersic profile can be added by placing an image of the source in a fits file. Mirage will then read in, scale, and add this image to the seed image.
It is assumed that the fits file contains an array in the 0th or 1st extension. The array can be any size. If it is larger than the field of view of the simulated data, then it is cropped by placing the center of the extended stamp image at the specified x,y or RA, Dec location on the detector, and cropping any areas that fall outside of the detector.
Each row of this catalog contains the name of a FITS file containing the image to use, along with the RA, Dec (or x,y) position of the source, the position angle to use, and the source’s magnitude. The position angle is the angle in degrees East of North of the stamp image. For example, if you have an image of a spiral galaxy extracted from a dataset where the angle from North to the y-axis of the array is 330 degrees, then 330 should go into the pos_angle column of the catalog (assuming you wish to keep the real orientation of the object in the simulated data). Mirage will combine this pos_angle value with the roll angle of the telescope in the simulated scene to properly rotate the stamp image. In the case where you have a stamp image of a generic spiral galaxy that you would like to have at various position angles in the simulated data, then you can modify the pos_angle value appropriately. If you do not wish the stamp image to be rotated, set the pos_angle column to “None”. In that case the provided image will be used exactly as-is without any change in orientation.
For stamp images where it may not make sense to specify a magnitude (such as a galaxy cluster), it is possible to specify ‘None’ as the magnitude. In this case the code assumes that the data contained in the fits file is in units of ADU per second, and will not rescale the data before adding to the seed image. However, the user can also adjust the signal rate of all extended sources through the use of the extendedScale field in the input yaml file. This is a multiplicative factor to apply to the data in the fits file prior to adding the source to the seed image.
#
#
#
#
# Columns 1 and 2 can be either x,y positions on the detector aperture (e.g.
# 0,0 is lower left corner of the full frame of the subarray used for the
# output) or RA,Dec location of the center of the source. If they are x,y
# positions, make the top line of the file '# position_pixel'
#
#
#
index x_or_RA y_or_Dec pos_angle nircam_f200w_clear_magnitude filename
1 359.65 0.0006 20 16.000 ring_nebula.fits
Non-sidereal Source¶
This catalog is used when creating non-sidereal simulated exposures. In this case, all targets other than that specified in this catalog will then trail through the field of view during the observation. This mode is meant to simulate observations of solar system targets with non-sidereal velocities. This catalog should contain only one entry, with RA, Dec or x, y position, as well as velocity values (arcsec/hour or pixels/hour) and object magnitude. An optional ephemeris_file column can list a Horizons-formatted ephemeris file. If an ephemeris file is given, the RA and Dec of the object, along with it’s RA and Dec velocities, will be calculated at runtime using the ephemeris file and the observation date and time provided in the input yaml file. In this case, any RA, Dec, and velocity values given in the catalog will be ignored. To avoid confusion, it is possible to set the RA, Dec and RA and Dec velocity entries to ‘nan’ in this case. If the ephemeris_file column is not present (or if the column is present in the catalog but set to ‘None’), then the provided RA, Dec and velocity values will be used. For a simulation containing mulitple non-sidereal sources (such as a planet and its moons), place the source that you wish to have JWST track in this catalog, and place the other non-sidereal sources in the Moving Point Source, Moving 2D Sersic or Moving Extended Source catalogs.
#
#
#
# abmag
#
# radius can also be in units of pixels or arcseconds. Put 'radius_pixels' at top of file
# to specify radii in pixels.
# position angle is given in degrees counterclockwise.
# An "object" value containing 'point' will be interpreted as a point source.
# Anything containing "sersic" will create a 2D sersic profile.
# Any other value will be interpreted as an extended source.
# x_or_RA_velocity is the proper motion of the target in units of arcsec (or pixels) per hour
# Y_or_Dec_velocity is the proper motion of the target in units of arcsec (or pixels) per hour
# if the units are pixels per hour, include 'velocity pixels' in line 2 above.
index object x_or_RA y_or_Dec x_or_RA_velocity y_or_Dec_velocity nircam_f200w_clear_magnitude ephemeris_file
1 pointSource 53.101 -27.801 2103840. 0.0 17. none
or, with a provided ephemeris file:
#
#
#
# abmag
#
# radius can also be in units of pixels or arcseconds. Put 'radius_pixels' at top of file
# to specify radii in pixels.
# position angle is given in degrees counterclockwise.
# An "object" value containing 'point' will be interpreted as a point source.
# Anything containing "sersic" will create a 2D sersic profile.
# Any other value will be interpreted as an extended source.
# x_or_RA_velocity is the proper motion of the target in units of arcsec (or pixels) per hour
# Y_or_Dec_velocity is the proper motion of the target in units of arcsec (or pixels) per hour
# if the units are pixels per hour, include 'velocity pixels' in line 2 above.
index object x_or_RA y_or_Dec x_or_RA_velocity y_or_Dec_velocity nircam_f200w_clear_magnitude ephemeris_file
1 pointSource nan nan nan nan 17. neptune_2030.txt
Moving Point Sources¶
The moving point source catalog contains a list of point sources to move through the field of view during the integration. Similar to the static point source catalog, the position of each object (at the beginning of the integration) in RA, Dec or x,y must be provided, along with the object’s magnitude in the filter used for the simulation. In addition, the velocity of the object must be specified. This can be done in one of two ways:
Provide the name of a Horizons-formatted ephemeris file in the optional ephemeris_file column. In this case, the source’s location and velocity will be calculated at runtime using the ephemeris file and the observation date and time provided in the input yaml file. This will override any values provided in x_or_RA, y_or_Dec, x_or_RA_velocity, and y_or_Dec_velocity columns. Note that it is possible to set the values in these columns to ‘nan’ in order to avoid any confustion.
If the ephemeris_file column is not present, or has a value of ‘None’, then the source’s velocity must be specified using the x_or_RA_velocity and y_or_Dec_velocity columns. The units for these columns can be arcsec/hour or pixels/hour. ‘velocity_pixels’ must be placed in one of the top 4 lines of the file if the provided velocities are in units of pixels/hour rather than arcseconds/hour.
Here is an example catalog:
#
#abmag
#
#
# List of point sources to create as moving targets (KBOs, asteroids, etc)
# position can be x,y or RA,Dec. If x,y, put the phrase 'position_pixels' in one
# of the top 4 lines of the file.
# Velocity can be in units of pix/hour or arcsec/hour.
# If using pix/hour, place 'velocity_pixels' in the second line of the file.
# Note that if using velocities of pix/hour, the results will not be
# strictly correct because in reality distortion will cause object's
# velocities to vary in pixels/hour. Velocities in arcsec/hour will be
# constant.
index x_or_RA y_or_Dec nircam_f200w_clear_magnitude x_or_RA_velocity y_or_Dec_velocity ephemeris_file
1 53.0985 -27.8015 14 180 180 None
2 nan nan 14 nan nan mars_2030.txt
Moving 2D Sersic Objects¶
This option may be useful for simulating moving moons around a primary target that is being tracked. Similar to the static galaxy inputs, each moving target in this catalog must have an initial position in RA, Dec or x,y specified, along with a radius in arcseconds or pixels, ellipticity, position angle, Sersic index, and magnitude. In addition, velocities in the RA, Dec or x,y directions must be specified in units of arcseconds or pixels per hour. See the moving point source section above for a detailed description of how to specify velocity using either an ephemeris file or manual velocities.
#
#
#
#abmag
# Columns 1 and 2 can be either x,y positions on the detector aperture (e.g.
# 0,0 is lower left corner of the full frame of the subarray used for the
# output) or RA,Dec location of the center of the source. If they are x,y
# positions, make the top line of the file '# position_pixels'
#
# radius is the half-light radius in pixels or arcseconds. If in pixels
# make the second line of the file '# radius_pixels
#
# pos_angle is the position angle of the semimajor axis, in degrees.
# 0 causes the semi-major axis to be horizontal.
index x_or_RA y_or_Dec radius ellipticity pos_angle sersic_index nircam_f200w_clear_magnitude x_or_RA_velocity y_or_Dec_velocity ephemeris_file
1 354.765 0.00064 1.0 0.25 20 2.0 16.000 -0.5 -0.02 None
2 nan nan 1.0 0.25 20 2.0 16.000 nan nan kbo.txt
Moving Extended Sources¶
Similar to the catalog of static extended targets, this catalog contains a fits filename for each source containing the stamp image to use for the object, along with an initial position in RA, Dec or x,y, the object’s magnitude, and position angle (of the array as read in from the fits file). In addition, velocities in the RA, Dec (arcsec/hour) or x,y directions (pixels/hour) must be specified. See the moving point source section above for a detailed description of how to specify velocity using either an ephemeris file or manual velocities.
#
#
#
#abmag
# List of stamp image files to read in and use to create moving targets.
# This is the method to use in order to create moving targets of
# extended sources, like planets, moons, etc.
# position can be x,y or RA,Dec. Velocity can be in units of pix/hour or arcsec/hour.
# If using pix/hour, place 'velocity_pixels' in one of the top 4 lines.
# Note that if using velocities of pix/hour, the results will not be
# strictly correct because in reality distortion will cause object's
# velocities to vary in pixels/sec. Velocities in arcsec/hour will be
# constant.
index filename x_or_RA y_or_Dec nircam_f200w_clear_magnitude pos_angle x_or_RA_velocity y_or_Dec_velocity ephemeris_file
1 ring_nebula.fits 0.007 0.003 12.0 0.0 -0.5 -0.02 None
2 my_targ.fits nan nan 12.0 0.0 nan nan targ_ephem.txt
Grism Time Series Sources¶
When creating Grism TSO data, this catalog should contain the information on the TSO source parent body only. Other (background) sources should be placed in a catalog appropriate to their source type (point source, galaxy, extended). The index number for the TSO source should be set to 99999, to help ensure it will take presidence over the background sources when constructing the segmentation map. See the TSO example notebook for an example of how to use Mirage’s catalog generation functionality to create a Grism TSO source catalog. As with the other catalog types, the source location can be given in RA, Dec or pixel x, y.
Most of the other columns in the catalog are specific to the Batman package, which Mirage uses when creating TSO data. See the Batman documentation for lists of the possible limb darkening models and associated coefficients. Make sure that all time related entries use the same units. Start_time and End_time are for the lightcurve relative to the start time of the exposure. Code development was done using a Start_time of 0.0 (i.e. the beginning of the exposure) and an End_time set to the maximum time the user wishes to simulate. If the End_time of the lightcurve comes before the end of the exposure, Mirage will automatically extend the lightcurve to fully encompass the exposure.
The Transmission_spectrum column should hold the name of an ascii file containing the transmission curve of the source. This is the Wavelength-dependent effective radius of the planet, in units of the stellar radius. Again, see the TSO example notebook for an example of this file.
Finally, this catalog contains magnitude columns similar to those in other catalog types. Note that along with this catalog, Grism TSO simulations require a file containing the SED of the star. If the SED in that file is given in flux density units, then the magnitudes in this catalog will be ignored. If the SED is normalized or scaled in some way, then the SED will be renormalized to the magnitude in this catalog corresponding to the filter of the TSO observation.
# position_RA_Dec
# vegamag
#
#
index x_or_RA y_or_Dec Semimajor_axis_in_stellar_radii Orbital_inclination_deg Eccentricity Longitude_of_periastron Limb_darkening_model Limb_darkening_coeffs Time_units Start_time End_time Time_of_inferior_conjunction Orbital_period Transmission_spectrum nircam_f444w_clear_magnitude nircam_f322w2_clear_magnitude
99999 66.37090333 -30.60044722 9.37 83.3 0.0 90.0 nonlinear "0.5, 0.1, 0.1, -0.1" second 0.0 580.0 280.0 3162.24 ./transmission_spectrum.txt 9.0 9.05
Imaging Time Series Sources¶
When creating Imaging TSO data, this catalog should contain the information on the TSO source parent body only. Other (background) sources should be placed in a catalog appropriate to their source type (point source, galaxy, extended). The index number for the TSO source should be set to 99999, to help ensure it will take presidence over the background sources when constructing the segmentation map. See the TSO example notebook for an example of how to use Mirage’s catalog generation functionality to create an Imaging TSO source catalog. As with the other catalog types, the source location can be given in RA, Dec or pixel x, y.
The lightcurve_file column should contain the name of an hdf5 file that contains a dataset with wavelength and flux arrays that describe the transit lightcurve at the wavelength range corresponding to the filter used in the observation. Again, see the TSO example notebook for an example of how to create this file.
Finally, this catalog contains magnitude columns similar to those in other catalog types.
# position_RA_Dec
# vegamag
#
#
index x_or_RA y_or_Dec lightcurve_file nircam_f182m_magnitude nircam_f210m_clear_magnitude nircam_f444w_f470n_magnitude
99999 66.37090333 -30.60044722 ./example_lightcurve.hdf5 10.0 9.5 9.0
Addition of Ghosts to Mirage Simulations¶
NIRCam¶
The addition of ghosts is not currently supported.
FGS¶
The addition of ghosts is not currently supported.
NIRISS¶
Optical ghosts in NIRISS have been observed and characterized in ground testing data. A detailed description of the ghosts is given in the Ghosts section of the NIRISS GR150 Grisms Jdox page.
Controlling the Addition of Ghosts¶
Optical ghosts can be added to simulated NIRISS imaging and WFSS observations. Their addition is optional, and is controlled by the sim_Signals:add_ghosts and sim_Signals:PSFConvolveGhosts keywords in the yaml input files.
If add_ghosts is True, Mirage will calculate, for each point source in your source catalog, the location and magnitude of the optical ghost associated with the optical elements being used. Within Mirage, once the ghost locations and magnitudes are known, the ghosts are treated as extended sources, which rely upon fits files containing stamp images. The stamp images are read in, scaled to the requested brightness, and added to the seed image. The location and brightness of the ghost are calculated using the “gap summary” file in the config directory of the Mirage repository. This file specifies the observed offsets, as well as count rate of a ghost for a given filter, pupil, and source location and count rate on the detector.
Tip
The current “gap summary” file allows for the addition of ghost sources for observations that use F090W, F115W, F140M, F150W, and F200W. For other filters, the ghost positions and/or magnitudes have not been characterized, and therefore ghosts will not be added. If you wish to run Mirage with a “gap summary” file other than that in the config directory, you can either replace the file in the config directory with your own, or update the value of NIRISS_GHOST_GAP_FILE in the constants.py file of the repository to point to your file. This may be useful for testing new ghost position calibration results.
Mirage currently carries one fits file in its collection of reference files that provides an image of an optical ghost from a point source on the NIRISS detector. This image was created from ground testing data. By default, this file is used when adding ghosts to the data. NOTE that this stamp image applies to point sources only. By default, no ghosts are added for galaxy or extended sources.
Users may add ghosts associated with galaxy or extended sources (or override the default stamp image for point sources) by supplying their own stamp images and specifying which stamp images are to be associated with each source within the source catalogs. An example of this is shown in the Specifying Ghost Stamp Images section below.
If the PSFConvolveGhosts keyword is True, then the stamp image will be convolved with the instrumental PSF prior to adding the ghost image to the data. By default, this is set to False. Users can control its value, along with that of add_ghosts, in the call to the yaml_generator.
Specifying Ghost Stamp Images¶
In this section we give a brief example of creating a source catalog containing a list of stamp images to be used in the creation of ghost sources. By default, a source catalog will not be created with a column for these stamp images. In such a case, Mirage will fall back to using the default stamp image within the collection of reference files for point sources only. The example below creates a source catalog for extended sources with ghost stamp images. This will allow Mirage to create a ghost source for each extended source. The niriss_ghost_stamp keyword can enable this behavior when creating any type of source catalog (e.g. PointSourceCatalog, GalaxyCatalog, MovingPointSourceCatalog, etc).
In this example we have a catalog with three sources. Stamp images of the sources themselves are contained in the files source1.fits, source2.fits, and source3.fits. Stamp images of the corresponding ghosts are in ghost1.fits, ghost2.fits, and ghost3.fits.
from mirage.catalogs import catalog_generator as cg
ra = [12.2, 12.2001, 12.2002]
dec = [23.1, 23.1001, 23.1002]
source_stamp_files = ['source1.fits', 'source2.fits', 'source3.fits']
position_angle = [0., 0., 0.]
magnitudes = [15.5, 16.3, 17.4]
ghost_stamp_files = ['ghost1.fits', 'ghost2.fits', 'ghost3.fits']
cat = cg.ExtendedCatalog(ra=ra, dec=dec, filenames=source_stamp_files,
position_angle=position_angle,
niriss_ghost_stamp=ghost_stamp_files)
cat.add_magnitude_column(magnitudes, instrument='niriss', filter_name='f380m')
cat.save('extended_sources_with_ghosts.cat')
Display the table:
cat.table
index x_or_RA y_or_Dec filename niriss_f380m_magnitude niriss_ghost_stamp pos_angle
int64 float64 float64 str12 float64 str11 float64
----- ------- -------- ------------ ---------------------- ------------------ ---------
1 12.2 23.1 source1.fits 15.5 ghost1.fits 0.0
2 12.2001 23.1001 source2.fits 16.3 ghost2.fits 0.0
3 12.2002 23.1002 source3.fits 17.4 ghost3.fits 0.0
Tools for Creating Source Catalogs¶
To help users create source catalogs in the proper format, Mirage contains the catalog generator and create catalog modules. catalog_generator.py contains classes for each of the types of catalogs accepted by Mirage. The create_catalog.py module contains functions that will query several astronomical databases and attempt to construct realistic point source and extragalactic catalogs for a given pointing. There is also a function for automated catalog creation from an input APT file. See the Catalog Generation Tools notebook in the Mirage repository for examples of all of this functionality.
Catalog_generator - Create catalogs from a list of sources¶
The classes in this module can be used to simplify Mirage catalog creation from lists of source coordinates and magnitudes. Generally the user instantiates the class for the desired type of catalog with a list of RA and Dec or detector x, y source positions. Magnitude columns can then be added to the catalog by providing a list of magnitudes as well as the instrument and filter name associated with the magnitudes. Catalogs may contain multiple magnitude columns covering multiple instruments and filters. Examples of how to use the catalog generator functionality are given in the Catalog Generation Tools notebook notebook. Below, we show examples for the most common catalogs: point sources and galaxies.
Point Source Catalog¶
The examples below create point source catalogs containing 10 sources. By supplying source locations via the ra and dec keywords, Mirage assumes that the locations are Right Ascention and Declination values given in decimal degrees. If you wish to specify source locations in units of row and column indexes on the detector, then the x and y keywords should be used, as seen in the second example below.
Columns containing AB magnitudes in the NIRCam F200W and F212N filters, as well as the NIRISS F090W filter are added to the catalog using the add_magnitude_column method. Note that the instrument and filter names associated with the magnitude values must be supplied in these calls. The filter value can be a string containing the filter wheel and pupil wheel values to be used, separated by a “/” (e.g. “F444W/F405N”, “F444W/CLEAR”, “F150W/WLM8”). As a shorthand, in cases where the CLEAR element in the pupil wheel is used, you can provide only the filter wheel value in the string (e.g. “F444W”, “F090W”). Similarly, for the “standard” narrow band filter combinations (defined here: XXXXXX), you can provide only the name of the narrow band filter, and Mirage will assume that you are using one of the standard cases (e.g. specifying “F405N” will have Mirage assume you mean F405N crossed with F444W). For non-standard filter crossing, use the “/” format above and specify both filters.
The magnitude_system keyword is optional and allows uses to specify the magnitude system. Allowed values are “abmag”, “stmag”, and “vegamag”. The default is “abmag”. Note that all magnitude columns in a given catalog MUST be in the same magnitude system.
The starting_index keyword is also optional. The value of the keyword is the first value to be placed in the “index” column of the catalog. Index values will increment upwards from this starting value for each source. With this keyword, users creating multiple catalogs to be input into a single simulation can be sure that every source going into the simulation has a unique index number. See the Source Index Numbers section for more details.
Once the catalog has been created and at least one magnitude column has been added, the table can be printed to the screen using the table method.
The save method will save the table in an ascii file in the appropriate format for Mirage to use.
import numpy as np
from mirage.catalogs import catalog_generator
ra_list = np.random.random(10) + 53.5
dec_list = np.random.random(10) - 67.2
nrc_f200w_mag = np.random.random(10) + 16.
nrc_f212n_mag = np.random.random(10) + 19.
nrc_f405n_mag = np.random.random(10) + 20.
nrc_f150_wlp8_mag = np.random.random(10) + 19.
nis_f090w_mag = np.random.random(10) + 15.5
ptsrc = catalog_generator.PointSourceCatalog(ra=ra_list, dec=dec_list, starting_index=1)
ptsrc.add_magnitude_column(nrc_f200w_mag, instrument='nircam', filter_name='F200W', magnitude_system='abmag')
ptsrc.add_magnitude_column(nrc_f212n_mag, instrument='nircam', filter_name='F212N', magnitude_system='abmag')
ptsrc.add_magnitude_column(nrc_f405n_mag, instrument='nircam', filter_name='F444W/F405N', magnitude_system='abmag')
ptsrc.add_magnitude_column(nrc_f150_wlp8_mag, intsrument='nircam', filter_name='F150W/WLP8', magnitude_system='abmag')
ptsrc.add_magnitude_column(nis_f090w_mag, instrument='niriss', filter_name='F090W', magnitude_system='abmag')
ptsrc.save('point_sources.cat')
ptsrc.table()
x_list = np.random.random(10) * 2048
y_list = np.random.random(10) * 2048
ptsrc = catalog_generator.PointSourceCatalog(x=x_list, y=y_list, starting_index=1)
ptsrc.add_magnitude_column(nrc_f200w_mag, instrument='nircam', filter_name='F200W', magnitude_system='abmag')
ptsrc.add_magnitude_column(nrc_f212n_mag, instrument='nircam', filter_name='F212N', magnitude_system='abmag')
ptsrc.add_magnitude_column(nis_f090w_mag, instrument='niriss', filter_name='F090W', magnitude_system='abmag')
ptsrc.save('point_sources_xy.cat')
Galaxy Catalog¶
The example below creates a galaxy catalog. The main difference compared to the point source catalog is that the user must provide values for the ellipticity, sersic index, and position angle when instantiating the class. In this example, we assume the galaxy catalog will be used in combination with the point source catalog above in a single simulation. Therefore, we set starting_index equal to 11, since the point source catalog contains sources 1 through 10. See the Source Index Numbers section for more details.
import numpy as np
from mirage.catalogs import catalog_generator
ra_list = np.random.random(10) + 53.5
dec_list = np.random.random(10) - 67.2
ellipticity = np.random.random(10) * 0.75
sersic_index = np.random.random(10) * 4.
position_angle = np.random.random(10) * 359.
nrc_f200w_mag = np.random.random(10) + 16.
nrc_f212n_mag = np.random.random(10) + 19.
nis_f090w_mag = np.random.random(10) + 15.5
gal = catalog_generator.GalaxyCatalog(ra=ra_list, dec=dec_list, ellipticity=ellipticity,
sersic_index=sersic_index, position_angle=position_angle,
starting_index=11)
gal.add_magnitude_column(nrc_f200w_mag, instrument='nircam', filter_name='F200W', magnitude_system='abmag')
gal.add_magnitude_column(nrc_f212n_mag, instrument='nircam', filter_name='F212N', magnitude_system='abmag')
gal.add_magnitude_column(nis_f090w_mag, instrument='niriss', filter_name='F090W', magnitude_system='abmag')
gal.save('galaxies.cat')
gal.table()
Create_catalog - create catalogs using online astronomical databases¶
The functions in this module use astroquery to search astronomical databases and retrieve source lists for a given pointing. In this way, a user can quickly generate reasonably realistic catalogs of point sources and galaxies for a given pointing.
The get_all_catalogs function takes the RA and Dec of a particular pointing along with the width in arcseconds of the area for which to produce the catalog, and queries multiple databases to produce a point source catalog. An example call to create a 120 x 120 arcsecond catalog is shown below. The resulting point source catalog can then be placed in the pointSource entry of the yaml input file. The besancon_catalog_file in the command below is the result from a query of the Besancon model. Details of how to query the model and download the result are shown in the Background Stars section below. Note that starting_index is also an optional keyword here, so that users can control the index values of the sources. See the Source Index Numbers section for more details.
By default, Mirage will query the ALLWISE source catalog as part of the call to get_all_catalogs(). However, in certain cases (e.g. for very bright sources), it may be better to query the WISE All-Sky source catalog. If you wish to switch to the WISE All-Sky source catalog, simply add the keyword: wise_catalog=’wise_all_sky’ to the call below.
from mirage.catalogs import create_catalog
ra = 80.4 # degrees
dec = -69.8 # degrees
box_width = 120 # arcseconds
filter_list = ['F444W', 'F480M', 'F150W/WLM8', 'F444W/F405N']
cat, mag_column_names = create_catalog.get_all_catalogs(ra, dec, box_width, besancon_catalog_file='besancon.cat',
instrument='NIRCAM', filters=filter_list,
starting_index=1)
Foreground Stars¶
A realistic list of foreground stars is compiled by querying the 2MASS, WISE, and GAIA catalogs using the given pointing. Using the retrieved magnitudes in the various bands associated with these surveys, Mirage converts these to magnitude values in the requested NIRCam or NIRISS filters. Note that these queries return stars only down to about V=16. For dimmer stars, you can query the Besancon model. This process is described in the Background Stars section below.
Background Stars¶
To obtain a list of stars dimmer than those returned in the Foreground Stars search, Mirage uses queries the Besancon model of stars in the Milky Way. This query returns a representative sample (in terms of luminosity distribution) of stars for a given pointing, including a realistic stellar density and realistic luminosity distribution. Note that these stars are not actual stars in the sky. Due to the way in which the model is queried and results are returned, the use of a Besancon-derived catalog is a two-step process. First, you must create an account on the Besancon model page. Once the account is activated, you can query the model using Mirage’s wrapper function, as shown below. A more complete example of this is given in the Example uses of Mirage catalog generators jupyter notebook.
from mirage.catalogs import create_catalog
ra = 224.2 # degrees
dec = -65.54 # degrees
box_width = 200 # arcseconds
create_catalog.besancon(ra, dec, box_width, username='hilbert', kmag_limits=(17, 30))
Once the query is complete, you will receive an email with a link to download the resulting ascii table. With the saved table in hand, you can then transform the source magnitudes from JHK to the JWST filters of interest, and combine the catalog with query results from GAIA/2MASS/WISE. This combined catalog can then be used as input to a Mirage simulation. Again, see the Example uses of Mirage catalog generators jupyter notebook for an example.
Background Galaxies¶
For a given pointing, Mirage can also generate a catalog containing a representative sample of background galaxies. Similar to the Besancon query described above, Mirage will generate a catalog containing a realistic density of galaxies across the field at reasonable magnitudes. To accomplish this, Mirage queries the GOODS-S catalog from 3DHST and extracts an appropriate number of galaxies to populate the catalog at a reasonable density. Currently this function will fail if the user requests a catalog with an area larger than the GOODS-S field: 606,909 arcsec 2. An example is shown below. The resulting file can then be placed in the galaxyListFile entry of the yaml input file. Note that the starting_index keyword is also available in this function. See the Source Index Numbers section for more details.
from mirage.catalogs import create_catalog
xml_file = 'my_nircam_program.xml'
pointing_file = xml_file.replace('.xml', '.pointing')
# Pointing information
ra = 224.2 # degrees
dec = -65.54 # degrees
box_width = 200 # arcseconds
roll_angle = 0. # degrees E of N
# Filters to include in the galaxy catalog
filter_list = ['F115W', 'F444W']
# Create the catalog
gal_cat, gal_seed = create_catalog.galaxy_background(ra, dec, roll_angle, box_width,
'nircam', filter_list, starting_index=1000)
galaxy_catalog = 'galaxies.cat'
gal_cat.save(galaxy_catalog)
# Run the yaml_generator In this case, assume a target name in the APT
# file of HD-9999-B1, and assume we have a point source catalog called
# ptsrcs.cat
cat_dict = {'HD-99999-B1': {'point_source':'ptsrcs.cat', 'galaxy': galaxy_catalog}}
background = 'low'
output_dir = 'yaml_files'
simulation_dir = 'sim_data'
datatype = 'raw'
yam = yaml_generator.SimInput(input_xml=xml_file, pointing_file=pointing_file,
catalogs=cat_dict,
background=background,
verbose=True, output_dir=output_dir,
simdata_output_dir=simulation_dir,
datatype=datatype)
yam.create_inputs()
Simulating Observations from an APT File¶
The easiest way to simulate a large amount of obsrvtions using Mirage is to begin with the APT file of a JWST proposal. Mirage contains functions that can (for imaging and WFSS modes) create the collection of yaml files necessary to simulate all of the observations contained in the APT file. Specifically, the yaml_generator.py module will take the APT files, along with several other inputs, and will output yaml files that Mirage can use to create all the observations contained in the APT files.
Examples of this functionality are shown in the notebooks in the Mirage repository. Below are step-by-step instructions explaining how to create Mirage input yaml files from an APT file.
Important
Currently Mirage is able to parse APT files containing any observation template, including all JWST-supported coordinated parallel combinations. However, it will only generate yaml input files for the observation templates listed below. It will silently skip over all other templates.
NIRCAM |
|---|
|
NIRISS |
|---|
|
FGS |
|---|
|
Wavefront Sensing and Control |
|---|
|
Export XML and Pointing files from APT¶
Open the proposal within APT and under the File menu, choose Export, and select the xml and pointing files. Only one file at a time can be exported, so the process must be repeated for each.
Additional Yaml Generator Inputs¶
In addition to the xml and pointing files, there are several user-inputs needed by the yaml generator. Examples shown below are python statements.
Source catalogs¶
The user must specify source catalogs for each instrument and target in the proposal. This is done using one of several possible input formats, as shown below.
If you do not wish to specify catalogs, and want to leave all catalog inputs as ‘none’, set your catalogs variable to None, or omit completely:
catalogs = None
To use the same catalogs for all instruments in a given target, create a nested dictionary where the top-level keys exactly match the target names in the APT file. In the second level of the dictionary, specify catalog filenames for each type of catalog. Note that you only need to include the catalog types that you want to use for your data. All 7 catalog types are shown here for completeness, but in most cases you will only specify 1-3 catalog types.
catalogs = {'NGC4242': {'point_source': 'target1_ptsrc.cat',
'galaxy': 'target1_galaxies.cat',
'extended': 'target1_ext.cat',
'moving_pointsource': 'target1_mt_ptsrc.cat',
'moving_sersic': 'target1_mt_gal.cat',
'moving_extended': 'target1_mt_ext.cat',
'moving_target_to_track': 'target1_mt_track.cat'
},
'LMC': {'point_source': 'lmc_ptsrc.cat',
'galaxy': 'lmc_galaxy.cat',
'extended': 'lmc_ex.cat',
'moving_pointsource': 'lmc_mt_ptsrc.cat',
'moving_sersic': 'lmc_mt_gal.cat',
'moving_extended': 'lmc_mt_ext.cat',
'moving_target_to_track': 'lmc_mt_track.cat'
}
}
For more fine-grained control, you can also specify catalogs for each instrument and target. This adds another level to the input dictionary. Again, note that the example below shows all three instruments for completeness. You only need to include the instruments used in the proposal. Instrument and catalog type dictionary keys are case-insensitive. Target name keys must match exactly those in the APT file. The dictionary currently does not break out separate catalogs for NIRCam shortwave and longwave channels. Note that Mirage contains catalog tools that can be used to combine catalogs. See the Catalog Generation notebook for examples.
catalogs = {'NGC1234': {'nircam': {'point_source': 'ngc1234_ptsrc_nrc.cat',
'galaxy': 'ngc1234_galaxy_nrc.cat',
'extended': 'ngc1234_ex_nrc.cat',
'moving_pointsource': 'ngc1234_mt_ptsrc_nrc.cat',
'moving_sersic': 'ngc1234_mt_gal_nrc.cat',
'moving_extended': 'ngc1234_mt_ext_nrc.cat',
'moving_target_to_track': 'ngc1234_mt_track_nrc.cat'
},
'niriss': {'point_source': 'ngc1234_ptsrc_nis.cat',
'galaxy': 'ngc1234_galaxy_nis.cat',
'extended': 'ngc1234_ex_nis.cat',
'moving_pointsource': 'ngc1234_mt_ptsrc_nis.cat',
'moving_sersic': 'ngc1234_mt_gal_nis.cat',
'moving_extended': 'ngc1234_mt_ext_nis.cat',
'moving_target_to_track': 'ngc1234_mt_track_nis.cat'
}
},
'LMC': {'nircam': {'point_source': 'lmc_ptsrc_nrc.cat',
'galaxy': 'lmc_galaxy_nrc.cat',
'extended': 'lmc_ex_nrc.cat',
'moving_pointsource': 'lmc_mt_ptsrc_nrc.cat',
'moving_sersic': 'lmc_mt_gal_nrc.cat',
'moving_extended': 'lmc_mt_ext_nrc.cat',
'moving_target_to_track': 'lmc_mt_track_nrc.cat'
},
'niriss': {'point_source': 'lmc_ptsrc_nis.cat',
'galaxy': 'lmc_galaxy_nis.cat',
'extended': 'lmc_ex_nis.cat',
'moving_pointsource': 'lmc_mt_ptsrc_nis.cat',
'moving_sersic': 'lmc_mt_gal_nis.cat',
'moving_extended': 'lmc_mt_ext_nis.cat',
'moving_target_to_track': 'lmc_mt_track_nis.cat'
}
},
}
Background Specification¶
Users may also supply information on the background levels to use for each instrument and observation. Allowed values for the background parameter are described in bkgdrate section of the Input Yaml Parameters page.
If dateobs_for_background is True, then the background for all exposures will be determined based on the value of the observation date (discussed below). The JWST backgrounds package will be used, and the background associated with the requested pointing and date will be used.
If dateobs_for_background is False, then the background parameter controls the calculation of the background. As with the catalogs above, there are several formats that can be used to supply the background information. Examples are shown below.
To use the default value for background (“low”), either omit this parameter altogether, or set background equal to None.
background = None
To specify a single value for background to be used across all observations and instruments, you may supply a single string (‘high’, ‘medium’, or ‘low’) or a single number. The strings correspond to the background levels from the JWST ETC and are detailed in the bkgdrate section of the Input Yaml Parameters page. If a single number is provided, it is interpreted as the background signal in units of DN/sec/pixel. This signal will be uniform across all pixels.
background = 'high'
background = 22.2
In order to use a different background in each observation of the proposal (if for example, your proposal will be broken into multiple epochs observed at different times of year), you can use a dictionary. The keys are the observation numbers in the proposal file. Note that these are three-character strings. The values can then be the same strings or numbers described above.
background = {'001': 'high', '002': 'medium', '003': 22.3}
For finer control, you can use a nested dictionary to specify the background signal in each instrument and observation. The top level keys of the dictionary are the observation numbers from the APT file (again, as 3-character strings). The values are then instrument names, which are keys into the second level of the dictionary. The values for these keys can be strings or numbers, as above, or in the case of NIRCam, a further nested dictionary that breaks out the background level by channel (shortwave detectors versus longwave detectors).
background = {'001': {'nircam': {'sw': 0.2, 'lw': 0.3},
'niriss': 0.4},
'fgs': 0.2},
'002': {'nircam': {'sw': 'medium', 'lw': 'high'},
'niriss': 'low'},
'fgs': 'high'},
'003': {'nircam': {'sw': 0.75, 'lw': 'high'},
'niriss': 0.2}}
'fgs': 0.1}}
Roll Angle¶
Another optional user input to the yaml_generator is the roll angle of the telescope. This is often referred to as PAV3 (or V3PA) as it is the position angle of the V3 axis in degrees east of north. We include this as a tunable parameter in Mirage to allow users to explore different orientations for their data, including the use of different roll angles for different observations within their proposals in order to simulate epochs.
In order to use the Mirage default value (roll angle = 0), simply do not provide a roll angle input to the yaml_generator, or explicitly set it to None.
roll_angle = None
To specify a single roll angle to be used in all observations, supply a single number in decimal degrees.
roll_angle = 34.5
In order to simulate epochs and break up your observations, supply a dictionary where the keys are the (3-character string) observation numbers from your APT file, and the values are the roll angles to use for those observations.
roll_angle = {'001': 34.5, '002': 154.5, '003': 37.8}
There are convenience functions to compute the default (i.e., no observatory roll and no special requirements) position angles for an observation on a given hypothetical observing date. These rely on the JWST general target visibility tool, which you must have installed to use this feature. There are functions to compute either a single value or a dictionary in the above format for multiple observations.
pointing_filename = "/path/to/your_program.pointing"
# Compute default PA for one observation
obs_num = 2
roll_angle = yaml_generator.default_obs_v3pa_on_date(pointing_filename, obs_num, date='2021-12-21')
# Compute default PA for all observations in that program
roll_angle = yaml_generator.default_obs_v3pa_on_date(pointing_filename, date='2022-01-01')
Observation Dates¶
You may also specify the observation date for each observation in your APT file. This may be used along with roll angle to help define epochs in your observations, or simply to associate a given dataset with a date. Mirage saves the input dates into the date-obs header keyword in the output files. Mirage does not check that a given roll angle and pointing are physically realizable on a given date. It is up to you to provide realistic values for these paramters if they are important to you. The JWST Target Visibility Tools (TVT) are useful for this. Note that in all cases below, Mirage will use the entered date (along with a default time if necessary) as the starting time of the first exposure in the observation. Mirage keeps track of exposure times and makes some guesses about overheads, and increments the observation time and date for each exposure.
To use the Mirage default for observation date (arbitrarily set to 2021-10-04), you can either not supply any date information, or explicitly use None.
dates = None
To use a single date for all observations, you can give a date string.
dates = '2022-5-25'
To specify a different date for each observation, use a dictionary where the keys are the (3-character string) observation numbers from your APT file, and the values are the date strings for each.
dates = {'001': '2022-06-25', '002': '2022-11-15', '003': '2023-03-14'}
For observations where you want to be more specific about the observation time (such as for non-sidereal targets), datetime strings can be provided:
dates = {'001': '2022-06-25T12:00:00', '002': '2022-11-15T12:34:56', '003': '2023-03-14T18:35:42'}
Cosmic Ray Rates¶
You may also customize the cosmic ray rates applied to Mirage’s outputs. There are two aspects of the cosmic ray behavior that can be controlled. The first is the name of the library of cosmic ray stamp images to use, and the second is a scaling factor that can be applied to that library. The three library options, in are SUNMAX, SUNMIN, and FLARE. Each library contains a different collection of cosmic ray images, and each has a default cosmic ray rate (cosmic rays per pixel per second) associated with it. The SUNMIN and SUNMAX labels refer to the solar activity, and the galactic cosmic ray contribution at L2 is reduced at solar maximum compared to solar minimum. The FLARE case is for the largest solar flare event on record and corresponds to conditions under which JWST would presumably not be operating. The table below give the cosmic ray probabilities for the three libraries. The cosmic ray libraries and default probabilties were taken from Robberto 2009.
Library |
Cosmic Ray Probability |
SUNMAX |
5.762e-06 |
SUNMIN |
1.587e-05 |
FLARE |
0.0098729 |
The second configurable aspect of the cosmic ray rate is a scaling factor. This is a multiplicative factor that will be applied to the probability from the selected library in order to determine the final cosmic ray probability.
To use Mirage’s default values of the SUNMAX library and a scaling factor of 1.0, simply do not provide any input, or explicitly set the cosmic ray rate to None.
cr = None
To specify a different library and scale from the default, and apply those to all observations in your proposal, provide a dictionary with ‘library’ and ‘scale’ keys set to your desired values.
cr = {'library': 'FLARE', 'scale': 44.0}
In order to use a different cosmic ray library and scaling factor for each observation, create a nested dictionary where the top-level keys are the (3-character string) observation numbers from your APT file. Each entry should then contain a dictionary with ‘library’ and ‘scale’ values.
cr = {'001': {'library': 'FLARE', 'scale': 1.2},
'002': {'library': 'SUNMIN', 'scale': 5.5},
'003': {'library': 'SUNMAX', 'scale': 0.1}}
JWST Calibration Reference Files¶
Mirage makes use of a handful of the reference file types used by the JWST calibration pipeline. This includes the bad pixel mask, saturation level map, superbias, gain, interpixel capacitance, linearity correction, distortion correction pixel to pixel flat field and photom files.
Mirage relies on the CRDS package from STScI to identify the appropriate reference files for a given exposure. These files are automatically downloaded to the user’s machine at one of two times:
When running yaml_generator.py if the
reffile_defaultsoption is set to ‘crds_full_names’ (see Run the Yaml Generator for more details).When running any of the three main parts of Mirage: the catalog seed generator, dark preparation, or observation generator.
Tip
In order to specify a location on your machine to store the downloaded reference files, you must have the CRDS_PATH environment variable set to that location. If this environment variable is not set, Mirage will default to use $HOME/crds_cache/
Tip
Even though the transmission image reference files are not currently hosted by CRDS, for the purposes of user input options, Mirage treats them as if they are. Placing “crds” for these files in the input yaml files will cause Mirage to search the collection of Mirage reference files for them, rather than searching CRDS. We anticipate the transmission image files will be moved into CRDS in the future, which is why we treat them in this way.
Important
Due to a limitation of the CRDS package, the CRDS_PATH and CRDS_SERVER_URL environment variables must be set BEFORE importing the CRDS package. If you are running Mirage using code that imports CRDS or any other packages that import CRDS (such as Mirage’s dark_prep module or the jwst package, which contains the calibration pipeline) prior to running Mirage, you should explicitly set the environment variables before importing those packages. If you do not, you will get the following error:
CRDS - ERROR - (FATAL) CRDS server connection and cache load FAILED. Cannot continue. CRDS - ERROR - See https://hst-crds.stsci.edu/docs/cmdline_bestrefs/ or https://jwst-crds.stsci.edu/docs/cmdline_bestrefs/ CRDS - ERROR - for more information on configuring CRDS, particularly CRDS_PATH and CRDS_SERVER_URL. : [Errno 2] No such file or directory: ‘$HOME/crds_cache/config/jwst/server_config’
If you wish to set the environment variables in your code, simply add lines such as these prior to importing jwst or CRDS:
os.environ[“CRDS_PATH”] = ‘{}/crds_cache’.format(os.environ.get(‘HOME’)) os.environ[“CRDS_SERVER_URL”] = “https://jwst-crds.stsci.edu”
If your code does not import any packages that rely on the CRDS package, then you may safely neglect setting the two environment variables, and Mirage will set them for you prior to importing CRDS.
It is also possible to specify that Mirage use reference files other than those downloaded from CRDS. In order to do this, you must supply a dictionary of filenames. Due to the varying number of selection criteria needed to uniquely identify the reference file that matches up with a particular exposure, this dictionary is composed of multiple levels of nested dictionaries. Not all possibilities are required. You may specify reference files only for the particular observing modes you are interested in. Any modes in your APT file that are not contained within the dictionary will revert to using the reference files identified by CRDS. Below is a dictionary showing all nesting required for all reference files. Note that the dictionary structure is instrument dependent since a reference file type for different instruments does not necessarily have the same selection criteria.
Important
If you choose to provide your own reference files, it is best to use these same reference files when running the JWST calibration pipeline on the simulated data files produced by Mirage. Not doing this can lead to systematic errors in your calibrated data.
Here is a view of the dictionary structure required when specifying reference files. The easiest way to see the set of allowed values for a particular property (e.g. exposure_type), go to the JWST Keyword Dictionary, and search for the appropriate keyword. Exposure type is EXP_TYPE, filter is FILTER, pupil is PUPIL, detctor_name is DETECTOR, readpattern is READPATT.
override = {'nircam': {'superbias': {detector_name: {readpattern: 'reffile_name.fits'}},
'linearity': {detector_name: 'reffile_name.fits'},
'saturation': {detector_name: 'reffile_name.fits'},
'gain': {detector_name: 'reffile_name.fits'},
'distortion': {detector_name: {filter: {exposure_type: 'reffile_name.asdf'}}},
'ipc': {detector_name: 'reffile_name.fits'},
'transmission': {detector_name: {filter: {pupil: 'reffile_name.fits'}}},
'badpixmask': {detector_name: 'reffile_name.fits'},
'pixelflat': {detector_name: {filter: {pupil: 'reffile_name.fits'}}},
'photom': {detector_name: 'reffile_name.fits'}
},
'niriss': {'superbias': {readpattern: 'reffile_name.fits'},
'linearity': 'reffile_name.fits',
'saturation': 'reffile_name.fits',
'gain': 'reffile_name.fits',
'distortion': {pupil: {exposure_type: 'reffile_name.fits'}},
'ipc': 'reffile_name.fits',
'transmission': {filter: {pupil: 'reffile_name.fits'}},
'badpixmask': 'reffile_name.fits',
'pixelflat': {filter: {pupil: 'reffile_name.fits'}},
'photom': {detector_name: 'reffile_name.fits'}
},
'fgs': {'superbias': {detector_name: {readpattern: 'reffile_name.fits'}},
'linearity': {detector_name: 'reffile_name.fits'},
'saturation': {detector_name: 'reffile_name.fits'},
'gain': {detector_name: 'reffile_name.fits'},
'distortion': {detector_name: {exposure_type: 'reffile_name.fits'}},
'ipc': {detector_name: 'reffile_name.fits'},
'transmission': {detector_name: 'reffile_name.fits'},
'badpixmask': {detector_name: {exposure_type: 'reffile_name.fits'}},
'pixelflat': {detector_name: {exposure_type: 'reffile_name.fits'}},
'photom': {detector_name: 'reffile_name.fits'}
}
Here we show an example dictionary for a particular set of observations.
override = {'nircam': {'superbias': {'nrcb5': {'bright2': 'my_reffiles/my_superbias_for_b5.fits',
'rapid': 'my_reffiles/my_superbias_for_b5.fits',
'shallow4': 'my_reffiles/my_superbias_for_b5.fits'
},
'nrcb4': {'rapid': 'my_reffiles/my_superbias_for_b4.fits'}
},
'linearity': {'nrcb5': 'my_reffiles/my_linearity_for_b5.fits',
'nrcb4': 'my_reffiles/my_linearity_for_b4.fits'},
'saturation': {'nrcb5': 'my_reffiles/my_saturation_for_b5.fits',
'nrcb4': 'my_reffiles/my_saturation_for_b4.fits'},
'gain': {'nrcb5': 'my_reffiles/my_gain_for_b5.fits',
'nrcb4': 'my_reffiles/my_gain_for_b4.fits'},
'distortion': {'nrcb5': {'f322w2': {'nrc_image': 'my_reffiles/my_distortion_for_b5.asdf'}},
'nrcb4': {'f444w': {'nrc_image': 'my_reffiles/my_distortion_for_b4.asdf'}}},
'ipc': {'nrcb5': 'my_reffiles/my_ipc_for_b5.fits',
'nrcb4': 'my_reffiles/my_ipc_for_b4.fits'},
'transmission': {'nrcb5': {'f356w': {'clear': 'my_reffiles/my_transmission_for_b5.fits'}}},
'badpixmask': {'nrcb5': 'my_reffiles/my_bpm_for_b5.fits',
'nrcb4': 'my_reffiles/my_bpm_for_b4.fits'},
'pixelflat': {'nrcb5': {'f322w2': {'clear': 'my_favorites/lw_flat.fits',
'grismr': 'my_favorites/lwR_flat.fits',
'grismc': 'my_favorites/lwC_flat.fits'
}
},
'nrcb4': {'f070w': {'clear': 'my_SW_favs/sw_flat.fits'}}
},
'photom': {'nrca5': 'my_favorites/photom_file.fits'}
}
}
Signal Limits for the Segmentation Map¶
Accompanying the noiseless seed image, Mirage will also produce a segmentation map containing all sources on the detector. Mirage makes use of this segmentation map only when producing spectroscopic data (i.e. WFSS or grism TSO mode). For all other modes, the segmentation map is informational only. In order to save time when making spectroscopic data, only pixels identified in the segmentation map as belonging to a source are run through the disperser code. Mirage uses a simple threshold value to determine which pixels to add to each source in the segmentation map. Pixles with signal values equal to or above the threshold are added. Mirage uses a default value of 0.031 ADU/sec as the floor value. Depending on the brightness and size of your targets, you may want to focus only on the brighter targets, or include everything. Therefore, this is a tunable parameter which users can specify in a variety of units. To specify your own segmentation map signal threshold when creating yaml files, provide your values to the yaml_generator. Note that if you omit these keywords, Mirage will use the default value.
minimum_signal = 0.08
minimum_signal_units = 'MJy/sr'
The allowed units for the threshold value are: ADU/sec, e/sec, MJy/sr, erg/cm2/A, erg/cm2/Hz Note that these are case insensitive.
Add Optical Ghosts to NIRISS Observations¶
If you will be creating NIRISS simulations and have point sources, you can instruct Mirage to add optical ghosts corresponding to your astronomical sources. This is controlled using the add_ghosts and convolve_ghosts_with_psf keywords. add_ghosts is a boolean parameter describing whether or not to add ghosts to the simulation. The default for this is True. convolve_ghosts_with_psf is also a boolean. This parameter describes whether a stamp image containing the ghost should be convolved with the instrumental PSF prior to adding the ghost to the data. The default for this parameter is False. Details on how the ghosts are added and their morphology are given on the ghosts page.
ghosts = True
convolve_ghosts = False
Run the Yaml Generator¶
With the XML and pointing files in hand, and additional inputs defined above, Mirage’s yaml_generator.py module can be called to create the associated yaml files. We specify a location for the oputput yaml files using the output_dir keyword. We also define the directory into which the final simulated data will be placed, using the simulated_output_dir keyword. This information will be placed into the constructed yaml files.
Setting the use_linearized_darks option to True will cause the yaml_generator to look for linearized dark current files to use with the simulations. These files may be present in the collection of Mirage reference files. If linearized darks are not present, leaving this option as False will cause Mirage to use raw dark current ramps as inputs.
The reffile_defaults keyword can have one of two values, which induce slightly different behavior. The best value to use depends upon your use case.
crds (default) - This option will place the string ‘crds’ in the yaml file entries for CRDS reference files. When Mirage (i.e. the seed generator, dark prep, or observation generator) is run, it will query CRDS for the best reference files to use, and download those files to your CRDS_PATH directory if they are not already present. This has the advantage that your yaml files will always have Mirage use the latest, best reference files whenever they are used.
crds_full_name - This option will cause yaml_generator.py to query CRDS, which will identify and download the best reference files if they are not already in your CRDS_PATH. It will then place the names of these best reference files into the yaml files being created. This creates yaml files that will always use the same reference files whenever they are run, meaning the outputs should be consistent every time. In this case, if new reference files are delivered to CRDS, the yaml files, and therefore Mirage, will not know that information.
Set reffile_overrides equal to the name of your nested reference file dictionary, if present.
Set parameter_defaults equal to the dictionary of parameter values to use.
from mirage.yaml import yaml_generator
yam = yaml_generator.SimInput(xml_file, pointing_file, catalogs=catalogs, verbose=True,
output_dir='/location/to/place/yaml_files',
simdata_output_dir='/location/to/place/simulated_data',
cosmic_rays=crs, background=background, roll_angle=pav3,
dates=dates, datatype='raw', dateobs_for_background=False,
reffile_defaults='crds', reffile_overrides=reffile_overrides,
segmap_flux_limit=minmum_signal, segmap_flux_limit_units=minimum_signal_units,
add_ghosts=ghosts, convolve_ghosts_with_psf=convolve_ghosts
)
yam.use_linearized_darks = True
yam.create_inputs()
The outptut from this will be the collection of yaml files needed to run Mirage and create all of the simulated observation files. An example yaml file shows all of the parameters necessary when simulating an exposure.
See the Imaging and WFSS notebooks in the Mirage repository for examples of yaml_generator use.
Run Mirage¶
The collection of yaml files can then be fed into Mirage one at a time.
from glob import glob
from mirage import imaging_simulator
yaml_files = glob('*.yaml')
for yfile in yaml_files:
im = imaging_simulator.ImgSim()
im.paramfile = yfile
im.create()
Example yaml Input File¶
Below is an example yaml input file for Mirage. The yaml file used as the primary input to mirage contains a large number of telescope and instrument settings, reference files, and catalogs. The inputs are grouped by section:
newRamp section
Inst: instrument: NIRCam #Instrument name mode: imaging #Observation mode (e.g. imaging, wfss) use_JWST_pipeline: False #Use pipeline in data transformations Readout: readpatt: DEEP8 #Readout pattern (RAPID, BRIGHT2, etc) overrides nframe,nskip unless it is not recognized ngroup: 6 #Number of groups in integration nint: 1 #Number of integrations per exposure array_name: NRCB5_FULL #Name of array (FULL, SUB160, SUB64P, etc) intermediate_aperture: None # Name of intermediate aperture used in NIRCam Grism time series obs. PPS_aperture: NRCB5_FULL # Original aperture value supplied by PPS. filter: F250M #Filter of simulated data (F090W, F322W2, etc) pupil: CLEAR #Pupil element for simulated data (CLEAR, GRISMC, etc) Reffiles: #Set to None or leave blank if you wish to skip that step dark: None #Dark current integration used as the base linearized_darkfile: $MIRAGE_DATA/nircam/darks/linearized/B5/Linearized_Dark_and_SBRefpix_NRCNRCBLONG-DARK-60090141241_1_490_SE_2016-01-09T02h46m50_uncal.fits # Linearized dark ramp to use as input. Supercedes dark above badpixmask: crds # If linearized dark is used, populate output DQ extensions using this file superbias: crds #Superbias file. Set to None or leave blank if not using linearity: crds #linearity correction coefficients saturation: crds #well depth reference files gain: crds #Gain map pixelflat: None illumflat: None #Illumination flat field file astrometric: crds #Astrometric distortion file (asdf) photom: crds #Cal pipeline photom reference file ipc: crds #File containing IPC kernel to apply invertIPC: True #Invert the IPC kernel before the convolution. True or False. Use True if the kernel is designed for the removal of IPC effects, like the JWST reference files are. occult: None #Occulting spots correction image transmission: crds # Transmission image containing fractional throughput map. (e.g. to imprint occulters into fov subarray_defs: config #File that contains a list of all possible subarray names and coordinates readpattdefs: config #File that contains a list of all possible readout pattern names and associated NFRAME/NSKIP values crosstalk: config #File containing crosstalk coefficients filtpupilcombo: config #File that lists the filter wheel element / pupil wheel element combinations. Used only in writing output file filter_wheel_positions: config #File that lists the filter wheel element / pupil wheel element combinations. Used only in writing output file flux_cal: config #File that lists flux conversion factor and pivot wavelength for each filter. Only used when making direct image outputs to be fed into the grism disperser code. filter_throughput: /Users/me/mirage/mirage/config/placeholder.txt #File containing filter throughput curve nonlin: limit: 60000.0 #Upper singal limit to which nonlinearity is applied (ADU) accuracy: 0.000001 #Non-linearity accuracy threshold maxiter: 10 #Maximum number of iterations to use when applying non-linearity robberto: False #Use Massimo Robberto type non-linearity coefficients cosmicRay: path: $MIRAGE_DATA/nircam/cosmic_ray_library/ #Path to CR library library: SUNMIN #Type of cosmic rayenvironment (SUNMAX, SUNMIN, FLARE) scale: 1.5 #Cosmic ray rate scaling factor suffix: IPC_NIRCam_B5 #Suffix of library file names seed: 2956411739 #Seed for random number generator simSignals: pointsource: my_point_sources.cat #File containing a list of point sources to add (x,y locations and magnitudes) psfpath: $MIRAGE_DATA/nircam/gridded_psf_library/ #Path to PSF library gridded_psf_library_row_padding: 4 # Number of outer rows and columns to avoid when evaluating library. RECOMMEND 4. psf_wing_threshold_file: config # File defining PSF sizes versus magnitude add_psf_wings: True # Whether or not to place the core of the psf from the gridded library into an image of the wings before adding. psfwfe: predicted #PSF WFE value ("predicted" or "requirements") psfwfegroup: 0 #WFE realization group (0 to 4) galaxyListFile: my_galaxies_catalog.list extended: None #Extended emission count rate image file name extendedscale: 1.0 #Scaling factor for extended emission image extendedCenter: 1024,1024 #x,y pixel location at which to place the extended image if it is smaller than the output array size PSFConvolveExtended: True #Convolve the extended image with the PSF before adding to the output image (True or False) movingTargetList: None #Name of file containing a list of point source moving targets (e.g. KBOs, asteroids) to add. movingTargetSersic: None #ascii file containing a list of 2D sersic profiles to have moving through the field movingTargetExtended: None #ascii file containing a list of stamp images to add as moving targets (planets, moons, etc) movingTargetToTrack: None #Catalog containing a single moving target which JWST will track during observation (e.g. a planet, moon, KBO, asteroid) This file will only be used if tracking is set to "non-sidereal" tso_imaging_catalog: None #Catalog listing TSO source to be used for imaging TSO simulations tso_grism_catalog: None #Catalog listing TSO source to be used for grism TSO observations zodiacal: None #Zodiacal light count rate image file zodiscale: 1.0 #Zodi scaling factor scattered: None #Scattered light count rate image file scatteredscale: 1.0 #Scattered light scaling factor bkgdrate: medium #Constant background count rate (ADU/sec/pixel in an undispersed image) or "high","medium","low" similar to what is used in the ETC poissonseed: 2012872553 #Random number generator seed for Poisson simulation) photonyield: True #Apply photon yield in simulation pymethod: True #Use double Poisson simulation for photon yield expand_catalog_for_segments: False # Expand catalog for 18 segments and use distinct PSFs use_dateobs_for_background: False # Use date_obs value to determine background. If False, bkgdrate is used. signal_low_limit_for_segmap: 0.031 # Lower signal limit for a pixel to be included in the segmentation map signal_low_limit_for_segmap_units: ADU/sec # Units of signal_low_limit_for_segmap. Can be: ADU/sec, e/sec, MJy/sr, ergs/cm2/a, ergs/cm2/hz add_ghosts: True # Add optical ghosts to simulation PSFConvolveGhosts: False # Convolve ghost sources with instrument PSF before adding Telescope: ra: 53.1 #RA of simulated pointing dec: -27.8 #Dec of simulated pointing rotation: 0.0 #y axis rotation (degrees E of N) tracking: sidereal #sidereal or non-sidereal Output: file: jw42424024002_01101_00001_nrcb5_uncal.fits # Output filename directory: ./ # Directory in which to place output files datatype: linear,raw # Type of data to save. 'linear' for linearized ramp. 'raw' for raw ramp. 'linear,raw' for both format: DMS # Output file format Options: DMS, SSR(not yet implemented) save_intermediates: False # Save intermediate products separately (point source image, etc) grism_source_image: False # Create an image to be dispersed? unsigned: True # Output unsigned integers? (0-65535 if true. -32768 to 32768 if false) dmsOrient: True # Output in DMS orientation (vs. fitswriter orientation). program_number: 42424 # Program Number title: Supernovae and Black Holes Near Hyperspatial Bypasses #Program title PI_Name: Doug Adams # Proposal PI Name Proposal_category: GO # Proposal category Science_category: Cosmology # Science category target_name: TARG1 # Name of target target_ra: 53.1001 # RA of the target, from APT file. target_dec: -27.799 # Dec of the target, from APT file. observation_number: '002' # Observation Number observation_label: Obs2 # User-generated observation Label visit_number: '024' # Visit Number visit_group: '01' # Visit Group visit_id: '42424024002' # Visit ID sequence_id: '1' # Sequence ID activity_id: '01' # Activity ID. Increment with each exposure. exposure_number: '00001' # Exposure Number obs_id: 'V42424024002P0000000001101' # Observation ID number date_obs: '2019-10-15' # Date of observation time_obs: '06:29:11.852' # Time of observation obs_template: 'NIRCam Imaging' # Observation template primary_dither_type: NONE # Primary dither pattern name total_primary_dither_positions: 1 # Total number of primary dither positions primary_dither_position: 1 # Primary dither position number subpix_dither_type: 2-POINT-MEDIUM-WITH-NIRISS #Subpixel dither pattern name total_subpix_dither_positions: 2 # Total number of subpixel dither positions subpix_dither_position: 2 # Subpixel dither position number xoffset: 344.284 # Dither pointing offset in x (arcsec) yoffset: 466.768 # Dither pointing offset in y (arcsec)
Instrument secton¶
This section of the input yaml file contains information about the instrument being simulated.
Instrument Name¶
Inst:instrument
The name of the JWST instrument to be simulated. The simulator will only function if ‘NIRCam’, ‘NIRISS’, or ‘FGS’ is placed in this field.
Observing mode¶
Inst:mode
The observing mode to be simulated. There are three valid options for this field. “imaging” will create imaging data, “wfss” will produce wide field slitless spectroscopic data. The other accepted input is “ami” when simulating NIRISS, although this mode is functionally identical to the use of “imaging”.
Create data using JWST pipeline¶
Inst:use_JWST_pipeline
True/False. Set to False if you wish to proceed without using any JWST pipeline functions. In this case, the input dark current exposure must already be linearized, as the pipeline is used for the linearization process. True is recommneded.
Readout section¶
This section of the yaml file contains inputs describing the details of the exposure, including the readout pattern, filter, subarray, etc to use.
Readout pattern¶
Readout:readpatt
This is the name of the readout timing pattern used for the output simulated exposure. Examples for NIRCam include RAPID, BRIGHT1, BRIGHT2, and DEEP8. Each pattern averages and skips a predefined number of frames when constructing each group of an integration. The list of possible readout patterns and their definitions is provided by an ascii file specified in the readpattdefs parameter in the Reffiles section of the input file. A more detailed description of readout patterns is given in the detector readout pages for NIRCam, NIRISS, and FGS.
Number of groups per integration¶
Readout:ngroup
This parameter lists the number of groups comprising each output integration.
Number of integrations per exposure¶
Readout:nint
The number of integrations in the output exposure. Each integration is composed of ngroup groups. Note that currently, any observation containing a moving target (non-sidereal observation with trailed sidereal objects, or vice versa) cannot have an nint value greater than 1. (IS THIS STILL TRUE?)
Number of detector resets between integrations¶
Readout:resets_bet_ints
The number of detector resets between integrations within a single exposure. For all instruments, this should be set to 1.
Array Name¶
Readout:array_name
This is the name of the aperture used for the simulated data. Generally, this is composed of the name of the detector combined with the name of the subarray used. For example, a full frame observation using NIRCam’s A1 detector has an array_name of ‘NRCA1_FULL’, while a full frame NIRISS observation will have an array_name of ‘NIS_CEN’. The list of possible array_name values are given in the subarray_defs input file described below. The array_name is used to identify several other characteristics of the simulated data, including the detector to use, as well as the proper array dimensions and location on the detector.
Intermediate Aperture¶
Readout:intermediate_aperture
This is the name of the intermediate aperture designated by APT to control the telescope pointing during Grism Time Series observations. The intermediate aperture specifies the location of the target prior to inserting the grism into the beam. When the grism is placed in the beam, the target’s trace will be offset from this location by a few tens of rows. The intermediate aperture is defined such that the target’s trace will land at the reference location of the user-specified aperture. This is only used for Grism Time Series observations, including the shortwave imaging mode data that accompany the dispersed longwave data. For all other observations, this will be set to None, and ignored.
PPS_aperture¶
Readout:PPS_aperture
This is the name of the aperture as supplied by PPS (i.e. in the APT pointing file). The value of this may include apertures that are not specific to one detector (e.g. NRCBS_FULL).
Filter¶
Readout:filter
The name of the filter wheel element to use for the simulated data. (e.g. F444W). The filter is used when scaling astronomical sources from the requested brightness in magnitudes to counts on the detector. For NIRCam simulations, the filter name is also used to determine whether the simulated data are to be produced using a shortwave or longwave detector. Lists of instrument filters can be found on the NIRCam, NIRISS, and FGS filter pages.
Pupil¶
Readout:pupil
The name of the pupil wheel element to use for the simulated data. Some filters for both NIRCam and NIRISS reside in their respective pupil wheels. Therefore this entry is checked when deciding upon scaling factors for simulated sources. Pupil wheel elements are desribed in the NIRCam, NIRISS, and FGS pupil wheel pages.
Reffiles section¶
This section of the input file lists the various reference files needed for the various steps of the simulator to run.
Dark current exposure¶
Reffiles:dark
The name of the raw dark current file that will be used as the basis for the simulated exposure. This file must be in raw format, such that no JWST calibration pipeline steps have been applied to the data. If an already-linearized dark current integration is to be used, that file name should be placed in the linearized_darkfile field below. Note that the linearized_darkfile entry will take precedence. Only if that is set to __None__ will the file listed in this field be used.
The dark current integration must have a readout pattern of either RAPID/NISRAPID/FGSRAPID or a value identical to that of the integration to be simulated. RAPID/NISRAPID/FGSRAPID data keep every readout frame with no averaging. From this, any other readout pattern can be simulated by averaging and skipping the appropriate frames. Other readout patterns cannot be translated in this way as their data are already averaged or missing some frames. However if simulating, for example a BRIGHT2 integration, then the input dark current integration can be a BRIGHT2 integration, as no translation is necessary in this case.
If a translation between RAPID and another readout pattern is necessary, then frames will be averaged/skipped as necessary. If the input dark current integration does not contain enough frames to be translated into the requested number of output groups, then the script creates enough additional dark current frames to make up the difference. These additional frames are created by making a copy of an appropriate number of existing initial dark current frames, and adding their signals to that in the final dark current frame. Note that this can lead to apparent double cosmic rays in pixels where a cosmic ray appeared in the dark current integration.
Hint
This input can only be used if use_JWST_pipeline is set to True.
Hint
The collection of reference files associated with Mirage contains a small library of raw dark current exposures that can be used.
Linearized dark current exposure¶
Reffiles:linearized_darkfile
The name of a linearized dark current integration to use as input for the simulated data. This file should contain a dark integration that has been processed through the superbias subtraction, reference pixel subtraction, and linearity steps of the JWST calibration pipeline. The resulting linearized signal must be saved in an extension with the name ‘SCI’. Also, the subtracted signal from the superbias and reference pixels must be saved in an extension called ‘SBANDREFPIX’. This output will be produced and saved for a given dark current file by Mirage.
Using this input rather than the uncalibrated dark above can save significant computing time, especially in the case of creating many output exposures.
Hint
This input can be used for use_JWST_pipeline set to True or False.
Hint
The collection of reference files associated with Mirage contains a small library of linearized dark current products that can be used.
Bad pixel mask¶
Reffiles:badpixmask
If a linearized dark current file is to be used and a linearized output file is requested, this optional bad pixel mask can be used to populate the data quality array in the output simulated data file. The file must be in the format for JWST bad pixel masks that is used by the JWST calibration pipeline.
Hint
Setting this entry equal to ‘crds’ will cause Mirage to query the Calibration Reference Database System (CRDS) for the appropriate file, and download that file if it is not already present in your CRDS cache.
Superbias¶
Reffiles:superbias
The superbias reference file for the detector of the simulation. This file must match the format of the JWST pipeline superbias reference file. If the input dark current integration is a raw file then this superbias file is used to subtract the superbias from the dark. If the input dark is already linearized, this superbias file is not used.
Hint
Setting this entry equal to ‘crds’ will cause Mirage to query the Calibration Reference Database System (CRDS) for the appropriate file, and download that file if it is not already present in your CRDS cache.
Linearity correction coefficients¶
Reffiles:linearity
Name of the reference file containing the linearity correction coefficients. This file must be in the format expected by the JWST calibration pipeline. If the input dark current integration is raw, the coefficients contained in this file are used to linearize the dark current after subtracting the superbias and reference pixel signal. These coefficients are also used to “unlinearize” the final simulated exposure if a raw simulated observation is requested.
In addition, the coefficients in this file are used to linearize the values in the saturation reference file, such that saturated signals in the linear simulated exposure can be found.
Hint
Setting this entry equal to ‘crds’ will cause Mirage to query the Calibration Reference Database System (CRDS) for the appropriate file, and download that file if it is not already present in your CRDS cache.
Saturation¶
Reffiles:saturaiton
Name of the reference file containing a map of the saturation signal level for all pixels. If the input dark current integration is raw, this file is used by the calibration pipeline to flag saturated pixels in the dark current integration prior to linearizing. The format of this file must match that used in the saturation flagging step of the JWST calibration pipeline.
This saturation map, after being linearized, is also used to search for saturated signal values in the combined dark current/simulated source exposure prior to unlinearizing.
Hint
Setting this entry equal to ‘crds’ will cause Mirage to query the Calibration Reference Database System (CRDS) for the appropriate file, and download that file if it is not already present in your CRDS cache.
Gain¶
Reffiles:gain
Name of the file containing the gain map appropriate for the detector being used. The gain is used to translate the cosmic rays, which are in units of electrons, to units of ADU prior to adding them to the simulated data. The format of the gain file must match that used by the JWST calibration pipeline.
Hint
Setting this entry equal to ‘crds’ will cause Mirage to query the Calibration Reference Database System (CRDS) for the appropriate file, and download that file if it is not already present in your CRDS cache.
Pixel-to-pixel flat field image¶
Reffiles:pixelflat
Name of the pixel flat file to use. Once the simulated integration is created, the result is multiplied by the pixel flat. This is done to un-flatten the image.
Illumination flat (L-flat)¶
Reffiles:illumflat
Name of the illumination flat to use. Once the simulated integration is created, the result is multiplied by the illumination flat.
Astrometric distortion file¶
Reffiles:astrometric
Name of the astrometric distortion reference file to use for including the effects of distortion in the simulated data. This file is used to translate input source locations between RA and Dec coordinates and pixel x and y coordinates, and vice versa. This file must be in asdf format and match that expected by the calibration pipeline.
Hint
Setting this entry equal to ‘crds’ will cause Mirage to query the Calibration Reference Database System (CRDS) for the appropriate file, and download that file if it is not already present in your CRDS cache.
Photom Reference File¶
Reffiles:photom
Name of the JWST calibration pipeline photom reference file to use. This file is used to translate the minimum flux level for pixel inclusion in the segmentation map to ADU/sec in the case where the user provides that value in MJy/sr.
Hint
Setting this entry equal to ‘crds’ will cause Mirage to query the Calibration Reference Database System (CRDS) for the appropriate file, and download that file if it is not already present in your CRDS cache.
Interpixel capacitance (IPC)¶
Reffiles:ipc
File containing the interpixel capacitance (IPC) kernel to apply to the simulated data in order to introduce IPC effects. After all simulated objects have been added to a count rate image, the image is convolved with the IPC kernel. The IPC file must be a fits file with the IPC kernel located in the first (rather than 0th) extension. Typical JWST IPC reference file kernels are a 3x3 array, but Mirage supports kernels of any odd-numbered size, as well as 4-dimensional kernels, where there is a separate 2-dimensional kernel for each pixel. In order to introduce, rather than remove, IPC effects, the kernel must be normalized and have a value in the central pixel which is less than 1.0. This is the inverse of the kernel used in the JWST calibration pipeline IPC removal step, where the central pixel has a value greater than 1.0, and negative values in surrounding pixels. For the simulator, the user can specify a JWST calibration pipeline-formatted kernel file, and then set the invertIPC flag below to True, in which case the kernel will be inverted before using.
Hint
Setting this entry equal to ‘crds’ will cause Mirage to query the Calibration Reference Database System (CRDS) for the appropriate file, and download that file if it is not already present in your CRDS cache.
Invert IPC¶
Reffiles:invertIPC
If set to True, the IPC kernel supplied through the ipc entry is inverted before convolving with the signal rate image. JWST IPC kernel reference files contain the kernel necessary to remove IPC from the data. Therefore these kernels must be inverted before they can add IPC effects to the data in the simulator.
Occulting spot image¶
Reffiles:occult
This feature is not yet supported and should be set to None.
Transmission Image¶
Reffiles:transmission
Fits file containing the transmission image for the detector/filter/pupil to be simulated. The values in this image are the transmission fraction for each pixel, and the image is multiplied in to the seed image (prior to dispersing if simulating WFSS data). This image is designed to contain occulters/masks that are present within the field of view.
Hint
Setting this entry equal to ‘crds’ will cause Mirage to get the appropriate file from the collection of Mirage reference files. The ultimate source of these files are the GRISM_NIRCAM and GRISM_NIRISS repositories, which must be cloned from github during the Mirage installation process and placed within the Mirage reference files directory structure. In the future, we anticipate that the transmission files will be hosted by the CRDS system, which is why they are treated similarly to the current CRDS reference files.
Subarray definition file¶
Reffiles:subarray_defs*
Name of a whitespace-delimited ascii file that lists all of the possible supported subarray apertures. This file is provided with the MIRAGE repository, in the config subdirectory.
Hint
To use the subarray definition files packaged with Mirage, set this to config in the input yaml file. This is the default when creating yaml files from an APT file using the yaml generator
For each subarray, the file must list the full aperture name (e.g. NRCA1_FULL) as well as the corresponding name used in proposal planning (e.g. FULL), as well as the number of amplifiers used to read out each aperture.
Readout pattern definition file¶
Reffiles:readpattdefs
Ascii file which gives the definitions of the possible readout patterns for the instrument. For each readout pattern, the number of frames averaged to create each group (nframe) and the number of frames skipped beteren each group (nskip) must be specified, as well as the maximum number of allowed groups. For a given readout pattern the simulator will search the entries in this file in order to determine the proper nframe and nskip values to use. The current lists of acceptable NIRCam and NIRISS readout patterns are given on the NIRCam and NIRISS detector readouts webpages. These files for all instruments are provided with the MIRAGE repository, in the config subdirectory.
Hint
To use the readout pattern definition files packaged with Mirage, set this to config in the input yaml file. This is the default when creating yaml files from an APT file using the yaml generator
Crosstalk¶
Reffiles:crosstalk
Ascii file containing crosstalk coefficients. Crosstalk is only applied to data read out through more than one amplifer. The file contains one row for each detector. Each row contains all of the coefficients necessary to fully describe crosstalk. This file is contained in the MIRAGE repository, in the config subdirectory.
Hint
To use the crosstalk coefficient files packaged with Mirage, set this to config in the input yaml file. This is the default when creating yaml files from an APT file using the yaml generator
Allowed filter/pupil combinations¶
Reffiles:filtpupilcombo
Name of an ascii file containing a list of the filter and pupil wheel elements in place when requesting simulated data for a given filter. This information is used to apply the appropriate conversion between magnitudes and counts when reading in source catalogs. This flux calibration is also added to the header of the seed image, as it is used when seed images are dispersed during the simulation of WFSS data. This file is present in the config subdirectory of the MIRAGE repository.
Hint
To use the filter and pupil wheel definition files packaged with Mirage, set this to config in the input yaml file. This is the default when creating yaml files from an APT file using the yaml generator
Filter/Pupil wheel resolver positions for each optical element¶
Reffiles:filter_wheel_positions
Name of an ascii file containing a list of all filter wheel and pupil wheel elements, along with the nominal wheel resolver positions for each. These values are in degrees. This information is passed directly to the header keywords FWCPOS and PWCPOS in the simulated data FITS files. This information is needed to compute the dispersion solution for NIRISS WFSS. Currently the header keywords are only populated for NIRISS observations.
Hint
To use the filter and pupil wheel position files packaged with Mirage, set this to config in the input yaml file. This is the default when creating yaml files from an APT file using the yaml generator
Flux calibration¶
Reffiles:flux_cal
Ascii file that lists flux conversion factors and the pivot wavelength associated with each filter. Conversion factors include ABMAG, STMAG, and VEGAMAG to counts per second, as well as FLAM (erg s -1 cm -2 Å -1 and FNU (erg s -1 cm -2 Hz -1 to counts per second. This file is used when producing seed images to be fed into the grism disperser code, as well as for translating catalog sources from magnitudes to counts per second. This file is provided with the MIRAGE repository, in the config subdirectory.
Hint
To use the flux calibration files packaged with Mirage, set this to config in the input yaml file. This is the default when creating yaml files from an APT file using the yaml generator
Filter Throughput¶
Reffiels:filter_throughput
Ascii files that contains the system throughput when using a particular filter. By default, the yaml generator will set this parameter to have a value of “placeholder.txt” in the input yaml files. Mirage will then locate the appropriate throughput file at runtime.
Nonlin section¶
The following input fields describe how non-linearity is treated in the input and simulated data.
Limiting Signal¶
nonlin:limit
Signal limit, in units of ADU, above which the linearity correction is not applied. Pixels with signals above this limit are considered saturated. This single value across the entire detector is only used if a saturation reference file is not provided.
Accuracy¶
nonlin:accuracy
When introducing non-linearity back into the linear data, the Newton-Raphson method is used to essentially run the JWST calibration pipline’s linearity correction step in reverse. The value of this accuracy parameter is the threshold below which the solution is considered to have converged. For example, an accuracy threshold of 0.000001 means that the unlinearization is considered complete when the ratio of the signal values from one iteration to the next is less than 1.000001.
Maximum number of iterations¶
nonlin:maxiter
The maximum number of iterations of the Newton-Raphson method to use when introducing non-linearity back into the data before declaring failure. Default is 10.
Robberto¶
nonlin:robberto
If set to False, the simulator assumes that the non-linearity correction function and coefficients match those used in the JWST calibration pipeline. If set to True, the script assumes an alternate linearity function, as defined in Robberto (2010 , 2011). Currently, no coefficients for the latter method exist, implying this parameter should be set to False.
Cosmic ray section¶
Input parameters in this section describe how cosmic rays are added to the simulated data.
Path to cosmic ray library¶
cosmicRay:path
Path of the location of the cosmic ray library to use. The code was developed around the cosmic ray library produced by Robberto (2009). This library is included in the collection of reference files associated with Mirage. After extracting the library from the tar file, set this path to point to the top level directory of the cosmic ray library.
Library¶
cosmicRay:library
Specification of which cosmic ray library to choose cosmic rays from. Options are SUNMIN, SUNMAX, FLARE, each of which assumes a different cosmic ray rate. Details on the three types of libraries are given in Robberto (2009).
Scaling value for rate¶
cosmicRay:scale
Scaling factor to apply to the cosmic ray rate. For example, to simulate cosmic rays at a rate twice as high as that in SUNMIN, set library to SUNMIN and scale to 2.0
Suffix¶
cosmicRay:suffix
Filename suffix of the cosmic ray library files. The code was developed around files with the suffix of ‘IPC_NIRCam_XX’ where XX is the detector (e.g. B5) for NIRCam, ‘IPC_NIRISS_NIS’ for NIRISS, and ‘IPC_FGS_GUIDERy’ where y is 1 or 2, for FGS. These cosmic ray files are included in Mirage’s reference file collection. This field will be automatically populated with the correct suffix when creating yaml files using the yaml generator.
Seed for random number generator¶
cosmicRay:seed
Random number generator seed to use when selecting cosmic rays to add.
simSignals section¶
This section of the input file describes how sources and other signals are added to the simulated data.
Point source catalog file¶
simSignals:pointsource
Name of an ascii catalog file listing point sources to add to the simulated image. An example point source catalog is provided on the Catalogs page.
PSF library path¶
simSignals:psfpath
Path name to the PSF library to be used for adding point sources to the data. The code was developed around a PSF library constructed using WebbPSF (Perrin, 2014). This PSF library is included in the collection of Mirage reference files . Once that package is downloaded and the data files extracted from the tar file, set this field to point to the top-level directory of the PSF library.
Gridded PSF Library Row Padding¶
The number of outer rows and columns to crop when evaluating the PSF library. This is done to avoid edge effects that can sometimes be present in the evaluated PSF. Recommended and default value is 4.
PSF Wing Threshold File¶
Ascii file that defines the overall size of the PSF (in pixels) versus magnitude. Through this file, the user can tune the size of the PSFs in the
simulated data. If it is important for your science to see far out into the wings, you can enable that here. These files are located in the config
directory of the repo. There is one file per instrument. The default value for this keyword is config. In this case, Mirage will know to look
for the file in the config directory.
Add PSF Wings¶
Boolean value stating whether or not to place the core of the psf from the gridded library into an image of the wings before adding.
PSF library wavefront error¶
simSignals:psfwfe
PSF wavefront error value to use when choosing PSF files from the PSF library. The current PSF libraries distributed with the Mirage reference files have two options for wavefront error: “predicted” and “requirements”. These two values represent the predicted in-flight wavefront errors, and the maximum allowed wavefront errors, respectively.
PSF realization number¶
simSignals:psfwfegroup
The current PSF library contains 5 different realizations for each filter/wavefront error-specified PSF. In this field, place the realization number to use. With 5 realizations present in the library, this field can have a value of 0 through 4.
Galaxy source catalog file¶
simSignals:galaxyListFile
Similar to the pointsource entry, this is an ascii catalog file containing a list of the galaxies to simulate in the data. See the galaxies entry on the catalogs page for an example of this file.
Extended source catalog file¶
simSignals:extended
Name of an ascii file containing a list of “extended images” to add to the simulated data. These are stamp image of sources, contained in small fits files. These stamp images are read in, scaled to the requested magnitude, and added to the seed image. This is a way to add objects other than point sources or 2D Sersic profiles to the data. The extended catalog section of the catalogs page shows an example extended source catalog.
Extended source scaling factor¶
simSignals:extendedScale
Multiplicative factor by which to scale the data in the extended image file before adding to the simulated data. The extended image is multiplied by this factor if the magnitude is set to None in the extended catalog file.
Extended source center location¶
simSignals:extendedCenter
In the case where a single extended source is provided, this entry can be set to the (x,y) pixel location at which to place the center of the exteded image. This functionality is largely replaced by specifying the RA, Dec or x, y of the extended image in the extended source catalog file.
Convolve extended sources with PSF¶
simSignals:PSFConvolveExtended
True/False. Convolve the extended image with the appropriate instrumental PSF prior to adding to the output image.
Moving target source catalog file¶
simSignals:movingTargetList
Similar to the point source list file, this is a file containing a list of targets to treat as moving (non-sidereal) targets. These sources will move through the field of view as the exposure progresses. This is the list to use if you wish to insert an asteroid or KBO that is moving through the field of view of your observation. See the moving point source section on the Catalogs page for an example.
2D Sersic profile moving target catalog file¶
simSignals:movingTargetSersic
Similar to the galaxy target list file, this file contains a list of galaxies (2D Sersic profiles) to be used as moving targets. These sources will move through the background of the simulated data. This may be useful for inserting a resolved moon/asteroid into the scene. An example file is shown in the Moving Sersic section of the Catalogs page.
Moving extended source catalog file¶
simSignals:movingTargetExtended
Similar to the extended target list, this is an ascii file listing extended targets to move through the background of the image. A description and example of this file are shown in the Moving Extended section of the Catalogs page.
Tracked non-sidereal target catalog file¶
simSignals:movingTargetToTrack
This ascii catalog file is used for what are traditionally (in HST jargon) called ‘moving targets’. Targets listed in this file are treated as non-sidereal targets that JWST will track during the simulated observation. In this case, the target listed in this file will appear static in the output data, but all other sources (e.g. those listed in pointSource, galaxyListFile, and extended) will all appear trailed through the data. A description and example of the file are shown in the Non-sidereal Source section on the Catalogs page.
TSO Imaging Catalog¶
simSignals:tso_imaging_catalog
Ascii catalog file containing information on the source to be used when creating imaging TSO observations. The catalog format is detailed in the Imaging TSO Catalog section section of the Source Catalog Formats page.
TSO Grism Catalog¶
simSignals:tso_grism_catalog
Ascii catalog file containing information on the source to be used when creating grism TSO observations. The catalog format is detailed in the Grism TSO Catalog section section of the Source Catalog Formats page.
Zodiacal light¶
simSignals:zodiacal
This keyword has been depricated in favor of obtaining the zodiacal light from the JWST backgrounds package.
Name of a file containing a 2 dimensional count rate image of zodiacal light. This file is read in, scaled by the zodiscale value, and added to the seed image. Leave as None to skip this step. The behaviors of this step and the scattered step below are very basic, and identical. There are no requirements on what the count rate images in these files must look like.
Tip
Note that the bkgdrate input parameter, when set to “high”, “medium”, or “low”, will return a background rate image that includes the contribution from zodiacal light, in which case this step should be set to None.
Scaling factor for zodiacal light image¶
simSignals:zodiscale
Scaling factor to multiply the zodiacal light count rate image by prior to adding to the output data.
Scattered light image¶
simSignals:scattered
This keyword is currently not supported.
Scattered light count rate image file. This file is assumed to contain a 2-dimensional array of signals in units of ADU per second. The file is read in, scaled by the scatteredscale value, and added to the seed image. Leave as None to skip this step.
Scattered light scaling factor¶
simSignals:scatteredscale
Scaling factor to multiply the scattered light count rate image by prior to adding to the seed image.
Background signal¶
simSignals:bkgdrate
This entry, in combination with the use_dateobs_for_background and date_obs parameters, controls the background signal that is added to simulations. The text below describes the way Mirage interprets the various input options:
Imaging Mode (both NIRCam and NIRISS)
Number: The input value is assumed to be in units of ADU/pixel/second. This constant background value is placed in all pixels.
“low”, “medium”, or “high”. If one of these options is used, the simulator uses the jwst_backgrounds repository to calculate the background rate to apply to the simulated data. The package calculates the background signal at the requested pointing on the sky for each night over the course of a year and creates a histogram of these values. If the requested background is “low” then the returned background level is equal to that of the 10th percentile in the histogram. A “medium” background corresponds to the 50th percentile value, and “high” is the 90th percentile value. In this case, the returned background rate includes contributions from zodiacal light and telescope thermal emission.
use_dateobs_for_background set to True: (NOTE: currently the bkgdrate value must be set to “low”, “medium”, or “high” when using this option. If it is set to a number, then that number will be used and use_dateobs_for_background will be ignored.) This is similar to the “low”, “medium”, “high” case above, but instead of calculating the background based on a percetile of the distribution of background values, Mirage will select the background value associated with the date in the date_obs parameter.
WFSS Mode
NIRCam
Number: Not supported
“low”, “medium”, or “high”. Similar to the imaging case above. In this case, the background spectrum matching the percentile value is kept. This is fed into the disperser software, which generates a 2D background image.
use_dateobs_for_background set to True. The background spectrum for the date in the date_obs parameter is fed into the disperser, which generates a 2D background image.
NIRISS
Number: The input number is assumed to be the desired background value in ADU/pixels/second in the undispersed view of the scene. To get the background value in the dispersed image, this number is multiplied by the throughput of the NIRISS grism, which is about 80%. The dispersed background image, which is in the collection of Mirage reference files, is then scaled such that the mean value is equal to the calculated dispersed background value.
“low”, “medium”, or “high”. Same as in the imaging case above. The calculated backrgound value will be multiplied by the throughput of the NIRISS grism, which is about 80%.
use_dateobs_for_background. Not supported
Note that background rates associated with the “low”, “medium”, and “high” values are calculated in the same way as when they are used in the JWST ETC.
Seed value for poisson noise generator¶
simSignals:poissonseed
Random number generator seed used for Poisson simulation
Photon Yield¶
simSignals:photonyield
This keyword is currently not used. T/F. Set this to True to include the effects of photon yield in the simulation outputs.
Photon yield method¶
simSignals:pymethod
This keyword is currently not used. T/F. Whether or not to use the double photon method when applying photon yield.
Expand catalog for segments¶
simSignals:expand_catalog_for_segments
This entry controls whether Mirage will look for a separate point source library for each of the mirror segments on the telescope. This mode is only used for certain wavefront sensing and control observations and should normally be set to False.
Use date_obs for background¶
simSignals:use_dateobs_for_background
This entry controls the way the background signal for the observation is calculated. If it is True, then the background value will be created by extracting the background spectrum assoicated with date_obs from the jwst_backgrounds package. If False, the background will be determined by calculating the background value at a certain percentile of the collection of backgrounds for the given pointing over 365 days. If bkgdrate is “low”, “medium”, “high”, then the percentiles used are 10th, 50th, and 90th, respectively. If it is a float, that value (in ADU/sec/pixel) will be added to all pixels.
Lower Signal Limit for the Segmentation Map¶
simSignals:signal_low_limit_for_segmap
When adding an astrophysical source to the seed image, this is the lower threshold value used to control which pixels in the source are added to the segmentation map. Pixels with signal rates below this threshold will not be included. Mirage uses the segmentation map for WFSS simulations. Only object pixels that are present in the segmentation map are dispersed. This is done, rather than dispersing all pixels, in order to save compute time. The value can be given in a number of units. See signal_low_limit_for_segmap_units below for a list of valid units.
Units for the Lower Signal Limit for the Segmentation Map¶
simSignals:signal_low_limit_for_segmap_units
This field gives the units associated with the value in the :ref: signal_low_limit_for_segmap <signal_low_limit_for_segmap> value above. Supported units include: ADU/sec, ADU/s, e/sec, e/s, MJy/sr, ergs/cm2/a, ergs/cm2/hz. Note that this field is case insensitive. If the signal limit is given in units other than ADU/sec, Mirage will convert the value to ADU/sec before comparing source signal levels and adding pixels to the segmentation map.
Add ghosts¶
simSignals:add_ghosts
If True, Mirage will add optical ghosts to the simulated data. Currently this is only supported for NIRISS F090W, F115W, F140M, F150W, and F200W. In simulations using other instruments or filters, this keyword will be ignored.
Convolve ghosts with PSF¶
simSignals:PSFConvolveGhosts
If True, optical ghosts sources will be convolved with the instrumental PSF before adding them to the simulation
Telescope section¶
Inputs in this section of the yaml file describe the telescope pointing to use for the simulation.
Right Ascension¶
Telescope:ra
Right ascension of the observation. This will be the RA at the reference location on the detector being used for the simulation. The reference location varies with the requested subarray, but is generally in the center of the field of view. This input can be a string “HH:MM:SS.sss”, or a float in decimal degrees.
Declination¶
Telescope:dec
Declination of the observation. This will be the Dec at the reference location on the detector. The reference location varies with the requested subarray, but is generally in the center of the field of view. This input can be a string “DD:MM:SS.sss” or a float in decimal degrees.
Rotation¶
Telescope:rotation
Rotation of the y-axis in degrees East of North. Currently this rotation is defined around the reference location of the chosen subarray.
Telescope tracking¶
Telescope:tracking
Either ‘sidereal’ or ‘non-sidereal’ depending on the type of exposure. If it is set to non-sidereal then the exposure will be created as if JWST is tracking on the source in the movingTargetToTrack catalog. Sources in the pointsource, galaxyListFile, and extended catalogs will trail across the field of view over the course of the exposure.
Output section¶
This section of the yaml file contains information about the output file, such as filename and location. In addition, this section contains a large number of fields that describe how this particular exposure fits within an observing program/proposal. This information is not used during the creation of the simulated data, but is placed in the header of the output file in order to be consistent with the contents of real JWST data files. In addition, level 3 of the JWST calibration pipeline, which is used to combine multiple exposures into mosaic images, does require some of this information. The easiest way to correctly populate this information in the simulator yaml files is to create the yaml files from an APT file via yaml_generator.py, in which case the fields are all populated automatically.
Output filename¶
Output:file
Filename of the output simulated file (e.g. jw42424024002_01101_00001_nrcb5_uncal.fits). If the linearized ramp is requested as output in the datatype field, it will be saved with ‘uncal’ replaced with ‘linear’ in the filename or if ‘uncal’ is not present, ‘linear’ will simply be appended to the filename. If the raw ramp is requested as output, the given filename will be used with no changes.
We recommend using filenames that end in ‘uncal.fits’ in order to be consistent with JWST file naming conventions. The filename is constructed from various pieces of information, including the program ID and visit number. If you wish to use this convention for the output filenames, the easiest way to accomplish this is to create the yaml files from an APT file, in which case the filenames will be generated automatically.
Output directory¶
Output:directory
The directory into which the output simulated data will be placed.
Data type¶
Output:datatype
List of the data format(s) of the output files. Options include: “linear”, where the output files will contain linearized signals with the superbias and reference pixel signals removed. Bad pixels will also be flagged if a bad pixel file is specified. These files are ready to be run through the jump detection and ramp fitting steps of the JWST calibration pipeline. “raw”, where the output files will be in an uncalibrated state. These files are ready to be run through the entirety of the calibration pipeline, beginning with calwebb_detector1. “linear,raw”, where both the raw and linearized versions of the output files will be saved.
Data format¶
Output:format
Format of the output file. Currently, only ‘DMS’ is supported, indicating that the fits file format, as well as header keywords, match those expected by the JWST calibration pipeline.
Save intermediate outputs¶
Output:save_intermediates
True/False. If True, intermediate products are saved to disk. These products are listed in the table below.
Module |
Suffix Appended to Output Filename |
Description |
|---|---|---|
Seed Image Generator |
_pointsources.list |
Ascii file listing point source x,y and RA, Dec positions as well as magnitude and count rate. |
_galaxySources.list |
Ascii file listing galaxy source x,y and RA, Dec positions, morphology parameters, magnitudes, and count rates. |
|
_extendedsources.list |
Ascii file listing extended source x,y and RA, Dec positions as well as magnitude and count rate. |
|
_pointSourceRateImage_elec_per_sec.fits |
Count rate image containing only added point sources |
|
_galaxyRateImage_elec_per_sec.fits |
Count rate image containing only added galaxies |
|
_extendedObject_elec_per_sec.fits |
Count rate image containing only extended objects |
|
_AddedSources_elec_per_sec.fits |
Count rate image containing all added sources |
|
Observation Generator |
_doNonLin_accuracy.fits |
Final accuracy map from the process where the linearized simulated exposure was “unlinearized” |
_xtalk_correction_image.fits |
Image of the crosstalk signal added to the exposure |
|
_cosmicrays.list |
Ascii file containing location and magnitude of added cosmic rays |
Grism output image¶
Output:grism_source_image
True/False. If True, the size of the output image is enlarged from the requested array size by a multiplicative factor in the x and y dimensions. For NIRCam this factor is √2, while it NIRISS it is 1.134. This extra area is required if the image is passed to the grism disperser software. In this case, the disperser software is able to include sources which fall just outside the nominal field of view but whose dispersed spectra fall into the nominal field of view.
Outputs in unsigned integers¶
Output:unsigned
T/F. If True, output signal values for raw data will be in units of unsigned integers. This matches the output of real JWST data.
Output data in DMS orientation¶
T/F. If True, data will be output in DMS orientation, as opposed to raw FITSwriter orientation. JWST data will be in DMS orientation.
Program number¶
Output:program_number
The proposal ID number. This is placed in the header of the output file in order to match the contents of real observation files.
Proposal title¶
Output:title
The title of the proposal. This placed in the header of the output file in order to match the contents of real observation files.
PI name¶
Output:PI_Name
Name of the proposal PI. This is placed in the header of the output file in order to match the contents of real observation files.
Proposal category¶
Output:proposal_category
Proposal category (e.g. GO, GTO). This is placed in the header of the output file in order to match the contents of real observation files.
Science category¶
Output:science_category
Science category of the proposal, as defined in the APT file. This is placed in the header of the output file in order to match the contents of real observation files.
Target Name¶
Output:target_name
Name of the target. For yaml files constructed from an APT file, this is the name of the target as input by the user. This value will be propagated into the TARGPROP keyword in the simulated data FITS files.
Target RA¶
Output:target_ra
RA of the target. For yaml files constructed from an APT file, this is the RA of the target as input by the user, translated to units of degrees. This value will be propagated into the TARG_RA keyword in the simulated data FITS files.
Target Dec¶
Output:target_dec
Declination of the target. For yaml files constructed from an APT file, this is the declination of the target as input by the user, translated to units of degrees. This value will be propagated into the TARG_DEC keyword in the simulated data FITS files.
Observation number¶
Output:observation_number
The observation number containing the output exposure, as defined in the program’s APT file. This is placed in the header of the output file in order to match the contents of real observation files.
Observation label¶
Output:observation_label
The observation label in the APT file under which the output exposure appears. This is placed in the header of the output file in order to match the contents of real observation files.
Visit number¶
Output:visit_number
The visit number, as defined in the APT file, within which the output exposure appears. This is placed in the header of the output file in order to match the contents of real observation files.
Visit group number¶
Output:visit_group
The visit group, as defined in the APT file, within which the output exposure appears. This is placed in the header of the output file in order to match the contents of real observation files.
Visit ID number¶
Output:visit_id
The visit identifier of the exposure. This can be created by combining the program ID, visit number, and observation number. This is placed in the header of the output file in order to match the contents of real observation files.
Sequence ID¶
Output:sequence_id
The parallel sequence identifier denotes whether the data were acquired during parallel observations, and with which instrument. Set to 0 for non-parallel observations, 1 for a parallel sequence using the primary instrument, or 2-5 for one of the non-prime instruments.
Activity ID¶
Output:activity_id
The activity identifier of the exposure is a base-36 number that is unique to each exposure in a proposal. This is placed in the header of the output file in order to match the contents of real observation files.
Exposure Number¶
Output:exposure_number
A five-character number used to identify the exposure within the current activity.
Observation ID¶
Output:obs_id
The observation ID is constructed from several of the other parameters. OBS_ID = ‘V’ + program_number + observation_id + visit_id + ‘P’ + parallel-program number + parallel-observation number + visit_group + parallel sequence identifier + activity_identifier.
Observation date¶
Output:date_obs
UTC date of the start of the exposure with format yyyy-mm-dd.
Observation time¶
Output:time_obs
UTC time of the start of the exposure with format hh:mm:ss.ssssss.
Observation template¶
Output:obs_template
The name of the observation template used for the exposure (e.g. NIRCam Imaging, NIRCam Time Series)
Primary dither type¶
Output:primary_dither_type
Name of the primary dither pattern in use when the data were obtained. For details, see the documentation pages on dither patterns for NIRCam, and NIRISS. (e.g. INTRAMODULEX, INTRASCA).
Number of primary dither positions¶
Output:total_primary_dither_positions
Total number of primary dither positions in the observation.
Primary dither position¶
Output:primary_dither_position
Primary dither position number of the exposure being simulated.
Subpixel dither type¶
Output:subpix_dither_type
Name of the subpixel dither pattern used for these data. Details on subpixel dither patterns can be found on the NIRCam subpixel dither patterns page.
Number of subpixel dither positions¶
Output:total_subpix_dither_positions
Total number of subpixel dither positions for this observation.
Subpixel dither position¶
Output:subpix_dither_position
The subpixel dither position number corresponding to the current exposure.
X offset¶
Output:xoffset
Offset in the x direction, in arcseconds, of the pointing used for the current exposure relative to the starting position of the dither pattern. This is used to populate header values only. It is not used to determine the pointing when creating the simulated data.
Y offset¶
Output:yoffset
Offset in the y direction, in arcseconds, of the pointing used for the current exposure relative to the starting position of the dither pattern. This is used to populate header values only. It is not used to determine the pointing when creating the simulated data.
Simulating Imaging data with Mirage¶
The simulation of imaging mode observations (with NIRCam, NIRISS, or FGS) with Mirage is done using the imaging_simulator.py module. As discussed on the 3 stages page, Mirage is broken down into three main stages. To streamline the creation of simulated data imaging_simulator.py wraps around these three stages. A single call to imaging_simulator.py with an yaml parameter file will create a single simulated exposure from one detector.
The imaging simulator has only two possible inputs: the yaml parameter file, and an option to input a dark current ramp of the proper format. Here is an example call:
from mirage.imaging_simulator import ImgSim
sim = ImgSim(paramfile='my_yaml_file.yaml')
sim.create()
If you have dark current products for your simulation from a previous run of Mirage, it is possible to provide these files to imaging_simulator.py as inputs using the override_dark parameter. In this case, the call to dark_prep.py will be skipped, which will save some computing time. Note that the dark current products must be specific to your exoposure, in that they must contain arrays of the proper shape. So in practice, this detail is useful if you are repeating a previous call to Mirage. In that case, the darks can be provided as shown below. In the case where a single dark file is needed, it can be provided as a string or a 1-element list. In cases where the exposure is broken into segments and there are multiple dark files, these files must be provided as a list.
from mirage.imaging_simulator import ImgSim
# Single dark file
dark = 'jw09996001001_01101_00001_nrcb5_uncal_linear_dark_prep_object.fits'
m = ImgSim(override_dark=dark)
m.paramfile = 'jw09996001001_01101_00001_nrcb5.yaml'
m.create()
# Exposure broken into multiple segments
dark = ['jw09996001001_01101_00001_nrcb5_uncal_seg001_linear_dark_prep_object.fits',
'jw09996001001_01101_00001_nrcb5_uncal_seg002_linear_dark_prep_object.fits']
m = ImgSim(override_dark=dark)
m.paramfile = 'jw09996001001_01101_00001_nrcb5.yaml'
m.create()
Tip
For more examples of calling the imaging simulator, see the Imaging Simulator example notebook, as well as the Simulations from input mosaic image notebook.
Simulating WFSS data with Mirage¶
Mirage can be used to simulate Wide Field Slitless Spectroscopy (WFSS) mode data for NIRCam and NIRISS, using the wfss_simulator.py module. To produce these simulations, Mirage constructs one or more imaging mode seed images, along with an associated segmentation map. These seed images and segmentation map are then passed to the disperser software, which is in the NIRCAM_Gsim package. The disperser then takes the seed images, along with an optional input file containing object spectra, and disperses the signal across the detector in the same manner as the NIRCam and NIRISS grisms.
This mode generally uses more input compared to the creation of imaging mode data, as spectral information on the input sources may be provided. There are two methods that spectral information
yaml file: must specify wfss mode and grism_source_image = True. The appropriate grism must be specified, as well as a crossing filter
Inputs¶
There are three types of inputs that can be used to create WFSS data. The first is the same yaml parameter file that is used when creating imaging mode data. Along with the yaml files, the appropriate ascii source catalogs must be provided. The third input, which is optional, is an hdf5 file that contains the spectra for some or all of the targets that are in the source catalogs. Below we describe how to use these inputs to create WFSS data.
Note that when using yaml files as inputs, at least one of these files must have the mode set to “wfss”, grism_source_image set to True, and a grism placed in the appropriate filter or pupil entry. This will be done automatically for the appropriate files during the yaml generation process if generating yaml files from an APT proposal.
Single yaml file¶
In the simplest case a single yaml parameter file is provided to the WFSS simulator module, along with a source catalog containing target magnitudes in a single filter. In this case, Mirage converts the provided magnitude values to flux densities, and the disperser assumes a flat continuum spanning the entire wavelength range of the filter/grism combination.
If the source catalog contains magnitudes in multiple filters, Mirage will, for each source, linearly interpolate the source magnitudes in order to construct a continuum spectrum. If the provided magnitudes do not cover the entire wavelength range necessary for the dispersion, then Mirage will optionally extrapolate the continuum spectrum to cover the full wavelength range.
Tip
In this case where only ascii source catalogs are provided, source spectra will not contain any features (emission, absorption), but will rather be smooth continuum spectra. In order to simulate sources with emission or absorption features, this information must be added via the hdf5 file described below.
Multiple yaml files¶
Another way to produce data with smooth continuum spectra is to provide multiple yaml files, where each yaml file will produce a seed image through a different filter. In this case, the mutiple seed images will be used to calculate source flux densities, rather than these calculations being done using the source catalogs as input. One of these yaml files must specify WFSS mode with the requested grism, as described above. The other yaml files should specify imaging mode. As this method produces the same output as that when a single yaml file and source catalog with multiple magnitude columns is provided, but with more calculations taking a longer time, we recommend against using this strategy.
Yaml file plus SED file¶
In order to create simulated data with more realistic spectra, users can provide an optional hdf5 file that contains spectra for some or all of the targets listed in the source catalogs. The spectra in this file can have any shape, and can include emission and absorption features. The spectrum for each source is contained in a “dataset”, which is read in as a python dictionary containing “wavelengths” and “fluxes” keys. The values associated with each of these fields is a list of floating point numbers. Units can be specified by adding a string as a dataset attribute. If no units are provided, Mirage assumes that wavelengths are in units of microns, and flux densisites are in units of F_lambda (erg/sec/cm:sup:2 /A). Mirage contains functions that can be used to create hdf5 files with their target SEDs.
Tip
The NIRISS WFSS example notebook shows examples of how to create your own hdf5 catalog file, and how to create WFSS data using the methods described above.
Skip the dark current prep step¶
Similar to the case for imaging mode simulations, if you have dark current products for your simulation from a previous run of Mirage, it is possible to provide these files to wfss_simulator.py as inputs using the override_dark parameter. In this case, the call to dark_prep.py will be skipped, which will save some computing time. Note that the dark current products must be specific to your exoposure, in that they must contain arrays of the proper shape. So in practice, this detail is useful if you are repeating a previous call to Mirage. In that case, the darks can be provided as shown below. In the case where a single dark file is needed, it can be provided as a string or a 1-element list. In cases where the exposure is broken into segments and there are multiple dark files, these files must be provided as a list.
from mirage.wfss_simulator import WFSSSim
# Single file
darks = ['jw01345007001_01101_00010_nrca5_uncal_linear_dark_prep_object.fits']
yfile = 'jw01345007001_01101_00010_nrca5.yaml'
c = WFSSSim(yfile, override_dark=darks)
c.create()
# File broken into multiple segments
darks = ['jw01345007001_01101_00010_nrca5_uncal_seg001_linear_dark_prep_object.fits',
'jw01345007001_01101_00010_nrca5_uncal_seg002_linear_dark_prep_object.fits']
yfile = 'jw01345007001_01101_00010_nrca5.yaml'
c = WFSSSim(yfile, override_dark=darks)
c.create()
Backgrounds in WFSS simulations¶
Dispersed background signals in WFSS exposures are created from pre-computed background images. First, Mirage will take the user-input date or low/medium/high values, and calculate a 1D background spectrum. (Mirage uses the observation date to determine the background only if the use_dateobs_for_background parameter in the input yaml file is set to True). This 1D background spectrum is then transformed into a 2D dispersed background image using the disperse_background_1d function in NIRCAM_Gsim . The 2D background image that will actually be used is then read in from the appropriate pre-computed file, and scaled by the maximum value in the 2D background image that was created from the 1D spectrum.
Simulating TSO data with Mirage¶
Mirage can be used to simulate both grism and imaging time series data for NIRCam. Grism Time Series Observations (TSO) are constructed using the grism_tso_simulator.py module. Imaging TSO data are produced using the same catalog_seed_image.py module that is used for standard imaging observations.
The easiest way to see how to create both types of TSO data is by reading through the TSO example notebook.
Yaml files¶
Generation of the appropriate input yaml files from an APT file is fully supported, as with other observing modes. Note that TSO observations often include a target acquisition (TA) exposure at the beginning of the observation. Mirage will generate a yaml file corresponding to the TA exposure in addition to the TSO exposure. Further, when simulating Grism Time Series observations, the two shortwave detectors with the save field of view as the lower half of the longwave detector will collect imaging time series data of the scene that is dispersed in the long wave detector. Mirage also generates yaml files for these accompanying observations.
User Inputs¶
As shown in the TSO example notebook, there are several required user inputs that are unique to TSO data. These inputs are listed in the subsections below. See the notebook for details on how to create each of these inputs.
Mirage uses two TSO-specific types of source catalogs. Descriptions of the **GrismTSOCatalog** and **ImagingTSOCatalog** are given on the catalogs page.
Note that Mirage relies on the Batman package to generate and work with lightcurves. Many of the input quantities required in the Grism TSO catalog are Batman-specific.
Both Grism and Imaging TSO data¶
Background source catalog: Point source, galaxy, or extended source catalog containing any background sources you wish to include in the simulation.
Grism TSO data¶
SED file of TSO target: This is the spectrum of the unocculted star associated with the TSO object.
Transmission spectrum of planet: Wavelength-dependent effective radius of the planet, in units of the stellar radius.
Grism TSO source catalog: Mirage-specific source catalog containing all information relevant to the source
Imaging TSO data¶
Lightcurve file: Tabulated list of the parent body’s flux versus time
Imaging TSO source catalog: Mirage-specific source catalog containing all information relevant to the source
Dark_prep files (optional)¶
If you have a fits file containing a dark current exposure that is the proper format (linearized, with the correct readout pattern and array size) for the simulated data you are creating, you can provide this via the override_dark keyword parameter. This will cause the dark current preparation step to be skipped, which will save some computing time. In practice, the only way to have a fits file with the properly formatted dark current exposure will be from previous runs of the imaging simulator (or dark prep step).
In the case where a single dark file is needed, it can be provided as a string or a 1-element list. In cases where the exposure is broken into segments and there are multiple dark files, these files must be provided as a list. For TSO observations, the latter case is much more likely.
For a grism TSO observation, darks can be provided as such:
from mirage.grism_tso_simulator import GrismTSO
sed_file = 'test_grism_tso_sed_file_wasp79.hdf5'
gr_tso_yaml_file = 'jw88888001001_01101_00002_nrca5.yaml'
darks_to_use = ['jw88888001001_01101_00002_nrca5_uncal_seg001_linear_dark_prep_object.fits',
'jw88888001001_01101_00002_nrca5_uncal_seg002_linear_dark_prep_object.fits']
m = GrismTSO(gr_tso_yaml_file, SED_file=sed_file, SED_normalizing_catalog_column=None,
final_SED_file=None, save_dispersed_seed=True, source_stamps_file=None,
extrapolate_SED=True, override_dark=None, disp_seed_filename=None, orders=["+1", "+2"])
m.create()
For an imaging TSO observation, the call the call looks the same as for regular imaging observations.
Mirage’s Three Stages¶
Mirage can be broadly divided into three stages. Mirage first creates an idealized, noiseless image of the astronomical scene. We refer to this as the “seed image”. It then reorganizes an existing dark current exposure to match the characteristics of the specified simulation. The seed image and the dark current exposure are then combined.
Below is a more detailed flow chart showing Mirage’s overall process for creating simulated data. Blue boxes show user inputs, and white boxes contain outputs.
Seed Images¶
The first stage is the creation of the “seed image”. This is a noiseless count rate image that contains signal only from the astronomical sources to be simulated. Seed images include instrument distortion effects, so that given RA, Dec values are properly converted to pixel x,y values, with the exception of tangent plane projection, which will be added soon. Mirage currently contains two methods for the construction of seed images:
Through the use of source catalog files
Extraction from a fits file containing a distortion-free image (e.g. HUDF, GOODS, CANDELS, etc)
For WFSS observations, multiple seed images or optional input spectra are then fed into the disperser software, which simulates the effects of the grism by dispersing the signal from the astronomical sources, creating a “grism seed image”.
For a detailed description of seed image creation, see the Seed Images page.
Dark Current Preparation¶
The second stage is the preparation of the dark current exposure to use for the simulation. The input dark current exposure is reorganized into the requested readout pattern and number of groups and cropped to the requested subarray size. Detector non-linearity effects are then removed using the initial steps of the JWST calibration pipeline.
By using actual dark current exposures from ground testing, Mirage is able to capture many effects which are specific to the instrument and detector being simulated. For example, the 1/f noise, bias structure, and hot pixel population.
Details of the dark current preparation are given on the Dark Preparation page.
Observation Generation¶
The final stage involves the combination of the seed image and the dark current in order to produce the output exposure. The seed image is expanded into integrations with groups that follow the requested readout pattern. Other effects are also added at this stage, including cosmic rays, interpixel capacitance (IPC) and crosstalk effects.
More details are provided on the Observation Generation page.
Creation of Seed Images¶
A “seed image” is a noiseless image that contains signal only from simulated astronomical sources, created during the first stage of Mirage. In addition to containing no noise, there are no cosmic rays nor dark current. The seed image can be used as a “truth image” for the purposes of quality checks on the final output data.
Mirage generates seed images in one of two ways:
Extraction from a fits file containing a distortion-free image (e.g. HUDF, GOODS, CANDELS, etc)
Seed images from source catalogs¶
To create a seed image from user-specified source catalogs, the catalog_seed_image.py module is used. There are two types of observations supported by this method, sidereal and non-sidereal, and several catalogs to go with each. The Catalogs page shows examples of each type of source catalog that can be used by Mirage.
For each type of target, Mirage creates or retrieves a small stamp image of the target based on the user-supplied parameters in the catalog. This stamp image is then placed into the seed image at the x,y or RA, Dec location specified by the user. Source locations on the detector take into account instrument distortion. If source locations in the catalogs are specified using RA and Dec, Mirage will use the distortion model from a distortion reference file if supplied in the input yaml file. If no reference file is given, Mirage will fall back to using distortion information in the Science Instrument Aperture File (SIAF).
For point sources, Mirage selects the appropriate PSF for each source from a PSF library. This PSF is scaled to the requested brightness, and then placed in the correct location on the detector.
Galaxies in Mirage are represented by 2 dimensional Seric profiles. These are created using astropy’s Sersic2D model, scaled to the requested brightness, and placed in the seed image.
“External sources” is the term used for stamp images that are read in from supplied fits files. In this case, the user-suppied fits file is read in and the image is scaled and placed in the seed image at the requested location. See the Catalogs page for more details.
After simulated astronomical sources are added to the seed image, background signals can be added using the bkgrnd entry in the input yaml file. Users can request a constant background in units of ADU/pixel/second, or a general background level. For the latter, Mirage uses the jwst_backgrounds package to add zodiacal and thermal background signal. In this case, “low”, “medium”, or “high” are the three values that can be used. The definition of these terms mirrors those used in APT and the JWST ETC.
For a given pointing, jwst_backgrounds will calculate the total expected background level (zodiacal plus thermal) for each day over the course of a year. A “low” background is defined as the background level corresponding to the 10th percentile of this distribution of background levels. “Medium” is 50th percentile, and “high” is 90th percentile.
Users can also provide a custom background through the use of an input fits file referenced in the “zodiacal” entry of the input yaml file. In this case, the contents of the file are read in and added to the seed image.
Seed images from distortion-free images¶
Another way to create a seed image is through the use of a large field-of-view image from a FITS file. This file should contain a north-up, distortion-free count rate image at an arbitrary pixel scale with a proper world coordinate system (WCS) in the header.
In this case, the portion of the image corresponding to the requested RA, Dec of the simulation is extracted from the input, and some of the JWST calibration pipeline functionality is used to resample it onto the appropriate JWST instrument pixel grid while also introducing the appropriate distortion for the requested detector. This functionality is a new implementation of Astrodrizzle’s blotting function, and works in essentially the same way. For more details on Astrodrizzle, see the DrizzlePac website. The simulator then saves this blotted image in the seed image format used by subsequent steps of MIRAGE.
In order to create a seed image from this input file, the simulator uses the fits_seed_image.py, function. For convenience, this function accepts the same yaml input file as the other parts of the simulator. To see an example of how to create a seed image from an input mosaic file, see the Simulated_data_from_mosaic_image.ipynb notebook in the examples directory of the Mirage repository.
Preparation of the Dark Current Exposure¶
After creating the seed image mirage moves on to preparing the input dark current exposure. This involves manipulating the input dark current exposure in several ways.
Adjust Number of Frames/Groups and Integrations¶
First the number of groups and integrations are adjusted to match the requested groups and integrations specified in the input yaml file. If the input dark has more groups or integrations than the requested output, the extras are removed. If additional groups or integrations are required above what is present in the dark file, then copies of existing groups/integrations are made and appended to the observation. In the case where extra groups are needed, copies of the existing groups are added to the signal in the final group of the existing dark, such that the dark current signals continue increasing in as they would in a longer integration.
Put into Requested Readout Pattern¶
Next, the readout pattern of the dark current exposure is adjusted to match the requested output. Effectively this only works in two cases. If the input dark and output exposure have the same readout pattern, then no action is taken. If the input dark is in the RAPID or NISRAPID (for NIRCam and NIRISS respectively) readout mode, where each group is composed of only one frame, then frames can be averaged/dropped in such a way to transform the dark into any other readout pattern. Input darks using readout patterns other than RAPID/NISRAPID cannot be translated into any other readout patterns, as all other readout patterns have groups composed of averaged frames, such that the original information from all frames is not accessible.
Calibration and Linearization¶
If the input dark current exposure is “raw”, meaning that no calibration steps have been performed on the data, then the reorganized dark is run through the superbias and reference pixel subtraction, as well as the non-linearity correction steps of the JWST calibration pipeline. The signal subtracted in the superbias and reference pixel subtraction steps is saved. Once the simulated data are created, this signal can be added back in order to create a “raw” exposure if requested.
There is also an option to provide a linearized, rather than raw, dark current exposure. In this case, the file containing the linearized dark current also must contain the subtracted superbias and reference pixel signal, and the pipeline steps are subsequently skipped. ]
Crop to Requested Subarray¶
Once the linearized dark current exposure is present, it is then optionally cut down to the requested subarray size. At this point, the dark current data are ready to have signals from simulated sources added.
Tip
Both raw and linearized dark current exposures are provided in the collection of reference files that accompany mirage.
Generate Observation¶
The final stage of the simulator is the observation generator. Here, the seed image is combined with the dark current exposure to create the final simulated data. This process includes several steps: addition of instrumental effects to the seed image, translation of the seed image from a count rate image to a signal ramp, the addition of noise sources, and finally summing the simulated signals with the dark current signals.
Translate Seed Image into a RAPID Exposure¶
The first task performed in the observation generator is to translate the seed image, which is a count rate image, into a “seed exposure” composed of multiple frames with signals in counts. Specifically, the seed image is translated into one or more integrations using the RAPID, NISRAPID, or FGSRAPID readout pattern. The exposure is the collection of these integrations.
Add Cosmic Rays¶
Cosmic rays are then added to the seed exposure. Cosmic ray details are controlled through the entries in the cosmicRay section of the input yaml file. The user must provide a pointer to the location of the cosmic ray library. The cosmic ray libraries that are included in Mirage’s reference files are based on the cosmic ray library created by Massimo Robberto, which is detailed in his 2009 technical report.
Three separate cosmic ray libraries are provided in the reference file collection. These libraries are accessed using the cosmicRay:library yaml file entry. The three options are SUNMIN, SUNMAX, and FLARE. Each library has an associated cosmic ray rate, shown below. Users can modify these rates through the multiplicative factor specified in the cosmicRay:scale entry.
Default cosmic ray rates (cosmic rays / pixel / second):
SUNMIN |
|
SUNMAX |
|
FLARE |
|
Add Poisson Noise¶
Mirage then adds Poisson noise to the seed exposure. This is done one frame at a time within the integration, with the signal for each frame dependent upon the signal and poisson noise from the previous frame. Through the simSignals:poissonseed entry in the input yaml file, users can provide a seed to the random number generator used to create the Poisson noise, in order to have reproducible results.
Flat Field Effects¶
If the user provides an illumination flat or pixel flat in the Reffiles:illumflat or Reffiles:pixelflat yaml file entries, these files are read in and multiplied into the seed exposure. This will adjust the sources’ brightness to account for pixel-to-pixel effects, including differences in pixel area across the detector.
IPC¶
Interpixel capacitance (IPC) effects are added to the seed exposure at this point. IPC is an electronic coupling between adjacent pixels that causes some of the signal generated in a pixel to be measured by a neighboring pixel (Donlon 2016). To add IPC, we convolve the image with a user-specified IPC kernel. Typically the kernel is a 3x3 array. The kernel lists the relative fraction of signal that will be measured in the 3x3 grid of pixels around a central pixel that sees some input flux. For example, in the case where 0.5% of the signal spreads to each of the 4 pixels immediately adjacent to the central pixel, we would use the kernel shown below.
0.0002 |
0.0050 |
0.0002 |
0.0050 |
0.9792 |
0.0050 |
0.0002 |
0.0050 |
0.0002 |
This is the inverse of the kernel that can be used by the JWST calibration pipeline to remove IPC effects, like that shown below. Therefore, Mirage has an option to accept a pipeline-formatted kernel that will then be inverted before being used to add IPC effects. It is also possible to use a file with a separate 3x3 kernel for each pixel. In this case the total kernel will have a shape of 2048 x 2048 x 3 x 3.
-0.0002 |
-0.0052 |
-0.0002 |
-0.0052 |
1.0214 |
-0.0052 |
-0.0002 |
-0.0052 |
-0.0002 |
The latest IPC kernels for all instruments and detectors are provided in the reference files assocaited with Mirage.
Add Seed Exposure to Dark Current Exposure¶
At this point, the seed exposure is added to the dark current exposure from the previous stage. At this point, both are in the RAPID, NISRAPID, or FGSRAPID readout pattern, depending on instrument. They are added frame by frame, resulting in a single observation exposure.
Crosstalk¶
Crosstalk effects can also be added to the observation. Crosstalk in this case is defined as the unintentional mirroring of a fraction of the measured signal across amplifiers. Bright sources located in one quadrant of a detector can create dimmed copies of themselves in the other three quadrants. Crosstalk on the NIRCam detectors has been quantified by Armin Rest. Results are detailed in his 2015 technical report. Crosstalk is added to the observation using an input file containing crosstalk coefficients. These coefficients control the magnitude and sign of the crosstalk signal between each combination of amplifiers. Currently, only the NIRCam detectors have non-zero crosstalk coefficients in the collection of Mirage reference files.
(Optionally) Return from Linearized to Raw State¶
At this point, the observation contains linearized, partially calibrated (superbias and reference pixel subtracted) signal. This can be directly saved to an output fits file in the format used by the JWST calibration pipeline. There is also an option to return the observation exposure to a completely uncalibrated (“raw”) state, such that the full calibration pipeline can be run on the data. The advantage of this is that pipeline parameters and settings can be customized/optimized by the user.
In order to return to a raw state, the linearization step of the pipeline is run in reverse on the data. The previously subtracted superbias and reference pixel signal is then added back in to the exposure.
Save Observation¶
With the construction of the observation exposure complete, the exposure is then saved in the requested format.
Hint
The output format of the data is controlled by the Output:datatype line in the input yaml file. Possible values for this parameter include “raw”, “linear”, or “raw,linear”. In the latter case, both the linearized and raw versions of the observation will be saved.
Logging with Mirage¶
Creation of log files¶
Currently, logging in Mirage is a 2 step process, born out of the desire to have a single log file for each module called, regardless of whether that is a call to catalog_seed_image.py or a call to imaging_simulator.py (which in turn calls catalog_seed_image.py, dark_prep.py, and obs_generator.py).
While Mirage is running, it will write log messages to a file called mirage_latest.log in the working directory. Upon successful completion, this log file will be copied into a new location. The new location depends on exactly which Mirage modules are being called. In cases where Mirage is using a yaml input file to create a simulated exposure (via imaging_simulator.py, catalog_seed_image.py, dark_prep.py, obs_generator.py, wfss_simulator.py, or grism_tso_simulator.py), the log file will be copied to a subdirectory called mirage_logs which will be located within the output directory specified in the yaml file. The name of the copied log file will also be updated to be the name of the yaml file with the suffix “.log”. In practice, this means that in order to find the log files, simply go to the directory containing your newly created simulated data, and look in the mirage_logs subdirectory.
If you are calling the yaml_generator.py in order to create input yaml files, then the log file will be copied to a subdirectory called mirage_logs located in the directory containing the APT xml and pointing files. In this case, the log file name will match the name of the APT xml file, with the suffix “log”.
Note that the initial log file, mirage_latest.log will remain in the working directory. In the event of a Mirage crash, mirage_latest.log will contain the traceback.
Customizing logging¶
The default beehavior is for Mirage to print all messages with a level of INFO or above to the screen, and to print all messages with a level of DEBUG or above into the log file. Users can customize the logs created by Mirage using the log configuration file contained in the Mirage repository. This file is located in mirage/logging/logging_config.yaml and specifies which types of logs Mirage creates and the message levels that are printed to the logs. By changing these values, users can customize the output log files.
Contributors¶
Mirage is based on early NIRISS simulator software written by Kevin Volk. It has been developed by a group of core contributors from STScI: