{ "cells": [ { "cell_type": "markdown", "metadata": { "nbsphinx": "hidden" }, "source": [ "This is an example of how to set up notebooks for use in Sphinx docs. You can choose to have some cells hidden; they execute in the notebook but are not shown in the HTML output. This is useful for imports and setup. \n", "\n", "To do this, in Jupyter go to the View menu, select the Cell Toolbar submenu, and select \"Edit Metadata\". This will make each cell have a new \"Edit Metadata\" button. The metadata is a dict for each cell. Add ` \"nbsphinx\": \"hidden\"` to a given cell and it will be hidden. This has been done for the above cell and for this one. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# MIRAGE Quick Start" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Table of Contents:*\n", "* Getting Started\n", "* [Create `mirage` input yaml files from an APT file](#make_yaml)\n", "* Single image simulation\n", " * [Create simulation with one command](#run_steps_together)\n", " * [Running simulator steps independently](#run_steps_independently)\n", "* [Running multiple simulations](#mult_sims)\n", " * [Running in series](#mult_sims_in_series)\n", " * [Running in parallel](#mult_sims_in_parallel)\n", "* [Example observation list file](#observation_example)\n", "* [Example yaml file](#yaml_example)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Getting Started\n", "\n", "
\n", "**Important:** \n", "Before proceeding, ensure you have set the MIRAGE_DATA environment variable to point to the directory that contains the reference files associated with MIRAGE.\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "# For examining outputs\n", "from glob import glob\n", "from scipy.stats import sigmaclip\n", "import numpy as np\n", "from astropy.io import fits\n", "from astropy.visualization import simple_norm\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# Import top level functions\n", "from mirage import imaging_simulator\n", "from mirage.apt import apt_inputs\n", "from mirage.yaml import yaml_generator\n", "\n", "# Import the three steps of the simulator.\n", "from mirage.seed_image import catalog_seed_image\n", "from mirage.dark import dark_prep\n", "from mirage.ramp_generator import obs_generator" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "### Create `mirage` input yaml files from an APT file" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For convenience, observing programs with multiple pointings \n", "and detectors can be simulated starting with the program's \n", "APT file. The xml and pointings files must be exported from \n", "APT, and are then used as input into a tool that will\n", "generate a series of yaml input files." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "apt_xml_file = 'my_apt_file.xml'\n", "apt_pointing_file = 'my_apt_file.pointing'" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "catalogs = {'nircam': {'sw': ['nrc_ptsrc_1.cat', 'nrc_ptsrc_2.cat'], \n", " 'lw': ['nrc_ptsrc_lw_1.cat', 'nrc_ptsrc_lw_2.cat']}, \n", " 'niriss': ['niriss_ptsrc_1.cat', 'niriss_ptsrc_2.cat']}\n", "backgrounds = ['low', 'medium']" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Create a series of data simulator input yaml files\n", "# from APT files\n", "yam = yaml_generator.SimInput()\n", "\n", "# Point to the xml and pointing files from your APT proposal\n", "yam.input_xml = apt_xml_file\n", "yam.pointing_file = apt_pointing_file\n", "\n", "# Output directory for the collection of yaml files\n", "yam.output_dir = './yaml_files'\n", "\n", "# Output directory for the simulated observations\n", "yam.simdata_output_dir = './imaging_data/'\n", "\n", "# Observation table that lists the source catalogs to use with each filter/observation\n", "yam.observation_table = observation_list_file\n", "\n", "# Output data type. \"raw\", \"linear\", or \"linear,raw\" for both\n", "yam.datatype = 'raw'\n", "\n", "# Optional parameters\n", "yam.use_JWST_pipeline = True\n", "yam.use_linearized_darks = True\n", "\n", "# Create the yaml files\n", "yam.reffile_setup()\n", "yam.create_inputs()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Look at the list of files generated\n", "yaml_files = glob('./yaml_files/*yaml')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Create simulation with one command" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### This will take several minutes to run. For a better idea of what Mirage is doing, skip down several cells to the [Running simulator steps independently](#run_steps_independently) section" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The imaging_simulator function will run all three steps of the simulator. This convenience function is useful when creating simulated imaging mode data. WFSS data will need to be run in a slightly different way." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "test_yaml_file = yaml_files[0]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# imaging_simulator is a wrapper around all 3 steps of Mirage\n", "img_sim = imaging_simulator.ImgSim()\n", "img_sim.paramfile = test_yaml_file\n", "img_sim.create()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## More detail on what's going on: Running simulation steps independently" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### First generate the \"seed image\" \n", "\n", "This is generally a 2D noiseless countrate image that contains only simulated astronomical sources.\n", "\n", "A seed image is generated based on a `.yaml` file that contains all the necessary parameters for simulating data. An example `.yaml` file is shown at the [bottom of this notebook](#yaml_example)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Generate the seed image\n", "cat = catalog_seed_image.Catalog_seed()\n", "cat.paramfile = test_yaml_file\n", "cat.make_seed()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Look at the seed image" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Need to flip the image vertically in order to match what ds9 would show\n", "def show(array, title ,min=None, max=None):\n", " plt.figure(figsize=(12,12))\n", " if min is None and max is None:\n", " norm = simple_norm(array, 'log', percent=99)\n", " else:\n", " if min is None:\n", " min = np.min(array)\n", " if max is None:\n", " max = np.max(array)\n", " norm = simple_norm(array, 'log', min_cut=min, max_cut=max)\n", " plt.imshow(array[::-1,:], norm=norm)\n", " plt.title(title)\n", " plt.colorbar().set_label('DN$^{-}$/s')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "show(cat.seedimage,'Seed Image', max=100, min=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Prepare the dark current exposure\n", "This will serve as the base of the simulated data.\n", "This step will linearize the dark current (if it \n", "is not already), and reorganize it into the \n", "requested readout pattern and number of groups." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "d = dark_prep.DarkPrep()\n", "d.paramfile = test_yaml_file\n", "d.prepare()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Look at the dark current " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# For this, we will look at an image of the final group minus the first group\n", "exptime = d.linDark.header['NGROUPS'] * cat.frametime\n", "diff = (d.linDark.data[0,-1,:,:] - d.linDark.data[0,0,:,:]) / exptime\n", "show(diff,'Dark Current Countrate', max=0.1, min=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Create the final exposure\n", "Turn the seed image into a exposure of the \n", "proper readout pattern, and combine it with the\n", "dark current exposure. Cosmic rays and other detector\n", "effects are added. \n", "\n", "The output can be either this linearized exposure, or\n", "a 'raw' exposure where the linearized exposure is \n", "\"unlinearized\" and the superbias and \n", "reference pixel signals are added, or the user can \n", "request both outputs. This is controlled from\n", "within the yaml parameter file." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "obs = obs_generator.Observation()\n", "obs.linDark = d.prepDark\n", "obs.seed = cat.seedimage\n", "obs.segmap = cat.seed_segmap\n", "obs.seedheader = cat.seedinfo\n", "obs.paramfile = test_yaml_file\n", "obs.create()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Examine the final output image\n", "Again, we will look at the last group minus the first group" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "with fits.open(obs.raw_output) as h:\n", " lindata = h[1].data\n", " header = h[0].header" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "exptime = header['EFFINTTM']\n", "diffdata = (lindata[0,-1,:,:] - lindata[0,0,:,:]) / exptime\n", "show(diffdata,'Simulated Data',min=0,max=20)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you have multiple exposures that will use the same dark current image (with the same readout pattern, subarray size, and number of groups), you can feed the output from the initial run of dark_prep into future runs of the obs_generator, to save time. This can be accomplished with the `imaging_simulator.py` code, as shown below.\n", "(Note that time savings are minimal in this case, where the readout pattern is RAPID and there are only a handful of groups. This means that no averaging/skipping of frames has to be done within `dark_prep.py`)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Now that the linearized dark product has been created, if you want to use it\n", "# when running the simulator with a different yaml file (or repeating the run\n", "# with the same yaml file) you can provide the filename of the dark product, and the\n", "# dark_prep step will be skipped. \n", "# NOTE: if you use the same dark product for multiple exposures, those exposures\n", "# will contain exactly the same dark signal. This may or may not be advisable, depending\n", "# on your goals for the simulated data.\n", "img_sim_same_dark = imaging_simulator.ImgSim()\n", "img_sim_same_dark.paramfile = second_yaml_file\n", "img_sim_same_dark.override_dark = 'jw44444001001_01101_00001_nrcb1_uncal_linear_dark_prep_object.fits'\n", "img_sim_same_dark.create()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Running Multiple Simulations" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Each yaml file will simulate an exposure for a single pointing using a single detector." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "### Function to simulate multiple detectors/pointings in series" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def make_sim(paramlist):\n", " '''Function to run many simulations in series\n", " '''\n", " for file in paramlist:\n", " m = imaging_simulator.ImgSim()\n", " m.paramfile = file\n", " m.create()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# THIS WILL RUN ALL SIMULATIONS AND CAN TAKE A LONG TIME DEPENDING ON HOW MANY YAML FILES YOU HAVE!!\n", "for yfile in yaml_files:\n", " make_sim(yfile)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "### Function to simulate multiple detectors/pointings in parallel" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from multiprocessing import Pool\n", "\n", "n_procs = 3 # number of cores available\n", "\n", "with Pool(n_procs) as pool:\n", " pool.map(make_sim, yaml_files)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "--- \n", "\n", "## Example yaml input file\n", "\n", "Entries listed as 'config' have default files that are present in the \n", "config directory of the repository. The scripts are set up to \n", "automatically find and use these files. The user can replace 'config'\n", "with a filename if they wish to override the default.\n", "\n", "In general, if 'None' is placed in a field, then the step that uses\n", "that particular file will be skipped.\n", "\n", "Note that the linearized_darkfile entry overrides the dark entry, unless\n", "linearized_darkfile is set to None, in which case the dark entry will be\n", "used.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```yaml\n", "Inst:\n", " instrument: NIRCam #Instrument name\n", " mode: imaging #Observation mode (e.g. imaging, WFSS, moving_target)\n", " use_JWST_pipeline: False #Use pipeline in data transformations\n", "\n", "Readout:\n", " readpatt: RAPID #Readout pattern (RAPID, BRIGHT2, etc) overrides nframe,nskip unless it is not recognized\n", " ngroup: 3 #Number of groups in integration\n", " nint: 1 #Number of integrations per exposure\n", " array_name: NRCB5_FULL #Name of array (FULL, SUB160, SUB64P, etc)\n", " filter: F250M #Filter of simulated data (F090W, F322W2, etc)\n", " pupil: CLEAR #Pupil element for simulated data (CLEAR, GRISMC, etc)\n", "\n", "Reffiles: #Set to None or leave blank if you wish to skip that step\n", " dark: None #Dark current integration used as the base\n", " linearized_darkfile: $MIRAGE_DATA/nircam/darks/linearized/B5/Linearized_Dark_and_SBRefpix_NRCNRCBLONG-DARK-60090141241_1_490_SE_2016-01-09T02h46m50_uncal.fits # Linearized dark ramp to use as input. Supercedes dark above\n", " badpixmask: $MIRAGE_DATA/nircam/reference_files/badpix/NRCB5_17161_BPM_ISIMCV3_2016-01-21_ssbspmask_DMSorient.fits # If linearized dark is used, populate output DQ extensions using this file\n", " superbias: $MIRAGE_DATA/nircam/reference_files/superbias/NRCB5_superbias_from_list_of_biasfiles.list.fits #Superbias file. Set to None or leave blank if not using\n", " linearity: $MIRAGE_DATA/nircam/reference_files/linearity/NRCBLONG_17161_LinearityCoeff_ADU0_2016-05-22_ssblinearity_v2_DMSorient.fits #linearity correction coefficients\n", " saturation: $MIRAGE_DATA/nircam/reference_files/saturation/NRCB5_17161_WellDepthADU_2016-03-10_ssbsaturation_wfact_DMSorient.fits #well depth reference files\n", " gain: $MIRAGE_DATA/nircam/reference_files/gain/NRCB5_17161_Gain_ISIMCV3_2016-02-25_ssbgain_DMSorient.fits #Gain map\n", " pixelflat: None \n", " illumflat: None #Illumination flat field file\n", " astrometric: $MIRAGE_DATA/nircam/reference_files/distortion/NRCB5_FULL_distortion.asdf #Astrometric distortion file (asdf)\n", " ipc: $MIRAGE_DATA/nircam/reference_files/ipc/NRCB5_17161_IPCDeconvolutionKernel_2016-03-18_ssbipc_DMSorient.fits #File containing IPC kernel to apply\n", " invertIPC: True #Invert the IPC kernel before the convolution. True or False. Use True if the kernel is designed for the removal of IPC effects, like the JWST reference files are.\n", " occult: None #Occulting spots correction image\n", " pixelAreaMap: $MIRAGE_DATA/nircam/reference_files/pam/NIRCam_B5_PAM_imaging.fits #Pixel area map for the detector. Used to introduce distortion into the output ramp.\n", " subarray_defs: config #File that contains a list of all possible subarray names and coordinates\n", " readpattdefs: config #File that contains a list of all possible readout pattern names and associated NFRAME/NSKIP values\n", " crosstalk: config #File containing crosstalk coefficients\n", " filtpupilcombo: config #File that lists the filter wheel element / pupil wheel element combinations. Used only in writing output file\n", " flux_cal: config #File that lists flux conversion factor and pivot wavelength for each filter. Only used when making direct image outputs to be fed into the grism disperser code.\n", " \n", "nonlin:\n", " limit: 60000.0 #Upper singal limit to which nonlinearity is applied (ADU)\n", " accuracy: 0.000001 #Non-linearity accuracy threshold\n", " maxiter: 10 #Maximum number of iterations to use when applying non-linearity\n", " robberto: False #Use Massimo Robberto type non-linearity coefficients\n", "\n", "cosmicRay:\n", " path: $MIRAGE_DATA/nircam/cosmic_ray_library/ #Path to CR library\n", " library: SUNMIN #Type of cosmic rayenvironment (SUNMAX, SUNMIN, FLARE)\n", " scale: 1.5 #Cosmic ray scaling factor\n", " suffix: IPC_NIRCam_B5 #Suffix of library file names\n", " seed: 2956411739 #Seed for random number generator\n", "\n", "simSignals:\n", " pointsource: my_point_sources.cat #File containing a list of point sources to add (x,y locations and magnitudes)\n", " psfpath: $MIRAGE_DATA/nircam/webbpsf_library/ #Path to PSF library\n", " psfbasename: nircam #Basename of the files in the psf library\n", " psfpixfrac: 0.25 #Fraction of a pixel between entries in PSF library (e.g. 0.25 = files for PSF centered at 0.25 pixel intervals within pixel)\n", " psfwfe: predicted #PSF WFE value (\"predicted\" or \"requirements\")\n", " psfwfegroup: 0 #WFE realization group (0 to 4)\n", " galaxyListFile: my_galaxies_catalog.list\n", " extended: None #Extended emission count rate image file name\n", " extendedscale: 1.0 #Scaling factor for extended emission image\n", " extendedCenter: 1024,1024 #x,y pixel location at which to place the extended image if it is smaller than the output array size\n", " PSFConvolveExtended: True #Convolve the extended image with the PSF before adding to the output image (True or False)\n", " movingTargetList: None #Name of file containing a list of point source moving targets (e.g. KBOs, asteroids) to add.\n", " movingTargetSersic: None #ascii file containing a list of 2D sersic profiles to have moving through the field\n", " movingTargetExtended: None #ascii file containing a list of stamp images to add as moving targets (planets, moons, etc)\n", " movingTargetConvolveExtended: True #convolve the extended moving targets with PSF before adding.\n", " movingTargetToTrack: None #File containing a single moving target which JWST will track during observation (e.g. a planet, moon, KBO, asteroid)\tThis file will only be used if mode is set to \"moving_target\" \n", " zodiacal: None #Zodiacal light count rate image file \n", " zodiscale: 1.0 #Zodi scaling factor\n", " scattered: None #Scattered light count rate image file\n", " scatteredscale: 1.0 #Scattered light scaling factor\n", " bkgdrate: 0.0 #Constant background count rate (electrons/sec/pixel)\n", " poissonseed: 2012872553 #Random number generator seed for Poisson simulation)\n", " photonyield: True #Apply photon yield in simulation\n", " pymethod: True #Use double Poisson simulation for photon yield\n", "\n", "Telescope:\n", " ra: 53.1 #RA of simulated pointing\n", " dec: -27.8 #Dec of simulated pointing\n", " rotation: 0.0 #y axis rotation (degrees E of N)\n", "\n", "newRamp:\n", " dq_configfile: config #config file used by JWST pipeline\n", " sat_configfile: config #config file used by JWST pipeline\n", " superbias_configfile: config #config file used by JWST pipeline\n", " refpix_configfile: config #config file used by JWST pipeline \n", " linear_configfile: config #config file used by JWST pipeline\n", "\n", "Output:\n", " file: jw44444024002_01101_00001_nrcb1_uncal.fits #Output filename\n", " directory: ./ # Directory in which to place output files\n", " datatype: linear,raw # Type of data to save. 'linear' for linearized ramp. 'raw' for raw ramp. 'linear,raw' for both\n", " format: DMS #Output file format Options: DMS, SSR(not yet implemented)\n", " save_intermediates: False #Save intermediate products separately (point source image, etc)\n", " grism_source_image: False # Create an image to be dispersed?\n", " unsigned: True #Output unsigned integers? (0-65535 if true. -32768 to 32768 if false)\n", " dmsOrient: True #Output in DMS orientation (vs. fitswriter orientation).\n", " program_number: 44444 #Program Number\n", " title: Supernovae and Black Holes Near Hyperspatial Bypasses #Program title\n", " PI_Name: Doug Adams #Proposal PI Name\n", " Proposal_category: GO #Proposal category\n", " Science_category: Cosmology #Science category\n", " observation_number: '024' #Observation Number\n", " observation_label: Obs2 #User-generated observation Label\n", " visit_number: '002' #Visit Number\n", " visit_group: '01' #Visit Group\n", " visit_id: '42424024002' #Visit ID\n", " sequence_id: '1' #Sequence ID\n", " activity_id: '01' #Activity ID. Increment with each exposure.\n", " exposure_number: '00001' #Exposure Number\n", " obs_id: 'V44444024002P0000000001101' #Observation ID number\n", " date_obs: '2019-10-15' #Date of observation\n", " time_obs: '06:29:11.852' #Time of observation\n", " obs_template: 'NIRCam Imaging' #Observation template\n", " primary_dither_type: NONE #Primary dither pattern name\n", " total_primary_dither_positions: 1 #Total number of primary dither positions\n", " primary_dither_position: 1 #Primary dither position number\n", " subpix_dither_type: 2-POINT-MEDIUM-WITH-NIRISS #Subpixel dither pattern name\n", " total_subpix_dither_positions: 2 #Total number of subpixel dither positions\n", " subpix_dither_position: 2 #Subpixel dither position number\n", " xoffset: 344.284 #Dither pointing offset in x (arcsec)\n", " yoffset: 466.768 #Dither pointing offset in y (arcsec)\n", " \n", " ```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "celltoolbar": "Edit Metadata", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.6" } }, "nbformat": 4, "nbformat_minor": 2 }