Reference/API

KeyHandler

class phangsPipeline.KeyHandler(master_key: str = 'key_templates/master_key.txt', dochecks: bool = True)

Class to handle data files that indicate the names and data sets associated with reducing a (generally) ALMA imaging project.

The KeyHandler generally just gets passed the path to the master key file (which tells the pipeline where to look for other relevant files, imaging recipes etc.), and then will be passed to other handlers for data processing.

Parameters:

master_key (str, optional) – Path to master key file. Defaults to ‘key_templates/master_key.txt’.
dochecks (bool, optional) – Whether to check for missing files. Defaults to True.

build_key_handler(master_key=None): Construct the key handler object.

check_dir_existence(imaging=True, postprocess=True, derived=True, release=True): Check the existence of the directories for imaging and post-processing.

check_key_existence(): Check file existence for the input keys defined in the master file.

check_ms_existence(): Check that the measurement sets in the ms key all exist in the specified directories.

check_sd_existence(): Check that the FITS files in the singledish key all exist in the specified directories.

get_all_configs(): Get a list of all interf and feather configs.

get_all_mosaic_targets(): Get all mosaic targets defined in “linearmosaic_definitions.txt” which have parts.

get_all_non_mosaic_targets(): Get all targets which are not mosaic targets.

get_all_targets(): Get all targets defined in “target_definitions.txt”, including mosaic targets and their parts and non mosaic targets.

get_alma_download_restrictions(target=None, product=None, config=None): Get any download restrictions for ALMA data

get_ang_res_dict(config=None, product=None): Return the angular resolutions for derived product creation for a combination of resolution and spectral product.

get_array_tags_for_config(config=None): Get the list of array tags associated with an interferometric configuration.

get_channel_width_for_cont_product(product=None): Get the channel width (in km/s) associated with a continuum product.

get_channel_width_for_line_product(product=None): Get the channel width (in km/s) associated with a line product.

get_clean_scales_for_config(config=None): Return the angular scales used for multiscale clean for an interferometric configuration.

get_cleanmask_dir_for_target(target=None, changeto=False): Return the release working directory given a target name. If changeto is true, then change directory to that location.

get_cleanmask_filename(target=None, product=None): Get the file name of the clean mask associated with a target and product.

get_continuum_products(only=None, skip=None)

Get a list of continuum ‘products’.

Modified by keywords only, skip. Will only return targets in only, skip targets in skip.

get_contsub_combinespw(product=None): Query whether the continuum subtraction should combine all SPWs.

get_contsub_excludefreqrange(product=None): Get the frequency range to force excluding for continuum subtraction for a line product.

get_contsub_fitorder(product=None): Get the fitorder to be used for continuum subtraction for a line product.

get_contsub_flagedgefraction(product=None): Get the frequency range to force excluding for continuum subtraction for a line product.

get_derived_dir_for_target(target=None, changeto=False): Return the derived working directory given a target name. If changeto is true, then change directory to that location.

get_derived_kwargs(config=None, product=None, kwarg_type='strictmask_kw'): Get the dictionary of keyword arguments from the derived key for masking or noise estimation. Valid kwarg_types are ‘strictmask_kw’, ‘broadmask_kw’, ‘noise_kw’ ‘mask_configs’, ‘ang_res’, ‘phys_res’, ‘noise_kw’, ‘strictmask_kw’, ‘broadmask_kw’, ‘convolve_kw’, ‘moments’, ‘shuffle_kw’, ‘flatstrictmask_kw’, ‘flatbroadmask_kw’

get_distance_for_target(target=None): Get the distance (in Mpc) associated with a target. If the target is part of a mosaic, return the distance to the whole galaxy.

get_feather_config_for_interf_config(interf_config=None): Get the interferometric configuration to go with a feather configuration.

get_feather_configs(only=None, skip=None)

Get a list of feathered single dish + interferometer array configruations.

Modified by keywords only, skip. Will only return targets in only, skip targets in skip.

get_field_for_input_ms(target=None, project=None, array_tag=None, obsnum=None): Return the science field given a target, project, array_tag, obsnum combination.

get_file_for_input_ms(target=None, project=None, array_tag=None, obsnum=None): Return the full file path to of an input measurement set given a target, project, array_tag, obsnum combination.

get_freq_ranges_for_cont_product(product=None): Get the frequency ranges (in GHz) associated with a continuum product.

get_imaging_dir_for_target(target=None, changeto=False): Return the imaging working directory given a target name. If changeto is true, then change directory to that location.

get_imaging_recipes(config=None, product=None, stage=None): Return the imaging recipe for the input config and product.

get_interf_config_for_feather_config(feather_config=None): Get the feather configuration to go with an interferometric configuration.

get_interf_configs(only=None, skip=None)

Get a list of interferometer array configurations

Modified by keywords only, skip. Will only return targets in only, skip targets in skip.

get_joint_imaging_dirs_for_singledish_config(config='tp'): Get joint_imaging_dirs.

get_joint_imaging_suffix_for_singledish_config(config='tp'): Get joint_imaging_suffix.

get_line_products(only=None, skip=None)

Get a list of line ‘products,’ i.e., line plus velocity resolution combinations.

Modified by keywords only, skip. Will only return targets in only, skip targets in skip.

get_line_tag_for_line_product(product=None): Get the line tag (in utilsLines) associated with a line data product.

get_lines_to_flag(product=None): Get the list of lines to flag when either constructing a continuum product or carrying out continuum subtraction on a line product.

get_linked_mask_configs(config=None, product=None): Return the list of linked configurations used in making hybrid masks.

get_linmos_targets(only=None, skip=None, first=None, last=None)

List all linear mosaics.

Modified by keywords only, skip, first, last. Will only return targets in only, skip targets in skip, and return targets alphabetically after first and before last.

get_moment_list(config=None, product=None): Return the list of moments to build for a config + product. masks.

get_mosaic_target_for_parts(target_part_name=None)

Get mosaic target name “ngc4321” given a part name like “ngc4321_1”. This is the inverted operation of get_parts_for_linmos.

This is also the same as is_target_in_mosaic(target_part_name, return_target_name=True).

get_overrides(key=None, param=None, default=None): Check the override dictionary for entries given some key, parameter pair.

get_params_for_moment(moment=None): Return parameter dictionary for a moment.

get_params_for_singledish(singledish_config=None): Return parameter dictionary for a singledish config.

get_parts_for_linmos(target=None): Return the parts for a linear mosaic.

get_path_for_casaversion(casa_version=None): Get the CASA path for a given version of CASA.

get_phasecenter_for_target(target=None): Return the strings (ra, dec) that define the phase center for a target in the target dictionary.

get_phys_res_dict(config=None, product=None): Return the physical resolutions for derived product creation for a combination of resolution and spectral product.

get_postprocess_dir_for_target(target=None, changeto=False): Return the postprocessing working directory given a target name. If changeto is true, then change directory to that location.

get_release_dir_for_target(target=None, changeto=False): Return the release working directory given a target name. If changeto is true, then change directory to that location.

get_require_tags_for_config(config=None): Get the list of required array tags associated with an interferometric configuration.

get_sd_filename(target=None, product=None, nocheck=False): Return the single dish filename for a target and product combination.

get_singledish_configs(only=None, skip=None)

Get a list of singledish configruations.

Modified by keywords only, skip. Will only return targets in only, skip targets in skip.

get_singledish_dir_for_target(target=None, changeto=False): Return the release working directory given a target name. If changeto is true, then change directory to that location.

get_statwt_edge_for_line_product(product=None): Get the velocity width of the edge region to use when running statwt on a line product.

get_system_velocity_and_velocity_width_for_target(target=None, check_parent=False)

Inputs

check_parent: bool, optional: If part of a linear mosaic, return the vsys, vwidth of the parent target.

get_targets(only=None, skip=None, first=None, last=None)

List the full set of targets.

Modified by keywords only, skip, first, last. Will only return targets in only, skip targets in skip, and return targets alphabetically after first and before last.

get_targets_in_ms_key(only=None, skip=None, first=None, last=None)

List all targets that have uv data.

Modified by keywords only, skip, first, last. Will only return targets in only, skip targets in skip, and return targets alphabetically after first and before last.

get_timebin_for_array_tag(array_tag=None): Get the timebin for an array tag. Returns 0s by default.

get_vfield_dir_for_target(target=None, changeto=False): Return the vfield working directory given a target name. If changeto is true, then change directory to that location.

get_vfield_file_for_target(target=None): Get the velocity field file associated with a target. If the target is part of a mosaic, return the velocity field file for the whole galaxy.

get_whole_targets(only=None, skip=None, first=None, last=None)

List only full galaxy names (no parts, e.g., _1 or _2). Very similar to the directory list.

Modified by keywords only, skip, first, last. Will only return targets in only, skip targets in skip, and return targets alphabetically after first and before last.

get_window_for_target(target=None): Get the velocity window (in km/s) associated with a target. If the target is part of a mosaic, return the velocity window to the whole galaxy.

has_data_for_config(target=None, config=None): Test whether a target has data for a configuration in the ms key. If “strict” is TRUE then require that a target has data from ALL arrays that make up the configuration.

has_overrides_for_key(key=None)

Check whether the override dictionary contains the input key or not.

The key is usually a file name or a galaxy name.

has_singledish(target=None, product=None): Return true or false indicating if this target and product combination has associated single dish data.

is_target_in_mosaic(target, return_target_name=False): Return true or false depending on whether the target is in a linear mosaic.

is_target_linmos(target=None): Return True or False based on whether the target is a linear mosaic. True means that this target is the OUTPUT of a linear mosaic operation.

loop_over_input_ms(target=None, config=None, project=None, check_linmos=False)

Loop over the target name, project tag, array tag, and obsnum for each input visibility file. If a target is supplied then restrict to that target, trying to match to a linear mosaic if the target is not represented in the dictionary. If a configuration is supplied, then only include array_tags that contribute to that configuration.

Note that the interaction with configs that contain multiple arrays is tricky. Use the ‘requires’ keyword in the config_definitions file to control this. By default, we require ALL arrays that make up the configuration, but this can be changed to an OR if you only need one of a certain combination, e.g.

interf_config 12m {‘array_tags’:[‘12m_1’,’12m_2’]} interf_config 12m {‘requires’:[‘12m_1|12m_2’]}

requires only one of 12m_1 or 12m_2 to be present. If an array is not in the ‘requires’ list, then it is assumed to be required.

make_missing_directories(ms_root=False, imaging=False, postprocess=False, derived=False, release=False): Make any missing imaging or postprocessing directories.

print_configs(): Print out the configurations for inspection.

print_derived(): Print out the information for derived products.

print_missing_distances(): Print out the missing distances

print_missing_targets(): Print the targets missing a definition in the target list key.

print_missing_window(): Print out the missing velocity window

print_products(): Print out the products for inspection.

set_dochecks(dochecks=True)

SingleDishHandler

class phangsPipeline.SingleDishHandler(key_handler=None, dry_run: bool = False, use_legacy_pipeline: bool = False)

Class to handle imaging single dish data.

Parameters:

key_handler – KeyHandler instance
dry_run (bool) – If True, will not modify files on disk. Defaults to False.
use_legacy_pipeline (bool) – If True, will use legacy pipeline. Kept in for legacy reasons. Defaults to False.

loop_singledish(do_all: bool = True, make_directories: bool = True)

Loops over the full set of targets, products, and configurations to run the postprocessing. Toggle the parts of the loop using the do_XXX booleans. Other choices affect the algorithms used.

Parameters:

do_all (bool) – If True, will actually run the singledish pipeline
make_directories (bool) – If True, will create missing directories

recipe_process_one_target(target=None, product=None)

task_execute_single_dish_pipeline(target, product='all', source='all', extra_ext_in='', extra_ext_out='', line_wing_kms=200.0): Execute single dish data reduction for one target.

VisHandler

class phangsPipeline.VisHandler(key_handler=None, dry_run: bool = False)

Class to manipulate calibrated ALMA visibility data (measurement sets), extracting lines, combining multiple data sets, and carrying out other steps in prepration for imaging.

loop_stage_uvdata(do_all: bool = False, do_copy: bool = False, do_remove_staging: bool = False, do_custom: bool = False, do_contsub: bool = False, do_extract_line: bool = False, do_extract_cont: bool = False, extra_ext: str = '', make_directories: bool = True, statwt_line: bool = True, statwt_cont: bool = True, intent=None, timebin=None, just_projects=None, require_full_line_coverage: bool = False, require_full_cont_coverage: bool = False, overwrite: bool = False, strict_config=None)

Loops over the full set of targets, products, and configurations to run the uv data processing. Toggle the parts of the loop using the do_XXX booleans. Other choices affect the algorithms used.

This strict_config option has now been deprecated in favour of more granular control. Use the ‘requires’ keyword in the config_definitions file instead. By default, we require ALL arrays that make up the configuration, but this can be changed to an OR if you only need one of a certain combination, e.g.

interf_config 12m {‘array_tags’:[‘12m_1’,’12m_2’]} interf_config 12m {‘requires’:[‘12m_1|12m_2’]}

requires only one of 12m_1 or 12m_2 to be present. If an array is not in the ‘requires’ list, then it is assumed to be required.

The require_full_line_coverage option sets whether to require a measurement set to completely cover a given line’s frequency range (True) or not (False).

Parameters:

do_all (bool) – Will switch do_copy, do_contsub, do_custom, do_extract_line, do_extract_cont, and do_remove_staging to True
do_copy – If True, will split out the raw data to the staging directory
do_remove_staging – If True, will remove the intermediate data after staging
do_custom – Not currently used
do_contsub – If True, will do continuum subtraction
do_extract_line – If True, will extract the line from the dataset
do_extract_cont – If True, will extract continuum from the dataset
make_directories – If True, will create missing directories
statwt_line – If True, will statwt the lines after concatenation
statwt_cont – If True, will statwt the continuum after concatenation
intent – If set, will only split out datasets with the specified intent (e.g. ‘OBSERVE_TARGET#ON_SOURCE’)
timebin – If set, will apply timebin in the splitting
just_projects – If set, will only run on the specified projects (list)
require_full_line_coverage – If True, will only run on datasets where an SPW completely contains the line in question
require_full_cont_coverage – If True, will only run on datasets where an SPW completely contains the continuum frequency range requested
overwrite – If True, will overwrite existing staged datasets
strict_config – Deprecated. If set, raise an error. Use the “requires” keywords in the config key file instead.

remove_staged_products(target=None, project=None, array_tag=None, obsnum=None, product=None, extra_ext=''): Remove ‘staged’ visibility products, which are intermediate between the calibrated data and the concated measurement sets that we begin processing on. Run this step after concat to reduct the disk footprint of the pipeline.

task_concat_uvdata(target=None, product=None, config=None, just_projects=None, extra_ext_in='', extra_ext_out='', overwrite=False): Concatenate all measurement sets for the supplied target+config+product combination.

task_contsub(target=None, project=None, array_tag=None, obsnum=None, product=None, extra_ext_in='', overwrite=False): Run u-v plane continuum subtraction on an individual input measurement set.

task_extract_continuum(target=None, product=None, config=None, exact=False, extra_ext_in='', extra_ext_out='', do_statwt=True, method='regrid_then_rebin', require_full_cont_coverage=False, overwrite=False): Extract continuum data from ms data for the input target, config, and product.

task_extract_line(target=None, product=None, config=None, exact=False, contsub='prefer', extra_ext_in='', extra_ext_out='', do_statwt=True, edge_for_statwt=None, method='regrid_then_rebin', require_full_line_coverage=False, overwrite=False): Extract spectral line data from ms data for the input target, config and product.

task_remove_concat(target=None, product=None, config=None, extra_ext_in='', suffixes=None): Remove any concatenated measurement sets. These are intermediate (though time consuming) products not needed for imaging. This procedure wipes them and saves disk space.

task_run_custom_scripts(target=None, product=None, config=None, extra_ext='')

task_split(target=None, project=None, array_tag=None, obsnum=None, product=None, intent=None, extra_ext_out='', do_statwt=False, timebin=None, use_symlink=True, require_full_line_coverage=False, overwrite=False): Copy visibility data for one target, project, array_tag, obsnum combination from their original location to the imaging directory for the target. Then optionally split out only the science targets.

ImagingHandler

class phangsPipeline.ImagingHandler(key_handler=None, dry_run=False)

Class to makes image cubes out of uv data from each spectral line and continuum of each galaxy.

loop_imaging(do_all=False, imaging_method='tclean', imaging_method_override=None, do_dirty_image=False, do_revert_to_dirty=False, do_read_clean_mask=False, do_multiscale_clean=False, do_revert_to_multiscale=False, do_singlescale_mask=False, singlescale_mask_high_snr=None, singlescale_mask_low_snr=None, singlescale_mask_absolute=False, do_singlescale_clean=False, do_revert_to_singlescale=False, do_export_to_fits=False, convergence_fracflux=0.01, convergence_noise_z_threshold=None, singlescale_threshold_value=1.0, extra_ext_in=None, suffix_in=None, extra_ext_out=None, recipe='phangsalma', make_directories=True, dynamic_sizing=True, force_square=False, export_dirty=False, export_multiscale=False, overwrite=False)

Loops over the full set of targets, products, and configurations to do the imaging. Toggle the parts of the loop using the do_XXX booleans. Other choices affect algorithms used.

The clean convergence is set by the convergence_fracflux and convergence_noise_z_threshold parameters.

convergence_fracflux is the fractional flux threshold between imaging loops.
Convergence is reached when the fractional flux change is less than this value.
convergence_noise_z_threshold is the noise threshold in z-scores.
Convergence is reached when the z-score test is less than this value. By default, this is disabled by setting to None. We suggest setting this to 2.0 when enabled as a conservative choice.

Parameters:

do_all (bool) – If True, do all steps.
imaging_method (str) – ‘tclean’ or ‘sdintimaging’
do_dirty_image (bool) – If True, make a dirty image.
do_revert_to_dirty (bool) – If True, revert to dirty image.
do_read_clean_mask (bool) – If True, read clean mask.
do_multiscale_clean (bool) – If True, multiscale clean.
do_revert_to_multiscale (bool) – If True, revert to multiscale clean.
do_singlescale_mask (bool) – If True, make singlescale mask.
singlescale_mask_high_snr (float) – High SNR threshold for singlescale mask.
singlescale_mask_low_snr (float) – Low SNR threshold for singlescale mask.
singlescale_mask_absolute (bool) – If True, use absolute threshold for singlescale mask.
do_singlescale_clean (bool) – If True, singlescale clean.
do_revert_to_singlescale (bool) – If True, revert to singlescale clean.
do_export_to_fits (bool) – If True, export ms data folders into fits-format image cube files.
convergence_fracflux (float) – Fractional flux convergence threshold for multiscale clean. Default is 0.01.
convergence_noise_z_threshold (float) – Noise convergence threshold for multiscale clean. Default is None. A suggested value to use is 2.0, i.e. the null hypothesis is rejected if the z-score is greater than 2.0.
singlescale_threshold_value (float) – Threshold value for singlescale clean. Default is 1.0.
extra_ext_in (str) – Extra extension for input files.
suffix_in (str) – Suffix for input files.
extra_ext_out (str) – Extra extension for output files.
recipe (str) – The recipe. Default is ‘phangsalma’.
make_directories (bool) – Create missing directories.
dynamic_sizing (bool) – Use dynamic sizing.
force_square (bool) – Force the image size to be square.
export_dirty (bool) – If True, export dirty images.
export_multiscale (bool) – If True, export multiscale images.
overwrite (bool) – Overwrite existing files.

recipe_phangsalma_imaging(target=None, product=None, config=None, extra_ext_in=None, suffix_in=None, extra_ext_out=None, imaging_method='tclean', imaging_method_override=None, do_dirty_image=True, do_revert_to_dirty=True, do_read_clean_mask=True, do_multiscale_clean=True, do_revert_to_multiscale=True, do_singlescale_mask=True, singlescale_mask_high_snr=None, singlescale_mask_low_snr=None, singlescale_mask_absolute=False, do_singlescale_clean=True, do_revert_to_singlescale=True, do_export_to_fits=True, convergence_fracflux=0.01, convergence_noise_z_threshold=None, singlescale_threshold_value=1.0, dynamic_sizing=True, force_square=False, export_dirty=False, export_multiscale=False, overwrite=False)

PHANGS-ALMA basic imaging recipe.

Major steps:

Dirty imaging
Align a broad clean mask (if present)
Lightly masked multiscale clean to S/N ~ 4
Heavily masked single scale clean until convergence
Export to FITS

Operates by passing a “clean_call” object between steps. Can restart from any individual step.

Input file names: {target}_{config}_{product}_{extra_ext_in}.ms{.suffix_in}

Output file names: {target}_{config}_{product}_{extra_ext_out}.image

imaging_method_override allows for some switches if you don’t want to use a particular algorithm for a particular setup. This should be supplied as a dictionary containing ‘target’, ‘config’, and ‘product’ keys that match up with a particular setup (these can be lists or ‘all’), and a ‘new_imaging_method’ key that the target/product/config setup should switch to (either tclean or sdintimaging)

task_assign_multiscales(**kwargs)

task_export_to_fits(**kwargs)

task_initialize_clean_call(target=None, config=None, product=None, imaging_method='tclean', extra_ext_in=None, suffix_in=None, extra_ext_out=None, stage='dirty'): Initialize a clean call object for a target, config, product combination and an imaging stage.

task_make_dirty_image(**kwargs)

task_multiscale_clean(**kwargs)

task_pick_cell_and_imsize(**kwargs)

task_read_clean_mask(**kwargs)

task_revert_to_imaging(**kwargs)

task_setup_sdintimaging(**kwargs)

task_singlescale_clean(**kwargs)

task_singlescale_mask(**kwargs)

ImagingChunkedHandler

class phangsPipeline.ImagingChunkedHandler(target, config, product, key_handler, dry_run=False, chunksize=10, imaging_method='tclean', recipe='phangsalma', set_cell_imsize_on_init=True, force_square=False, oversamp=5, make_temp_dir=True, temp_key=None, temp_path=None, copy_ms_to_temp=False)

Class to makes image cubes out of uv data for imaging a single spectral line.

Parameters:

target (str) – The target name.
config (str) – The configuration name.
product (str) – The product name.
key_handler (dict) – The key handler dictionary.
dry_run (bool, optional) – If True, do not actually make the images. Defaults to False.
chunksize (int, optional) – The number of channels per image cube. Defaults to 10.
imaging_method (str, optional) – The imaging method. Defaults to ‘tclean’.
recipe (str, optional) – The recipe. Defaults to ‘phangsalma’.
set_cell_imsize_on_init (bool, optional) – If True, set the cell size on initialization. Defaults to True.
force_square (bool, optional) – If True, force the image size to be square. Defaults to False.
oversamp (int, optional) – The oversampling factor of pixels per beam FWHM. Defaults to 5.
make_temp_dir (bool, optional) – If True, make a temporary directory. Defaults to True.
temp_key (str, optional) – The temporary key. Defaults to None.
temp_path (str, optional) – The temporary path. Defaults to None. Set this to specify an independent path for the temporary directory that is unassociated with the default imaging directory.

configure_chunk_parameters(): Create dictionaries with the required meta data for tracking the

recipe_phangsalma_imaging(chunk_num=None, extra_ext_in=None, suffix_in=None, extra_ext_out=None, do_dirty_image=True, do_revert_to_dirty=True, do_read_clean_mask=True, do_multiscale_clean=True, do_revert_to_multiscale=True, do_singlescale_mask=True, singlescale_mask_high_snr=None, singlescale_mask_low_snr=None, singlescale_mask_absolute=False, skip_singlescale_if_mask_empty=True, do_singlescale_clean=True, do_revert_to_singlescale=True, do_recombine_cubes=False, do_export_to_fits=True, do_cleanup=True, convergence_fracflux=0.01, convergence_noise_z_threshold=None, singlescale_threshold_value=1.0, dynamic_sizing=True, force_square=False, export_multiscale=False, overwrite=False)

PHANGS-ALMA basic imaging recipe.

Major steps:

Dirty imaging
Align a broad clean mask (if present)
Lightly masked multiscale clean to S/N ~ 4
Heavily masked single scale clean until convergence
Export to FITS

Operates by passing a “clean_call” object between steps. Can restart from any individual step.

Input file names: {target}_{config}_{product}_{extra_ext_in}.ms{.suffix_in}

Output file names: {target}_{config}_{product}_{extra_ext_out}.image

imaging_method_override allows for some switches if you don’t want to use a particular algorithm for a particular setup. This should be supplied as a dictionary containing ‘target’, ‘config’, and ‘product’ keys that match up with a particular setup (these can be lists or ‘all’), and a ‘new_imaging_method’ key that the target/product/config setup should switch to (either tclean or sdintimaging)

return_valid_chunks(chunk_num=None)

Allow specifying and validating running on a subset of chunk numbers.

Parameters:: chunk_num (int, list or None) – If None, will loop through all chunks. If a list or integer are given, this will loop through only the specified chunk numbers defined in ImagingChunkedHandler.chunk_params.
Returns:: chunks_iter – List of chunk numbers to be considered in a task.
Return type:: list

run_imaging(chunk_num=None, do_all=False, do_dirty_image=False, do_revert_to_dirty=False, do_read_clean_mask=False, do_multiscale_clean=False, do_revert_to_multiscale=False, do_singlescale_mask=False, singlescale_mask_high_snr=None, singlescale_mask_low_snr=None, singlescale_mask_absolute=False, skip_singlescale_if_mask_empty=True, do_singlescale_clean=False, do_revert_to_singlescale=False, do_recombine_cubes=False, do_export_to_fits=True, do_cleanup=True, convergence_fracflux=0.01, convergence_noise_z_threshold=None, singlescale_threshold_value=1.0, extra_ext_in=None, suffix_in=None, extra_ext_out=None, dynamic_sizing=True, force_square=False, overwrite=False)

Run the imaging process.

The clean convergence is set by the convergence_fracflux and convergence_noise_z_threshold parameters.

convergence_fracflux is the fractional flux threshold between imaging loops.
Convergence is reached when the fractional flux change is less than this value.
convergence_noise_z_threshold is the noise threshold in z-scores.
Convergence is reached when the z-score test is less than this value. By default, this is disabled by setting to None. We suggest setting this to 2.0 when enabled as a conservative choice.

Parameters:

do_all (bool) – If True, do all steps.
do_dirty_image (bool) – If True, make a dirty image.
do_revert_to_dirty (bool) – If True, revert to dirty image.
do_read_clean_mask (bool) – If True, read clean mask.
do_multiscale_clean (bool) – If True, multiscale clean.
do_revert_to_multiscale (bool) – If True, revert to multiscale clean.
do_singlescale_mask (bool) – If True, make singlescale mask.
singlescale_mask_high_snr (float) – High SNR threshold for singlescale mask.
singlescale_mask_low_snr (float) – Low SNR threshold for singlescale mask.
singlescale_mask_absolute (bool) – If True, use absolute threshold for singlescale mask.
do_singlescale_clean (bool) – If True, singlescale clean.
do_revert_to_singlescale (bool) – If True, revert to singlescale clean.
do_export_to_fits (bool) – If True, export ms data folders into fits-format image cube files.
convergence_fracflux (float) – Fractional flux convergence threshold for multiscale clean. Default is 0.01.
convergence_noise_z_threshold (float) – Noise convergence threshold for multiscale clean. Default is None. A suggested value to use is 2.0, i.e. the null hypothesis is rejected if the z-score is greater than 2.0.
singlescale_threshold_value (float) – Threshold value for singlescale clean. Default is 1.0.
extra_ext_in (str) – Extra extension for input files.
suffix_in (str) – Suffix for input files.
extra_ext_out (str) – Extra extension for output files.
recipe (str) – The recipe. Default is ‘phangsalma’.
make_directories (bool) – Create missing directories.
dynamic_sizing (bool) – Use dynamic sizing.
force_square (bool) – Force the image size to be square.
export_dirty (bool) – If True, export dirty images.
export_multiscale (bool) – If True, export multiscale images.
overwrite (bool) – Overwrite existing files.

set_channel_num(spw=0): Set number of channels from the linked MS.

set_cube_cell_and_imsize(force_square=False, oversamp=5): Set a single cell and imsize for all chunks.

task_cleanup(tag=None, chunk_num=None)

Cleanup imaging products per chunk.

Parameters:

tag (str or None) – A tag to append to the image root. Defaults to None.
chunk_num (int or None, optional) – The chunk number to clean up.

task_complete_gather_into_cubes(**kwargs)

task_export_to_fits(**kwargs)

task_gather_into_cube(root_name=None, remove_chunks=False, overwrite=False): Gather the chunked data into a final cube.

task_initialize_clean_call(chunk_num, stage='dirty'): Initialize a clean call object for a target, config, product combination and an imaging stage.

task_make_dirty_image(**kwargs)

task_multiscale_clean(**kwargs)

task_read_clean_mask(**kwargs)

task_revert_to_imaging(**kwargs)

task_setup_sdintimaging(**kwargs)

task_singlescale_clean(**kwargs)

task_singlescale_mask(**kwargs)

task_split_cube_to_chunks(imagename, overwrite=False, chunk_num=None, specaxis_name='Frequency')

Split a given cube into the channel chunks.

Parameters:

overwrite (bool) – Overwrite an existing MS file for the chunk.
chunk_num (int, list or None) – If None, will loop through all chunks. If a list or integer are given, this will loop through only the specified chunk numbers defined in ImagingChunkedHandler.chunk_params.

property tempdir_exists: Return True if a temporary imaging directory was created and still exists on disk.

PostProcessHandler

class phangsPipeline.PostProcessHandler(key_handler=None, dry_run=False, raise_exception_mosaic_part_missing=False)

Class to handle post-processing of ALMA data. Post-processing here begins with the results of imaging and proceeds through reduced, science-ready data cubes.

loop_postprocess(imaging_method: str = 'tclean', do_all: bool = False, do_prep: bool = False, do_feather: bool = False, do_mosaic: bool = False, do_cleanup: bool = False, do_summarize: bool = False, trim_coarse_beam_edge_channels: bool = False, feather_apod: bool = False, feather_noapod: bool = False, feather_before_mosaic: bool = False, make_directories: bool = True)

Loops over the full set of targets, products, and configurations to run the postprocessing. Toggle the parts of the loop using the do_XXX booleans. Other choices affect the algorithms used.

Parameters:

imaging_method (str, optional) – Imaging method used. Should be one of ‘tclean’, ‘sdintimaging’. Defaults to ‘tclean’.
do_all (bool, optional) – If True, run all steps. Defaults to False.
do_prep (bool, optional) – If True, run the preparation steps. Defaults to False.
do_feather (bool, optional) – If True, run the feathering steps. Defaults to False.
do_mosaic (bool, optional) – If True, run the mosaicking steps. Defaults to False.
do_cleanup (bool, optional) – If True, run the cleanup steps. Defaults to False.
do_summarize (bool, optional) – If True, run the summarization steps. Defaults to False.
trim_coarse_beam_edge_channels (bool, optional) – If True, will trim out edge channels in the cube with lower resolution beams. Defaults to False
feather_apod (bool, optional) – If True, will do feathering with apodization. Defaults to False.
feather_noapod (bool, optional) – If True, will do feathering without apodization. Defaults to False.
feather_before_mosaic (bool, optional) – If True, will feather each tile before mosaicking. Defaults to False.
make_directories (bool, optional) – If True, will make directories that don’t already exist. Defaults to True.

recipe_cleanup_one_target(target=None, product=None, config=None, check_files=True, ext_ext=''): Recipe that cleans up the output for one target, converting to Kelvin, compressing and trimming the cube and then exporting as a FITS file.

recipe_convolve_to_scale(target=None, product=None, config=None, ext_ext='', export_to_fits=True, check_files=True): Convolve a target, product, config combination to a succession of angulars scale using the task that convolves to a round beam.

recipe_mosaic_one_target(target=None, product=None, config=None, imaging_method='tclean', check_files=True, extra_ext_in='', extra_ext_out=''): Linearly mosaic a single target, performing the convolution, alignment, and mosaicking steps.

recipe_prep_one_target(target=None, product=None, config=None, check_files=True, imaging_method='tclean', trim_coarse_beam_edge_channels=False)

Recipe that takes data from imaging through all steps that come before feathering and/or mosaicking. This means copying the data, primary beam correction, convolution to a round beam, importing and aligning the single dish data, and making weight files for targets that are part of linear mosaicks.

The recipe assumes a lot of file name conventions so just needs the target/product/config defined.

task_align_for_mosaic(target=None, product=None, config=None, in_tags=['linmos_commonres', 'weight', 'prepped_sd', 'sd_weight'], out_tags=['linmos_aligned', 'weight_aligned', 'sd_aligned', 'sd_weight_aligned'], extra_ext_in='', extra_ext_out='', check_files=True): For one target, config, product combination that is a linear mosaic, align all parts of the mosaic to a common astrometric grid for combination into a single image.

task_compress(target=None, product=None, config=None, imaging_method='tclean', in_tag='pbcorr_round', out_tag='pbcorr_trimmed', do_trimrind=True, do_pb_too=True, in_pb_tag='pb', out_pb_tag='pb_trimmed', extra_ext_in='', extra_ext_out='', check_files=True): For one target, product, config combination, compress the cube to the smallest reasonable volume. Also align the primary beam file out onto this grid.

task_convert_units(target=None, product=None, config=None, imaging_method='tclean', in_tag='pbcorr_trimmed', out_tag='pbcorr_trimmed_k', extra_ext_in='', extra_ext_out='', check_files=True): For one target, config, product combination convert the units from Jy/beam to Kelvin.

task_convolve_parts_for_mosaic(target=None, product=None, config=None, in_tag='pbcorr_round', out_tag='linmos_commonres', extra_ext_in='', extra_ext_out='', check_files=True): For one target, config, product combination that is a linear mosaic, convolve all of the parts of the mosaic to share a common angular resolution, appropriate for gridding together into a single image.

task_export_to_fits(target=None, product=None, config=None, imaging_method='tclean', in_tag='pbcorr_trimmed_k', out_tag='pbcorr_trimmed_k_fits', do_pb_too=True, in_pb_tag='trimmed_pb', out_pb_tag='trimmed_pb_fits', extra_ext_in='', extra_ext_out='', check_files=True): For one target, config, product combination export to FITS. Optionally also export the primary beam files.

task_feather(target=None, product=None, config=None, interf_tag='pbcorr_round', sd_tag='prepped_sd', out_tag='pbcorr_round', extra_ext_in='', extra_ext_out='', apodize=False, apod_ext='pb', copy_weights=True, check_files=True): For one target, product, config combination, feather together a single dish and interferometric data set. Note that apodization is exposed as an option. Also note that the configuration of the input and output will differ (an interferometric configuration comes in, a feather configuration comes out). Optionally, propagate the weights from the interferometric side to become the weights for the new feathered data.

task_linear_mosaic(target=None, product=None, config=None, image_tag='linmos_aligned', weight_tag='weight_aligned', out_tag='pbcorr_round', extra_ext_in='', extra_ext_out='', check_files=True): For one target, config, product combination that is a linear mosaic and has already been aligned and convolved, execute the linear mosaic. Needs to be run separately for single dish and interferometer data.

task_make_interf_weight(target=None, product=None, config=None, imaging_method='tclean', image_tag='pbcorr_round', in_tag='pb', input_type='pb', scale_by_noise=True, out_tag='weight', extra_ext_in='', extra_ext_out='', check_files=True): For one target, product, config combination, make a ‘weight’ image for use in linearly mosaicking the cube with other, overlapping cubes. This task targets interferometric dish data.

task_make_singledish_weight(target=None, product=None, config=None, image_tag='prepped_sd', out_tag='sd_weight', extra_ext_in='', extra_ext_out='', check_files=True): For one target, product, config combination, make a ‘weight’ image for use in linearly mosaicking the cube with other, overlapping cubes. This task targets single dish data.

task_pbcorr(target=None, product=None, config=None, imaging_method='tclean', in_tag='orig', out_tag='pbcorr', extra_ext_in='', extra_ext_out='', check_files=True): For one target, product, config combination primary beam correct the interferometer data.

task_rename_sdintimaging(target=None, product=None, config=None, imaging_method='sdintimaging')

task_round_beam(target=None, product=None, config=None, imaging_method='tclean', in_tag='pbcorr', out_tag='pbcorr_round', extra_ext_in='', extra_ext_out='', force_beam_as=None, check_files=True): For one target, product, config combination, convolve the cube to have a round beam. Note that via the force_beam_as keyword this task can also be used to convolve data to a fixed (round) angular resolution.

task_stage_interf_data(target=None, product=None, config=None, imaging_method='tclean', extra_ext_in='', extra_ext_out='', check_files=True, trim_coarse_beam_edge_channels=False): For one target, product, config combination copy the interferometric cube and primary beam file to the working postprocessing directory.

task_stage_singledish(target=None, product=None, config=None, template_tag='pbcorr_round', out_tag='prepped_sd', extra_ext_in='', extra_ext_out='', check_files=True): For one target, product, config combination, copy the single dish data and align it to the interferometric grid.

DerivedHandler

class phangsPipeline.DerivedHandler(key_handler=None, dry_run=False)

Class to create signal masks based on image cubes, and then apply the masks to make moment maps. This is done for each galaxy at multiple spatial/angular scales.

loop_derive_products(do_all: bool = False, do_convolve: bool = False, do_noise: bool = False, do_strictmask: bool = False, do_broadmask: bool = False, do_moments: bool = False, do_secondary: bool = False, do_vfield: bool = False, do_shuffling: bool = False, do_flatmask: bool = False, do_flatmaps: bool = False, make_directories: bool = True, overwrite: bool = True)

Loops over the full set of targets, spectral products (note the dual definition of “product” here), and configurations to do the imaging. Toggle the parts of the loop using the do_XXX booleans. Other choices affect algorithms used.

Parameters:

do_all (bool, optional) – If True, will create all derived product types. Defaults to False.
do_convolve (bool, optional) – If True, will do convolution to different resolutions. Defaults to False.
do_noise (bool, optional) – If True, will do noise estimation. Defaults to False.
do_strictmask (bool, optional) – If True, will build strict masks. Defaults to False.
do_broadmask (bool, optional) – If True, will build broad masks. Defaults to False.
do_moments (bool, optional) – If True, will generate moment maps. Defaults to False.
do_secondary (bool, optional) – If True, will generate secondary moment maps. Defaults to False.
do_vfield (bool, optional) – If True, will generate velocity field. Defaults to False.
do_shuffling (bool, optional) – If True, will generate shuffled cubes. Defaults to False.
do_flatmask (bool, optional) – If True, will generate flat masks. Defaults to False.
do_flatmaps (bool, optional) – If True, will generate flat maps. Defaults to False.
make_directories (bool, optional) – If True, will make the directories if they don’t already exist. Defaults to True.
overwrite (bool, optional) – If True, will overwrite existing files. Defaults to True.

task_build_broad_mask(target=None, config=None, product=None, res_tag=None, extra_ext='', overwrite=False): Estimate the noise associated with a data cube and save it to disk.

task_build_flat_broad_mask(target=None, config=None, product=None, res_tag=None, extra_ext='', overwrite=False): Construct the flat broad mask and save it to disk.

task_build_flat_strict_mask(target=None, config=None, product=None, res_tag=None, extra_ext='', overwrite=False): Construct the flat strict mask and save it to disk.

task_build_strict_mask(target=None, config=None, product=None, res_tag=None, extra_ext='', overwrite=False): Estimate the noise associated with a data cube and save it to disk.

task_build_vfield(target=None, config=None, product=None, res_tag=None, extra_ext='', overwrite=False): Generate a combined velocity field from a list of mom1 maps.

task_convolve(target=None, config=None, product=None, res_tag=None, res_value=None, res_type='ang', just_copy=False, extra_ext_in='', extra_ext_out='', overwrite=False, tol=0.1, nan_treatment='interpolate'): Convolve data to lower resolutions. Defaults to copying in some cases.

task_estimate_noise(target=None, config=None, product=None, res_tag=None, extra_ext='', overwrite=False): Estimate the noise associated with a data cube and save it to disk.

task_generate_flatmaps(target=None, config=None, product=None, res_tag=None, extra_ext='', overwrite=False): Generate moment maps.

task_generate_moments(target=None, config=None, product=None, res_tag=None, extra_ext='', overwrite=False): Generate moment maps.

task_generate_secondary_moments(target=None, config=None, product=None, res_tag=None, extra_ext='', overwrite=False): Generate moment maps.

task_shuffle_cube(target=None, config=None, product=None, res_tag=None, extra_ext='', overwrite=False): Construct shuffled cube and save it to disk.