flint.prefect.common.imaging¶
This contains common utilities to enable components of the prefect imaging flowws. The majority of the items here are the task decorated functions. Effort should be made to avoid putting in too many items that are not going to be directly used by prefect imaging flows.
Attributes¶
Functions¶
|
Convolve images to a specified resolution |
|
An internal function that launches the convolution to a common resolution |
Derive the appropriate set of beam shapes and then produce corresponding |
|
|
|
|
Compute a common beam size that all input images will be convoled to. |
|
Compute a common beam size that all input images will be convoled to. |
|
Compute a common beam size that all input images will be convoled to. |
|
Compute a common beam size that all input images will be convoled to. |
Apply a ao-calibrate solutions file to the subject measurement set. |
|
|
Convolve images to a specified resolution |
|
Convolve images to a specified resolution |
Smooth the linmos input image to a fixed resolution |
|
|
Create a mask for an image, with the intention of providing it as a clean mask |
|
Create a multi-panel figure highlighting the RMS, flux scale and astrometry of a field |
Create a set of validation tables that can be used to assess the |
|
|
Extract a clean mask for a beam from a larger clean mask (e.g. derived from a field) |
|
Extract the solution path from a calibrate command. This is often required when |
|
|
|
Perform self-calibration using CASA gaincal and applycal. |
|
|
|
Compute a common beam size for input cubes. |
|
Linmos together a set of input images. |
|
|
Run BANE and Aegean against a FITS image. |
|
|
Run the wsclean imager against an input measurement set |
|
Zip a measurement set |
|
Construct the validation plot and validation table items for the imaged field. |
Module Contents¶
- flint.prefect.common.imaging.convolve_image_set(image_set: flint.imager.wsclean.ImageSet, beam_shape: flint.convol.BeamShape, cutoff: float = 60, mode: str = 'image', filter_str: str | None = None, convol_suffix_str: str = 'conv', remove_original_images: bool = False) list[pathlib.Path][source]¶
Convolve images to a specified resolution
- Parameters:
wsclean_result (WSCleanResult) – Collection of output images from wsclean that will be convolved
beam_shape (BeamShape) – The shape images will be convolved to
cutoff (float, optional) – Maximum major beam axis an image is allowed to have before it will not be convolved. Defaults to 60.
filter_str (Optional[str], optional) – This string must be contained in the image path for it to be convolved. Defaults to None.
convol_suffix_str (str, optional) – The suffix added to the convolved images. Defaults to ‘conv’.
remove_original_images (bool, optional) – If True remove the original image after they have been convolved. Defaults to False.
- Returns:
Path to the output images that have been convolved.
- Return type:
Collection[Path]
- flint.prefect.common.imaging.convolve_then_linmos(wsclean_results: Collection[flint.imager.wsclean.WSCleanResult], beam_shape: flint.convol.BeamShape, field_options: flint.options.FieldOptions | flint.options.SubtractFieldOptions, linmos_suffix_str: str | None, field_summary: flint.summary.FieldSummary | None = None, convol_mode: str = 'image', convol_filter: str = '.MFS.', convol_suffix_str: str = 'conv', trim_linmos_fits: bool = True, remove_original_images: bool = False, cleanup_linmos: bool = False) flint.coadd.linmos.LinmosResult[source]¶
An internal function that launches the convolution to a common resolution and subsequent linmos of the wsclean residual images.
- Parameters:
wsclean_results (Collection[WSCleanResult]) – Collection of wsclean imaging results, with residual images described in the attached
ImageSetbeam_shape (BeamShape) – The beam shape that residual images will be convolved to
field_options (FieldOptions) – Options related to the processing of the field
linmos_suffix_str (str) – The suffix string passed to the linmos parset name
field_summary (Optional[FieldSummary], optional) – The summary of the field, including (importantly) to orientation of the third-axis. Defaults to None.
convol_mode (str, optional) – The mode passed to the convol task to describe the images to extract. Support image or residual. Defaults to image.
convol_filter (str, optional) – A text file applied when assessing images to co-add. Defaults to ‘.MFS.’.
convol_suffix_str (str, optional) – The suffix added to the convolved images. Defaults to ‘conv’.
trim_linmos_fits (bool, optional) – Attempt to trim the output linmos files of as much empty space as possible. Defaults to True.
remove_original_images (bool, optional) – If True remove the original image after they have been convolved. Defaults to False.
cleanup_linmos (bool, optional) – Clean up items created throughout linmos, including the per-channel weight text files for each input image. Defaults to False.
- Returns:
Resulting linmos command parset
- Return type:
- flint.prefect.common.imaging.create_convol_linmos_images(wsclean_results: Collection[flint.imager.wsclean.WSCleanResult], field_options: flint.options.FieldOptions, field_summary: flint.summary.FieldSummary | None = None, additional_linmos_suffix_str: str | None = None) list[flint.coadd.linmos.LinmosResult][source]¶
Derive the appropriate set of beam shapes and then produce corresponding convolved and co-added images
- Parameters:
wsclean_results (Collection[WSCleanResult]) – Set of wsclean commands that have been executed
field_options (FieldOptions) – Set of field imaging options, containing details of the beam/s
field_summary (Optional[FieldSummary], optional) – Summary of the MSs, importantly containing their third-axis rotation. Defaults to None.
additional_linmos_suffix_str (Optional[str], optional) – An additional string added to the end of the auto-generated linmos base name. Defaults to None.
- Returns:
The collection of linmos commands executed.
- Return type:
List[LinmosResult]
- flint.prefect.common.imaging.create_convolve_linmos_cubes(wsclean_results: Collection[flint.imager.wsclean.WSCleanResult], field_options: flint.options.FieldOptions, current_round: int | None = None, additional_linmos_suffix_str: str | None = 'cube')[source]¶
- flint.prefect.common.imaging.get_common_beam_from_image_set(image_set: flint.imager.wsclean.ImageSet, cutoff: float = 25, filter_str: str | None = None, fixed_beam_shape: list[float] | None = None) flint.convol.BeamShape[source]¶
Compute a common beam size that all input images will be convoled to.
- Parameters:
wsclean_results (Collection[WSCleanResult]) – Input images whose restoring beam properties will be considered
cutoff (float, optional) – Major axis larger than this valur, in arcseconds, will be ignored. Defaults to 25.
filter_str (Optional[str], optional) – Only include images when considering beam shape if this string is in the file path. Defaults to None.
fixed_beam_shape (Optional[List[float]], optional) – Specify the final beamsize of linmos field images in (arcsec, arcsec, deg). If None it is deduced from images. Defaults to None;
- Returns:
The final convolving beam size to be used
- Return type:
- flint.prefect.common.imaging.get_common_beam_from_image_sets(image_sets: list[flint.imager.wsclean.ImageSet], cutoff: float = 25, filter_str: str | None = None, fixed_beam_shape: list[float] | None = None) flint.convol.BeamShape[source]¶
Compute a common beam size that all input images will be convoled to.
- Parameters:
wsclean_results (Collection[WSCleanResult]) – Input images whose restoring beam properties will be considered
cutoff (float, optional) – Major axis larger than this valur, in arcseconds, will be ignored. Defaults to 25.
filter_str (Optional[str], optional) – Only include images when considering beam shape if this string is in the file path. Defaults to None.
fixed_beam_shape (Optional[List[float]], optional) – Specify the final beamsize of linmos field images in (arcsec, arcsec, deg). If None it is deduced from images. Defaults to None;
- Returns:
The final convolving beam size to be used
- Return type:
- flint.prefect.common.imaging.get_common_beam_from_images(image_paths: list[pathlib.Path], cutoff: float = 25, filter_str: str | None = None, fixed_beam_shape: list[float] | None = None) flint.convol.BeamShape[source]¶
Compute a common beam size that all input images will be convoled to.
- Parameters:
image_paths (list[Path]) – Input images whose restoring beam properties will be considered
cutoff (float, optional) – Major axis larger than this valur, in arcseconds, will be ignored. Defaults to 25.
filter_str (Optional[str], optional) – Only include images when considering beam shape if this string is in the file path. Defaults to None.
fixed_beam_shape (Optional[List[float]], optional) – Specify the final beamsize of linmos field images in (arcsec, arcsec, deg). If None it is deduced from images. Defaults to None;
- Returns:
The final convolving beam size to be used
- Return type:
- flint.prefect.common.imaging.get_common_beam_from_results(wsclean_results: list[flint.imager.wsclean.WSCleanResult], cutoff: float = 25, filter_str: str | None = None, fixed_beam_shape: list[float] | None = None) flint.convol.BeamShape[source]¶
Compute a common beam size that all input images will be convoled to.
- Parameters:
wsclean_results (Collection[WSCleanResult]) – Input images whose restoring beam properties will be considered
cutoff (float, optional) – Major axis larger than this valur, in arcseconds, will be ignored. Defaults to 25.
filter_str (Optional[str], optional) – Only include images when considering beam shape if this string is in the file path. Defaults to None.
fixed_beam_shape (Optional[List[float]], optional) – Specify the final beamsize of linmos field images in (arcsec, arcsec, deg). If None it is deduced from images. Defaults to None;
- Returns:
The final convolving beam size to be used
- Return type:
- flint.prefect.common.imaging.task_bandpass_create_apply_solutions_cmd(ms: flint.ms.MS, calibrate_cmd: flint.calibrate.aocalibrate.CalibrateCommand, container: pathlib.Path) flint.calibrate.aocalibrate.ApplySolutions[source]¶
Apply a ao-calibrate solutions file to the subject measurement set.
- Parameters:
ms (MS) – The measurement set that will have the solutions file applied
calibrate_cmd (CalibrateCommand) – A resulting ao-calibrate command and meta-data item
container (Path) – The container that can apply the ao-calibrate style solutions file to the measurement set
- Returns:
The resulting apply solutions command and meta-data
- Return type:
- flint.prefect.common.imaging.task_common_beam_convolve_linmos(wsclean_results: list[flint.imager.wsclean.WSCleanResult], field_options: flint.options.SubtractFieldOptions, convol_mode: str, convol_filter: str | None = None, convol_suffix_str: str | None = None, remove_original_images: bool = False, trim_linmos_fits: bool = False, cleanup_linmos: bool = False, linmos_suffix_str: str | None = None, field_summary: flint.summary.FieldSummary | None = None) flint.coadd.linmos.LinmosResult[source]¶
- flint.prefect.common.imaging.task_convolve_cube(wsclean_result: flint.imager.wsclean.WSCleanResult, beam_shapes: list[flint.convol.BeamShape], cutoff: float = 60, mode: Literal['image'] = 'image', convol_suffix_str: str = 'conv') Collection[pathlib.Path][source]¶
Convolve images to a specified resolution
- Parameters:
wsclean_result (WSCleanResult) – Collection of output images from wsclean that will be convolved
beam_shapes (BeamShape) – The shape images will be convolved to
cutoff (float, optional) – Maximum major beam axis an image is allowed to have before it will not be convolved. Defaults to 60.
convol_suffix_str (str, optional) – The suffix added to the convolved images. Defaults to ‘conv’.
- Returns:
Path to the output images that have been convolved.
- Return type:
Collection[Path]
- flint.prefect.common.imaging.task_convolve_image(wsclean_result: flint.imager.wsclean.WSCleanResult, beam_shape: flint.convol.BeamShape, cutoff: float = 60, mode: str = 'image', filter_str: str | None = None, convol_suffix_str: str = 'conv', remove_original_images: bool = False) Collection[pathlib.Path][source]¶
Convolve images to a specified resolution
- Parameters:
wsclean_result (WSCleanResult) – Collection of output images from wsclean that will be convolved
beam_shape (BeamShape) – The shape images will be convolved to
cutoff (float, optional) – Maximum major beam axis an image is allowed to have before it will not be convolved. Defaults to 60.
filter_str (Optional[str], optional) – This string must be contained in the image path for it to be convolved. Defaults to None.
convol_suffix_str (str, optional) – The suffix added to the convolved images. Defaults to ‘conv’.
remove_original_images (bool, optional) – If True remove the original image after they have been convolved. Defaults to False.
- Returns:
Path to the output images that have been convolved.
- Return type:
Collection[Path]
- flint.prefect.common.imaging.task_convolve_linmos_to_fixed_shape(linmos_result: flint.coadd.linmos.LinmosResult, field_options: flint.options.FieldOptions) flint.coadd.linmos.LinmosResult[source]¶
Smooth the linmos input image to a fixed resolution
- Parameters:
linmos_result (LinmosResult) – An existing linmos results set representing a created image
field_options (FieldOptions) – A field options instance with an set fixed beeam shape set
- Returns:
A smoothed version of the input linmos image
- Return type:
- flint.prefect.common.imaging.task_create_image_mask_model(image: flint.coadd.linmos.LinmosResult | flint.imager.wsclean.ImageSet | flint.imager.wsclean.WSCleanResult, image_products: flint.source_finding.aegean.AegeanOutputs, update_masking_options: dict[str, Any] | None = None) flint.naming.FITSMaskNames[source]¶
Create a mask for an image, with the intention of providing it as a clean mask to an appropriate imager.
- Parameters:
linmos_parset (LinmosResult) – Linmos command and associated meta-data
image_products (AegeanOutputs) – Images of the RMS and BKG
update_masking_options (Optional[Dict[str,Any]], optional) – Updated options supplied to the default MaskingOptions. Defaults to None.
- Raises:
ValueError – Raised when
image_productsare not known- Returns:
Clean mask where all pixels below a S/N are masked
- Return type:
- flint.prefect.common.imaging.task_create_validation_plot(field_summary: flint.summary.FieldSummary, aegean_outputs: flint.source_finding.aegean.AegeanOutputs, reference_catalogue_directory: pathlib.Path, upload_artifact: bool = True) pathlib.Path[source]¶
Create a multi-panel figure highlighting the RMS, flux scale and astrometry of a field
- Parameters:
aegean_outputs (AegeanOutputs) – Output aegean products
reference_catalogue_directory (Path) – Directory containing NVSS, SUMSS and ICRS reference catalogues. These catalogues are reconginised internally and have expected names.
upload_artifact (bool, optional) – If True the validation plot will be uploaded to the prefect service as an artifact. Defaults to True.
- Returns:
Path to the output figure created
- Return type:
Path
- flint.prefect.common.imaging.task_create_validation_tables(field_summary: flint.summary.FieldSummary, aegean_outputs: flint.source_finding.aegean.AegeanOutputs, reference_catalogue_directory: pathlib.Path, upload_artifacts: bool = True) flint.validation.ValidationTables[source]¶
Create a set of validation tables that can be used to assess the correctness of an image and associated source catalogue.
- Parameters:
processed_ms_paths (List[Path]) – The processed MS files that were used to create the source catalogue
rms_image_path (Path) – The RMS fits image the source catalogue was constructed against.
source_catalogue_path (Path) – The source catalogue.
output_path (Path) – The output path of the figure to create
reference_catalogue_directory (Path) – The directory that contains the reference catalogues installed
- Returns:
The tables that were created
- Return type:
- flint.prefect.common.imaging.task_extract_beam_mask_image(linmos_mask_names: flint.naming.FITSMaskNames, wsclean_result: flint.imager.wsclean.WSCleanResult) flint.naming.FITSMaskNames[source]¶
Extract a clean mask for a beam from a larger clean mask (e.g. derived from a field)
- Parameters:
linmos_mask_names (FITSMaskNames) – Mask that will be drawn from to form a smaller clean mask (e.g. for a beam)
wsclean_result (WSCleanResult) – Wsclean command and meta-data. This is used to draw from the WCS to create an appropriate pixel-to-pixel mask
- Returns:
Clean mask for a image
- Return type:
- flint.prefect.common.imaging.task_extract_solution_path(calibrate_cmd: flint.calibrate.aocalibrate.CalibrateCommand) pathlib.Path[source]¶
Extract the solution path from a calibrate command. This is often required when interacting with a
PrefectFuturewrappedCalibrateCommandresult.- Parameters:
calibrate_cmd (CalibrateCommand) – The subject calibrate command the solution file will be extracted from
- Returns:
Path to the solution file
- Return type:
Path
- flint.prefect.common.imaging.task_flag_ms_aoflagger(ms: FlagMS, container: pathlib.Path | None) FlagMS[source]¶
- flint.prefect.common.imaging.task_gaincal_applycal_ms(ms: flint.ms.MS | flint.imager.wsclean.WSCleanResult, selfcal_round: int, casa_container: pathlib.Path, update_gain_cal_options: dict[str, Any] | None = None, archive_input_ms: bool = False, skip_selfcal: bool = False, rename_ms: bool = False, archive_cal_table: bool = False) flint.ms.MS[source]¶
Perform self-calibration using CASA gaincal and applycal.
- Parameters:
ms (Union[MS, WSCleanResult]) – A resulting wsclean output. This is used purely to extract the
.msattribute.selfcal_round (int) – Counter indication which self-calibration round is being performed. A name is included based on this.
casa_container (Path) – A path to a singularity container with CASA tooling.
update_gain_cal_options (Optional[Dict[str, Any]], optional) – Options used to overwrite the default
gaincaloptions. Defaults to None.archive_input_ms (bool, optional) – If True the input measurement set is zipped. Defaults to False.
skip_selfcal (bool, optional) – Should this self-cal be skipped. If True, the a new MS is created but not calibrated the appropriate new name and returned.
rename_ms (bool, optional) – It True simply rename a MS and adjust columns appropriately (potentially deleting them) instead of copying the complete MS. If True archive_input_ms is ignored. Defaults to False.
archive_cal_table (bool, optional) – Archive the output calibration table in a tarball. Defaults to False.
- Raises:
ValueError – Raised when a
.msattribute can not be obtained- Returns:
Self-calibrated measurement set
- Return type:
- flint.prefect.common.imaging.task_get_channel_images_from_paths(paths: list[pathlib.Path]) list[pathlib.Path][source]¶
- flint.prefect.common.imaging.task_get_cube_common_beam(wsclean_results: Collection[flint.imager.wsclean.WSCleanResult], cutoff: float = 25) list[flint.convol.BeamShape][source]¶
Compute a common beam size for input cubes.
- Parameters:
wsclean_results (Collection[WSCleanResult]) – Input images whose restoring beam properties will be considered
cutoff (float, optional) – Major axis larger than this valur, in arcseconds, will be ignored. Defaults to 25.
- Returns:
The final convolving beam size to be used per channel in cubes
- Return type:
List[BeamShape]
- flint.prefect.common.imaging.task_linmos_images(image_list: list[pathlib.Path], container: pathlib.Path, linmos_options: flint.coadd.linmos.LinmosOptions, field_summary: flint.summary.FieldSummary | None = None, suffix_str: str | None = None, parset_output_path: str | None = None) flint.coadd.linmos.LinmosResult[source]¶
Linmos together a set of input images.
The
linmos_options.image_output_nameis updated to be a name based on the commonn unique fields ofimage_list.- Parameters:
image_list (list[Path]) – Collection of images to coadd
container (Path) – Path to a yandasoft singularity container
linmos_options (LinmosOptions) – Collection of linmos options
field_summary (FieldSummary | None, optional) – Description of the field, used to get the
pol_axisof the field. Defaults to None.suffix_str (str | None, optional) – Additional suffix str to add when generating the output file names. Defaults to None.
parset_output_path (str | None, optional) – The output parameter set that will be generated. Defaults to None.
- Returns:
Collection of linmos items generated
- Return type:
- flint.prefect.common.imaging.task_potato_peel(ms: flint.ms.MS, potato_container: pathlib.Path, update_potato_config_options: dict[str, Any] | None = None, update_potato_peel_options: dict[str, Any] | None = None, update_wsclean_options: dict[str, Any] | None = None) flint.ms.MS[source]¶
- flint.prefect.common.imaging.task_run_bane_and_aegean(image: flint.imager.wsclean.WSCleanResult | flint.coadd.linmos.LinmosResult, aegean_container: pathlib.Path, update_bane_options: dict[str, Any] | None = None, update_aegean_options: dict[str, Any] | None = None) flint.source_finding.aegean.AegeanOutputs[source]¶
Run BANE and Aegean against a FITS image.
Notes
It has been noted that BANE can sometimes get caught in a interpolation error which haults execution. The
timelimit_secondswill attempt to detect long runnings BANE processes and raise an error. The retry functionality of prefect should then restart the task. Since this task is pure (e.g. no last dataproducts, modification to data etc) simply restarting should be fine.- Parameters:
image (Union[WSCleanResult, LinmosResult]) – The image that will be searched
aegean_container (Path) – Path to a singularity container containing BANE and aegean
update_bane_options (dict[str, Any] | None, optional) – Over-ride any default options of BANEOptions. If None defaults are used. Defaults to None.
update_aegean_options (dict[str, Any] | None, optional) – Over-ride any default options of AegeanOptions. If None defaults are used. Defaults to None.
- Raises:
ValueError – Raised when
imageis not a supported type- Returns:
Output BANE and aegean products, including the RMS and BKG images
- Return type:
- flint.prefect.common.imaging.task_wsclean_imager(in_ms: flint.calibrate.aocalibrate.ApplySolutions | flint.ms.MS, wsclean_container: pathlib.Path, update_wsclean_options: dict[str, Any] | None = None, fits_mask: flint.naming.FITSMaskNames | None = None, channel_range: tuple[int, int] | None = None, scan_range: tuple[int, int] | None = None, make_cube_from_subbands: bool = True) flint.imager.wsclean.WSCleanResult[source]¶
Run the wsclean imager against an input measurement set
The channel_range and scan_range options over-write the update_wsclean_options, as some flow programmatically create these.
- Parameters:
in_ms (Union[ApplySolutions, MS]) – The measurement set that will be imaged
wsclean_container (Path) – Path to a singularity container with wsclean packages
update_wsclean_options (Optional[Dict[str, Any]], optional) – Options to update from the default wsclean options. Defaults to None.
fits_mask (Optional[FITSMaskNames], optional) – A path to a clean guard mask. Defaults to None.
channel_range (Optional[Tuple[int,int]], optional) – Add to the wsclean options the specific channel range to be imaged. Defaults to None.
scan_range (Optional[Tuple[int,int]], optional) – Add to the wsclean options the specific scan range to be imaged. Defaults to None.
- Returns:
A resulting wsclean command and resulting meta-data
- Return type:
- flint.prefect.common.imaging.task_zip_ms(in_item: flint.imager.wsclean.WSCleanResult) pathlib.Path[source]¶
Zip a measurement set
- Parameters:
in_item (WSCleanResult) – The inpute item with a
.msattribute of typeMS.- Returns:
Output path of the zipped measurement set
- Return type:
Path
- flint.prefect.common.imaging.validation_items(field_summary: flint.summary.FieldSummary, aegean_outputs: flint.source_finding.aegean.AegeanOutputs, reference_catalogue_directory: pathlib.Path)[source]¶
Construct the validation plot and validation table items for the imaged field.
Internally these are submitting the prefect task versions of: - task_create_validation_plot - task_create_validation_tables
- Parameters:
field_summary (FieldSummary) – Container representing the SBID being imaged and its populated characteristics
aegean_outputs (AegeanOutputs) – Source finding results
reference_catalogue_directory (Path) – Location of directory containing the reference known NVSS, SUMSS and ICRS catalogues
- flint.prefect.common.imaging.task_copy_and_preprocess_casda_askap_ms: prefect.Task[P, R] = None[source]¶
- flint.prefect.common.imaging.task_get_common_beam_from_image_set: prefect.Task[P, R] = None[source]¶