flint.imager.wsclean ==================== .. py:module:: flint.imager.wsclean .. autoapi-nested-parse:: Simple interface into wsclean Some notes around the file naming. A certain filenaming scheme is required for FITS files to perform leakage correction when co-adding them together in the yandasoft linmos application. The stokes field needs to be encoded as ``.i.``. For example: >>> `SB1234.RACS_0123+43.beam01.i.image.fits` The wsclean formatted output string appends something denoted with ``-``. Within this code there is and attempt to rename the wsclean outputs to replace the ``-`` with a ``.``. Classes ------- .. autoapisummary:: flint.imager.wsclean.ImageSet flint.imager.wsclean.ResolvedCLIResult flint.imager.wsclean.WSCleanOptions flint.imager.wsclean.WSCleanResult Functions --------- .. autoapisummary:: flint.imager.wsclean._make_pols flint.imager.wsclean._rename_wsclean_file flint.imager.wsclean._rename_wsclean_title flint.imager.wsclean._resolve_wsclean_key_value_to_cli_str flint.imager.wsclean._wsclean_output_callback flint.imager.wsclean.cli flint.imager.wsclean.combine_image_set_to_cube flint.imager.wsclean.combine_images_to_cube flint.imager.wsclean.create_wsclean_cmd flint.imager.wsclean.create_wsclean_name_argument flint.imager.wsclean.delete_wsclean_outputs flint.imager.wsclean.get_parser flint.imager.wsclean.get_wsclean_output_names flint.imager.wsclean.get_wsclean_output_source_list_path flint.imager.wsclean.image_set_from_result flint.imager.wsclean.merge_image_sets flint.imager.wsclean.merge_image_sets_from_results flint.imager.wsclean.rename_wsclean_prefix_in_image_set flint.imager.wsclean.rotate_cube flint.imager.wsclean.run_wsclean_imager flint.imager.wsclean.split_and_get_image_set flint.imager.wsclean.split_image_set flint.imager.wsclean.wsclean_cleanup_files flint.imager.wsclean.wsclean_imager Module Contents --------------- .. py:class:: ImageSet(/, **data: Any) Bases: :py:obj:`flint.options.BaseOptions` A structure to represent the images and auxiliary products produced by wsclean .. py:attribute:: dirty :type: list[pathlib.Path] | None :value: None Dirty images. .. py:attribute:: image :type: list[pathlib.Path] Images produced. .. py:attribute:: model :type: list[pathlib.Path] | None :value: None Model images. .. py:attribute:: prefix :type: str Prefix of the images and other output products. This should correspond to the -name argument from wsclean .. py:attribute:: psf :type: list[pathlib.Path] | None :value: None References to the PSFs produced by wsclean. .. py:attribute:: residual :type: list[pathlib.Path] | None :value: None Residual images. .. py:attribute:: source_list :type: pathlib.Path | None :value: None Path to a source list that accompanies the image data .. py:class:: ResolvedCLIResult Bases: :py:obj:`NamedTuple` Mapping results to provide to wsclean .. py:attribute:: bindpath :type: pathlib.Path | None :value: None A path to bind to when called within a container .. py:attribute:: cmd :type: str | None :value: None The argument value pair to place on the CLI. .. py:attribute:: ignore :type: bool :value: False Ignore this CLIResult if True .. py:attribute:: unknown :type: Any | None :value: None Unknown options that could not be converted .. py:class:: WSCleanOptions(/, **data: Any) Bases: :py:obj:`flint.options.BaseOptions` A basic container to handle WSClean options. These attributes should conform to the same option name in the calling signature of wsclean Basic support for environment variables is available. Should a value start with `$` it is assumed to be a environment variable, it is will be looked up. Some basic attempts to determine if it is a path is made. Should the `temp_dir` options be specified then all images will be created in this location, and then moved over to the same parent directory as the imaged MS. This is done by setting the wsclean `-name` argument. Options that start with `flint_` are ignored when generating the wsclean command and generally denote some other flint specific operation. .. py:attribute:: abs_mem :type: int :value: 100 Memory wsclean should try to limit itself to .. py:attribute:: auto_mask :type: float | None :value: 3.5 How deep the construct clean mask is during each cycle .. py:attribute:: auto_threshold :type: float | None :value: 0.5 How deep to clean once initial clean threshold reached .. py:attribute:: beam_fitting_size :type: float | None :value: 1.25 Use a fitting box the size of times the theoretical beam size for fitting a Gaussian to the PSF. .. py:attribute:: channel_range :type: tuple[int, int] | None :value: None Image a channel range between a lower (inclusive) and upper (exclusive) bound .. py:attribute:: channels_out :type: int :value: 4 Number of output channels .. py:attribute:: data_column :type: str :value: 'CORRECTED_DATA' Which column in the MS to image .. py:attribute:: deconvolution_channels :type: int | None :value: None The channels out will be averaged down to this many sub-band images during deconvolution .. py:attribute:: fit_spectral_pol :type: int | None :value: None Number of spectral terms to include during sub-band subtraction .. py:attribute:: fits_mask :type: pathlib.Path | None :value: None Path to a FITS file that encodes a cleaning mask .. py:attribute:: flint_make_cube_inplace :type: bool :value: True Rotate the cube for the linmos axis ordering in place, or do it via a temporary file that then gets deleted. Good thing to turn off when getting weird OSErrors on file writing .. py:attribute:: flint_no_log_wsclean_output :type: bool :value: False If True do not log the wsclean output .. py:attribute:: force_mask_rounds :type: int | None :value: None Round of force masked derivation .. py:attribute:: gridder :type: str | None :value: 'wgridder' Use the wgridder kernel in wsclean (instead of the default w-stacking method) .. py:attribute:: interval :type: tuple[int, int] | None :value: None Image a set of scans between a lower (inclusive) and upper (exclusive) bound .. py:attribute:: join_channels :type: bool :value: True Collapse the sub-band images down to an MFS image when peak-finding .. py:attribute:: join_polarizations :type: bool :value: False Perform deconvolution by searching for peaks in the sum of squares of the polarizations, but subtract components from the individual images. Only possible when imaging two or four Stokes or linear parameters. Default: off. .. py:attribute:: local_rms :type: bool :value: True Whether a local rms map is computed .. py:attribute:: local_rms_window :type: int | None :value: None Size of the window used to estimate rms noise .. py:attribute:: maxw :type: float | None :value: None A percentage specifying the maximum w-term to be gridded, relative to the max w-term being considered .. py:attribute:: mgain :type: float :value: 0.7 Major cycle gain .. py:attribute:: minuv_l :type: float | None :value: None The minimum lambda length that the visibility data needs to meet for it to be selected for imaging .. py:attribute:: minuvw_m :type: float | None :value: None A (u,v) selection command, where any baselines shorter than this will be ignored during imaging .. py:attribute:: multiscale :type: bool :value: False Enable multiscale deconvolution .. py:attribute:: multiscale_fits_mask :type: pathlib.Path | None :value: None Path to a FITS file that contains per-scale clean masks encoded as per-pixel bit values .. py:attribute:: multiscale_gain :type: float | None :value: None Size of step made in the subminor loop of multi-scale. Default currently 0.2, but shows sign of instability. A value of 0.1 might be more stable. .. py:attribute:: multiscale_scale_bias :type: float :value: 0.75 Multiscale bias term .. py:attribute:: multiscale_scales :type: tuple[int, Ellipsis] | None :value: None Scales used for multi-scale deconvolution .. py:attribute:: niter :type: int :value: 750000 Maximum number of minor cycles .. py:attribute:: nmiter :type: int :value: 15 Maximum number of major cycles to perform .. py:attribute:: no_mf_weighting :type: bool :value: False Opposite of -ms-weighting; can be used to turn off MF weighting in -join-channels mode .. py:attribute:: no_reorder :type: bool :value: False If True turn off the reordering of the MS at the beginning of wsclean .. py:attribute:: no_small_inversion :type: bool :value: False Disables an optimisation of wsclean's w-gridder mode. This might improve accuracy of the w-gridder. .. py:attribute:: no_update_model_required :type: bool :value: False Will instruct wsclean not to create the MODEL_DATA column .. py:attribute:: nwlayers :type: int | None :value: None Number of w-layers to use if the gridder mode is w-stacking .. py:attribute:: parallel_deconvolution :type: int | None :value: None If not none, the maximum sub-image size in pixels used during deconvolution (suggestion between to between 1024-4096) .. py:attribute:: parallel_gridding :type: int | None :value: None If not none, then this is the number of channel images that will be gridded in parallel .. py:attribute:: pol :type: str :value: 'i' The polarisation to be imaged .. py:attribute:: save_source_list :type: bool :value: False Saves the found clean components as a BBS/DP3 text sky model .. py:attribute:: scale :type: str :value: '2.5asec' Pixel scale size .. py:attribute:: size :type: int :value: 10128 Image size, only a single dimension is required. Note that this means images will be squares. .. py:attribute:: squared_channel_joining :type: bool :value: False Use with -join-channels to perform peak finding in the sum of squared values over channels, instead of the normal sum. This is useful for imaging QU polarizations with non-zero rotation measures, for which the normal sum is insensitive. .. py:attribute:: temp_dir :type: str | pathlib.Path | None :value: None The path to a temporary directory where files will be written. .. py:attribute:: threshold :type: float | None :value: None Threshold in Jy to stop cleaning .. py:attribute:: weight :type: str :value: 'briggs -0.5' Robustness of the weighting used .. py:attribute:: wgridder_accuracy :type: float | None :value: None The accuracy requested of the wgridder (should it be used), compared as the RMS error when compred to a DFT .. py:class:: WSCleanResult(/, **data: Any) Bases: :py:obj:`flint.options.BaseOptions` Simple container for a wsclean command. .. py:attribute:: bind_dirs :type: tuple[pathlib.Path, Ellipsis] Paths that should be binded to when executing the command .. py:attribute:: cleanup :type: bool :value: True Will clean up the dirty images/psfs/residuals/models when the imaging has completed .. py:attribute:: cmd :type: str The constructede wsclean command that would be executed. .. py:attribute:: image_prefix_str :type: str | None :value: None The prefix of the images that will be created .. py:attribute:: image_set :type: ImageSet | None :value: None The set of images produced by wsclean .. py:attribute:: move_hold_directories :type: tuple[pathlib.Path, pathlib.Path] Paths that should be moved to and from when executing the command .. py:attribute:: ms :type: flint.options.MS The measurement sets that have been included in the wsclean command. .. py:attribute:: options :type: WSCleanOptions The set of wslean options used for imaging .. py:function:: _make_pols(pol_str: str) -> tuple[str, Ellipsis] | None Create a tuple of polarisations from a polarisation string :param pol_str: Polarisation string to convert :type pol_str: str :returns: Tuple of polarisations :rtype: tuple[str, ...] | None .. py:function:: _rename_wsclean_file(input_path: pathlib.Path, rename_file: bool = False) -> pathlib.Path Rename a file with wsclean appended string information to convert its `-` separation markers with `.`. This should handle skipping the field name of the target field observed. :param input_path: The file path that would be examined and modified :type input_path: Path :param clean_parts: Which parts of a split string will be modified. Defaults to -2. :type clean_parts: Union[int, Tuple[int, ...]], optional :param rename_file: Whether the file should be moved / renamed. Defaults to False. :type rename_file: bool, optional :returns: Path to the renamed file :rtype: Path .. py:function:: _rename_wsclean_title(name_str: str) -> str Construct and apply a regular expression that aims to identify the wsclean appended properties string within a file and replace the `-` separator with a `.`. A simple replace of all `-` with `.` may not be ideal if the character has been used on purpose. :param name_str: The name that will be extracted and modified :type name_str: str :returns: The modified string if a wsclean string was matched, otherwise the input `name-str` :rtype: str .. py:function:: _resolve_wsclean_key_value_to_cli_str(key: str, value: Any) -> ResolvedCLIResult An internal function intended to map a key-value pair to the appropriate form to pass to a CLI call into wsclean. :param key: The wsclean argument name to consider. Underscores will be converted to hyphens, as expected by wsclean :type key: str :param value: The value of the argument that should be converted to the appropriately formatted string :type value: Any :returns: Converted CLI output, including paths to bind to and unknown conversions :rtype: ResolvedCLIResult .. py:function:: _wsclean_output_callback(line: str) -> None Call back function used to detect clean divergence .. py:function:: cli() -> None .. py:function:: combine_image_set_to_cube(image_set: ImageSet, remove_original_images: bool = False, inplace: bool = True) -> ImageSet Combine wsclean subband channel images into a cube. Each collection attribute of the input `image_set` will be inspected. The MFS images will be ignored. A output file name will be generated based on the prefix and mode (e.g. `image`, `residual`, `psf`, `dirty`). :param image_set: Collection of wsclean image productds :type image_set: ImageSet :param remove_original_images: If True, images that went into the cube are removed. Defaults to False. :type remove_original_images: bool, optional :param inplace: If True, modify the file in-place. If False, write to a temporary file and :type inplace: bool, optional :param then replace the original. Default True: :returns: Updated iamgeset describing the new outputs :rtype: ImageSet .. py:function:: combine_images_to_cube(images: list[pathlib.Path], prefix: str, mode: str, remove_original_images: bool = False, inplace: bool = True) -> pathlib.Path Combine wsclean subband channel images into a cube. Each collection attribute of the input `image_set` will be inspected. The MFS images will be ignored. A output file name will be generated based on the prefix and mode (e.g. `image`, `residual`, `psf`, `dirty`). :param image_set: Collection of wsclean image productds :type image_set: ImageSet :param remove_original_images: If True, images that went into the cube are removed. Defaults to False. :type remove_original_images: bool, optional :param inplace: If True, modify the file in-place. If False, write to a temporary file and :type inplace: bool, optional :param then replace the original. Default True: :returns: Updated iamgeset describing the new outputs :rtype: ImageSet .. py:function:: create_wsclean_cmd(ms: flint.options.MS, wsclean_options: WSCleanOptions) -> WSCleanResult Create a wsclean command from a WSCleanOptions container For the most part these are one-to-one mappings to the wsclean CLI with the exceptions being: #. the `-name` argument will be generated and supplied to the CLI string and will default to the parent directory and name of the supplied measurement set #. If `wsclean_options.temp_dir` is specified this directory is used in place of the measurement sets parent directory If `container` is supplied to immediately execute this command then the output wsclean image products will be moved from the `temp-dir` to the same directory as the measurement set. :param ms: The measurement set to be imaged :type ms: MS :param wsclean_options: WSClean options to image with :type wsclean_options: WSCleanOptions :param container: If a path to a container is provided the command is executed immediately. Defaults to None. :type container: Optional[Path], optional :raises ValueError: Raised when a option has not been successfully processed :returns: The wsclean command to run :rtype: WSCleanResult .. py:function:: create_wsclean_name_argument(wsclean_options: WSCleanOptions, ms: flint.options.MS) -> pathlib.Path Create the value that will be provided to wsclean -name argument. This has to be generated. Among things to consider is the desired output directory of imaging files. This by default will be alongside the measurement set. If a `temp_dir` has been specified then output files will be written here. :param wsclean_options: Set of wsclean options to consider :type wsclean_options: WSCleanOptions :param ms: The measurement set to be imaged :type ms: MS :returns: Value of the -name argument to provide to wsclean :rtype: Path .. py:function:: delete_wsclean_outputs(prefix: str, output_type: str = 'image', ignore_mfs: bool = True, no_subbands: bool = False) -> list[pathlib.Path] Attempt to remove elected wsclean output files If ``no_subbands`` is True (as in ``channels_out`` is 1) then nothing is deleted. :param prefix: The prefix of the files to remove. This would correspond to the -name of wsclean. :type prefix: str :param output_type: What type of wsclean output to try to remove. Defaults to 'image'. :type output_type: str, optional :param ignore_mfs: If True, do not remove MFS outputs (attempt to, at least). Defaults to True. :type ignore_mfs: bool, optional :param no_subbands: If True, nothing is deleted. Defaults to False. :type no_subbands: bool, Optional :returns: The paths that were removed (or at least attempted to be removed)/ :rtype: List[Path] .. py:function:: get_parser() -> argparse.ArgumentParser .. py:function:: get_wsclean_output_names(prefix: str, subbands: int | None = None, pols: str | tuple[str] | tuple[str, Ellipsis] | None = None, verify_exists: bool = False, include_mfs: bool = True, output_types: str | Collection[str] = ('image', 'dirty', 'residual', 'model', 'psf'), check_exists_when_adding: bool = False) -> ImageSet Attempt to generate the file names and paths that would be created by an imaging run of wsclean. This is done using a the known (expected) naming format of modern wsclean versions. Checks can be made to ensure a file exists before adding it to the output ``ImageSet``. This might be important as some wsclean image products might be deleted in order to preserve disk space. :param prefix: The prefix of the imaging run (akin to -name option in wsclean call) :type prefix: str :param subbands: Number of subbands that were imaged :type subbands: int :param pols: The polarisation of the image. If None are provided then this is not used. Multiple polarisation may be supplied. If multiple pols are given in an iterable, each will be produced. Defaults to None. :type pols: Optional[Union[str,Tuple[str]]], optional :param verify_exists: Ensures that each generated path corresponds to an actual file. Defaults to False. :type verify_exists: bool, optional :param include_mfs: Include the MFS images produced by wsclean. Defaults to True. :type include_mfs: bool, optional :param output_types: Include files of this type, including image, dirty, residual, model, psf. Defaults to ('image','dirty','residual','model', 'psf'). :type output_types: Union[str,Collection[str]] :param check_exists_when_adding: Only add a generated file if it exists. Although related to the ``verify_exists``, when this optional is ``True`` files will silently be ignored if they are not found. if this and ``verify_exists`` are both ``True`` then no errors on missing files will be raised. Defaults to False. :type check_exists_when_adding: bool, optional :raises FileExistsError: Raised when a file does not exist and verify_exists is True. :returns: The file paths that wsclean should create/has created. :rtype: ImageSet .. py:function:: get_wsclean_output_source_list_path(name_path: str | pathlib.Path, pol: str | None = None) -> pathlib.Path WSClean can produce a text file that describes the components that it cleaned, their type, scale and brightness. These are placed in a file that is: >> {name}.{pol}-sources.txt where ``name`` represented the `-name` component. Given an input measurement set path or this `-name` value return the expected source list text file. ``pol`` is the stokes that the source is expected. :param name_path: Value of the ``-name`` option. If `str` converted to a ``Path`` :type name_path: Union[str,Path] :param pol: The polarisation to add to the name. If None the -source.txt suffix is simply appended. Defaults to None. :type pol: Optional[str], optional :returns: Path to the source list text file :rtype: Path .. py:function:: image_set_from_result(wsclean_result: WSCleanResult) -> ImageSet | None .. py:function:: merge_image_sets(image_sets: list[ImageSet]) -> ImageSet Merge the image sets from multiple wsclean results into a single image set. :param wsclean_results: The results of multiple wsclean imaging runs :type wsclean_results: List[WSCleanResult] :returns: The merged image set :rtype: ImageSet .. py:function:: merge_image_sets_from_results(wsclean_results: list[WSCleanResult]) -> ImageSet Merge the image sets from multiple wsclean results into a single image set. :param wsclean_results: The results of multiple wsclean imaging runs :type wsclean_results: List[WSCleanResult] :returns: The merged image set :rtype: ImageSet .. py:function:: rename_wsclean_prefix_in_image_set(input_image_set: ImageSet) -> ImageSet Given an input image_set, rename the files contained in it to remove the `-` separator that wsclean uses and replace it with `.`. Files will be renamed on disk appropriately. :param input_image_set: The collection of output wsclean products :type input_image_set: ImageSet :returns: The updated image_set after replacing the separator and renaming files :rtype: ImageSet .. py:function:: rotate_cube(output_cube_path: str | pathlib.Path, inplace: bool = True) -> pathlib.Path Rotate the FITS cube axes to a shape of (chan, pol, dec, ra) which is what yandasoft linmos tasks expect. :param output_cube_path: Path to the FITS cube to rotate. :type output_cube_path: str | Path :param inplace: If True, modify the file in-place. If False, write to a temporary file and then replace the original. Default True :type inplace: bool, optional :returns: Path to the rotated FITS cube. :rtype: Path .. py:function:: run_wsclean_imager(wsclean_result: WSCleanResult, container: pathlib.Path, make_cube_from_subbands: bool = True) -> ImageSet Run a provided wsclean command. Optionally will clean up files, including the dirty beams, psfs and other assorted things. An `ImageSet` is constructed that attempts to capture the output wsclean image products. If `image_prefix_str` is specified the image set will be created by (ordered by preference): #. Adding the `image_prefix_str` to the `move_directory` #. Guessing it from the path of the measurement set from `wsclean_result.ms.path` :param wsclean_result: The command to run, and other properties (cleanup.) :type wsclean_result: WSCleanResult :param container: Path to the container with wsclean available in it :type container: Path :param bind_dirs: Additional directories to include when binding to the wsclean container. Defaults to None. :type bind_dirs: Optional[Tuple[Path, ...]], optional :param move_hold_directories: The `move_directory` and `hold_directory` passed to the temporary context manager. If None no `hold_then_move_into` manager is used. Defaults to None. :type move_hold_directories: Optional[Tuple[Path,Optional[Path]]], optional :param make_cube_from_subbands: Form a single FITS cube from the set of sub-band images wsclean produces. Defaults to False. :type make_cube_from_subbands: bool, optional :param image_prefix_str: The name used to search for wsclean outputs. If None, it is guessed from the name and location of the MS. Defaults to None. :type image_prefix_str: Optional[str], optional :returns: The executed wsclean output products. :rtype: ImageSet .. py:function:: split_and_get_image_set(image_set: ImageSet, get: str, by: str = 'pol', mode: str = 'image') -> list[pathlib.Path] Split an ImageSet by a given field and return the images that match the field of interest. :param image_set: The ImageSet to split :type image_set: ImageSet :param get: The field to extract from the split images :type get: str :param by: The field to split by. Defaults to "pol". :type by: str, optional :param mode: The mode to extract from ImageSet. Defaults to "image". :type mode: str, optional :raises NamingException: If the field to split by is not found in the ImageSet :returns: The images that match the field of interest :rtype: list[Path] .. py:function:: split_image_set(image_set: ImageSet, by: str = 'pol', mode: str = 'image') -> dict[str, list[pathlib.Path]] Split an ImageSet by a given field. :param image_set: The ImageSet to split :type image_set: ImageSet :param by: Field to split images by. Defaults to "pol". :type by: str, optional :param mode: Type of images to extract from ImageSet. Defaults to "image". :type mode: str, optional :raises NamingException: If the field to split by is not found in the ImageSet :returns: _description_ :rtype: dict[str, list[Path]] .. py:function:: wsclean_cleanup_files(prefix: str | pathlib.Path, output_types: tuple[str, Ellipsis] | None = ('dirty', 'psf', 'model', 'residual'), single_channel: bool = False) -> tuple[pathlib.Path, Ellipsis] Clean up (i.e. delete) files from wsclean. :param prefix: The prefix used to search for files. This is generally the -name :type prefix: Union[str, Path] :param output_types: Which type of output wsclean products to delete. Defaults to ("dirty", "psf", "model", "residual"). :type output_types: Optional[Tuple[str]], optional :param single_channel: Whether there is the subband part of the wsclean file names to consider. Defaults to False. :type single_channel: bool, optional :returns: Set of files that were deleted :rtype: Tuple[Path, ...] .. py:function:: wsclean_imager(ms: pathlib.Path | flint.options.MS, wsclean_container: pathlib.Path, update_wsclean_options: dict[str, Any] | None = None, make_cube_from_subbands: bool = True) -> WSCleanResult Create and run a wsclean imager command against a measurement set. :param ms: Path to the measurement set that will be imaged :type ms: Union[Path,MS] :param wsclean_container: Path to the container with wsclean installed :type wsclean_container: Path :param update_wsclean_options: Additional options to update the generated WscleanOptions with. Keys should be attributes of WscleanOptions. Defaults to None. :type update_wsclean_options: Optional[Dict[str, Any]], optional :returns: _description_ :rtype: WSCleanResult