flint.masking ============= .. py:module:: flint.masking .. autoapi-nested-parse:: Utility functions to make image based masks from images, with the initial thought being towards FITS images. Classes ------- .. autoapisummary:: flint.masking.MaskingOptions flint.masking.SkewResult Functions --------- .. autoapisummary:: flint.masking._adaptive_minimum_absolute_clip flint.masking._create_signal_from_rmsbkg flint.masking._get_signal_image flint.masking._minimum_absolute_clip flint.masking._need_to_make_signal flint.masking._verify_set_positive_seed_clip flint.masking.beam_shape_erode flint.masking.cli flint.masking.consider_beam_mask_round flint.masking.convolve_image_by_scale flint.masking.create_beam_mask_kernel flint.masking.create_boxcar_skew_mask flint.masking.create_convolved_erosion_mask flint.masking.create_multi_scale_erosion flint.masking.create_snr_mask_from_fits flint.masking.extract_beam_mask_from_mosaic flint.masking.fft_binary_erosion flint.masking.get_parser flint.masking.grow_low_snr_mask flint.masking.minimum_absolute_clip flint.masking.reverse_negative_flood_fill Module Contents --------------- .. py:class:: MaskingOptions(/, **data: Any) Bases: :py:obj:`flint.options.BaseOptions` Contains options for the creation of clean masks from some subject image. Clipping levels specified are in units of RMS (or sigma). They are NOT in absolute units. .. py:attribute:: base_snr_clip :type: float :value: 4 A base clipping level to be used should other options not be activated .. py:attribute:: beam_shape_erode :type: bool :value: False Erode the mask using the shape of the restoring beam .. py:attribute:: beam_shape_erode_minimum_response :type: float :value: 0.6 The minimum response of the beam that is used to form the erode structure shape .. py:attribute:: beam_shape_erode_scales :type: tuple[int, Ellipsis] | None :value: None The multi-scale convolution sizes, in pixels, to perform a binary erosion with. Output scales are encoded as a bitmapped value (e.g. n'th scale is n'th bit) .. py:attribute:: convolve_first :type: bool :value: False Attempt to construct mask across scales by first convolving the input image by a scale kernel, then run the island construction stage .. py:attribute:: flood_fill :type: bool :value: False Whether to attempt to flood fill when constructing a mask. This should be `True` for ``grow_low_snr_islands`` and ``suppress_artefacts`` to have an effect. .. py:attribute:: flood_fill_positive_flood_clip :type: float :value: 1.5 Clipping level used to grow seeded islands down to .. py:attribute:: flood_fill_positive_seed_clip :type: float :value: 4.5 The clipping level to seed islands that will be grown to lower signal metric .. py:attribute:: flood_fill_use_mac :type: bool :value: False If True, the clipping levels are used as the `increase_factor` when using a minimum absolute clip .. py:attribute:: flood_fill_use_mac_adaptive_max_depth :type: int | None :value: None Determines the number of adaptive boxcar scales to use when constructing seed mask. If None no adaptive boxcar sizes .. py:attribute:: flood_fill_use_mac_adaptive_skew_delta :type: float :value: 0.2 A box is consider too small for a pixel if the fractional proportion of positive pixels is larger than the deviation away of (0.5 + frac). This threshold is therefore 0 to 0.5 .. py:attribute:: flood_fill_use_mac_adaptive_step_factor :type: float :value: 2.0 Stepping size used to increase box by should adaptive detect poor boxcar statistics .. py:attribute:: flood_fill_use_mac_box_size :type: int :value: 75 The size of the mac box size should mac be used .. py:attribute:: grow_low_snr_island :type: bool :value: False Whether to attempt to grow a mask to capture islands of low SNR (e.g. diffuse emission) .. py:attribute:: grow_low_snr_island_clip :type: float :value: 1.75 The minimum significance levels of pixels to be to seed low SNR islands for consideration .. py:attribute:: grow_low_snr_island_size :type: int :value: 768 The number of pixels an island has to be for it to be accepted .. py:class:: SkewResult Bases: :py:obj:`NamedTuple` .. py:attribute:: box_size :type: int Size of the boxcar window applies .. py:attribute:: positive_pixel_frac :type: numpy.ndarray The fraction of positive pixels in a boxcar function .. py:attribute:: skew_delta :type: float The test threshold for skew .. py:attribute:: skew_mask :type: numpy.ndarray Mask of pixel positions indicating which positions failed the skew test .. py:function:: _adaptive_minimum_absolute_clip(image: numpy.ndarray, increase_factor: float = 2.0, box_size: int = 100, adaptive_max_depth: int = 3, adaptive_box_step: float = 2.0, adaptive_skew_delta: float = 0.2) -> numpy.ndarray .. py:function:: _create_signal_from_rmsbkg(image: pathlib.Path | numpy.ndarray, rms: pathlib.Path | numpy.ndarray, bkg: pathlib.Path | numpy.ndarray) -> numpy.ndarray .. py:function:: _get_signal_image(image: numpy.ndarray | None = None, rms: numpy.ndarray | None = None, background: numpy.ndarray | None = None, signal: numpy.ndarray | None = None) -> numpy.ndarray .. py:function:: _minimum_absolute_clip(image: numpy.ndarray, increase_factor: float = 2.0, box_size: int = 100) -> numpy.ndarray Given an input image or signal array, construct a simple image mask by applying a rolling boxcar minimum filter, and then selecting pixels above a cut of the absolute value value scaled by `increase_factor`. This is a pixel-wise operation. :param image: The input array to consider :type image: np.ndarray :param increase_factor: How large to scale the absolute minimum by. Defaults to 2.0. :type increase_factor: float, optional :param box_size: Size of the rolling boxcar minimum filtr. Defaults to 100. :type box_size: int, optional :returns: The mask of pixels above the locally varying threshold :rtype: np.ndarray .. py:function:: _need_to_make_signal(masking_options: MaskingOptions) -> bool Isolated functions to consider whether a signal image is needed .. py:function:: _verify_set_positive_seed_clip(positive_seed_clip: float, signal: numpy.ndarray) -> float Ensure that the positive seed clip is handled appropriately .. py:function:: beam_shape_erode(mask: numpy.ndarray, fits_header: astropy.io.fits.Header, minimum_response: float = 0.6, scales: list[int] | tuple[int, Ellipsis] | None = None) -> numpy.typing.NDArray[numpy.bool] | numpy.typing.NDArray[numpy.int32] Construct a kernel representing the shape of the restoring beam at a particular level, and use it as the basis of a binary erosion of the input mask. The ``fits_header`` is used to construct the beam shape that matches the same pixel size The ``scales`` parameters outlines the scales being used to clean at during some multi-scale deconvolution algorithm. If provided the binary eroision is carried out at each scale (the beam is convolved by a circular gaussian at this scale). The outputs are stored as a bitmask per pixel where the n'th scales is stored as the n'th bit. Args: mask (np.ndarray): The current mask that will be eroded based on the beam shape fits_header (fits.Header): The fits header of the mask used to generate the beam kernel shape minimum_response (float, optional): The minimum response of the main restoring beam to craft the shape from. Defaults to 0.6. scales (list[int] | tuple[int, ...] | None, optional): Defines the scales that are being used during multi-scale clean. Perform the beam erosion at each of these scales. Defaults to None. Returns: NDArray[np.bool] | NDArray[np.int32]: The eroded beam shape. If a no/single scale provide it is a bool return, otherwise int32. .. py:function:: cli() .. py:function:: consider_beam_mask_round(current_round: int, mask_rounds: str | Collection[int] | int, allow_beam_masks: bool = True) -> bool Evaluate whether a self-calibration round should have a beam clean mask constructed. Rules are: - if `mask_rounds` is a string and is "all", all rounds will have a beam mask - if 'mask_rounds' is a single integer, so long as `current_round` is larger it will have a beam mask - if `mask_rounds` is iterable and contains `current_round` it will have a beam mask - if `allow_beam_masks` is False a False is returned. Otherwise options above are considered. :param current_round: The current self-calibration round that is being performed :type current_round: int :param mask_rounds: The rules to consider whether a beam mask is needed :type mask_rounds: Union[str, Collection[int], int] :param allow_beam_masks: A global allow / deny. This should be `True` for other rules to be considered. Defaults to True. :type allow_beam_masks: bool, optional :returns: Whether per beam mask should be performed :rtype: bool .. py:function:: convolve_image_by_scale(image_data: numpy.typing.NDArray[numpy.floating], scale: int) -> numpy.typing.NDArray[numpy.floating] Generate a convolved version of the input image based on some convolved scale. Note that here the scale refers to the scales used by wsclean. Per the multiscale documentation, where considering gaussians: >> fwhm = 0.45 * scale Also not that: >> fwhm = 2.355 * sigma :param image_data: The imaged to be convolved :type image_data: NDArray[np.floating] :param scale: The pixel scale to smooth with :type scale: int :returns: Smoother version of the input image :rtype: NDArray[np.floating] .. py:function:: create_beam_mask_kernel(fits_header: astropy.io.fits.Header, kernel_size: int | tuple[int, int] = 100, minimum_response: float = 0.6, pixel_scale: int | None = None, auto_resize: bool = True) -> numpy.ndarray Make a mask using the shape of a beam in a FITS Header object. The beam properties in the ehader are used to generate the two-dimensional Gaussian main lobe, from which a cut is made based on the minimum power. :param fits_header: The FITS header to create beam from :type fits_header: fits.Header :param kernel_size: Size of the output kernel in pixels. Will be a square. Defaults to 100. :type kernel_size: int | tuple[int, int], optional :param minimum_response: Minimum response of the beam shape for the mask to be constructed from. Defaults to 0.6. :type minimum_response: float, optional :param pixel_scale: Convolve the restoring beam by this scale. The input scale is in pixel units. Defaults to None. :type pixel_scale: int | None, optional :param auto_resize: If True and the kernel is evaluated to all 1's, enlarge the ``kernel_size`` by a factor of 2. Defaults to True. :type auto_resize: bool, optional :raises KeyError: Raised if CDELT1 and CDELT2 missing :returns: Boolean mask of the kernel shape :rtype: np.ndarray .. py:function:: create_boxcar_skew_mask(image: numpy.ndarray, skew_delta: float, box_size: int) -> numpy.ndarray .. py:function:: create_convolved_erosion_mask(fits_image_path: pathlib.Path, masking_options: MaskingOptions) -> flint.naming.FITSMaskNames Create a per-scale clean mask by smoothing the input image to some scale, and then applying a reverse flood fill with the minimum absolute clip statistic. After the initial mask per scale is constructed a binary erosion is then performed to further refine the mask. :param fits_image_path: The path to the FITS image to load. :type fits_image_path: Path :param masking_options: Options to use throughout the convolve and erosion process. :type masking_options: MaskingOptions :returns: Clean mask written to a FITS file. Should multiple scales be specified they are encoded bitwise per pixel. :rtype: FITSMaskNames .. py:function:: create_multi_scale_erosion(mask: numpy.typing.NDArray[numpy.bool], fits_header: astropy.io.fits.Header, scale: int, minimum_response: float = 0.6) -> numpy.typing.NDArray[numpy.int32] Specialised encode per-scale clean masks as bitmapped per-pixel integer values. :param mask: The current mask that will be eroded based on the beam shape :type mask: np.ndarray :param fits_header: The fits header of the mask used to generate the beam kernel shape :type fits_header: fits.Header :param scales: Defines the scales that are being used during multi-scale clean. Perform the beam erosion at each of these scales. Defaults to None. :type scales: list[int] | None, optional :param minimum_response: The minimum response of the main restoring beam to craft the shape from. Defaults to 0.6. :type minimum_response: float, optional :returns: _description_ :rtype: NDArray[np.int32] .. py:function:: create_snr_mask_from_fits(fits_image_path: pathlib.Path, masking_options: MaskingOptions, fits_rms_path: pathlib.Path | None, fits_bkg_path: pathlib.Path | None, create_signal_fits: bool = False, overwrite: bool = True) -> flint.naming.FITSMaskNames Create a mask for an input FITS image based on a signal to noise given a corresponding pair of RMS and background FITS images. Internally should a signal image be needed it is computed as something akin to: > signal = (image - background) / rms This is done in a staged manner to minimise the number of (potentially large) images held in memory. Each of the input images needs to share the same shape. This means that compression features offered by some tooling (e.g. BANE --compress) can not be used. Depending on the `MaksingOptions` used the signal image may not be needed. Once the signal map as been computed, all pixels below ``min_snr`` are flagged. :param fits_image_path: Path to the FITS file containing an image :type fits_image_path: Path :param masking_options: Configurables on the masking operation procedure. :type masking_options: MaskingOptions :param fits_rms_path: Path to the FITS file with an RMS image corresponding to ``fits_image_path``. Defaults to None. :type fits_rms_path: Optional[Path], optional :param fits_bkg_path: Path to the FITS file with an background image corresponding to ``fits_image_path``. Defaults to None. :type fits_bkg_path: Optional[Path], optional :param create_signal_fits: Create an output signal map. Defaults to False. :type create_signal_fits: bool, optional :param overwrite: Passed to `fits.writeto`, and will overwrite files should they exist. Defaults to True. :type overwrite: bool :returns: Container describing the signal and mask FITS image paths. If ``create_signal_path`` is None, then the ``signal_fits`` attribute will be None. :rtype: FITSMaskNames .. py:function:: extract_beam_mask_from_mosaic(fits_beam_image_path: pathlib.Path, fits_mosaic_mask_names: flint.naming.FITSMaskNames) -> flint.naming.FITSMaskNames Extract a region based on an existing FITS image from a masked FITS image. Here a masked FITS image is intended to be one created by ``create_snr_mask_from_fits``. A simple cutout (e.g. ``CutOut2D``) might not work as intended, as when creating the field image (the intended use case here) might be regridded onto a unique pixel grid that does not correspond to one that would be constructed by wsclean. :param fits_beam_image_path: The template image with a valid WCS. This region is used to extract the masked region :type fits_beam_image_path: Path :param fits_mosaic_mask_names: The previously masked image :type fits_mosaic_mask_names: FITSMaskNames :returns: _description_ :rtype: FITSMaskNames .. py:function:: fft_binary_erosion(mask: numpy.typing.NDArray[numpy.bool] | numpy.typing.NDArray[numpy.floating], kernel: numpy.typing.NDArray[numpy.bool]) -> numpy.typing.NDArray[numpy.bool] Attempt to perform a binary erosion using convolution therom. FFT the input mask, FFT the input kernel, multiply, FFT the result. This would give the running sum of the kernel moved across the image, which in the case of boolean-type values will be the number of overlapping 1's. The returned mask is after ensuring the kernel total sum matches the running sum. :param mask: The mask that will be eroded :type mask: NDArray[np.bool] | NDArray[np.floating] :param kernel: The kernel structure used for the erosion :type kernel: NDArray[np.bool] :returns: The eroded mask :rtype: NDArray[np.bool] .. py:function:: get_parser() -> argparse.ArgumentParser .. py:function:: grow_low_snr_mask(image: numpy.ndarray | None = None, rms: numpy.ndarray | None = None, background: numpy.ndarray | None = None, signal: numpy.ndarray | None = None, grow_low_snr: float = 2.0, grow_low_island_size: int = 512, region_mask: numpy.ndarray | None = None) -> numpy.ndarray There may be cases where normal thresholding operations based on simple pixel-wise SNR cuts fail to pick up diffuse, low surface brightness regions of emission. When some type of masking operation is used there may be instances where these regions are never cleaned. Sometimes smoothing can help to pick up these features, but when attempting to pass such a mask through to the imagery of choice the islands may be larger than the features at their native resolution, unless some other more sophisticated filtering is performed. This function attempts to grow masks to capture islands of contiguous pixels above a low SNR cut that would otherwise go uncleaned. :param Args: image (Optional[np.ndarray], optional): The total intensity pixels to have the mask for. Defaults to None. rms (Optional[np.ndarray], optional): The noise across the image. Defaults to None. background (Optional[np.ndarray], optional): The background acros the image. If None, zeros are assumed. Defaults to None. signal(Optional[np.ndarray], optional): A signal map. Defaults to None. grow_low_snr (float, optional): The SNR pixekls have to be above. Defaults to 2. grow_low_island_size (int, optional): The minimum number of pixels an island should be for it to be considered valid. Defaults to 512. region_mask (Optional[np.ndarray], optional): Whether some region should be masked out before the island size constraint is applied. Defaults to None. :returns: The derived mask of objects with low-surface brightness :rtype: np.ndarray .. py:function:: minimum_absolute_clip(image: numpy.ndarray, increase_factor: float = 2.0, box_size: int = 100, adaptive_max_depth: int | None = None, adaptive_box_step: float = 2.0, adaptive_skew_delta: float = 0.2) -> numpy.ndarray Implements minimum absolute clip method. A minimum filter of a particular boxc size is applied to the input image. The absolute of the output is taken and increased by a guard factor, which forms the clipping level used to construct a clean mask: >>> image > (absolute(minimum_filter(image, box)) * factor) The idea is only valid for zero mean and normally distributed pixels, with positive definite flux, making it appropriate for Stokes I. Larger box sizes and guard factors will make the mask more conservative. Should the boxcar be too small relative to some feature it is aligned it is possible that an excess of positive pixels will produce an less than optimal clipping level. An adaptive box size mode, if activated, attempts to use a larger box around these regions. The basic idea being detecting regions where the boxcar is too small is around the idea that there should be a similar number of positive to negative pixels. Should there be too many positive pixels in a region it is likely there is an :param image: Image to create a mask for :type image: np.ndarray :param increase_factor: The guard factor used to inflate the absolute of the minimum filter. Defaults to 2.0. :type increase_factor: float, optional :param box_size: Size of the box car of the minimum filter. Defaults to 100. :type box_size: int, optional :param adaptive_max_depth: The maximum number of rounds that the adaptive mode is allowed to perform when rescaling boxcar results in certain directions. Defaults to None. :type adaptive_max_depth: Optional[int], optional :param adaptive_box_step: A multiplicative factor to increase the boxcar size by each round. Defaults to 2.0. :type adaptive_box_step: float, optional :param adaptive_skew_delta: Minimum deviation from 0.5 that needs to be met to classify a region as skewed. Defaults to 0.2. :type adaptive_skew_delta: float, optional :returns: Final mask :rtype: np.ndarray .. py:function:: reverse_negative_flood_fill(base_image: numpy.ndarray, masking_options: MaskingOptions, pixels_per_beam: float | None = None) -> numpy.ndarray Attempt to: * seed masks around bright regions of an image and grow them to lower significance thresholds * remove regions of negative and positive islands that surround bright sources. An initial set of islands (and masks) are constructed by first using the `positive_seed_clip` to create an initial SNR based mask. These islands then are binary dilated to grow the islands to adjacent pixels at a lower significance level (see `scipy.ndimage.binary_dilation`). Next an attempt is made to remove artefacts around bright sources, where there are likely to be positive and negative artefacts that originate from calibration errors, deconvolution errors, or residual structure from an incomplete clean. This operation will search for islands of _negative_ pixels above a threshold. These pixels are then grown after a guard mask has been constructed around bright pixels. The assumptions that go into this process: * the no genuine source of negative sky emission * if there are bright negative artefacts there are likely bright positive artefacts nearby * such negative pixels are ~10% level artefacts from a presumed bright sources Optionally, the `grow_low_snr_mask` may also be considered via the `grow_low_snr` and `grow_low_island_size` parameters. :param base_image: The base image or signal map that is used throughout the fill procedure. :type base_image: np.ndarray :param masking_options: Options to carry out masking. :type masking_options: MaskingOptions :param pixels_per_beam: The number of pixels that cover a beam. Defaults to None. :type pixels_per_beam: Optional[float], optional :returns: Mask of the pixels to clean :rtype: np.ndarray