Creating Clean Masks¶

The imaging performance of CLEAN and its derivatives can be improved through the construction of reliable clean mask regions. These regions restrict the set of pixels that are involved during the peak finding stage of CLEAN. Here the aim of the game is to optimally select pixels that contain genuine emission. Issues such as clean bias. sub-optimal self-calibration and clean divergence can be minimised or outright eliminated provided a clean mask has been reliable constructed.

Flint provides functionality to construct such clean masks, where the output is a pixel-wise mask of arbitrary shapes that either allow of deny cleaning to occur at certain pixel locations. These are intended to be evaluated against a restored FITS image. Pixels in the output clean mask are either 0 (cleaning is not allowed here) or 1 (cleaning is allowed here).

Available statistics¶

Flint currently supports two statistics to identify pixels of significance that should be cleaned.

Signal-to-noise (SNR)¶

The obvious method is based on signal-to-noise (SNR). After constructing background (\(\mu\)) and noise (\(\sigma\)) measures across an image a direction-dependent SNR (\(\mathrm{SNR}\)) across the image (\(\mathrm{img}\)) can be expressed:

\(\mathrm{SNR}(x,y) = \frac{\left(\mathrm{img}-\mu\right)}{\sigma}.\)

In the simplest case constant values may be set across the extent of the image \(\mu=0\), and \(\sigma=\mathrm{SD}(\mathrm{img})\). These can be replaced with more sophisticated schemes that compute position dependent metrics that also incorporate iterative outlier clipping (e.g. Background and Noise Estimator) or robust statistics (e.g. Selavy).

At their heart these SNR based processes are assuming Gaussian distributed zero-mean noise, where pixel intensities that are unlikely to occur by chance are assumed to be genuine emission. That is to say, pixels that are \(\gg5\sigma\) are likely to be real and should be included in a mask for cleaning.

Minimum absolute clip¶

SNR based measures have a clear statistical foundation when identifying bright pixels. However, there are situations when such a metric could be perturbed or otherwise corrupted, including:

Calibration errors that produce imaging artefacts,
Deconvolution errors that accumulate over minor/major iterations,
Misshandling of the w-term so that the 2D Fourier transform approximation breaks down,
Extended diffuse structures that represents a large fraction of the region used to calculate the local \(\mu\) or \(\sigma\) quantities,
Combinations of the above, or
Gremlins lurking around in the data.

Ultimately, if the regions being considered in the derivation of the noise do not appear Gaussian like, the robustness of methods to calculate the local noise level are suspect. Additionally, it is unclear whether an accurate noise estimation is even an appropriate basis for such regions. For instance, artefacts around a bright sources produced by phase error could be \(\gg\sigma\) if the genuine source is sufficiently bright.

We introduce the Minimum Absolute Clip (\(\mathrm{MAC}\)) as an alternative metric. By assuming:

approximately zero-mean distributed Gaussian noise,
the sky brightness is positive definite, and
should there be a significantly bright negative component there will be a positive one of comparable brightness nearby.

These assumptions are fair for Stokes-I images. A masking function can therefore be constructed, where:

\(\mathrm{MAC}(x, y) = \mathrm{img} > P \times |\mathrm{RollingMinimum}\left(\mathrm{img}, \mathrm{BoxSize}(\mathrm{img}, x, y)\right)|\)

where \(\mathrm{RollingMinimum}\) is a minimum boxcar filter of dimensions return by \(\mathrm{BoxSize}\)pixels, which could be turned in a position dependent way. The absolute value of the \(\mathrm{RollingMinimum}\) function is increased by a padding factor \(P\) to increase the minimum positive threshold. The \(\mathrm{MAC}\) is a simply and efficient statistic to compute.

The choice of an appropriate boxcar size is an important consideration. A larger size enforces more conservative behaviour from the \(\mathrm{MAC}\) process. Should too small a region be supplied than typically large multi-scale features are less reliably assessed. Extensions can be made to detect when an the rolling box filter size is too small (e.g. by an imbalance of positive to negative pixels). A basic implementation of this adaptive boxcar size is implemented via the MaskingOptions.flood_fill_use_mbd_adaptive parameters.

Reverse flood filling¶

Cleaning source with features that are both faint and diffuse is difficult. On a per-pixel basis it is often a reality that a minimum signal-to-noise threshold is not met. Should the cleaning thresholds (set in either absolute \(\mathrm{Jy}/\mathrm{beam}\) units or in terms of \(\sigma\) units) be too high this diffuse emission is never cleaned. In flint we include a reverse flood fill procedure that grows islands of pixels above an initial criteria out to adjacent pixels that met a secondary lower level. This process is often the initial step of most source finders.

There are two steps to this process:

Initial siginficant island detection: The statistic of choice (see above) is executed and all pixels above the threshold are marketed as pixels to clean. We can set this via MaskingOptions.flood_fill_positive_seed_clip
Growing the initial set of islands; Islands that are identified as significant go through a dilation process, where the dilation process is restricted to limit it only to pixels above the lower level threshold. This lower level clip is set via MaskingOptions.flood_fill_positive_flood_clip.

Activating the flood fill procedure is ‘opt in’ and activated via MaskingOptions.flood_fill. Further, the \(\mathrm{MAC}\) statistic is only exposed when the flood fill is enabled.

Eroding output islands¶

Output binary clean masks are naturally at the resolution of the input image (at least in flint). All pixels that reside in an island are candidate pixels where cleaning is allowed to occur. Consider an unresolved source that is perfectly aligned to the discrete pixel grid. In principal its clean component should be contained entirely to a single pixel. However, as the source is sufficiently bright the resulting mask is made up of many pixels (i.e. the shape the restoring beam). Should deep cleaning be employed it is possible that small perturbations from the noise underlying a source would also be cleaned.

A binary erosion process with the erosision structure set to the shape of the restoring beam at a particular power level can be used to ‘contract’ all pixel masks. The output should better reflect where clean components ought to be placed. See the MaskingOptions.beam_shape_erode and MaskingOptions.beam_shape_erode_minimum_response options to activate and control this process. The minimum response should be set between 0 to 1, where numbers closer to 0 represent a larger binary erosion structure shape. That is to say islands need to be larger for any pixels to remain as MaskingOptions.beam_shape_erode_minimum_response approaches 0.

Multi-scale beam erosion¶

basic routines to enable per-scale beam shape erosion are provided via MaskingOptions.beam_shape_eroison_scales. For each specified scale the restoring beam (the base shape in this erosion) is convolved with a circular gaussian of the specified scale (each scale is FWHM in pixels). The resulting beam is then evaluated in the basic process outlined above, eroding the clean islands at the native resolution to the clean islands at the specified scale. The clean mask at each scale is stored as ‘bitmapped’ value - except since these are floats the n’th scale is stored as 2**n for each pixel.

This is intended to provide a mechanism to ensure cleaning at larger scales occurs in well defined regions. An example of the erosion process for 0 1 2 4 8 16 32 64 128 256 512 1024 2056 is shown below. Point source islands are still in the mask, it is just that since they are at the initial scale defined, they are stored as 2**0 = 1 values.

`--convolve-first` option¶

Historically for extract galactic fields the reverse flood fill process described above using a minimum absolute clip as the threshold statistic produced reliable clean masks. A subsequent binary erosion process to attempt to characterise per-scale clean masks produced meaningful results. For regions like the Galactic Plane, however, the reverse flood fill process would often miss diffuse emission that was both extended and below the flood fill clip level. Reducing this lower level clip would not consistently capture such faint extended structures, and pixels boosted by noise would often leak into the clean mask from legitimate islands (flying spaghetti monsters).

The --convol-first option activates a routine where the base image is first put through an open filter before being convolved to some desired scale. Significance thresholding operations are then applied to this filtered image. Importantly, the open filter attempts to isolate small structures from larger ones. This is implemented through the combined usage of a minimum filter followed by a maximum filter (could be considered a high-pass filter).

Should this option be activated be aware that the seed and flood clipping threshold change - it should be possible to use lower limits. For example flood_fill_positive_seed_clip=1.02 and flood_file_positive_flood_clip=0.15 when the minimum absolute clip statistic is used.

Accessing via the CLI¶

The masking utility may be accessed via a CLI entrypoint:

Simple utility functions to create masks from FITS images.

usage: flint_masking [-h] {mask,extractmask} ...

Positional Arguments¶

mode

Possible choices: mask, extractmask

Operation mode of flint_bandpass

Sub-commands¶

mask¶

Create a mask for an image, potentially using its RMS and BKG images (e.g. outputs from BANE). Output FITS image will default to the image with a mask suffix.

flint_masking mask [-h] [--base-snr-clip BASE_SNR_CLIP] [--flood-fill]
                   [--flood-fill-positive-seed-clip FLOOD_FILL_POSITIVE_SEED_CLIP]
                   [--flood-fill-positive-flood-clip FLOOD_FILL_POSITIVE_FLOOD_CLIP]
                   [--flood-fill-use-mac]
                   [--flood-fill-use-mac-box-size FLOOD_FILL_USE_MAC_BOX_SIZE]
                   [--flood-fill-use-mac-adaptive-step-factor FLOOD_FILL_USE_MAC_ADAPTIVE_STEP_FACTOR]
                   [--flood-fill-use-mac-adaptive-skew-delta FLOOD_FILL_USE_MAC_ADAPTIVE_SKEW_DELTA]
                   [--flood-fill-use-mac-adaptive-max-depth FLOOD_FILL_USE_MAC_ADAPTIVE_MAX_DEPTH]
                   [--grow-low-snr-island]
                   [--grow-low-snr-island-clip GROW_LOW_SNR_ISLAND_CLIP]
                   [--grow-low-snr-island-size GROW_LOW_SNR_ISLAND_SIZE]
                   [--beam-shape-erode]
                   [--beam-shape-erode-minimum-response BEAM_SHAPE_ERODE_MINIMUM_RESPONSE]
                   [--beam-shape-erode-scales BEAM_SHAPE_ERODE_SCALES [BEAM_SHAPE_ERODE_SCALES ...]]
                   [--convolve-first] [--rms-fits RMS_FITS]
                   [--bkg-fits BKG_FITS] [--save-signal]
                   image

Positional Arguments¶

image: Path to the input image.

Named Arguments¶

--rms-fits

Path to the RMS of the input image.

--bkg-fits

Path to the BKG of the input image.

--save-signal

Save the signal image internally generated (should it be generated)

Default: False

Inputs for MaskingOptions¶

--base-snr-clip

A base clipping level to be used should other options not be activated

Default: 4

--flood-fill

Whether to attempt to flood fill when constructing a mask. This should be True for grow_low_snr_islands and suppress_artefacts to have an effect.

Default: False

--flood-fill-positive-seed-clip

The clipping level to seed islands that will be grown to lower signal metric

Default: 4.5

--flood-fill-positive-flood-clip

Clipping level used to grow seeded islands down to

Default: 1.5

--flood-fill-use-mac

If True, the clipping levels are used as the increase_factor when using a minimum absolute clip

Default: False

--flood-fill-use-mac-box-size

The size of the mac box size should mac be used

Default: 75

--flood-fill-use-mac-adaptive-step-factor

Stepping size used to increase box by should adaptive detect poor boxcar statistics

Default: 2.0

--flood-fill-use-mac-adaptive-skew-delta

A box is consider too small for a pixel if the fractional proportion of positive pixels is larger than the deviation away of (0.5 + frac). This threshold is therefore 0 to 0.5

Default: 0.2

--flood-fill-use-mac-adaptive-max-depth

Determines the number of adaptive boxcar scales to use when constructing seed mask. If None no adaptive boxcar sizes

--grow-low-snr-island

Whether to attempt to grow a mask to capture islands of low SNR (e.g. diffuse emission)

Default: False

--grow-low-snr-island-clip

The minimum significance levels of pixels to be to seed low SNR islands for consideration

Default: 1.75

--grow-low-snr-island-size

The number of pixels an island has to be for it to be accepted

Default: 768

--beam-shape-erode

Erode the mask using the shape of the restoring beam

Default: False

--beam-shape-erode-minimum-response

The minimum response of the beam that is used to form the erode structure shape

Default: 0.6

--beam-shape-erode-scales

The multi-scale convolution sizes, in pixels, to perform a binary erosion with. Output scales are encoded as a bitmapped value (e.g. n’th scale is n’th bit)

--convolve-first

Attempt to construct mask across scales by first convolving the input image by a scale kernel, then run the island construction stage

Default: False

extractmask¶

Extract a beam FITS masked region from a larger FITS mask mosaic image.

flint_masking extractmask [-h] beam_image mosaic_image

Positional Arguments¶

beam_image: The FITS image with the WCS that will be used to extract the mask region from
mosaic_image: The field mosaic image that will be used as a basis to extract a region mask from

The `MaskingOptions` class¶

Embedded below is the flint Options class used to construct clean masks. Input values are validated by pydantic to ensure they are appropriately typed.

class MaskingOptions(BaseOptions):
    """Contains options for the creation of clean masks from some subject
    image. Clipping levels specified are in units of RMS (or sigma). They
    are NOT in absolute units.
    """

    base_snr_clip: float = 4
    """A base clipping level to be used should other options not be activated"""
    flood_fill: bool = False
    """Whether to attempt to flood fill when constructing a mask. This should be `True` for ``grow_low_snr_islands`` and ``suppress_artefacts`` to have an effect. """
    flood_fill_positive_seed_clip: float = 4.5
    """The clipping level to seed islands that will be grown to lower signal metric"""
    flood_fill_positive_flood_clip: float = 1.5
    """Clipping level used to grow seeded islands down to"""
    flood_fill_use_mac: bool = False
    """If True, the clipping levels are used as the `increase_factor` when using a minimum absolute clip"""
    flood_fill_use_mac_box_size: int = 75
    """The size of the mac box size should mac be used"""
    flood_fill_use_mac_adaptive_step_factor: float = 2.0
    """Stepping size used to increase box by should adaptive detect poor boxcar statistics"""
    flood_fill_use_mac_adaptive_skew_delta: float = 0.2
    """A box is consider too small for a pixel if the fractional proportion of positive pixels is larger than the deviation away of (0.5 + frac). This threshold is therefore 0 to 0.5"""
    flood_fill_use_mac_adaptive_max_depth: int | None = None
    """Determines the number of adaptive boxcar scales to use when constructing seed mask. If None no adaptive boxcar sizes"""
    grow_low_snr_island: bool = False
    """Whether to attempt to grow a mask to capture islands of low SNR (e.g. diffuse emission)"""
    grow_low_snr_island_clip: float = 1.75
    """The minimum significance levels of pixels to be to seed low SNR islands for consideration"""
    grow_low_snr_island_size: int = 768
    """The number of pixels an island has to be for it to be accepted"""
    beam_shape_erode: bool = False
    """Erode the mask using the shape of the restoring beam"""
    beam_shape_erode_minimum_response: float = 0.6
    """The minimum response of the beam that is used to form the erode structure shape"""
    beam_shape_erode_scales: tuple[int, ...] | None = None
    """The multi-scale convolution sizes, in pixels, to perform a binary erosion with. Output scales are encoded as a bitmapped value (e.g. n'th scale is n'th bit)"""
    convolve_first: bool = False
    """Attempt to construct mask across scales by first convolving the input image by a scale kernel, then run the island construction stage"""

Creating Clean Masks¶

Available statistics¶

Signal-to-noise (SNR)¶

Minimum absolute clip¶

Reverse flood filling¶

Eroding output islands¶

Multi-scale beam erosion¶

--convolve-first option¶

Accessing via the CLI¶

Positional Arguments¶

Sub-commands¶

mask¶

Positional Arguments¶

Named Arguments¶

Inputs for MaskingOptions¶

extractmask¶

Positional Arguments¶

The MaskingOptions class¶

`--convolve-first` option¶

The `MaskingOptions` class¶