Continuum imaging and self-calibration¶

flint allows for a very flexible, but powerful, set of self-calibration options. The full suite of options are enumerated below and can be run using the flint_flow_continuum_pipeline entry point.

As a user, you will need to set and consider some of the following options:

science_path: Directory containing your input science visibilities
split-path: Where your output calibrated visibiltiies will be placed
calibrated-bandpass-path: Set either where your bandpass solutions are (see bandpass calibration).
imaging-strategy: Path to your ‘strategy’ YAML file (see below).

Note that flint supports passing in a config file to specify CLI options via --cli-config (see Configuration). This is particularly useful for sharing a common set of options between multiple runs of the pipeline.

Skipping bandpass calibration¶

flint supports the imaging of CASDA deposited measurement sets whose visibilities produced by the operational ASKAP pipeline. These measurement sets are already bandpass calibrated, and often have gone through multiple rounds of self-calibration. In such a situation bandpass solutions are not needed. Should flint detect that the measurement sets specified by science_path be appear to be from CASDA, the flint_flow_continuum_pipeline will not attempt to apply any bandpass set of solutions, and will appropriately pre-process the visibilities accordingly.

Imaging strategy¶

To keep track of options across rounds of self-calibration we use a ‘strategy’ file in a YAML format. We give details of this in Configuration. You can generate a minimal strategy file using flint_config create. You can specify a ‘global’ set of options under defaults, which will be overwritten by any options set in rounds of selfcal.

By way of example, the following strategy file appears to work well for RACS-style data. We do not recommend using this verbatim for any/all sets of data. The flint_flow_selfcal_pipeline workflow is referenced by the selfcal operation.

version: 0.2
defaults:
  wsclean:
    temp_dir: $MEMDIR
    abs_mem: 100
    local_rms_window: 30
    size: 6128
    local_rms: true
    auto_mask: 10
    auto_threshold: 0.75
    threshold: null
    channels_out: 18
    mgain: 0.7
    nmiter: 14
    niter: 200000
    multiscale: true
    multiscale_scale_bias: 0.6
    multiscale_scales: !!python/tuple
    - 0
    - 4
    - 8
    - 16
    - 24
    - 32
    - 48
    - 64
    - 92
    - 128
    - 196
    fit_spectral_pol: 5
    weight: briggs 0.5
    data_column: CORRECTED_DATA
    scale: 2.5asec
    gridder: wgridder
    nwlayers: null
    wgridder_accuracy: 0.0001
    join_channels: true
    minuv_l: 200
    minuvw_m: null
    maxw: null
    no_update_model_required: false
    no_small_inversion: false
    beam_fitting_size: 1.25
    fits_mask: null
    deconvolution_channels: 6
    parallel_gridding: 36
    pol: i
  gaincal:
    solint: 60s
    calmode: p
    round: 0
    minsnr: 0.0
    uvrange: '>235m'
    selectdata: true
    gaintype: G
    nspw: 1
  masking:
    base_snr_clip: 4
    flood_fill: true
    flood_fill_positive_seed_clip: 6
    flood_fill_positive_flood_clip: 1.25
    flood_fill_use_mac_adaptive_max_depth: 4
    flood_fill_use_mac_adaptive_skew_delta: 0.025
    flood_fill_use_mac_adaptive_step_factor: 4
    grow_low_snr_island: false
    grow_low_snr_island_clip: 1.75
    grow_low_snr_island_size: 12046
  archive:
    tar_file_re_patterns: !!python/tuple
    - .*round4.*MFS.*(image|residual|model,cube)\.fits
    - .*linmos.*
    - .*weight\.fits
    - .*yaml
    - .*\.txt
    - .*png
    - .*beam[0-9]+\.ms\.(zip|tar)
    - .*beam[0-9]+\.ms
    - .*\.caltable
    - .*\.tar
    - .*\.csv
    - .*comp\.fits
    copy_file_re_patterns: !!python/tuple
    - .*linmos.*fits
    - .*weight\.fits
    - .*png
    - .*csv
    - .*caltable\.tar
    - .*txt
    - .*comp\.fits
    - .*yaml
selfcal:
  0:
    wsclean:
      auto_mask: 8
      auto_threshold: 3
      multiscale_scale_bias: 0.8
  1:
    wsclean:
      auto_mask: 5
      auto_threshold: 1.5
      local_rms: False
      nmiter: 9
    gaincal:
      solint: 60s
      calmode: p
      uvrange: '>400m'
      nspw: 2
    masking:
      flood_fill_use_mac: true
      flood_fill_positive_seed_clip: 1.5
      flood_fill_positive_flood_clip: 1.2
      flood_fill_use_mac_box_size: 400
  2:
    wsclean:
      auto_mask: 2
      auto_threshold: 1.0
      local_rms: false
      nmiter: 11
    gaincal:
      solint: 30s
      calmode: p
      uvrange: '>400m'
      nspw: 4
    masking:
      flood_fill_use_mac: true
      flood_fill_positive_seed_clip: 1.2
      flood_fill_positive_flood_clip: 1.1
      flood_fill_use_mac_box_size: 300
  3:
    wsclean:
      auto_mask: 2.0
      auto_threshold: 0.5
      local_rms: false
      nmiter: 16
    gaincal:
      solint: 480s
      calmode: ap
      uvrange: '>400m'
      nspw: 2
    masking:
      flood_fill_use_mac: true
      flood_fill_positive_seed_clip: 1.2
      flood_fill_positive_flood_clip: 0.8
      flood_fill_use_mac_box_size: 60
  4:
    wsclean:
      auto_mask: 2.0
      auto_threshold: 0.5
      local_rms: False
    gaincal:
      solint: 480s
      calmode: ap
      uvrange: '>400m'
      nspw: 2
    masking:
      flood_fill_use_mac: true
      flood_fill_positive_seed_clip: 1.2
      flood_fill_positive_flood_clip: 0.7
      flood_fill_use_mac_box_size: 60
stokesv:
  wsclean:
    pol: v
    no_update_model_required: true
    nmiter: 6

Other notes¶

Should --stokes-v-imaging be invoked than after the last round of self-calibration each measurement set will be imaged in Stokes V. Settings around the imaging parameters for the Stokes V imaging are specified by the stokesv operation.

Should --coadd-cubes be invoked than the spectral Stokes-I cubes produced by wsclean after the final imaging round are co-addede together to form a field image at different channel ranges. This can be used to investigate the spectral variation of sources. Each channel will be convolved to a common resolution for that channel. In this mode a single linmos task is invoked to do the co-adding, which may mean a single long running task should wsclean produce many output channels. Be mindful of memory requirements here, as this modde of operation will attempt to load the entirety of all cubes and weights into memory.

Accessing via the CLI¶

The primary entry point for the continuum and self-calibration and imaging pipeline in flint is the flint_flow_continuum_pipeline:

A prefect based pipeline that:

will perform bandpass calibration with PKS B1934-638 data, or from a derived sky-model
copy and apply to science field
image and self-calibration the science fields
run aegean source finding

usage: flint_flow_continuum_pipeline [-h] [--cli-config CLI_CONFIG]
                                     [--split-path SPLIT_PATH]
                                     [--calibrated-bandpass-path CALIBRATED_BANDPASS_PATH]
                                     [--cluster-config CLUSTER_CONFIG]
                                     [--flagger-container FLAGGER_CONTAINER]
                                     [--calibrate-container CALIBRATE_CONTAINER]
                                     [--casa-container CASA_CONTAINER]
                                     [--expected-ms EXPECTED_MS]
                                     [--wsclean-container WSCLEAN_CONTAINER]
                                     [--yandasoft-container YANDASOFT_CONTAINER]
                                     [--potato-container POTATO_CONTAINER]
                                     [--holofile HOLOFILE] [--rounds ROUNDS]
                                     [--skip-selfcal-on-rounds SKIP_SELFCAL_ON_ROUNDS]
                                     [--zip-ms] [--run-aegean]
                                     [--aegean-container AEGEAN_CONTAINER]
                                     [--no-imaging]
                                     [--reference-catalogue-directory REFERENCE_CATALOGUE_DIRECTORY]
                                     [--linmos-residuals]
                                     [--beam-cutoff BEAM_CUTOFF]
                                     [--fixed-beam-shape FIXED_BEAM_SHAPE FIXED_BEAM_SHAPE FIXED_BEAM_SHAPE]
                                     [--pb-cutoff PB_CUTOFF]
                                     [--use-preflagger] [--use-smoothed]
                                     [--use-beam-masks]
                                     [--use-beam-masks-from USE_BEAM_MASKS_FROM]
                                     [--use-beam-masks-rounds USE_BEAM_MASKS_ROUNDS]
                                     [--imaging-strategy IMAGING_STRATEGY]
                                     [--sbid-archive-path SBID_ARCHIVE_PATH]
                                     [--sbid-copy-path SBID_COPY_PATH]
                                     [--rename-ms] [--stokes-v-imaging]
                                     [--coadd-cubes]
                                     [--update-model-data-with-source-list]
                                     [--use-jolly-tukey-tractor]
                                     science_path

Positional Arguments¶

science_path: Path to directories containing the beam-wise science measurementsets that will have solutions copied over and applied.

Named Arguments¶

--cli-config

Path to configuration file

--split-path

Location to write field-split MSs to. Will attempt to use the parent name of a directory when writing out a new MS.

Default: .

--calibrated-bandpass-path

Path to directory containing the uncalibrated beam-wise measurement sets that contain the bandpass calibration source. If None then the ‘–sky-model-directory’ should be provided.

--cluster-config

Path to a cluster configuration file, or a known cluster name.

Default: 'petrichor'

Inputs for FieldOptions¶

--flagger-container

Path to the singularity aoflagger container

--calibrate-container

Path to the singularity calibrate container

--casa-container

Path to the singularity CASA container

--expected-ms

The expected number of measurement set files to find

Default: 36

--wsclean-container

Path to the singularity wsclean container

--yandasoft-container

Path to the singularity yandasoft container

--potato-container

Path to the singularity potato peel container

--holofile

Path to the holography FITS cube that will be used when co-adding beams

--rounds

Number of required rouds of self-calibration and imaging to perform

Default: 2

--skip-selfcal-on-rounds

Do not perform the derive and apply self-calibration solutions on these rounds

--zip-ms

Whether to zip measurement sets once they are no longer required

Default: False

--run-aegean

Whether to run the aegean source finding tool

Default: False

--aegean-container

Path to the singularity aegean container

--no-imaging

Whether to skip the imaging process (including self-calibration)

Default: False

--reference-catalogue-directory

Path to the directory container the reference catalogues, used to generate validation plots

--linmos-residuals

Linmos the cleaning residuals together into a field image

Default: False

--beam-cutoff

Cutoff in arcseconds to use when calculating the common beam to convol to

Default: 150

--fixed-beam-shape

Specify the final beamsize of linmos field images in (arcsec, arcsec, deg)

--pb-cutoff

Primary beam attenuation cutoff to use during linmos

Default: 0.1

--use-preflagger

Whether to apply (or search for solutions with) bandpass solutions that have gone through the preflagging operations

Default: False

--use-smoothed

Whether to apply (or search for solutions with) a bandpass smoothing operation applied

Default: False

--use-beam-masks

Construct beam masks from MFS images to use for the next round of imaging.

Default: False

--use-beam-masks-from

If use_beam_masks is True, this sets the round where beam masks will be generated from

Default: 1

--use-beam-masks-rounds

If use_beam_masks is True, this sets which rounds should have a mask applied

--imaging-strategy

Path to a FLINT imaging yaml file that contains settings to use throughout imaging

--sbid-archive-path

Path that SBID archive tarballs will be created under. If None no archive tarballs are created. See ArchiveOptions.

--sbid-copy-path

Path that final processed products will be copied into. If None no copying of file products is performed. See ArchiveOptions.

--rename-ms

Rename MSs throughout rounds of imaging and self-cal instead of creating copies. This will delete data-columns throughout.

Default: False

--stokes-v-imaging

Specifies whether Stokes-V imaging will be carried out after the final round of imagine (whether or not self-calibration is enabled).

Default: False

--coadd-cubes

Co-add cubes formed throughout imaging together. Cubes will be smoothed channel-wise to a common resolution. Only performed on final set of images

Default: False

--update-model-data-with-source-list

Attempt to update a MSs MODEL_DATA column with a source list (e.g. source list output from wsclean)

Default: False

--use-jolly-tukey-tractor

Use the jolly roger tukey tractor. See the TukeyTractorOptions and the jolly-roger package for more details.

Default: False

The `FieldOptions` class¶

Embedded below is the flint Options class used to drive the flint_flow_continuum_pipeline workflow. Input values are validated by pydantic to ensure they are appropriately typed.

class FieldOptions(BaseOptions):
    """Container that represents the flint related options that
    might be used throughout components related to the actual
    pipeline.

    In its present form this `FieldOptions` class is not intended
    to contain properties of the data that are being processed,
    rather how those data will be processed.

    These settings are not meant to be adjustable throughout
    rounds of self-calibration.
    """

    flagger_container: Path | None = None
    """Path to the singularity aoflagger container"""
    calibrate_container: Path | None = None
    """Path to the singularity calibrate container"""
    casa_container: Path | None = None
    """Path to the singularity CASA container"""
    expected_ms: int = 36
    """The expected number of measurement set files to find"""
    wsclean_container: Path | None = None
    """Path to the singularity wsclean container"""
    yandasoft_container: Path | None = None
    """Path to the singularity yandasoft container"""
    potato_container: Path | None = None
    """Path to the singularity potato peel container"""
    holofile: Path | None = None
    """Path to the holography FITS cube that will be used when co-adding beams"""
    rounds: int = 2
    """Number of required rouds of self-calibration and imaging to perform"""
    skip_selfcal_on_rounds: list[int] | None = None
    """Do not perform the derive and apply self-calibration solutions on these rounds"""
    zip_ms: bool = False
    """Whether to zip measurement sets once they are no longer required"""
    run_aegean: bool = False
    """Whether to run the aegean source finding tool"""
    aegean_container: Path | None = None
    """Path to the singularity aegean container"""
    no_imaging: bool = False
    """Whether to skip the imaging process (including self-calibration)"""
    reference_catalogue_directory: Path | None = None
    """Path to the directory container the reference catalogues, used to generate validation plots"""
    linmos_residuals: bool = False
    """Linmos the cleaning residuals together into a field image"""
    beam_cutoff: float = 150
    """Cutoff in arcseconds to use when calculating the common beam to convol to"""
    fixed_beam_shape: tuple[float, float, float] | None = None
    """Specify the final beamsize of linmos field images in (arcsec, arcsec, deg)"""
    pb_cutoff: float = 0.1
    """Primary beam attenuation cutoff to use during linmos"""
    use_preflagger: bool = False
    """Whether to apply (or search for solutions with) bandpass solutions that have gone through the preflagging operations"""
    use_smoothed: bool = False
    """Whether to apply (or search for solutions with) a bandpass smoothing operation applied"""
    use_beam_masks: bool = False
    """Construct beam masks from MFS images to use for the next round of imaging. """
    use_beam_masks_from: int = 1
    """If `use_beam_masks` is True, this sets the round where beam masks will be generated from"""
    use_beam_masks_rounds: list[int] | None = None
    """If `use_beam_masks` is True, this sets which rounds should have a mask applied"""
    imaging_strategy: Path | None = None
    """Path to a FLINT imaging yaml file that contains settings to use throughout imaging"""
    sbid_archive_path: Path | None = None
    """Path that SBID archive tarballs will be created under. If None no archive tarballs are created. See ArchiveOptions. """
    sbid_copy_path: Path | None = None
    """Path that final processed products will be copied into. If None no copying of file products is performed. See ArchiveOptions. """
    rename_ms: bool = False
    """Rename MSs throughout rounds of imaging and self-cal instead of creating copies. This will delete data-columns throughout. """
    stokes_v_imaging: bool = False
    """Specifies whether Stokes-V imaging will be carried out after the final round of imagine (whether or not self-calibration is enabled). """
    coadd_cubes: bool = False
    """Co-add cubes formed throughout imaging together. Cubes will be smoothed channel-wise to a common resolution. Only performed on final set of images"""
    update_model_data_with_source_list: bool = False
    """Attempt to update a MSs MODEL_DATA column with a source list (e.g. source list output from wsclean)"""
    use_jolly_tukey_tractor: bool = False
    """Use the jolly roger tukey tractor. See the TukeyTractorOptions and the jolly-roger package for more details."""

Continuum imaging and self-calibration¶

Skipping bandpass calibration¶

Imaging strategy¶

Other notes¶

Accessing via the CLI¶

Positional Arguments¶

Named Arguments¶

Inputs for FieldOptions¶

The FieldOptions class¶

The `FieldOptions` class¶