Configuration

CLI Configuration file

To help manage (and avoid) long CLI calls to configure flint, most command line options may be dumped into a new-line delimited text file which can then be set as the --cli-config option of some workflows. See the configargparse python utility to read up on more on how options may be overridden if specified in both the text file and CLI call.

Strategy file

To help wrangling the many options available in flint we use a ‘strategy’ file. These options are those that would be used by stages throughout a workflow, and are used to help track how options may evolve across operations that may be repeated (e.g. imaging and self-calibration).

This file is written in YAML and has the following rough layout:

defaults:
    mode:
        option1: X
        option2: Y

operation1:
    mode:
        option1: Z

operation2:
    mode:
        option2: A

The defaults section sets the ‘global’ default for a given option, which can be updated in given context e.g. a particular round of self-calibration.

An ‘operation’ refers to flow context in which a particular tool is being usef. As of version 0.2, the following ‘operations’ are supported:

  • selfcal

  • stokesv

  • subtractcube

  • polarisation

A ‘mode’ refers to a set of options for a given tool. As of version 0.2, the following ‘modes’ are supported:

  • wsclean : This corresponds to flint.imager.wsclean.WSCleanOptions

  • gaincal : This corresponds to flint.selfcal.casa.GainCalOptions

  • masking : This corresponds to eye_patch.options.MaskingOptions (see Creating Clean Masks)

  • archive : This corresponds to flint.options.ArchiveOptions

  • bane : This corresponds to flint.source_finding.aegean.BANEOptions

  • aegean : This corresponds to flint.source_finding.aegean.AegeanOptions

  • potatopeel: This corresponds to flint.peel.potato.PotatoPeelOptions

  • fitscube: This correspond to flint.options.FitsCubeOptions

  • dp3preflagger, dp3aoflagger, dp3ddecal, dp3gaincal, dp3applycal: These correspond to the options classes in flint.dp3 (see DP3)

To see all the available options you can run flint_{mode} -h on the command-line (eye-patch mask -h for masking). All attributes available in the corresponding Options class listed above may be provided in the strategy YAML file.

As a general rule, the following hierarchy is used to set a value for a given option:

  • The value given in a CLI call

  • The specific context in a given ‘mode’ (e.g. round of self-cal)

  • The value in defaults

  • The default value in the Options class

  • The default value in a calling function

  • The default value in the external tool

Note that not all options are available in all of the above locations.

wsclean

Imaging XX and YY

Setting pol: xx,yy has wsclean deconvolve the XX and YY polarisations separately. flint then forms Stokes I = (XX + YY) / 2 from them itself:

selfcal:
  3:
    wsclean:
      pol: xx,yy
      join_polarizations: true
    gaincal:
      calmode: ap

With pol: i wsclean predicts the same Stokes I model into both the XX and YY correlations of MODEL_DATA. That is wrong for bright sources away from the beam centre, where the X and Y primary beams differ. Amplitude self-calibration then absorbs the difference into the antenna gains. With pol: xx,yy each correlation is predicted from its own model.

  • The Stokes I images are named exactly as a pol: i run names them, so every later stage (cubing, convolution, linmos, source finding) is unchanged. The XX and YY images are deleted once Stokes I is formed. Set flint_keep_linear_images: true to keep them, for example to inspect the beam’s ellipticity.

  • join_polarizations: true is recommended. Without it XX and YY are cleaned independently, each with √2 more noise than Stokes I, so each gets a shallower model with its own mask. With it peaks are found in XX² + YY², and the components of both share positions.

  • wsclean grids both polarisations, so expect up to twice the imaging time and temporary disk use.

  • save_source_list cannot be used with pol: xx,yy. wsclean only saves a source list when a single polarisation is imaged (Stokes I, or a pseudo Stokes I such as XX alone), and refuses to start otherwise. The list would not help in any case: it holds one I flux per component, so update_model_data_with_source_list would replace the separate XX and YY models with a single unpolarised one.

  • The subtract-cube flow reads the source list from the continuum imaging. If it will be run, keep the final self-cal round at pol: i with save_source_list, and use pol: xx,yy on the earlier rounds.

  • No other feed products are supported. xx alone, all four correlations, and circular feeds are refused when the strategy is verified.

  • The polarisation operation is unaffected. It always sets pol from the polarisation being imaged.

fitscube

In the self-cal flows (continuum_pipeline, racs_all_continuum_selfcal), fitscube options are looked up per round like wsclean/gaincal/ masking: selfcal.<round>.fitscube in the strategy file overrides defaults.fitscube for that round — with one deliberate exception, compress (see below).

subtract_cube_pipeline and polarisation_pipeline are not self-cal pipelines so a single value from defaults (or the subtractcube/polarisation operation block) is used throughout.

compress

FitsCubeOptions.compress compresses a finished cube, in the way set by compress_method:

  • gzip / pgzip gzip the whole file to .fits.gz.

  • tile-gzip (lossless) and tile-rice (lossy, compress_quantize_level steps per noise sigma) write a tile-compressed .fits.fz, the format CFITSIO’s fpack writes. Single channels can be read without decompressing the whole cube, and tile-rice is several times smaller than gzip on noise-dominated cubes.

The SPICE stage always compresses, with SpiceOptions.compress_method (default tile-gzip). Under tile-rice the weight and background cubes are still compressed losslessly, since quantizing smooth cubes makes them bigger.

Compression is only safe for a cube that is never reopened later in the pipeline: astropy cannot memmap a compressed file (.gz or .fz), so the memmapped reads flint and rm-lite use would decompress the whole cube into memory. split_cube_into_planes and RM-synthesis refuse compressed cubes for exactly this reason (see flint.cube.split_cube_into_planes).

How to read, decompress and handle compressed cubes, and the pitfalls of quantizing more than once, are covered in fitscube’s README. In short: read .fits.fz cubes with hdul[1].section[...] rather than .data, decompress with fitscube decompress, and quantize at most once.

In the self-cal flows, each round reassigns the wsclean_results variable to that round’s own output, so only the final round’s per-beam cube survives the loop and feeds create_convolve_linmos_cubes, which splits it into planes to co-add across beams. Earlier rounds’ per-beam cubes are discarded once the next round starts and are never split. So compress cannot simply follow the usual per-round lookup: get_selfcal_round_fitscube_options (flint.configuration) only forces compress=False for the final round’s per-beam cube; every other round’s fitscube options, including compress, are drawn from the strategy file exactly as written, with no overriding. The separate, co-added cube produced by create_convolve_linmos_cubes at the end of the loop is unaffected by this and honours compress: true set under defaults.fitscube (or a selfcal.<round>.fitscube override for the final round) as normal. If you set compress: true for the final round’s per-beam cube, it is ignored and a warning is logged.

Configuration based settings in Python API

Most settings within flint are stored in immutable option classes, e.g. WSCleanOptions, GainCalOptions. Once such an option class has been created, any new option values may only be set by creating a new instance. In such cases there is an appropriate .with_options method that might be of use. This ‘nothing changes unless explicitly done so’ was adopted early as a way to avoid confusion when moving to a distributed multi-node execution environment.

The added benefit is that it has defined very clear interfaces into key stages throughout flints calibration and imaging stages. The flint_config program can be used to create template yaml file that lists default values of these option classes that are expected to be user-tweakable, and provides the ability to change values of options throughout initial imaging and subsequent rounds of self-calibration.

In a nutshell, the currently supported option classes that may be tweaked through this template method are:

  • WSCleanOptions (shorthand wsclean)

  • GainCalOptions (shorthand gaincal)

  • MaskingOptions (shorthand masking)

  • ArchiveOptions (shorthand archive)

  • BANEOptions (shorthand bane)

  • AegeanOptions (shorthand aegean)

  • FitsCubeOptions (shorthand fitscube)

All attributes supported by these options may be set in this template format. Not that these options would have to be retrieved within a particular flow and passed to the appropriate functions - they are not (currently) automatically accessed.

The defaults scope sets all of the default values of these classes. The initial scope overrides the default imaging wsclean options to be used with the first round of imaging before self-calibration.

The selfcal scope contains a key-value mapping, where an integer key relates the options to that specific round of masking, imaging and calibration options for that round of self-calibration. Again, options set here override the corresponding options defined in the defaults scope.

flint_config can be used to generate a template file, which can then be tweaked. The template file uses YAML to define scope and settings. So, use the YAML standard when modifying this file. There are primitive verification functions to ensure the modified template file is correctly form.

BaseOptions class

All (or most) of flint’s Options classes are derived from flint.options.BaseOptions. This uses pydantic in order to validate upon class initialisation that the provided values match the types they are listed as having. For instance, if a float of a value 3.141 was provided to an option with type int it will be converted appropriately. This is also true for values described in a strategy file.

If the values are unable to be coerced into a correct type than pydantic will raise an error.

All flint Option classes should be derived from BaseOptions:

from flint.options import BaseOptions


class PirateOptions(BaseOptions):
    """An Options class used to create a new Pirate"""

    name: str
    """Name of the pirate we will create"""
    age: int = 34
    """The age, in years, of the pirate"""
    weaknesses: list[str] | None = None
    """Should the pirate have any weaknessess they go here"""

Should PirateOptions(name="Jack Sparrow", age=34.23, weaknesses=None) be invoked, the age=34.23 input, which is a float, will be cast to an int.