flint.options ============= .. py:module:: flint.options .. autoapi-nested-parse:: Contains the core of the option class containers that are used to hold stateful properties throughout the flint codebase. Attributes ---------- .. autoapisummary:: flint.options.DEFAULT_COPY_RE_PATTERNS flint.options.DEFAULT_TAR_RE_PATTERNS flint.options.T flint.options.U Classes ------- .. autoapisummary:: flint.options.AddModelSubtractFieldOptions flint.options.ArchiveOptions flint.options.BandpassOptions flint.options.BaseOptions flint.options.FieldOptions flint.options.FitsCubeOptions flint.options.MS flint.options.PolFieldOptions flint.options.SubtractFieldOptions Functions --------- .. autoapisummary:: flint.options._create_argparse_options flint.options.add_options_to_parser flint.options.create_options_from_parser flint.options.dump_field_options_to_yaml flint.options.options_to_dict Module Contents --------------- .. py:class:: AddModelSubtractFieldOptions(/, **data: Any) Bases: :py:obj:`BaseOptions` Options related to predicting a continuum model during the SubtractFieldOptions workflow. Specifically these options deal with identifying the wsclean produced source list model, which may be used by ``admodel`` to predict model visibilities. See utilities around the ``aocalibrate`` functions and routines. .. py:attribute:: addmodel_cluster_config :type: pathlib.Path | None :value: None Specify a new cluster configuration file different to the preferred on. If None, drawn from preferred cluster config .. py:attribute:: calibrate_container :type: pathlib.Path | None :value: None Path to the container with the calibrate software (including addmodel) .. py:attribute:: wsclean_pol_mode :type: list[str] :value: ['i'] The polarisation of the wsclean model that was generated .. py:class:: ArchiveOptions(/, **data: Any) Bases: :py:obj:`BaseOptions` Container for options related to archiving products from flint workflows .. py:attribute:: copy_file_re_patterns :type: tuple[str, Ellipsis] :value: ('.*linmos.*fits', '.*weight\\.fits', '.*png', '.*csv') Regular-expressions used to identify files to copy into a final location (not tarred) .. py:attribute:: tar_file_re_patterns :type: tuple[str, Ellipsis] :value: ('.*MFS.*image\\.fits', '.*linmos.*', '.*weight\\.fits', '.*yaml', '.*\\.txt', '.*png',... Regular-expressions to use to collect files that should be tarballed .. py:class:: BandpassOptions(/, **data: Any) Bases: :py:obj:`BaseOptions` Container that represents the flint related options that might be used throughout the processing of bandpass calibration data. In its present form this `BandpassOptions` class is not intended to contain properties of the data that arebeing processed, rather how these data will be processed. These settings are not meant to be adjustabled throughout a single bandpass pipeline run .. py:attribute:: calibrate_container :type: pathlib.Path | None :value: None Path to the singularity calibrate container .. py:attribute:: expected_ms :type: int :value: 36 The expected number of measurement set files to find .. py:attribute:: flag_calibrate_rounds :type: int :value: 3 The number of times the bandpass will be calibrated, flagged, then recalibrated .. py:attribute:: flagger_container :type: pathlib.Path | None :value: None Path to the singularity aoflagger container .. py:attribute:: minuv :type: float | None :value: None The minimum baseline length, in meters, for data to be included in bandpass calibration stage .. py:attribute:: preflagger_ant_mean_tolerance :type: float :value: 0.2 Tolerance that the mean x/y antenna gain ratio test before the antenna is flagged .. py:attribute:: preflagger_jones_max_amplitude :type: float | None :value: None Flag Jones matrix if any amplitudes with a Jones are above this value .. py:attribute:: preflagger_mesh_ant_flags :type: bool :value: False Share channel flags from bandpass solutions between all antenna .. py:attribute:: smooth_polynomial_order :type: int :value: 4 The polynomial order used by the Savgol filter when smoothing the bandpass solutions .. py:attribute:: smooth_solutions :type: bool :value: False Will activate the smoothing of the bandpass solutions .. py:attribute:: smooth_window_size :type: int :value: 16 The width of the smoothing window used to smooth the bandpass solutions .. py:class:: BaseOptions(/, **data: Any) Bases: :py:obj:`pydantic.BaseModel` A base class that Options style flint classes can inherit from. This is derived from ``pydantic.BaseModel``, and can be used for validation of supplied values. Class derived from ``BaseOptions`` are immutable by default, and have the docstrings of attributes extracted. .. py:method:: _asdict() -> dict[str, Any] .. py:method:: with_options(/, **kwargs) -> T .. py:attribute:: model_config Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict]. .. py:class:: FieldOptions(/, **data: Any) Bases: :py:obj:`BaseOptions` Container that represents the flint related options that might be used throughout components related to the actual pipeline. In its present form this `FieldOptions` class is not intended to contain properties of the data that are being processed, rather how those data will be processed. These settings are not meant to be adjustable throughout rounds of self-calibration. .. py:attribute:: aegean_container :type: pathlib.Path | None :value: None Path to the singularity aegean container .. py:attribute:: beam_cutoff :type: float :value: 150 Cutoff in arcseconds to use when calculating the common beam to convol to .. py:attribute:: calibrate_container :type: pathlib.Path | None :value: None Path to the singularity calibrate container .. py:attribute:: casa_container :type: pathlib.Path | None :value: None Path to the singularity CASA container .. py:attribute:: coadd_cubes :type: bool :value: False Co-add cubes formed throughout imaging together. Cubes will be smoothed channel-wise to a common resolution. Only performed on final set of images .. py:attribute:: expected_ms :type: int :value: 36 The expected number of measurement set files to find .. py:attribute:: fixed_beam_shape :type: tuple[float, float, float] | None :value: None Specify the final beamsize of linmos field images in (arcsec, arcsec, deg) .. py:attribute:: flagger_container :type: pathlib.Path | None :value: None Path to the singularity aoflagger container .. py:attribute:: holofile :type: pathlib.Path | None :value: None Path to the holography FITS cube that will be used when co-adding beams .. py:attribute:: imaging_strategy :type: pathlib.Path | None :value: None Path to a FLINT imaging yaml file that contains settings to use throughout imaging .. py:attribute:: linmos_residuals :type: bool :value: False Linmos the cleaning residuals together into a field image .. py:attribute:: no_imaging :type: bool :value: False Whether to skip the imaging process (including self-calibration) .. py:attribute:: pb_cutoff :type: float :value: 0.1 Primary beam attenuation cutoff to use during linmos .. py:attribute:: potato_container :type: pathlib.Path | None :value: None Path to the singularity potato peel container .. py:attribute:: reference_catalogue_directory :type: pathlib.Path | None :value: None Path to the directory container the reference catalogues, used to generate validation plots .. py:attribute:: rename_ms :type: bool :value: False Rename MSs throughout rounds of imaging and self-cal instead of creating copies. This will delete data-columns throughout. .. py:attribute:: rounds :type: int :value: 2 Number of required rouds of self-calibration and imaging to perform .. py:attribute:: run_aegean :type: bool :value: False Whether to run the aegean source finding tool .. py:attribute:: sbid_archive_path :type: pathlib.Path | None :value: None Path that SBID archive tarballs will be created under. If None no archive tarballs are created. See ArchiveOptions. .. py:attribute:: sbid_copy_path :type: pathlib.Path | None :value: None Path that final processed products will be copied into. If None no copying of file products is performed. See ArchiveOptions. .. py:attribute:: skip_selfcal_on_rounds :type: list[int] | None :value: None Do not perform the derive and apply self-calibration solutions on these rounds .. py:attribute:: stokes_v_imaging :type: bool :value: False Specifies whether Stokes-V imaging will be carried out after the final round of imagine (whether or not self-calibration is enabled). .. py:attribute:: update_model_data_with_source_list :type: bool :value: False Attempt to update a MSs MODEL_DATA column with a source list (e.g. source list output from wsclean) .. py:attribute:: use_beam_masks :type: bool :value: False Construct beam masks from MFS images to use for the next round of imaging. .. py:attribute:: use_beam_masks_from :type: int :value: 1 If `use_beam_masks` is True, this sets the round where beam masks will be generated from .. py:attribute:: use_beam_masks_rounds :type: list[int] | None :value: None If `use_beam_masks` is True, this sets which rounds should have a mask applied .. py:attribute:: use_jolly_tukey_tractor :type: bool :value: False Use the jolly roger tukey tractor. See the TukeyTractorOptions and the jolly-roger package for more details. .. py:attribute:: use_preflagger :type: bool :value: False Whether to apply (or search for solutions with) bandpass solutions that have gone through the preflagging operations .. py:attribute:: use_smoothed :type: bool :value: False Whether to apply (or search for solutions with) a bandpass smoothing operation applied .. py:attribute:: wsclean_container :type: pathlib.Path | None :value: None Path to the singularity wsclean container .. py:attribute:: yandasoft_container :type: pathlib.Path | None :value: None Path to the singularity yandasoft container .. py:attribute:: zip_ms :type: bool :value: False Whether to zip measurement sets once they are no longer required .. py:class:: FitsCubeOptions(/, **data: Any) Bases: :py:obj:`BaseOptions` Container of opptions used to combine images into a single cube using the `fitscube` package. This is particularly useful to manage the larger concatenations. .. py:attribute:: bounding_box :type: bool :value: False Whether to attempt to trim images when combining .. py:attribute:: invalidate_zeros :type: bool :value: True Set pixels whose values are exactly 0.0 to not-a-number (nan) .. py:attribute:: max_workers :type: int :value: 4 The number of concurrent workers (readers/writers) that are permitted at a time .. py:class:: MS Bases: :py:obj:`NamedTuple` Helper to keep track of measurement set information This is the class that should be used when describing a measurement set that will be operated on. .. py:method:: cast(ms: MS | pathlib.Path) -> MS :classmethod: Create/return a MS instance given either a Path or MS. If the input is neither a MS instance or Path, the object will be checked to see if it has a `.ms` attribute. If it does then this will be used. :param ms: The input type to consider :type ms: Union[MS, Path] :raises MSError: Raised when the input ms can not be cast to an MS instance :returns: A normalised MS :rtype: MS .. py:method:: with_options(**kwargs) -> MS Create a new MS instance with keywords updated :returns: New MS instance with updated attributes :rtype: MS .. py:attribute:: beam :type: int | None :value: None The beam ID of the MS within an ASKAP field .. py:attribute:: column :type: str | None :value: None Column that should be operated against .. py:attribute:: field :type: str | None :value: None The field name of the data .. py:attribute:: model_column :type: str | None :value: None The column name of the most recently MODEL data .. py:property:: ms :type: MS .. py:attribute:: path :type: pathlib.Path Path to the measurement set that is being represented .. py:attribute:: spw :type: int | None :value: None Intended to be used with ASKAP high-frequency resolution modes, where the MS is divided into SPWs .. py:class:: PolFieldOptions(/, **data: Any) Bases: :py:obj:`BaseOptions` Container that represents the flint related options that might be used throughout components related to the actual pipeline. In its present form this `PolFieldOptions` class is not intended to contain properties of the data that are being processed, rather how those data will be processed. These settings are not meant to be adjustable across different polarisations. .. py:attribute:: beam_cutoff :type: float :value: 150 Cutoff in arcseconds to use when calculating the common beam to convol to .. py:attribute:: casa_container :type: pathlib.Path | None :value: None Path to the singularity CASA container .. py:attribute:: expected_ms :type: int :value: 36 The expected number of measurement set files to find .. py:attribute:: fixed_beam_shape :type: tuple[float, float, float] | None :value: None Specify the final beamsize of linmos field images in (arcsec, arcsec, deg) .. py:attribute:: holofile :type: pathlib.Path | None :value: None Path to the holography FITS cube that will be used when co-adding beams .. py:attribute:: imaging_strategy :type: pathlib.Path | None :value: None Path to a FLINT imaging yaml file that contains settings to use throughout imaging .. py:attribute:: pb_cutoff :type: float :value: 0.1 Primary beam attenuation cutoff to use during linmos .. py:attribute:: sbid_copy_path :type: pathlib.Path | None :value: None Path that final processed products will be copied into. If None no copying of file products is performed. See ArchiveOptions. .. py:attribute:: trim_linmos_fits :type: bool :value: False Trim the linmos fits files to remove the padding that is added. If True, the output fits files will be smaller but might be different shapes .. py:attribute:: wsclean_container :type: pathlib.Path | None :value: None Path to the singularity wsclean container .. py:attribute:: yandasoft_container :type: pathlib.Path | None :value: None Path to the singularity yandasoft container .. py:class:: SubtractFieldOptions(/, **data: Any) Bases: :py:obj:`BaseOptions` Container for options related to the continuum-subtracted pipeline .. py:attribute:: attempt_subtract :type: bool :value: False Attempt to subtract the model column from the nominated data column .. py:attribute:: beam_cutoff :type: float :value: 150 Cutoff in arcseconds to use when calculating the common beam to convol to .. py:attribute:: channelwise_image :type: bool :value: False Perform channel-wise imaing of the residuals .. py:attribute:: data_column :type: str :value: 'CORRECTED_DATA' Describe the column that should be imaed and, if requested, have model subtracted from .. py:attribute:: expected_ms :type: int :value: 36 The number of measurement sets that should exist .. py:attribute:: fitscube_remove_original_images :type: bool :value: False Remove the images that go into forming the fitscube .. py:attribute:: holofile :type: pathlib.Path | None :value: None Path to the holography FITS cube that will be used when co-adding beams .. py:attribute:: imaging_strategy :type: pathlib.Path | None :value: None Path to a FLINT imaging yaml file that contains settings to use throughout imaging .. py:attribute:: linmos_residuals :type: bool :value: False Linmos the cleaning residuals together into a field image .. py:attribute:: max_intervals :type: int :value: 500 The maximum number of scans/channels to consider .. py:attribute:: pb_cutoff :type: float :value: 0.1 Primary beam attenuation cutoff to use during linmos .. py:attribute:: predict_wsclean_model :type: bool :value: False Search for the continuum model produced by wsclean and subtract .. py:attribute:: stagger_delay_seconds :type: float | None :value: None The delay, in seconds, that should be used when submitting items in batches (e.g. looping over channels) .. py:attribute:: subtract_data_column :type: str :value: 'DATA' Should the continuum model be subtracted, where to store the output. This will update the column to be imaged. .. py:attribute:: subtract_model_data :type: bool :value: False Subtract the MODEL_DATA column from the nominated data column .. py:attribute:: subtract_only :type: bool :value: False Only perform the continuum subtraction .. py:attribute:: timestep_image :type: bool :value: False Perform timestep imaging after subtraction .. py:attribute:: use_addmodel :type: bool :value: False Invoke the ``addmodel`` visibility prediction, including the search for the ``wsclean`` source list .. py:attribute:: use_crystalball :type: bool :value: False Attempt to predict the model visibilities using ``crystalball`` .. py:attribute:: wsclean_container :type: pathlib.Path Path to the container with wsclean .. py:attribute:: yandasoft_container :type: pathlib.Path Path to the container with yandasoft .. py:function:: _create_argparse_options(name: str, field: pydantic.fields.FieldInfo) -> tuple[str, dict[str, Any]] Convert a pydantic Field into ``dict`` to splate into ArgumentParser.add_argument() .. py:function:: add_options_to_parser(parser: argparse.ArgumentParser, options_class: type[BaseOptions], description: str | None = None) -> argparse.ArgumentParser Given an established argument parser and a class derived from a ``pydantic.BaseModel``, populate the argument parser with the model properties. :param parser: Parser that arguments will be added to :type parser: ArgumentParser :param options_class: A ``Options`` style class derived from ``BaseOptions`` :type options_class: type[BaseModel] :returns: Updated argument parser :rtype: ArgumentParser .. py:function:: create_options_from_parser(parser_namespace: argparse.Namespace, options_class: type[U]) -> U Given a ``BaseOptions`` derived class, extract the corresponding arguments from an ``argparse.nNamespace``. These options correspond to ones generated by ``add_options_to_parser``. :param parser_namespace: The argument parser corresponding to those in the ``BaseOptions`` class :type parser_namespace: Namespace :param options_class: A ``BaseOptions`` derived class :type options_class: U :returns: An populated options class with arguments drawn from CLI argument parser :rtype: U .. py:function:: dump_field_options_to_yaml(output_path: pathlib.Path, field_options: FieldOptions | PolFieldOptions | SubtractFieldOptions, overwrite: bool = False) -> pathlib.Path Dump the supplied instance of `FieldOptions` to a yaml file for record keeping. The parent directory of the `output_path` will be created if it does not already exist. :param output_path: Path of the output file. :type output_path: Path :param field_options: The `FieldOptions` class to write. :type field_options: FieldOptions :param overwrite: Overwrite the file if it exists. Defaults to False. :type overwrite: bool, optional :raises FileExistsError: Raise if `output_path` already exists and `overwrite` is `False` :returns: Output path written to. :rtype: Path .. py:function:: options_to_dict(input_options: Any) -> dict Helper function to convert an `Options` type class to a dictionary. Most of `flint` `Option` and `Result` classes used `typing.NamedTuples`, which carry with it a `_asdict` method to convert them to a dictionary. Future roadmap plans to move over to pydantic type models. This is a place holder function to help transition to this. :param input_options: Item to convert to a dictionary :type input_options: Any :raises TypeError: Raised if the conversion to a dictionary was not successful :returns: The dictionary version of the input options :rtype: Dict .. py:data:: DEFAULT_COPY_RE_PATTERNS :value: ('.*linmos.*fits', '.*weight\\.fits', '.*png', '.*csv') .. py:data:: DEFAULT_TAR_RE_PATTERNS :value: ('.*MFS.*image\\.fits', '.*linmos.*', '.*weight\\.fits', '.*yaml', '.*\\.txt', '.*png',... .. py:data:: T .. py:data:: U