flint.calibrate.aocalibrate =========================== .. py:module:: flint.calibrate.aocalibrate .. autoapi-nested-parse:: Code to use AO calibrate s Classes ------- .. autoapisummary:: flint.calibrate.aocalibrate.AOSolutions flint.calibrate.aocalibrate.ApplySolutions flint.calibrate.aocalibrate.CalibrateCommand flint.calibrate.aocalibrate.CalibrateOptions flint.calibrate.aocalibrate.FlaggedAOSolution Functions --------- .. autoapisummary:: flint.calibrate.aocalibrate.apply_solutions_to_ms flint.calibrate.aocalibrate.calibrate_apply_ms flint.calibrate.aocalibrate.calibrate_options_to_command flint.calibrate.aocalibrate.cli flint.calibrate.aocalibrate.create_apply_solutions_cmd flint.calibrate.aocalibrate.create_calibrate_cmd flint.calibrate.aocalibrate.fill_between_flags flint.calibrate.aocalibrate.find_existing_solutions flint.calibrate.aocalibrate.flag_aosolutions flint.calibrate.aocalibrate.get_parser flint.calibrate.aocalibrate.load_aosolutions_file flint.calibrate.aocalibrate.plot_solutions flint.calibrate.aocalibrate.run_apply_solutions flint.calibrate.aocalibrate.run_calibrate flint.calibrate.aocalibrate.save_aosolutions_file flint.calibrate.aocalibrate.select_aosolution_for_ms flint.calibrate.aocalibrate.select_refant Module Contents --------------- .. py:class:: AOSolutions Bases: :py:obj:`NamedTuple` Structure to load an AO-style solutions file .. py:method:: load(path: pathlib.Path) -> AOSolutions :classmethod: Load in an AO-stule solution file. See `load_solutions_file`, which is internally used. .. py:method:: plot_solutions(ref_ant: int | None = 0) -> Iterable[pathlib.Path] Plot the solutions of all antenna for the first time-interval in the aosolutions file. The XX and the YY will be plotted. :param ref_ant: Reference antenna to use. If None is specified there is no division by a reference antenna. Defaults to 0. :type ref_ant: Optional[int], optional :returns: Path to the phase and amplited plots created. :rtype: Iterable[Path] .. py:method:: save(output_path: pathlib.Path) -> pathlib.Path Save the instance of AOSolution to a standard aosolution binary file :param output_path: Location to write the file to :type output_path: Path :returns: Location the file was written to :rtype: Path .. py:attribute:: bandpass :type: numpy.ndarray Complex data representing the antenna Jones. Shape is (nsol, nant, nchan, npol) .. py:attribute:: nant :type: int Number of antenna in the solution file .. py:attribute:: nchan :type: int Number of channels in the solution file .. py:attribute:: npol :type: int Number of polarisations in the file .. py:attribute:: nsol :type: int Number of time solutions .. py:attribute:: path :type: pathlib.Path Path of the solutions file loaded .. py:class:: ApplySolutions(/, **data: Any) Bases: :py:obj:`flint.options.BaseOptions` The applysolutions command to execute .. py:attribute:: cmd :type: str The command that will be executed .. py:attribute:: ms :type: flint.options.MS The measurement set that will have the solutions applied to .. py:attribute:: solution_path :type: pathlib.Path Location of the solutions file to apply .. py:class:: CalibrateCommand(/, **data: Any) Bases: :py:obj:`flint.options.BaseOptions` The AO Calibrate command and output path of the corresponding solutions file .. py:attribute:: cmd :type: str The calibrate command that will be executed .. py:attribute:: model :type: pathlib.Path Path to the model that would be used to calibrate against .. py:attribute:: ms :type: flint.options.MS The measurement set to have solutions derived for .. py:attribute:: preflagged :type: bool :value: False Indicates whether the solution file has gone through preflagging routines. .. py:attribute:: solution_path :type: pathlib.Path The output path of the solutions file .. py:class:: CalibrateOptions(/, **data: Any) Bases: :py:obj:`flint.options.BaseOptions` Structure used to represent options into the `calibrate` program These attributes have the same names as options into the `calibrate` command. .. py:attribute:: datacolumn :type: str The name of the datacolumn that will be calibrates .. py:attribute:: i :type: int | None :value: 100 The number of iterations that may be performed .. py:attribute:: m :type: pathlib.Path The path to the model file used to calibtate .. py:attribute:: maxuv :type: float | None :value: None The maximum distance in meters that is .. py:attribute:: minuv :type: float | None :value: None The minimum distance in meters that is .. py:attribute:: p :type: tuple[pathlib.Path, pathlib.Path] | None :value: None Plot output names for the amplitude gain and phases .. py:class:: FlaggedAOSolution Bases: :py:obj:`NamedTuple` Hold the final set of flagged solutions and generated plots .. py:attribute:: bandpass :type: numpy.ndarray The bandpass solutions after flagging, as saved in the solutions file .. py:attribute:: path :type: pathlib.Path Path to the final set of flagged solutions .. py:attribute:: plots :type: Collection[pathlib.Path] Contains paths to the plots generated throughout the flagging and smoothing procedure .. py:function:: apply_solutions_to_ms(ms: pathlib.Path | flint.options.MS, solutions_path: pathlib.Path, container: pathlib.Path, data_column: str = 'DATA') -> ApplySolutions .. py:function:: calibrate_apply_ms(ms_path: pathlib.Path, model_path: pathlib.Path, container: pathlib.Path, data_column: str = 'DATA') -> ApplySolutions Will create and run a calibration command using AO calibrator, and then apply these solutions. :param ms_path: The measurement set that will be calibrated :type ms_path: Path :param model_path: The model file containing sources to calibrate against :type model_path: Path :param container: Container that has the AO calibtate and applysolutions file. :type container: Path :param data_column: The name of the column containing the data to calibrate. Defaults to "DATA". :type data_column: str, optional :returns: The command, solution binary path and new measurement set structure :rtype: Applysolutions .. py:function:: calibrate_options_to_command(calibrate_options: CalibrateOptions, ms_path: pathlib.Path, solutions_path: pathlib.Path) -> str Generate a `calibrate` command given an input option set :param calibrate_options: The set of `calibrate` options to use :type calibrate_options: CalibrateOptions :param ms: Path to the measurement set that will be calibrated :type ms: Path :param solutions_path: Output path of the solutions file :type solutions_path: Path :returns: The command string to execute :rtype: str .. py:function:: cli() -> None .. py:function:: create_apply_solutions_cmd(ms: flint.options.MS, solutions_file: pathlib.Path, output_column: str | None = None, container: pathlib.Path | None = None) -> ApplySolutions Construct the command to apply calibration solutions to a MS using an AO calibrate style solutions file. The `applysolutions` program does not appear to have the ability to set a desured output column name. If the `output_column` specified matches the nominated column in `ms`, then `applysolutions` will simply overwrite the column with updated data. Otherwise, a `CORRECTED_DATA` column is produced. NOTE: Care to be taken when the nominated column is `CORRECTED_DATA`. :param ms: Measurement set to have solutions applied to :type ms: MS :param solutions_file: Path to the solutions file to apply :type solutions_file: Path :param output_column: The desired output column name. See notes above. Defaults to None. :type output_column: Optional[str], optional :param container: If a path to a container is supplied the calibrate command is executed immediately. Defaults to None. :type container: Optional[Path], optional :returns: Description of applysolutions command, solutions file path and updated MS :rtype: ApplySolutions .. py:function:: create_calibrate_cmd(ms: pathlib.Path | flint.options.MS, calibrate_model: pathlib.Path, solution_path: pathlib.Path | None = None, container: pathlib.Path | None = None, update_calibrate_options: dict[str, Any] | None = None, calibrate_data_column: str | None = None) -> CalibrateCommand Generate a typical ao calibrate command. Any extra keyword arguments are passed through as additional options to the `calibrate` program. :param ms: The measurement set to calibrate. There needs to be a nominated data_column. :type ms: Union[Path,MS] :param calibrate_model: Path to a generated calibrate sky-model :type calibrate_model: Path :param solution_path: The output path of the calibrate solutions file. If None, a default suffix of "calibrate.bin" is used. Defaults to None. :type solution_path: Path, optional :param container: If a path to a container is supplied the calibrate command is executed immediately. Defaults to None. :type container: Optional[Path], optional :param update_calibrate_options: Additional options to update the generated CalibrateOptions with. Keys should be attributes of CalibrationOptions. Defaults to None. :type update_calibrate_options: Optional[Dict[str, Any]], optional :param calibrate_data_column: The name of the column to calibrate, overwriting the nominated column set in the MS. If None, the MS.column attribute is used. Defaults to None. :type calibrate_data_column: Optional[str], optional :raises FileNotFoundError: Raised when calibrate_model can not be found. :returns: The calibrate command to execute and output solution file :rtype: CalibrateCommand .. py:function:: fill_between_flags(ax: matplotlib.pyplot.Axes, flags: numpy.ndarray, values: numpy.ndarray | None = None, direction: str = 'x') -> None Plot vertical or horizontal lines where data are flagged. NOTE: This is pretty inefficient and not intended for regular use. :param ax: Axes object to plot lines on :type ax: plt.Axes :param flags: Flags to consider. If `True`, plot. :type flags: np.ndarray :param values: The values to plot at. Useful if the position does not map to location. Defaults to None. :type values: Optional[np.ndarray], optional :param direction: If `x` use axvline, if `y` use axhline. Defaults to "x". :type direction: str, optional .. py:function:: find_existing_solutions(bandpass_directory: pathlib.Path, use_preflagged: bool = True, use_smoothed: bool = False) -> list[CalibrateCommand] Given a directory that contains a collection of bandpass measurement sets, attempt to identify a corresponding set of calibrate binary solution file. This search only supports the use of the default or known preflagger suffix. Limited support is provided to specify the expected calibrate suffix. These bandpass measurement sets should be processed already - which means just the B1936-638 field has been split out of the larger raw MS, flagged and calibrated. These steps are expected in order to get to the calibrate stage. :param bandpass_directory: Directory to search for split bandpass measurement sets :type bandpass_directory: Path :param use_preflagged: Add the pre-flag suffix when searching for solution files. This uses the expected suffix for preflagged solutions. Defaults to True. :type use_preflagged: bool, optional :param use_smoothed: Add the smoothed bandpass suffix when searching for solution files. This uses the expected suffix for smoothed solutions. Defaults to False. :type use_smoothed: bool, optional :returns: Collection of the calibrate command structures that are intended to be used to map the bandpass measurement sets to solution files. :rtype: List[CalibrateCommand] .. py:function:: flag_aosolutions(solutions_path: pathlib.Path, ref_ant: int = -1, flag_cut: float = 3, plot_dir: pathlib.Path | None = None, out_solutions_path: pathlib.Path | None = None, smooth_solutions: bool = False, plot_solutions_throughout: bool = True, smooth_window_size: int = 16, smooth_polynomial_order: int = 4, mean_ant_tolerance: float = 0.2, mesh_ant_flags: bool = False, max_gain_amplitude: float | None = None) -> FlaggedAOSolution Will open a previously solved ao-calibrate solutions file and flag additional channels and antennae. There are a number of distinct operations applied to the data, which are presented in order they are applied. If `mesh_ant_flags` is `True`, channels flagged from on channel on a single antenna will be applied to all (unless an antenna is completely flagged). This happens before any other operation,. If `max_gain_amplitude` is not `None` than any Jones with an element whose amplitude is above the set value will be flagged. Next, an attempt is made to search for channels where the the phase of the gain solution are outliers. The phase over frequency is first unwrapped (delay solved for) before the flagging statistics are computed. If an antenna is over 80% flagged then it is completely removed. A low order polynomial (typically order 5) is fit to the amplitudes of the Gx and Gy, and if the residuals are sufficiently high then the antenna will be flagged. If the mean ratio of the Gx and Gy amplitudes for an antenna are higher then `mean_ant_tolerance` then the antenna will be flagged. Keywords that with the `smooth` prefix are passed to the `smooth_bandpass_complex_gains` function. :param solutions_path: Location of the solutions file to examine and flag. :type solutions_path: Path :param ref_ant: Reference antenna to use, which is important when searching for phase-outliers and to smooth the bandpass. If ref_ant < 0, then an optimal one is selected. Defaults to -1. :type ref_ant: int, optional :param flag_cut: Significance of a phase-outlier from the mean (or median) before it should be flagged. Defaults to 3. :type flag_cut: float, optional :param plot_dir: Where diagnostic flagging plots should be written. If None, no plots will be produced. Defaults to None. :type plot_dir: Optional[Path], optional :param out_solutions_path: The output path of the flagged solutions file. If None, the solutions_path provided is used. Defaults to None. :type out_solutions_path: Optional[Path], optional :param smooth_solutions: Smooth the complex gain solutions after flaggined. Defaults to False. :type smooth_solutions: blool, optional :param plot_solutions_throughout: If True, the solutions will be plotted at different stages of processing. Defaults to True. :type plot_solutions_throughout: bool, Optional :param smooth_window_size: The size of the window function of the savgol filter. Passed directly to savgol. Defaults to 16. :type smooth_window_size: int, optional :param smooth_polynomial_order: The order of the polynomial of the savgol filter. Passed directly to savgol. Defaults to 4. :type smooth_polynomial_order: int, optional :param mean_ant_tolerance: Tolerance of the mean x/y antenna gain ratio test before the antenna is flagged. Defaults to 0.2. :type mean_ant_tolerance: float, optional :param mesh_ant_flags: If True, a channel is flagged across all antenna if it is flagged for any antenna. Performed before other flagging operations. Defaults to False. :type mesh_ant_flags: bool, optional :param max_gain_amplitude: If not None, flag the Jones if an antenna has a amplitude gain above this value. Defaults to 10. :type max_gain_amplitude: Optional[float], optional :returns: Path to the updated solutions file, intermediate solution files and plots along the way :rtype: FlaggedAOSolution .. py:function:: get_parser() -> argparse.ArgumentParser .. py:function:: load_aosolutions_file(solutions_path: pathlib.Path) -> AOSolutions Load in an AO-style solutions file :param solutions_path: The path of the solutions file to load :type solutions_path: Path :returns: Structure container the deserialized solutions file :rtype: AOSolutions .. py:function:: plot_solutions(solutions: pathlib.Path | AOSolutions, ref_ant: int | None = 0) -> Collection[pathlib.Path] Plot solutions for AO-style solutions :param solutions: Path to the solutions file :type solutions: Path :param ref_ant: Reference antenna to use. If None is specified there is no division by a reference antenna. Defaults to 0. :type ref_ant: Optional[int], optional :returns: Collection[Path] -- The paths of the two plots createda .. py:function:: run_apply_solutions(apply_solutions_cmd: ApplySolutions, container: pathlib.Path) -> None Will execute the applysolutions command inside the specified singularity container. :param apply_solutions_cmd: The constructed applysolutions command :type apply_solutions_cmd: ApplySolutions :param container: Location of the existing solutions file :type container: Path .. py:function:: run_calibrate(calibrate_cmd: CalibrateCommand, container: pathlib.Path) -> None Execute a calibrate command within a singularity container :param calibrate_cmd: The constructed calibrate command :type calibrate_cmd: CalibrateCommand :param container: Location of the container :type container: Path .. py:function:: save_aosolutions_file(aosolutions: AOSolutions, output_path: pathlib.Path) -> pathlib.Path Save a AOSolutions file to the ao-standard binary format. :param aosolutions: Instance of the solutions to save :type aosolutions: ApplySolutions :param output_path: Output path to write the files to :type output_path: Path :returns: Path the file was written to :rtype: Path .. py:function:: select_aosolution_for_ms(calibrate_cmds: list[CalibrateCommand], ms: flint.options.MS | pathlib.Path) -> pathlib.Path Attempt to select an AO-style solution file for a measurement set. This can be expanded to include a number of criteria, but at present it only searches for a matching beam number between the input set of CalibrationCommands and the input MS. :param calibrate_cmds: Set of calibration commands, which includes the solution file path and the corresponding MS, as attributes. :type calibrate_cmds: List[CalibrateCommand] :param ms: The measurement sett that needs a solutions file. :type ms: Union[MS, Path] :raises ValueError: Raised when not matching AO-solution file found. :returns: Path to solution file to apply. :rtype: Path .. py:function:: select_refant(bandpass: numpy.ndarray) -> int Attempt to select an optimal reference antenna. This works in a fairly simple way, and simply selects the antenna which is select based purely on the number of valid/unflagged solutions in the bandpass aosolutions file. :param bandpass: The aosolutions file that has been :type bandpass: np.ndarray :param solved for: :returns: The index of the reference antenna that should be used. :rtype: int