flint.calibrate.aocalibrate¶

Code to use AO calibrate s

Classes¶

`AOSolutions`	Structure to load an AO-style solutions file
`ApplySolutions`	The applysolutions command to execute
`CalibrateCommand`	The AO Calibrate command and output path of the corresponding solutions file
`CalibrateOptions`	Structure used to represent options into the calibrate program
`FlaggedAOSolution`	Hold the final set of flagged solutions and generated plots

Functions¶

`apply_solutions_to_ms`(→ ApplySolutions)
`calibrate_apply_ms`(→ ApplySolutions)	Will create and run a calibration command using AO calibrator, and then apply these solutions.
`calibrate_options_to_command`(→ str)	Generate a calibrate command given an input option set
`cli`(→ None)
`create_apply_solutions_cmd`(→ ApplySolutions)	Construct the command to apply calibration solutions to a MS
`create_calibrate_cmd`(→ CalibrateCommand)	Generate a typical ao calibrate command. Any extra keyword arguments
`fill_between_flags`(→ None)	Plot vertical or horizontal lines where data are flagged.
`find_existing_solutions`(→ list[CalibrateCommand])	Given a directory that contains a collection of bandpass measurement
`flag_aosolutions`(→ FlaggedAOSolution)	Will open a previously solved ao-calibrate solutions file and flag additional channels and antennae.
`get_parser`(→ argparse.ArgumentParser)
`load_aosolutions_file`(→ AOSolutions)	Load in an AO-style solutions file
`plot_solutions`(→ Collection[pathlib.Path])	Plot solutions for AO-style solutions
`run_apply_solutions`(→ None)	Will execute the applysolutions command inside the specified singularity
`run_calibrate`(→ None)	Execute a calibrate command within a singularity container
`save_aosolutions_file`(→ pathlib.Path)	Save a AOSolutions file to the ao-standard binary format.
`select_aosolution_for_ms`(→ pathlib.Path)	Attempt to select an AO-style solution file for a measurement
`select_refant`(→ int)	Attempt to select an optimal reference antenna. This works in

Module Contents¶

class flint.calibrate.aocalibrate.AOSolutions[source]¶

Bases: NamedTuple

Structure to load an AO-style solutions file

classmethod load(path: pathlib.Path) → AOSolutions[source]¶: Load in an AO-stule solution file. See load_solutions_file, which is internally used.

plot_solutions(ref_ant: int | None = 0) → Iterable[pathlib.Path][source]¶

Plot the solutions of all antenna for the first time-interval in the aosolutions file. The XX and the YY will be plotted.

Parameters:: ref_ant (Optional[int], optional) – Reference antenna to use. If None is specified there is no division by a reference antenna. Defaults to 0.
Returns:: Path to the phase and amplited plots created.
Return type:: Iterable[Path]

save(output_path: pathlib.Path) → pathlib.Path[source]¶

Save the instance of AOSolution to a standard aosolution binary file

Parameters:: output_path (Path) – Location to write the file to
Returns:: Location the file was written to
Return type:: Path

bandpass: numpy.ndarray[source]¶: Complex data representing the antenna Jones. Shape is (nsol, nant, nchan, npol)

nant: int[source]¶: Number of antenna in the solution file

nchan: int[source]¶: Number of channels in the solution file

npol: int[source]¶: Number of polarisations in the file

nsol: int[source]¶: Number of time solutions

path: pathlib.Path[source]¶: Path of the solutions file loaded

class flint.calibrate.aocalibrate.ApplySolutions(/, **data: Any)[source]¶

Bases: flint.options.BaseOptions

The applysolutions command to execute

cmd: str[source]¶: The command that will be executed

ms: flint.options.MS[source]¶: The measurement set that will have the solutions applied to

solution_path: pathlib.Path[source]¶: Location of the solutions file to apply

class flint.calibrate.aocalibrate.CalibrateCommand(/, **data: Any)[source]¶

Bases: flint.options.BaseOptions

The AO Calibrate command and output path of the corresponding solutions file

cmd: str[source]¶: The calibrate command that will be executed

model: pathlib.Path[source]¶: Path to the model that would be used to calibrate against

ms: flint.options.MS[source]¶: The measurement set to have solutions derived for

preflagged: bool = False[source]¶: Indicates whether the solution file has gone through preflagging routines.

solution_path: pathlib.Path[source]¶: The output path of the solutions file

class flint.calibrate.aocalibrate.CalibrateOptions(/, **data: Any)[source]¶

Bases: flint.options.BaseOptions

Structure used to represent options into the calibrate program

These attributes have the same names as options into the calibrate command.

datacolumn: str[source]¶: The name of the datacolumn that will be calibrates

i: int | None = 100[source]¶: The number of iterations that may be performed

m: pathlib.Path[source]¶: The path to the model file used to calibtate

maxuv: float | None = None[source]¶: The maximum distance in meters that is

minuv: float | None = None[source]¶: The minimum distance in meters that is

p: tuple[pathlib.Path, pathlib.Path] | None = None[source]¶: Plot output names for the amplitude gain and phases

class flint.calibrate.aocalibrate.FlaggedAOSolution[source]¶

Bases: NamedTuple

Hold the final set of flagged solutions and generated plots

bandpass: numpy.ndarray[source]¶: The bandpass solutions after flagging, as saved in the solutions file

path: pathlib.Path[source]¶: Path to the final set of flagged solutions

plots: Collection[pathlib.Path][source]¶: Contains paths to the plots generated throughout the flagging and smoothing procedure

flint.calibrate.aocalibrate.apply_solutions_to_ms(ms: pathlib.Path | flint.options.MS, solutions_path: pathlib.Path, container: pathlib.Path, data_column: str = 'DATA') → ApplySolutions[source]¶

flint.calibrate.aocalibrate.calibrate_apply_ms(ms_path: pathlib.Path, model_path: pathlib.Path, container: pathlib.Path, data_column: str = 'DATA') → ApplySolutions[source]¶

Will create and run a calibration command using AO calibrator, and then apply these solutions.

Parameters:

ms_path (Path) – The measurement set that will be calibrated
model_path (Path) – The model file containing sources to calibrate against
container (Path) – Container that has the AO calibtate and applysolutions file.
data_column (str, optional) – The name of the column containing the data to calibrate. Defaults to “DATA”.

Returns:

The command, solution binary path and new measurement set structure

Return type:

Applysolutions

flint.calibrate.aocalibrate.calibrate_options_to_command(calibrate_options: CalibrateOptions, ms_path: pathlib.Path, solutions_path: pathlib.Path) → str[source]¶

Generate a calibrate command given an input option set

Parameters:

calibrate_options (CalibrateOptions) – The set of calibrate options to use
ms (Path) – Path to the measurement set that will be calibrated
solutions_path (Path) – Output path of the solutions file

Returns:

The command string to execute

Return type:

str

flint.calibrate.aocalibrate.cli() → None[source]¶

flint.calibrate.aocalibrate.create_apply_solutions_cmd(ms: flint.options.MS, solutions_file: pathlib.Path, output_column: str | None = None, container: pathlib.Path | None = None) → ApplySolutions[source]¶

Construct the command to apply calibration solutions to a MS using an AO calibrate style solutions file.

The applysolutions program does not appear to have the ability to set a desured output column name. If the output_column specified matches the nominated column in ms, then applysolutions will simply overwrite the column with updated data. Otherwise, a CORRECTED_DATA column is produced.

NOTE: Care to be taken when the nominated column is CORRECTED_DATA.

Parameters:

ms (MS) – Measurement set to have solutions applied to
solutions_file (Path) – Path to the solutions file to apply
output_column (Optional[str], optional) – The desired output column name. See notes above. Defaults to None.
container (Optional[Path], optional) – If a path to a container is supplied the calibrate command is executed immediately. Defaults to None.

Returns:

Description of applysolutions command, solutions file path and updated MS

Return type:

ApplySolutions

flint.calibrate.aocalibrate.create_calibrate_cmd(ms: pathlib.Path | flint.options.MS, calibrate_model: pathlib.Path, solution_path: pathlib.Path | None = None, container: pathlib.Path | None = None, update_calibrate_options: dict[str, Any] | None = None, calibrate_data_column: str | None = None) → CalibrateCommand[source]¶

Generate a typical ao calibrate command. Any extra keyword arguments are passed through as additional options to the calibrate program.

Parameters:

ms (Union[Path,MS]) – The measurement set to calibrate. There needs to be a nominated data_column.
calibrate_model (Path) – Path to a generated calibrate sky-model
solution_path (Path, optional) – The output path of the calibrate solutions file. If None, a default suffix of “calibrate.bin” is used. Defaults to None.
container (Optional[Path], optional) – If a path to a container is supplied the calibrate command is executed immediately. Defaults to None.
update_calibrate_options (Optional[Dict[str, Any]], optional) – Additional options to update the generated CalibrateOptions with. Keys should be attributes of CalibrationOptions. Defaults to None.
calibrate_data_column (Optional[str], optional) – The name of the column to calibrate, overwriting the nominated column set in the MS. If None, the MS.column attribute is used. Defaults to None.

Raises:

FileNotFoundError – Raised when calibrate_model can not be found.

Returns:

The calibrate command to execute and output solution file

Return type:

CalibrateCommand

flint.calibrate.aocalibrate.fill_between_flags(ax: matplotlib.pyplot.Axes, flags: numpy.ndarray, values: numpy.ndarray | None = None, direction: str = 'x') → None[source]¶

Plot vertical or horizontal lines where data are flagged.

NOTE: This is pretty inefficient and not intended for regular use.

Parameters:

ax (plt.Axes) – Axes object to plot lines on
flags (np.ndarray) – Flags to consider. If True, plot.
values (Optional[np.ndarray], optional) – The values to plot at. Useful if the position does not map to location. Defaults to None.
direction (str, optional) – If x use axvline, if y use axhline. Defaults to “x”.

flint.calibrate.aocalibrate.find_existing_solutions(bandpass_directory: pathlib.Path, use_preflagged: bool = True, use_smoothed: bool = False) → list[CalibrateCommand][source]¶

Given a directory that contains a collection of bandpass measurement sets, attempt to identify a corresponding set of calibrate binary solution file.

This search only supports the use of the default or known preflagger suffix. Limited support is provided to specify the expected calibrate suffix.

These bandpass measurement sets should be processed already - which means just the B1936-638 field has been split out of the larger raw MS, flagged and calibrated. These steps are expected in order to get to the calibrate stage.

Parameters:

bandpass_directory (Path) – Directory to search for split bandpass measurement sets
use_preflagged (bool, optional) – Add the pre-flag suffix when searching for solution files. This uses the expected suffix for preflagged solutions. Defaults to True.
use_smoothed (bool, optional) – Add the smoothed bandpass suffix when searching for solution files. This uses the expected suffix for smoothed solutions. Defaults to False.

Returns:

Collection of the calibrate command structures that are intended to be used to map the bandpass measurement sets to solution files.

Return type:

List[CalibrateCommand]

flint.calibrate.aocalibrate.flag_aosolutions(solutions_path: pathlib.Path, ref_ant: int = -1, flag_cut: float = 3, plot_dir: pathlib.Path | None = None, out_solutions_path: pathlib.Path | None = None, smooth_solutions: bool = False, plot_solutions_throughout: bool = True, smooth_window_size: int = 16, smooth_polynomial_order: int = 4, mean_ant_tolerance: float = 0.2, mesh_ant_flags: bool = False, max_gain_amplitude: float | None = None) → FlaggedAOSolution[source]¶

Will open a previously solved ao-calibrate solutions file and flag additional channels and antennae.

There are a number of distinct operations applied to the data, which are presented in order they are applied.

If mesh_ant_flags is True, channels flagged from on channel on a single antenna will be applied to all (unless an antenna is completely flagged). This happens before any other operation,.

If max_gain_amplitude is not None than any Jones with an element whose amplitude is above the set value will be flagged.

Next, an attempt is made to search for channels where the the phase of the gain solution are outliers. The phase over frequency is first unwrapped (delay solved for) before the flagging statistics are computed.

If an antenna is over 80% flagged then it is completely removed.

A low order polynomial (typically order 5) is fit to the amplitudes of the Gx and Gy, and if the residuals are sufficiently high then the antenna will be flagged.

If the mean ratio of the Gx and Gy amplitudes for an antenna are higher then mean_ant_tolerance then the antenna will be flagged.

Keywords that with the smooth prefix are passed to the smooth_bandpass_complex_gains function.

Parameters:

solutions_path (Path) – Location of the solutions file to examine and flag.
ref_ant (int, optional) – Reference antenna to use, which is important when searching for phase-outliers and to smooth the bandpass. If ref_ant < 0, then an optimal one is selected. Defaults to -1.
flag_cut (float, optional) – Significance of a phase-outlier from the mean (or median) before it should be flagged. Defaults to 3.
plot_dir (Optional[Path], optional) – Where diagnostic flagging plots should be written. If None, no plots will be produced. Defaults to None.
out_solutions_path (Optional[Path], optional) – The output path of the flagged solutions file. If None, the solutions_path provided is used. Defaults to None.
smooth_solutions (blool, optional) – Smooth the complex gain solutions after flaggined. Defaults to False.
plot_solutions_throughout (bool, Optional) – If True, the solutions will be plotted at different stages of processing. Defaults to True.
smooth_window_size (int, optional) – The size of the window function of the savgol filter. Passed directly to savgol. Defaults to 16.
smooth_polynomial_order (int, optional) – The order of the polynomial of the savgol filter. Passed directly to savgol. Defaults to 4.
mean_ant_tolerance (float, optional) – Tolerance of the mean x/y antenna gain ratio test before the antenna is flagged. Defaults to 0.2.
mesh_ant_flags (bool, optional) – If True, a channel is flagged across all antenna if it is flagged for any antenna. Performed before other flagging operations. Defaults to False.
max_gain_amplitude (Optional[float], optional) – If not None, flag the Jones if an antenna has a amplitude gain above this value. Defaults to 10.

Returns:

Path to the updated solutions file, intermediate solution files and plots along the way

Return type:

FlaggedAOSolution

flint.calibrate.aocalibrate.get_parser() → argparse.ArgumentParser[source]¶

flint.calibrate.aocalibrate.load_aosolutions_file(solutions_path: pathlib.Path) → AOSolutions[source]¶

Load in an AO-style solutions file

Parameters:: solutions_path (Path) – The path of the solutions file to load
Returns:: Structure container the deserialized solutions file
Return type:: AOSolutions

flint.calibrate.aocalibrate.plot_solutions(solutions: pathlib.Path | AOSolutions, ref_ant: int | None = 0) → Collection[pathlib.Path][source]¶

Plot solutions for AO-style solutions

Parameters:

solutions (Path) – Path to the solutions file
ref_ant (Optional[int], optional) – Reference antenna to use. If None is specified there is no division by a reference antenna. Defaults to 0.

Returns:

Collection[Path] – The paths of the two plots createda

flint.calibrate.aocalibrate.run_apply_solutions(apply_solutions_cmd: ApplySolutions, container: pathlib.Path) → None[source]¶

Will execute the applysolutions command inside the specified singularity container.

Parameters:

apply_solutions_cmd (ApplySolutions) – The constructed applysolutions command
container (Path) – Location of the existing solutions file

flint.calibrate.aocalibrate.run_calibrate(calibrate_cmd: CalibrateCommand, container: pathlib.Path) → None[source]¶

Execute a calibrate command within a singularity container

Parameters:

calibrate_cmd (CalibrateCommand) – The constructed calibrate command
container (Path) – Location of the container

flint.calibrate.aocalibrate.save_aosolutions_file(aosolutions: AOSolutions, output_path: pathlib.Path) → pathlib.Path[source]¶

Save a AOSolutions file to the ao-standard binary format.

Parameters:

aosolutions (ApplySolutions) – Instance of the solutions to save
output_path (Path) – Output path to write the files to

Returns:

Path the file was written to

Return type:

Path

flint.calibrate.aocalibrate.select_aosolution_for_ms(calibrate_cmds: list[CalibrateCommand], ms: flint.options.MS | pathlib.Path) → pathlib.Path[source]¶

Attempt to select an AO-style solution file for a measurement set. This can be expanded to include a number of criteria, but at present it only searches for a matching beam number between the input set of CalibrationCommands and the input MS.

Parameters:

calibrate_cmds (List[CalibrateCommand]) – Set of calibration commands, which includes the solution file path and the corresponding MS, as attributes.
ms (Union[MS, Path]) – The measurement sett that needs a solutions file.

Raises:

ValueError – Raised when not matching AO-solution file found.

Returns:

Path to solution file to apply.

Return type:

Path

flint.calibrate.aocalibrate.select_refant(bandpass: numpy.ndarray) → int[source]¶

Attempt to select an optimal reference antenna. This works in a fairly simple way, and simply selects the antenna which is select based purely on the number of valid/unflagged solutions in the bandpass aosolutions file.

Parameters:

bandpass (np.ndarray) – The aosolutions file that has been
for (solved)

Returns:

The index of the reference antenna that should be used.

Return type:

int