flint.bptools.preflagger¶

Tooling to attempt to flag components related to bandpasses. The idea being that poorly calibration channels in the antenna-based gains should be removed.

At this point there are no attempts to smooth or interpolate these flagged components of the bandpass.

Classes¶

PhaseOutlierResults

Results from the attempt to identify outlier complex gains in

Functions¶

`complex_gain_model`(→ numpy.ndarray)	Simulate a simple set of complex gains. No consideration made to amplitudes,
`construct_jones_over_max_amp_flags`(→ numpy.ndarray)	Construct and return a mask that would flag an entire Jones
`construct_mesh_ant_flags`(→ numpy.ndarray)	Construct a mask that will accumulate the flags across
`fit_complex_gain_model`(*args)	A fudge to help the curve_fit along.
`flag_mean_residual_amplitude`(→ bool)	Calculate the median or mean of the residual amplitudes of the complex gains
`flag_mean_xxyy_amplitude_ratio`(→ bool)	Will robust compute through an iterative sigma-clipping procedure the
`flag_outlier_phase`(→ PhaseOutlierResults)	This procedure attempts to identify channels in the bandpass solutions to
`flags_over_threshold`(→ bool)	Given a set of flags for an antenna across frequency, consider how much is flagged, indicated
`plot_mean_amplitudes`(→ pathlib.Path)	A simply plot to examine the polynomial fit to the residual amplitude data.
`plot_phase_outlier`(→ pathlib.Path)	Create a simple diagnostic plot highlighting how the outlier

Module Contents¶

class flint.bptools.preflagger.PhaseOutlierResults[source]¶

Bases: NamedTuple

Results from the attempt to identify outlier complex gains in the bandpass solutions. This procedure is concerned with identifying channel-wise outliers by first unwrapping any uncorrected delay term in the complex_gains, fitting to the unwrapped phase components, and then fitting.

complex_gains: numpy.ndarray[source]¶: The input gains to plot

fit_model_gains: numpy.ndarray[source]¶: The complex gain model fit made against the unwrapped gains (i.e. complex_gains / init_model_gains)

fit_model_params: tuple[float, float][source]¶: The fitted model parameters constrained against the unwrapped gains

flag_cut: float[source]¶: The adopted significance level that a outlier should be before outlier_mask is set to True

init_model_gains: numpy.ndarray[source]¶: The initial model of the complex_gains

init_model_params: tuple[float, float][source]¶: The initial guess (gradient, offset) model parameters to represent the phase component of the complex_gains

outlier_mask: numpy.ndarray[source]¶: Boolean mask of equal length to complex_gain, where True represents outliers that should be flagged

unwrapped_residual_mean: float[source]¶: The mean of the residual unwrapped phases in radians

unwrapped_residual_std: float[source]¶: The std. of the residual unwrapped phases in radians

flint.bptools.preflagger.complex_gain_model(xs: numpy.ndarray, gradient: float, phase_offset: float) → numpy.ndarray[source]¶

Simulate a simple set of complex gains. No consideration made to amplitudes, only considering phase

Parameters:

xs (np.ndarray) – Positions to evaluate model at.
gradient (float) – Gradient of the phase-slope, rate of wrapping
phase_offset (float) – Initial phasor offset of the model

Returns:

Equal length to input xs of complex numbers representing phase-ramp

Return type:

np.ndarray

flint.bptools.preflagger.construct_jones_over_max_amp_flags(complex_gains: numpy.ndarray, max_amplitude: float) → numpy.ndarray[source]¶

Construct and return a mask that would flag an entire Jones should there be an element whose amplitude is above a flagging threshold

Parameters:

complex_gains (np.ndarray) – Complex gains that will have a mask constructed
max_amplitude (float) – The flagging threshold, any Jones with a member above this will be flagged

Returns:

Boolean array of equal shape to complex_gains, with True indicating a flag

Return type:

np.ndarray

flint.bptools.preflagger.construct_mesh_ant_flags(mask: numpy.ndarray) → numpy.ndarray[source]¶

Construct a mask that will accumulate the flags across all antennas. The input mask array should be boolean and of shape (ant, channels, pol), where True means flagged.

If an antenna is completely flagged it is ignored as the statistics are collected

Parameters:: mask (np.ndarray) – Input array denoting which items are flagged.
Returns:: Output array where antennas have common sets of flags
Return type:: np.ndarray

flint.bptools.preflagger.fit_complex_gain_model(*args)[source]¶: A fudge to help the curve_fit along.

flint.bptools.preflagger.flag_mean_residual_amplitude(complex_gains: numpy.ndarray, use_robust: bool = True, polynomial_order: int = 5, plot_path: pathlib.Path | None = None, plot_title: str | None = None) → bool[source]¶

Calculate the median or mean of the residual amplitudes of the complex gains after fitting a polynomial of order polynomial_order.

If this median/mean is above 0.1 a value of True is returned, indicating that the antenna should be flagged.

Internally this is fitting a simple polynomial to the amplitude spectrum. This method is not as robust as the moving polynomial harmonic fit as implemented in the bptools package. The method current;y implemented might not be robust to sudden changes in the solutions that might be present on the boundaries of beam forming intervals. In some instances this might be avoided or minimised if the input complex gains have been divided by a reference antenna.

Parameters:

complex_gains (np.ndarray) – The set of complex gains to be considered
use_robust (bool, optional) – Whether to use robust statistics (median, MAD) or mean/std to calculate the statistic against. Defaults to True.
polynomial_order (int, optional) – The order of the polynomial (numpy.polyfit) to use to compute the baseline. Defaults to 5.
plot_path (Path) – The location to save the output file to
plot_title (Optional[str], optional) – Title to add to the figure. Defaults to None.

Returns:

Whether the data should be considered bad. True if it is bad, False if otherwise.

Return type:

bool

flint.bptools.preflagger.flag_mean_xxyy_amplitude_ratio(xx_complex_gains: numpy.ndarray, yy_complex_gains, tolerance: float = 0.1) → bool[source]¶

Will robust compute through an iterative sigma-clipping procedure the mean XX and YY gain amplitudes. The ratio of these means are computed, and if the are:

>>> ratio < (1. / fraction) or ratio > fraction

the data are considered bad and a True is returned (indicating that they should be flagged).

Parameters:

xx_complex_gains (np.ndarray) – The XX complex gains to be considered
yy_complex_gains (_type_) – The YY complex gains to be considered
tolerance (float, optional) – The tolerance used used to distinguish a critical mean ratio threshold. Defaults to 0.10.

Returns:

Whether data should be flagged (True) or not (False)

Return type:

bool

flint.bptools.preflagger.flag_outlier_phase(complex_gains: numpy.ndarray, flag_cut: float, use_mad: bool = False, plot_title: str | None = None, plot_path: pathlib.Path | None = None) → PhaseOutlierResults[source]¶

This procedure attempts to identify channels in the bandpass solutions to flag but searching for gains with outlier phases. Typically, ASKAP solutions have a phase-slope across the band (i.e. a delay). This phase-slope first needs to be ‘unwrapped’ in order to correctly identify outliers reliably.

Internally this function constructs an initial model of the unknown phase-slope. It estimates this by looking at the bulk set of gradients on a channel-to-channel basis, and looking at the builk distribution after removing large jumps (possibly RFI, possible a wrap). The initial phase offset is taken as the phase of the first valid gain.

The initial model is used to unwrap the data, allowing a lest-squares fitter to more reliably fit. Once the fitter has been executed, the final cuts are applied against the unwrapped phase residuals.

Experience shows that best results are achieved when the input complex-gains have been normalised against a reference antenna. There may be complex structures when the raw antenna phase vs frequency plots that can not be reliably fit for in this manner. These structures (I believe) arise from the beam-wise spectral sub-windows. See BPTools for a more thorough explanation.

Parameters:

complex_gains (np.ndarray) – The comples-gains as a function of frequency.
flag_cut (float) – The significance a point should be before flagged as outlier
use_mad (bool, optional) – Use the median and MAD when selecting outlier. if False, use mean and std. Defaults to False.
plot_title (str, optional) – Title to add to the plot. Defaults to None.
plot_path (Path, optional) – If not None, a simple diagnostic plot will be created and written to this path. Defaults to None.

Returns:

Collection of results from this phase outlier flagging routine

Return type:

PhaseOUtlierResults

flint.bptools.preflagger.flags_over_threshold(flags: numpy.ndarray, thresh: float = 0.8, ant_idx: int | None = None) → bool[source]¶

Given a set of flags for an antenna across frequency, consider how much is flagged, indicated by a value of True, and return whether it was over a threshold. The intent is to return whether an entire antenna should be flagged.

Parameters:

flags (np.ndarray) – Array of flags to consider
thresh (float, optional) – Threshold as a fraction that has to be meet before considered bad. Defaults to 0.8.
ant_idx (Optional[int], optional) – Number of the antenna being considered. Defaults to None.

Returns:

Whether the number of flags has reached a threshold

Return type:

bool

flint.bptools.preflagger.plot_mean_amplitudes(amplitudes: numpy.ndarray, model: numpy.ndarray, mean: float, std: float, output_path: pathlib.Path, plot_title: str | None = None) → pathlib.Path[source]¶

A simply plot to examine the polynomial fit to the residual amplitude data.

Parameters:

amplitudes (np.ndarray) – The amplitudes being fit to
model (np.ndarray) – The evaluated polynomial that was fit to the data
mean (float) – The mean that was evaluated, which is used to plot onto the residual
std (float) – The deviation that was evaluated, which is used to highlight the dispersion on the residual
output_path (Path) – The location to save the output file to
plot_title (Optional[str], optional) – Title to add to the figure. Defaults to None.

Returns:

Location the figure was saved to

Return type:

Path

flint.bptools.preflagger.plot_phase_outlier(phase_outlier_results: PhaseOutlierResults, output_path: pathlib.Path, title: str | None = None) → pathlib.Path[source]¶

Create a simple diagnostic plot highlighting how the outlier channels and their phases were selected.

Parameters:

phase_outlier_results (PhaseOutlierResults) – Results from the outlier phase flagging method
output_path (Path) – Location to write the output plot to
title (str, optional) – Title to add to the figure. Defaults to None.

Returns:

Path of the output image file

Return type:

Path