flint.bptools.preflagger ======================== .. py:module:: flint.bptools.preflagger .. autoapi-nested-parse:: Tooling to attempt to flag components related to bandpasses. The idea being that poorly calibration channels in the antenna-based gains should be removed. At this point there are no attempts to smooth or interpolate these flagged components of the bandpass. Classes ------- .. autoapisummary:: flint.bptools.preflagger.PhaseOutlierResults Functions --------- .. autoapisummary:: flint.bptools.preflagger.complex_gain_model flint.bptools.preflagger.construct_jones_over_max_amp_flags flint.bptools.preflagger.construct_mesh_ant_flags flint.bptools.preflagger.fit_complex_gain_model flint.bptools.preflagger.flag_mean_residual_amplitude flint.bptools.preflagger.flag_mean_xxyy_amplitude_ratio flint.bptools.preflagger.flag_outlier_phase flint.bptools.preflagger.flags_over_threshold flint.bptools.preflagger.plot_mean_amplitudes flint.bptools.preflagger.plot_phase_outlier Module Contents --------------- .. py:class:: PhaseOutlierResults Bases: :py:obj:`NamedTuple` Results from the attempt to identify outlier complex gains in the bandpass solutions. This procedure is concerned with identifying channel-wise outliers by first unwrapping any uncorrected delay term in the complex_gains, fitting to the unwrapped phase components, and then fitting. .. py:attribute:: complex_gains :type: numpy.ndarray The input gains to plot .. py:attribute:: fit_model_gains :type: numpy.ndarray The complex gain model fit made against the unwrapped gains (i.e. complex_gains / init_model_gains) .. py:attribute:: fit_model_params :type: tuple[float, float] The fitted model parameters constrained against the unwrapped gains .. py:attribute:: flag_cut :type: float The adopted significance level that a outlier should be before outlier_mask is set to True .. py:attribute:: init_model_gains :type: numpy.ndarray The initial model of the complex_gains .. py:attribute:: init_model_params :type: tuple[float, float] The initial guess (gradient, offset) model parameters to represent the phase component of the complex_gains .. py:attribute:: outlier_mask :type: numpy.ndarray Boolean mask of equal length to complex_gain, where True represents outliers that should be flagged .. py:attribute:: unwrapped_residual_mean :type: float The mean of the residual unwrapped phases in radians .. py:attribute:: unwrapped_residual_std :type: float The std. of the residual unwrapped phases in radians .. py:function:: complex_gain_model(xs: numpy.ndarray, gradient: float, phase_offset: float) -> numpy.ndarray Simulate a simple set of complex gains. No consideration made to amplitudes, only considering phase :param xs: Positions to evaluate model at. :type xs: np.ndarray :param gradient: Gradient of the phase-slope, rate of wrapping :type gradient: float :param phase_offset: Initial phasor offset of the model :type phase_offset: float :returns: Equal length to input xs of complex numbers representing phase-ramp :rtype: np.ndarray .. py:function:: construct_jones_over_max_amp_flags(complex_gains: numpy.ndarray, max_amplitude: float) -> numpy.ndarray Construct and return a mask that would flag an entire Jones should there be an element whose amplitude is above a flagging threshold :param complex_gains: Complex gains that will have a mask constructed :type complex_gains: np.ndarray :param max_amplitude: The flagging threshold, any Jones with a member above this will be flagged :type max_amplitude: float :returns: Boolean array of equal shape to `complex_gains`, with `True` indicating a flag :rtype: np.ndarray .. py:function:: construct_mesh_ant_flags(mask: numpy.ndarray) -> numpy.ndarray Construct a mask that will accumulate the flags across all antennas. The input mask array should be boolean and of shape (ant, channels, pol), where `True` means flagged. If an antenna is completely flagged it is ignored as the statistics are collected :param mask: Input array denoting which items are flagged. :type mask: np.ndarray :returns: Output array where antennas have common sets of flags :rtype: np.ndarray .. py:function:: fit_complex_gain_model(*args) A fudge to help the curve_fit along. .. py:function:: flag_mean_residual_amplitude(complex_gains: numpy.ndarray, use_robust: bool = True, polynomial_order: int = 5, plot_path: pathlib.Path | None = None, plot_title: str | None = None) -> bool Calculate the median or mean of the residual amplitudes of the complex gains after fitting a polynomial of order polynomial_order. If this median/mean is above 0.1 a value of True is returned, indicating that the antenna should be flagged. Internally this is fitting a simple polynomial to the amplitude spectrum. This method is not as robust as the moving polynomial harmonic fit as implemented in the bptools package. The method current;y implemented might not be robust to sudden changes in the solutions that might be present on the boundaries of beam forming intervals. In some instances this might be avoided or minimised if the input complex gains have been divided by a reference antenna. :param complex_gains: The set of complex gains to be considered :type complex_gains: np.ndarray :param use_robust: Whether to use robust statistics (median, MAD) or mean/std to calculate the statistic against. Defaults to True. :type use_robust: bool, optional :param polynomial_order: The order of the polynomial (numpy.polyfit) to use to compute the baseline. Defaults to 5. :type polynomial_order: int, optional :param plot_path: The location to save the output file to :type plot_path: Path :param plot_title: Title to add to the figure. Defaults to None. :type plot_title: Optional[str], optional :returns: Whether the data should be considered bad. True if it is bad, False if otherwise. :rtype: bool .. py:function:: flag_mean_xxyy_amplitude_ratio(xx_complex_gains: numpy.ndarray, yy_complex_gains, tolerance: float = 0.1) -> bool Will robust compute through an iterative sigma-clipping procedure the mean XX and YY gain amplitudes. The ratio of these means are computed, and if the are: >>> ratio < (1. / fraction) or ratio > fraction the data are considered bad and a `True` is returned (indicating that they should be flagged). :param xx_complex_gains: The XX complex gains to be considered :type xx_complex_gains: np.ndarray :param yy_complex_gains: The YY complex gains to be considered :type yy_complex_gains: _type_ :param tolerance: The tolerance used used to distinguish a critical mean ratio threshold. Defaults to 0.10. :type tolerance: float, optional :returns: Whether data should be flagged (True) or not (False) :rtype: bool .. py:function:: flag_outlier_phase(complex_gains: numpy.ndarray, flag_cut: float, use_mad: bool = False, plot_title: str | None = None, plot_path: pathlib.Path | None = None) -> PhaseOutlierResults This procedure attempts to identify channels in the bandpass solutions to flag but searching for gains with outlier phases. Typically, ASKAP solutions have a phase-slope across the band (i.e. a delay). This phase-slope first needs to be 'unwrapped' in order to correctly identify outliers reliably. Internally this function constructs an initial model of the unknown phase-slope. It estimates this by looking at the bulk set of gradients on a channel-to-channel basis, and looking at the builk distribution after removing large jumps (possibly RFI, possible a wrap). The initial phase offset is taken as the phase of the first valid gain. The initial model is used to unwrap the data, allowing a lest-squares fitter to more reliably fit. Once the fitter has been executed, the final cuts are applied against the unwrapped phase residuals. Experience shows that best results are achieved when the input complex-gains have been normalised against a reference antenna. There may be complex structures when the raw antenna phase vs frequency plots that can not be reliably fit for in this manner. These structures (I believe) arise from the beam-wise spectral sub-windows. See BPTools for a more thorough explanation. :param complex_gains: The comples-gains as a function of frequency. :type complex_gains: np.ndarray :param flag_cut: The significance a point should be before flagged as outlier :type flag_cut: float :param use_mad: Use the median and MAD when selecting outlier. if False, use mean and std. Defaults to False. :type use_mad: bool, optional :param plot_title: Title to add to the plot. Defaults to None. :type plot_title: str, optional :param plot_path: If not None, a simple diagnostic plot will be created and written to this path. Defaults to None. :type plot_path: Path, optional :returns: Collection of results from this phase outlier flagging routine :rtype: PhaseOUtlierResults .. py:function:: flags_over_threshold(flags: numpy.ndarray, thresh: float = 0.8, ant_idx: int | None = None) -> bool Given a set of flags for an antenna across frequency, consider how much is flagged, indicated by a value of True, and return whether it was over a threshold. The intent is to return whether an entire antenna should be flagged. :param flags: Array of flags to consider :type flags: np.ndarray :param thresh: Threshold as a fraction that has to be meet before considered bad. Defaults to 0.8. :type thresh: float, optional :param ant_idx: Number of the antenna being considered. Defaults to None. :type ant_idx: Optional[int], optional :returns: Whether the number of flags has reached a threshold :rtype: bool .. py:function:: plot_mean_amplitudes(amplitudes: numpy.ndarray, model: numpy.ndarray, mean: float, std: float, output_path: pathlib.Path, plot_title: str | None = None) -> pathlib.Path A simply plot to examine the polynomial fit to the residual amplitude data. :param amplitudes: The amplitudes being fit to :type amplitudes: np.ndarray :param model: The evaluated polynomial that was fit to the data :type model: np.ndarray :param mean: The mean that was evaluated, which is used to plot onto the residual :type mean: float :param std: The deviation that was evaluated, which is used to highlight the dispersion on the residual :type std: float :param output_path: The location to save the output file to :type output_path: Path :param plot_title: Title to add to the figure. Defaults to None. :type plot_title: Optional[str], optional :returns: Location the figure was saved to :rtype: Path .. py:function:: plot_phase_outlier(phase_outlier_results: PhaseOutlierResults, output_path: pathlib.Path, title: str | None = None) -> pathlib.Path Create a simple diagnostic plot highlighting how the outlier channels and their phases were selected. :param phase_outlier_results: Results from the outlier phase flagging method :type phase_outlier_results: PhaseOutlierResults :param output_path: Location to write the output plot to :type output_path: Path :param title: Title to add to the figure. Defaults to None. :type title: str, optional :returns: Path of the output image file :rtype: Path