flint.prefect.clusters¶

Some utility functions around the creation of Prefect task funners.

For this work we will be using Dask backed workers to perform the compute operations.

Functions¶

`get_cluster_spec`(→ dict[Any, Any])	Given a cluster name, obtain the appropriate SLURM configuration
`get_dask_runner`(→ prefect_dask.DaskTaskRunner)	Creates and returns a DaskTaskRunner configured to established a SLURMCluster instance

Module Contents¶

flint.prefect.clusters.get_cluster_spec(cluster: str | pathlib.Path) → dict[Any, Any][source]¶

Given a cluster name, obtain the appropriate SLURM configuration file appropriate for use with SLURMCluster.

This cluster spec is expected to be consistent with the cluster_class and cluster_kwargs parameters that are used by dask_jobqueue based specifications.

Parameters:: cluster (Union[str,Path]) – Name of cluster or path to a configuration to look up for processing
Raises:: ValueError – Raised when cluster is not in KNOWN_CLUSTERS and has not corresponding YAML file.
Returns:: Dictionary of know options/parameters for dask_jobqueue.SLURMCluster
Return type:: dict[Any, Any]

flint.prefect.clusters.get_dask_runner(cluster: str | pathlib.Path = 'galaxy_small', extra_cluster_kwargs: dict[str, Any] | None = None) → prefect_dask.DaskTaskRunner[source]¶

Creates and returns a DaskTaskRunner configured to established a SLURMCluster instance to manage a set of dask-workers. The SLURMCluster is currently configured only for Galaxy.

Keyword Arguments:: cluster (Union[str,Path]) – The cluster name that will be used to search for a cluster specification file. This could be the name of a known cluster, or the name of a yaml file installed among the cluster_configs directory of the aces module.
Returns:: A dask task runner capable of being used as a task_runner for a prefect flow
Return type:: DaskTaskRunner