Tune Multi-Timestamp Downscaler

Configuration

s3lst_ds.downscaling.tune.tune_config.TuneConfig dataclass

Configurations for wrangling the data of timestamps of interest, batching it into cross-validation and test datasets, performing an optimized cross-validated hyperparameter tuning of a multi-timestamp downscaling model and subsequently training and testing it. Moreover, configurations for returning or writing the results to files are also defined.

Batching is done with respect to the issued timestamps, and, therefore, the whole data associated with a timestamp (a scene) is fully contained within a single batch. The resulting batch datasets are: - "test" dataset: timestamps for which there is Landsat data. - "cross_val" dataset: random split of the remaining timestamps into n_cross_val_folds folds, stratified with respect to a categorical variable var_cross_val_strat (if issued).

The optimized cross-validated hyperparameter tuning is done using optuna by trying suggested hyperparameter values within the search space defined in the issued params_tune_getter. The estimator of the downscaler (the downscaler before de-transformation and residual correction) is cross-validated for each suggested hyperparameter combination and the best one is selected based on the issued scoring metric best_scorer. Cross-validation is done by training the downscaler estimator on all cross-validation folds except one and scoring it on the latter, rotating the scoring fold until all are considered. The overall cross-validation score is computed from the arithmetic mean of the scores of each iteration.

Training of the tuned downscaler is done using the whole cross-validation data.

Testing is done by scoring the trained downscaler on the test dataset (for both Sentinel-3 and Landsat data).

WARNING: Note that for the sake of efficiency, during tuning, solely tuner-specific multiprocessing (set through present tune_n_jobs parameter) is considered . Base model-specific multi-processing (set through the respective n_jobs parameter) is subsequently considered. Note that for the case of the MLPRegressor, tune_n_jobs is forcefully set to 1, regardless of the issued value, as the MLPRegressor, by default, always use all available processors - and, therefore, a value of tune_n_jobs greater than 1 would impair the process.

WARNING: Note that the units of the computed cross-validation score in the hyperparameter tuning are based on the ones of the transformed target (whose transform corresponds to the one set in the downscaler). For example, if the transform corresponds to "standardize" and best_scorer is set to "rmse", the computed score corresponds to the RMSE of the standardized target, which is unitless.

Attributes:

Name	Type	Description
data_batcher	DataBatcher or Path or None, default=None	Data batcher or a path to a Joblib file containing it. If not issued, a data batcher is created from scratch using the data_batcher_-prefixed parameters of batcher is created from scratch using the data_batcher_-prefixed parameters of the present TuneConfig instance. Note that transform parameter of the downscaler is in any case enforced (therefore, transforming/re-transforming the wrangled data) on the one of the data batcher's data wrangler regardless of the previous value. Also, data_batcher_data_wrangler_max_workers parameter of the present TuneConfig instance is also in any case enforced.
data_batcher_data_wrangler_path_sentinel3	Path or None, default=None	If data_batcher is not issued: path to directory containing Sentinel-3 product folders whose data is to be wrangled. Each of such folders must contain georeferenced Sentinel-3 SLSTR Level-2 LST product file (https://sentiwiki.copernicus.eu/web/slstr-products#S3-SLSTR-Products-L2-LST-Products) as well as a georeferenced Sentinel-3 Synergy Level-2 product file (https://sentiwiki.copernicus.eu/web/synergy-products#SYNERGYProducts-L2SYNSDRprocessingS3-Synergy-Products-L2-SYN-SDR-processing). Furthermore, the name of such folders must correspond to the respective start sensing time in the format "YYYYMMDDTHHMMSS".
data_batcher_data_wrangler_path_spatial_pred	Path or None, default=None	If data_batcher is not issued: path to a NetCDF file with the spatial predictor data whose data is to be wrangled. If not set, no spatial predictor data is considered.
data_batcher_data_wrangler_aoi	str or Path or None, default=None	If data_batcher is not issued: WKT string or path to AOI geometry file to mask out the data. The data wrangler will add the AOI to the wrangled data as variable "aoi". If not set, no such variable is defined and no masking is applied.
data_batcher_data_wrangler_path_landsat	Path or None, default=None	If data_batcher is not issued: Path to the directory containing Landsat 8/9 folders whose data is to be wrangled. Each of such folders must contain a LST.TIF file with georeferenced Landsat 8/9 Level-2 LST data (https://www.usgs.gov/centers/eros/science/usgs-eros-archive-landsat-archives-landsat-8-9-olitirs-collection-2-level-2), having a resolution of 30 m. In the wrangling, such data and Sentinel-3's will be "matched" if the respective folders have the same name (it is implied here that the user had analysed the acquisitions obtained by the two platforms and set the names of the Landsat 8/9 data folders as the ones of Sentinel-3's (start sensing times) whose start sensing times and spatial extents are approximately the same). Note that the Landsat data will be solely used for testing.
data_batcher_data_wrangler_vars	list[str] or None, default=None	If data_batcher is not issued: aliases of the variables to be wrangled besides the target (such as predictor, sample_weight and visualization variables). If vars is not issued, but downscaler is, it will be set to the aliases of the predictors (cols_X) considered by the latter. Otherwise, if downscaler is not issued but downscaler_X is, it will be set to downscaler_X, or, if not, to all aliases of the predictors (X) considered by a default DataVars instance (s3lst_ds.utilities.var_utils.DataVars).
data_batcher_data_wrangler_max_workers	int, default=1	Number of simultaneous multiple processes to be considered by the data wrangler in wrangling. Note that if negative, one has the following conditions: - -1: all processors are used; - -k: all processors except k-1 are used. This parameter is enforced regardless of the data wrangler being issued or created from scratch.
data_batcher_n_cross_val_folds	int, default=5	If data_batcher is not issued: number of cross-validation folds.
data_batcher_var_cross_val_strat	str or None, default=None	If data_batcher is not issued: metadata categorical variable with respect to which stratification in the cross-validation data splitting into folds is to be performed. If not defined, no stratification is considered.
data_batcher_rnd_seed	int or np.random.RandomState or None = None	If data_batcher is not issued: random seed number considered in the cross-validation data splitting into folds. If not defined, no such number is regarded.
downscaler	Downscaler or Path or None, default=None	Downscaler or a path to a Joblib file containing it. If not issued, a downscaler is created from scratch using the downscaler_-prefixed parameters of the present TuneConfig. Note that downscaler_masks and downscaler_max_workers are in any case enforced, regardless of the downscaler being issued or created from scratch.
downscaler_base_model	Regressor, default=LinearRegression()	If downscaler is not issued: the regression model to be used as the base model of the downscaler to be created. If not issued, it is set to LinearRegression() by default.
downscaler_X	list[str] or None, default=["FVC", "NDWI"]	If downscaler is not issued: aliases of the predictors to be considered by the downscaler to be created. If not issued, it is set to ["FVC", "NDWI"].
downscaler_masks	list[str] or None, default=None	Aliases of the mask variables (e.g. ["aoi"]) to regard (wherever the variables have nan values, the respective data records are masked out). If not issued, it is set to [] and no masking is considered by the downscaler. This parameter is enforced regardless of the downscaler being issued or created from scratch.
downscaler_scale	{"standardize", "min_max_normalize", None}, default="standardize"	If downscaler is not issued: the scaling method to apply to numerical predictors.
downscaler_encode	{"one_hot", "dummy", None}, default="dummy"	If downscaler is not issued: the encoding method to apply to the categorical predictors: - "one_hot": to one-hot encode the categorical predictors; - "dummy": to dummy encode the categorical predictors (one-hot encoding with the first component dropped); - None: to regard the categorical predictors raw (no encoding). Note that dummy encoding is usually considered in place of one-hot to avoid multicollinearity problems (one may show that a component of a one-hot encoding vector is fully determined by all the other components making it redundant).
downscaler_transform	{None, "center", "standardize"}, default=None	If downscaler is not issued: the transform to apply on the coarse target and coarse and fine spatio-temporal predictors from a copy of the wrangled data in each SingleDataWrangler instance of the data batcher's data_wrangler by using coarse data statistics. The transformations are set in SingleDataWrangler'sdatawith the same names as the original columns with the substring"_trans"suffixed to them. Note that the transformations are timestamp-specific, that is, the computed statistics and the applied transformations in each timestamp solely concern the data of that timestamp. The possible values fordownscaler_transformare: -None: not transforming the data; -"center": subtracting the mean from the data; -"standardize"`: subtracting the mean from the data and dividing the result by the standard deviation. Note that such transforms are redundant for the case of the single-timestamp architecture. They only take effect for the multi-timestamp architecture.
downscaler_lasso_sel	bool, default=False	If downscaler is not issued: whether to use a Lasso regression for selecting the scaled-encoded downscaler_X predictors downstream of the preprocessor. Lasso selection is such that solely the input predictors associated with coefficients of the fitted Lasso regression model having absolute values larger than 1e-5 are selected. Note that the non-encoded downscaler_X predictors are regardlessly considered downstream of the preprocessor.
downscaler_lasso_alpha	float, default=1.0	If downscaler is not issued: the regularization strength of the Lasso regression model used for selecting the scaled-encoded downscaler_X predictors downstream of the preprocessor. Such regularization strength is the multiplying constant of the weight vector L1-norm (sum of the absolute values of the components) in the Lasso regression objective function. The larger the value, the stronger the regularization. Note that this parameter only takes effect if downscaler_lasso_sel is True.
downscaler_max_workers	int, default=1	Number of simultaneous multiple processes to be considered by the downscaler (in training, prediction and scoring). Note that if negative, one has the following conditions: - -1: all processors are used; - -k: all processors except k-1 are used. This parameter is enforced regardless of the downscaler being issued or created from scratch.
timestamps	list[pd.Timestamp] or list[str] or None, default=None	If data_batcher is not issued: timestamps of interest either as pd.Timestamp values or in any format parsable by pd.Timestamp (e.g. "YYYY-MM-DD HH:MM:SS"). These must correspond to the start sensing times of the respective acquisitions. Furthermore, they must be part of those regarded by data_batcher_data_wrangler_path_sentinel3. If neither timestamps nor data_batcher are issued, timestamps will be set to all of those regarded by such path. If timestamps is not issued but data_batcher is, timestamps will be set to all of those regarded by the batcher.
sample_weight_fit	str or None, default=None	Alias of the variable to be regarded as sample weight for cross-validation and training of the downscaler. Note that in cross-validation, sample_weight_fit is used for both training and scoring. If not issued, no sample weight in training and cross-validation is considered.
sample_weight_score	str or None, default=None	Alias of the variable to be regarded as sample weight for scoring the downscaler in training and testing (but not in cross-validation). If not issued, no sample weight in such scoring is considered.
scorers	list[str], default=["r2", "rmse", "mae", "mbe"]	Aliases of the scorers to consider in training and testing.
best_scorer	str, default="rmse"	Alias of the metric to consider in the selection of the best hyperparameter combination in the hyperparameter tuning of the downscaler estimator (that is, the downscaler before de-transformation and residual correction). WARNING: Note that since the estimator is the object tuned and not the downscaler itself, the units of the computed cross-validation metric in the tuning are based on the ones of the transformed target (whose transform corresponds to the one set in the downscaler). For example, if the transform corresponds to "standardize" and best_scorer is set to "rmse", the computed metric corresponds to the RMSE of the standardized target, which is unitless.
correct	bool, default=True	Whether to correct the predicted fine raw target for each image (from fine predictors and masks, X_and_mask_fine) using the finely-resampled residual for the prediction of the coarse raw target (from coarse predictors and masks, X_and_mask_coarse).
params_tune_getter	dict[str, dict[str, Any]]	Dictionary of search spaces of the downscaler estimator hyperparameters to be tuned. The keys must correspond to the hyperparameters' full access paths with each step separated by double underscores (e.g. "base_model__formula" in which "base_model" is a parameter of the downscaler estimator and "formula" is a parameter of the former). The values must be dictionaries with keys: - "suggest_method": alias of the optuna's suggest method to be used in the hyperparameter tuning of the respective hyperparameter: - "float": for continuous hyperparameters; - "int": for integer hyperparameters; - "categorical": for categorical hyperparameters. - "suggest_kwargs": dictionary of keyword arguments (except name - as this one is inferred from the key) to be passed to the suggest method (e.g., {"low": 0.0, "high": 1.0} for "float").
tune_rnd_seed	int or np.random.RandomState or None = None	Random seed number for the sampler of hyperparameter values during the optimised hyperparameter tuning. If not defined, no such number is regarded.
tune_n_trials	int, default=100	Number of trials to perform in the optimised hyperparameter tuning.
tune_n_jobs	int, default=1	Number of simultaneous multiple processes to be considered in the optimised hyperparameter tuning. Note that if negative, one has the following conditions: - -1: all processors are used; - -k: all processors except k-1 are used.
hparam_rnd_seed	int or np.random.RandomState or None = None	Random seed number for the sampler of hyperparameter values during tuning. If not defined, no such number is regarded.
path_out	Path or None, default=None	The directory path to save the tuned downscaler, the obtained scores, and the data batcher. If not issued, the results are instead returned.
out_data_batcher	bool, default=True	Whether to return or write (if path_out is issued) the data batcher.
log_mode	{None, "console", "file", "both"}, default="both"	The logging mode for wrangling, batching, tuning, training, testing and writing: - None: No logging is done; - "console": Logging is done to console only; - "file": Logging is done to a log file only; - "both": Logging is done to both console and a log file. Note the log file would be defined as tune.log at out_dir.

Methods:

Name	Description
__init__

Source code in src/s3lst_ds/downscaling/tune/tune_config.py

@dataclass
class TuneConfig:
    """
    Configurations for wrangling the data of timestamps of interest, batching it into
    cross-validation and test datasets, performing an optimized cross-validated
    hyperparameter tuning of a multi-timestamp downscaling model and subsequently
    training and testing it. Moreover, configurations for returning or writing the
    results to files are also defined.

    Batching is done with respect to the issued `timestamps`, and, therefore, the whole
    data associated with a timestamp (a scene) is fully contained within a single batch.
    The resulting batch datasets are:
        - `"test"` dataset: timestamps for which there is Landsat data.
        - `"cross_val"` dataset: random split of the remaining timestamps into
        `n_cross_val_folds` folds, stratified with respect to a categorical variable
        `var_cross_val_strat` (if issued).

    The optimized cross-validated hyperparameter tuning is done using
    [`optuna`](https://optuna.readthedocs.io/en/stable/) by trying suggested
    hyperparameter values within the search space defined in the issued
    `params_tune_getter`. The estimator of the downscaler (the downscaler before
    de-transformation and residual correction) is cross-validated for each suggested
    hyperparameter combination and the best one is selected based on the issued scoring
    metric `best_scorer`. Cross-validation is done by training the downscaler estimator
    on all cross-validation folds except one and scoring it on the latter, rotating the
    scoring fold until all are considered. The overall cross-validation score is
    computed from the arithmetic mean of the scores of each iteration.


    Training of the tuned downscaler is done using the whole cross-validation data.

    Testing is done by scoring the trained downscaler on the test dataset (for both
    Sentinel-3 and Landsat data).

    WARNING: Note that for the sake of efficiency, during tuning, solely tuner-specific
    multiprocessing (set through present `tune_n_jobs` parameter) is considered . Base
    model-specific multi-processing (set through the respective `n_jobs` parameter) is
    subsequently considered. Note that for the case of the MLPRegressor, `tune_n_jobs`
    is forcefully set to `1`, regardless of the issued value, as the MLPRegressor, by
    default, always use all available processors - and, therefore, a value of
    `tune_n_jobs` greater than `1` would impair the process.

    WARNING: Note that the units of the computed cross-validation score in the
    hyperparameter tuning are based on the ones of the transformed target (whose
    transform corresponds to the one set in the downscaler). For example, if the
    transform corresponds to `"standardize"` and `best_scorer` is set to `"rmse"`, the
    computed score corresponds to the RMSE of the standardized target, which is
    unitless.

    Attributes
    ----------

    data_batcher : DataBatcher or Path or None, default=None
        Data batcher or a path to a Joblib file containing it. If not issued, a data
        batcher is created from scratch using the `data_batcher_`-prefixed parameters of
        batcher is created from scratch using the `data_batcher_`-prefixed parameters of
        the present `TuneConfig` instance. Note that `transform` parameter of the
        downscaler is in any case enforced (therefore, transforming/re-transforming the
        wrangled data) on the one of the data batcher's data wrangler regardless of the
        previous value. Also, `data_batcher_data_wrangler_max_workers` parameter of the
        present `TuneConfig` instance is also in any case enforced.

    data_batcher_data_wrangler_path_sentinel3 : Path or None, default=None
        If `data_batcher` is not issued: path to directory containing Sentinel-3 product
        folders whose data is to be wrangled. Each of such folders must contain
        georeferenced Sentinel-3 SLSTR Level-2 LST product file
        (https://sentiwiki.copernicus.eu/web/slstr-products#S3-SLSTR-Products-L2-LST-Products)
        as well as a georeferenced Sentinel-3 Synergy Level-2 product file
        (https://sentiwiki.copernicus.eu/web/synergy-products#SYNERGYProducts-L2SYNSDRprocessingS3-Synergy-Products-L2-SYN-SDR-processing).
        Furthermore, the name of such folders must correspond to the respective start
        sensing time in the format "YYYYMMDDTHHMMSS".

    data_batcher_data_wrangler_path_spatial_pred : Path or None, default=None
        If `data_batcher` is not issued: path to a NetCDF file with the spatial
        predictor data whose data is to be wrangled. If not set, no spatial predictor
        data is considered.

    data_batcher_data_wrangler_aoi : str or Path or None, default=None
        If `data_batcher` is not issued: WKT string or path to AOI geometry file to mask
        out the data. The data wrangler will add the AOI to the wrangled data as
        variable `"aoi"`. If not set, no such variable is defined and no masking is
        applied.

    data_batcher_data_wrangler_path_landsat : Path or None, default=None
        If `data_batcher` is not issued: Path to the directory containing Landsat 8/9
        folders whose data is to be wrangled. Each of such folders must contain a
        `LST.TIF` file with georeferenced Landsat 8/9 Level-2 LST data
        (https://www.usgs.gov/centers/eros/science/usgs-eros-archive-landsat-archives-landsat-8-9-olitirs-collection-2-level-2),
        having a resolution of 30 m. In the wrangling, such data and Sentinel-3's will
        be "matched" if the respective folders have the same name (it is implied here
        that the user had analysed the acquisitions obtained by the two platforms and
        set the names of the Landsat 8/9 data folders as the ones of Sentinel-3's (start
        sensing times) whose start sensing times and spatial extents are approximately
        the same). Note that the Landsat data will be solely used for testing.

    data_batcher_data_wrangler_vars : list[str] or None, default=None
        If `data_batcher` is not issued: aliases of the variables to be wrangled besides
        the target (such as predictor, sample_weight and visualization variables). If
        `vars` is not issued, but `downscaler` is, it will be set to the aliases of the
        predictors (`cols_X`) considered by the latter. Otherwise, if `downscaler` is
        not issued but `downscaler_X` is, it will be set to `downscaler_X`, or, if not,
        to all aliases of the predictors (`X`) considered by a default `DataVars`
        instance (`s3lst_ds.utilities.var_utils.DataVars`).

    data_batcher_data_wrangler_max_workers : int, default=1
        Number of simultaneous multiple processes to be considered by the data wrangler
        in wrangling. Note that if negative, one has the following conditions:
            - `-1`: all processors are used;
            - `-k`: all processors except k-1 are used.
        This parameter is enforced regardless of the data wrangler being issued or
        created from scratch.

    data_batcher_n_cross_val_folds: int, default=5
        If `data_batcher` is not issued: number of cross-validation folds.

    data_batcher_var_cross_val_strat: str or None, default=None
        If `data_batcher` is not issued: metadata categorical variable with respect to
        which stratification in the cross-validation data splitting into folds is to be
        performed. If not defined, no stratification is considered.

    data_batcher_rnd_seed: int or np.random.RandomState or None = None
        If `data_batcher` is not issued: random seed number considered in the
        cross-validation data splitting into folds. If not defined, no such number is
        regarded.

    downscaler : Downscaler or Path or None, default=None
        Downscaler or a path to a Joblib file containing it. If not issued, a downscaler
        is created from scratch using the `downscaler_`-prefixed parameters of the
        present `TuneConfig`. Note that `downscaler_masks` and `downscaler_max_workers`
        are in any case enforced, regardless of the downscaler being issued or created
        from scratch.

    downscaler_base_model : Regressor, default=LinearRegression()
        If `downscaler` is not issued: the regression model to be used as the base model
        of the downscaler to be created. If not issued, it is set to
        `LinearRegression()` by default.

    downscaler_X : list[str] or None, default=["FVC", "NDWI"]
        If `downscaler` is not issued: aliases of the predictors to be considered by the
        downscaler to be created. If not issued, it is set to `["FVC", "NDWI"]`.

    downscaler_masks: list[str] or None, default=None
        Aliases of the mask variables (e.g. `["aoi"]`) to regard (wherever the variables
        have `nan` values, the respective data records are masked out). If not issued,
        it is set to `[]` and no masking is considered by the downscaler. This parameter
        is enforced regardless of the downscaler being issued or created from scratch.

    downscaler_scale : {"standardize", "min_max_normalize", None}, default="standardize"
        If `downscaler` is not issued: the scaling method to apply to numerical
        predictors.

    downscaler_encode : {"one_hot", "dummy", None}, default="dummy"
        If `downscaler` is not issued: the encoding method to apply to the categorical
        predictors:
            - `"one_hot"`: to one-hot encode the categorical predictors;
            - `"dummy"`: to dummy encode the categorical predictors (one-hot
            encoding with the first component dropped);
            - `None`: to regard the categorical predictors raw (no encoding).

        Note that dummy encoding is usually considered in place of one-hot to avoid
        multicollinearity problems (one may show that a component of a one-hot encoding
        vector is fully determined by all the other components making it redundant).

    downscaler_transform : {None, "center", "standardize"}, default=None
        If `downscaler` is not issued: the transform to apply on the coarse target and
        coarse and fine spatio-temporal predictors from a copy of the wrangled `data` in
        each `SingleDataWrangler` instance of the data batcher's `data_wrangler` by
        using coarse data statistics. The transformations are set in
        `SingleDataWrangler's `data` with the same names as the original columns with
        the substring `"_trans"` suffixed to them. Note that the transformations are
        timestamp-specific, that is, the computed statistics and the applied
        transformations in each timestamp solely concern the data of that timestamp. The
        possible values for `downscaler_transform` are:
            - `None`: not transforming the data;
            - `"center"`: subtracting the mean from the data;
            - `"standardize"`: subtracting the mean from the data and dividing the
            result by the standard deviation.
        Note that such transforms are redundant for the case of the single-timestamp
        architecture. They only take effect for the multi-timestamp architecture.

    downscaler_lasso_sel : bool, default=False
        If `downscaler` is not issued: whether to use a Lasso regression for selecting
        the scaled-encoded `downscaler_X` predictors downstream of the preprocessor.
        Lasso selection is such that solely the input predictors associated with
        coefficients of the fitted Lasso regression model having absolute values larger
        than `1e-5` are selected. Note that the non-encoded `downscaler_X` predictors
        are regardlessly considered downstream of the preprocessor.

    downscaler_lasso_alpha : float, default=1.0
        If `downscaler` is not issued: the regularization strength of the Lasso
        regression model used for selecting the scaled-encoded `downscaler_X` predictors
        downstream of the preprocessor. Such regularization strength is the multiplying
        constant of the weight vector L1-norm (sum of the absolute values of the
        components) in the Lasso regression objective function. The larger the value,
        the stronger the regularization. Note that this parameter only takes effect if
        `downscaler_lasso_sel` is `True`.

    downscaler_max_workers : int, default=1
        Number of simultaneous multiple processes to be considered by the downscaler (in
        training, prediction and scoring). Note that if negative, one has the following
        conditions:
            - `-1`: all processors are used;
            - `-k`: all processors except k-1 are used.
        This parameter is enforced regardless of the downscaler being issued or created
        from scratch.

    timestamps : list[pd.Timestamp] or list[str] or None, default=None
        If `data_batcher` is not issued: timestamps of interest either as `pd.Timestamp`
        values or in any format parsable by `pd.Timestamp` (e.g. `"YYYY-MM-DD
        HH:MM:SS"`). These must correspond to the start sensing times of the respective
        acquisitions. Furthermore, they must be part of those regarded by
        `data_batcher_data_wrangler_path_sentinel3`.  If neither `timestamps` nor
        `data_batcher` are issued, `timestamps` will be set to all of those regarded by
        such path. If `timestamps` is not issued but `data_batcher` is, `timestamps`
        will be set to all of those regarded by the batcher.

    sample_weight_fit: str or None, default=None
        Alias of the variable to be regarded as sample weight for cross-validation and
        training of the downscaler. Note that in cross-validation, `sample_weight_fit`
        is used for both training and scoring. If not issued, no sample weight in
        training and cross-validation is considered.

    sample_weight_score: str or None, default=None
        Alias of the variable to be regarded as sample weight for scoring the downscaler
        in training and testing (but not in cross-validation). If not issued, no sample
        weight in such scoring is considered.

    scorers : list[str], default=["r2", "rmse", "mae", "mbe"]
        Aliases of the scorers to consider in training and testing.

    best_scorer : str, default="rmse"
        Alias of the metric to consider in the selection of the best hyperparameter
        combination in the hyperparameter tuning of the downscaler estimator (that is,
        the downscaler before de-transformation and residual correction).

        WARNING: Note that since the estimator is the object tuned and not the
        downscaler itself, the units of the computed cross-validation metric in the
        tuning are based on the ones of the transformed target (whose transform
        corresponds to the one set in the downscaler). For example, if the transform
        corresponds to `"standardize"` and `best_scorer` is set to `"rmse"`, the
        computed metric corresponds to the RMSE of the standardized target, which is
        unitless.

    correct : bool, default=True
        Whether to correct the predicted fine raw target for each image (from fine
        predictors and masks, `X_and_mask_fine`) using the finely-resampled residual for
        the prediction of the coarse raw target (from coarse predictors and masks,
        `X_and_mask_coarse`).

    params_tune_getter: dict[str, dict[str, Any]]
        Dictionary of search spaces of the downscaler estimator hyperparameters to be
        tuned. The keys must correspond to the hyperparameters' full access paths with
        each step separated by double underscores (e.g. `"base_model__formula"` in which
        `"base_model"` is a parameter of the downscaler estimator and `"formula"` is a
        parameter of the former). The values must be dictionaries with keys:
            - `"suggest_method"`: alias of the `optuna`'s [suggest
            method](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.trial.Trial.html#optuna.trial.Trial)
            to be used in the hyperparameter tuning of the respective hyperparameter:
                - [`"float"`](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.trial.Trial.html#optuna.trial.Trial.suggest_float):
                for continuous hyperparameters;
                - [`"int"`](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.trial.Trial.html#optuna.trial.Trial.suggest_int):
                for integer hyperparameters;
                - [`"categorical"`](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.trial.Trial.html#optuna.trial.Trial.suggest_categorical):
                for categorical hyperparameters.
            - `"suggest_kwargs"`: dictionary of keyword arguments (except `name` - as
            this one is inferred from the key) to be passed to the suggest method (e.g.,
            `{"low": 0.0, "high": 1.0}` for `"float"`).

    tune_rnd_seed: int or np.random.RandomState or None = None
        Random seed number for the sampler of hyperparameter values during the optimised
        hyperparameter tuning. If not defined, no such number is regarded.

    tune_n_trials: int, default=100
        Number of trials to perform in the optimised hyperparameter tuning.

    tune_n_jobs: int, default=1
        Number of simultaneous multiple processes to be considered in the optimised
        hyperparameter tuning. Note that if negative, one has the following conditions:
            - `-1`: all processors are used;
            - `-k`: all processors except k-1 are used.

    hparam_rnd_seed: int or np.random.RandomState or None = None
        Random seed number for the sampler of hyperparameter values during tuning. If
        not defined, no such number is regarded.

    path_out : Path or None, default=None
        The directory path to save the tuned downscaler, the obtained scores, and the
        data batcher. If not issued, the results are instead returned.

    out_data_batcher : bool, default=True
        Whether to return or write (if `path_out` is issued) the data batcher.

    log_mode : {None, "console", "file", "both"}, default="both"
        The logging mode for wrangling, batching, tuning, training, testing and writing:
            - `None`: No logging is done;
            - `"console"`: Logging is done to console only;
            - `"file"`: Logging is done to a log file only;
            - `"both"`: Logging is done to both console and a log file. Note the log
            file would be defined as `tune.log` at `out_dir`.
    """

    params_tune_getter: Callable[
        [optuna.trial.Trial | optuna.trial.FrozenTrial], dict[str, Any]
    ]
    data_batcher: DataBatcher | Path | None = None
    data_batcher_data_wrangler_path_sentinel3: Path | None = None
    data_batcher_data_wrangler_path_spatial_pred: Path | None = None
    data_batcher_data_wrangler_aoi: str | Path | None = None
    data_batcher_data_wrangler_path_landsat: Path | None = None
    data_batcher_data_wrangler_vars: list[str] | None = None
    data_batcher_data_wrangler_max_workers: int = 1
    data_batcher_n_cross_val_folds: int = 5
    data_batcher_var_cross_val_strat: str | None = None
    data_batcher_rnd_seed: int | np.random.RandomState | None = None
    downscaler: Downscaler | Path | None = None
    downscaler_base_model: Regressor = field(default_factory=lambda: LinearRegression())
    downscaler_X: list[str] = field(default_factory=lambda: ["FVC", "NDWI"])
    downscaler_masks: list[str] | None = None
    downscaler_scale: Literal["standardize", "min_max_normalize"] | None = "standardize"
    downscaler_encode: Literal["one_hot", "dummy"] | None = "dummy"
    downscaler_transform: Literal["center", "standardize"] | None = None
    downscaler_lasso_sel: bool = False
    downscaler_lasso_alpha: float = 1.0
    downscaler_max_workers: int = 1
    timestamps: list[pd.Timestamp] | list[str] | None = None
    sample_weight_fit: str | None = None
    sample_weight_score: str | None = None
    scorers: list[str] = field(default_factory=lambda: ["r2", "rmse", "mae", "mbe"])
    best_scorer: str = "rmse"
    correct: bool = True
    tune_rnd_seed: int | np.random.RandomState | None = None
    tune_n_trials: int = 100
    tune_n_jobs: int = 1
    path_out: Path | None = None
    out_data_batcher: bool = True
    log_mode: Literal["console", "file", "both"] | None = "both"

best_scorer class-attribute instance-attribute

best_scorer: str = 'rmse'

correct class-attribute instance-attribute

correct: bool = True

data_batcher class-attribute instance-attribute

data_batcher: DataBatcher | Path | None = None

data_batcher_data_wrangler_aoi class-attribute instance-attribute

data_batcher_data_wrangler_aoi: str | Path | None = None

data_batcher_data_wrangler_max_workers class-attribute instance-attribute

data_batcher_data_wrangler_max_workers: int = 1

data_batcher_data_wrangler_path_landsat class-attribute instance-attribute

data_batcher_data_wrangler_path_landsat: Path | None = None

data_batcher_data_wrangler_path_sentinel3 class-attribute instance-attribute

data_batcher_data_wrangler_path_sentinel3: Path | None = None

data_batcher_data_wrangler_path_spatial_pred class-attribute instance-attribute

data_batcher_data_wrangler_path_spatial_pred: Path | None = None

data_batcher_data_wrangler_vars class-attribute instance-attribute

data_batcher_data_wrangler_vars: list[str] | None = None

data_batcher_n_cross_val_folds class-attribute instance-attribute

data_batcher_n_cross_val_folds: int = 5

data_batcher_rnd_seed class-attribute instance-attribute

data_batcher_rnd_seed: int | RandomState | None = None

data_batcher_var_cross_val_strat class-attribute instance-attribute

data_batcher_var_cross_val_strat: str | None = None

downscaler class-attribute instance-attribute

downscaler: Downscaler | Path | None = None

downscaler_X class-attribute instance-attribute

downscaler_X: list[str] = field(default_factory=lambda: ['FVC', 'NDWI'])

downscaler_base_model class-attribute instance-attribute

downscaler_base_model: Regressor = field(default_factory=lambda: LinearRegression())

downscaler_encode class-attribute instance-attribute

downscaler_encode: Literal['one_hot', 'dummy'] | None = 'dummy'

downscaler_lasso_alpha class-attribute instance-attribute

downscaler_lasso_alpha: float = 1.0

downscaler_lasso_sel class-attribute instance-attribute

downscaler_lasso_sel: bool = False

downscaler_masks class-attribute instance-attribute

downscaler_masks: list[str] | None = None

downscaler_max_workers class-attribute instance-attribute

downscaler_max_workers: int = 1

downscaler_scale class-attribute instance-attribute

downscaler_scale: Literal['standardize', 'min_max_normalize'] | None = 'standardize'

downscaler_transform class-attribute instance-attribute

downscaler_transform: Literal['center', 'standardize'] | None = None

log_mode class-attribute instance-attribute

log_mode: Literal['console', 'file', 'both'] | None = 'both'

out_data_batcher class-attribute instance-attribute

out_data_batcher: bool = True

params_tune_getter instance-attribute

params_tune_getter: Callable[[Trial | FrozenTrial], dict[str, Any]]

path_out class-attribute instance-attribute

path_out: Path | None = None

sample_weight_fit class-attribute instance-attribute

sample_weight_fit: str | None = None

sample_weight_score class-attribute instance-attribute

sample_weight_score: str | None = None

scorers class-attribute instance-attribute

scorers: list[str] = field(default_factory=lambda: ['r2', 'rmse', 'mae', 'mbe'])

timestamps class-attribute instance-attribute

timestamps: list[Timestamp] | list[str] | None = None

tune_n_jobs class-attribute instance-attribute

tune_n_jobs: int = 1

tune_n_trials class-attribute instance-attribute

tune_n_trials: int = 100

tune_rnd_seed class-attribute instance-attribute

tune_rnd_seed: int | RandomState | None = None

__init__

__init__(
    params_tune_getter: Callable[[Trial | FrozenTrial], dict[str, Any]],
    data_batcher: DataBatcher | Path | None = None,
    data_batcher_data_wrangler_path_sentinel3: Path | None = None,
    data_batcher_data_wrangler_path_spatial_pred: Path | None = None,
    data_batcher_data_wrangler_aoi: str | Path | None = None,
    data_batcher_data_wrangler_path_landsat: Path | None = None,
    data_batcher_data_wrangler_vars: list[str] | None = None,
    data_batcher_data_wrangler_max_workers: int = 1,
    data_batcher_n_cross_val_folds: int = 5,
    data_batcher_var_cross_val_strat: str | None = None,
    data_batcher_rnd_seed: int | RandomState | None = None,
    downscaler: Downscaler | Path | None = None,
    downscaler_base_model: Regressor = (lambda: LinearRegression())(),
    downscaler_X: list[str] = (lambda: ["FVC", "NDWI"])(),
    downscaler_masks: list[str] | None = None,
    downscaler_scale: Literal["standardize", "min_max_normalize"]
    | None = "standardize",
    downscaler_encode: Literal["one_hot", "dummy"] | None = "dummy",
    downscaler_transform: Literal["center", "standardize"] | None = None,
    downscaler_lasso_sel: bool = False,
    downscaler_lasso_alpha: float = 1.0,
    downscaler_max_workers: int = 1,
    timestamps: list[Timestamp] | list[str] | None = None,
    sample_weight_fit: str | None = None,
    sample_weight_score: str | None = None,
    scorers: list[str] = (lambda: ["r2", "rmse", "mae", "mbe"])(),
    best_scorer: str = "rmse",
    correct: bool = True,
    tune_rnd_seed: int | RandomState | None = None,
    tune_n_trials: int = 100,
    tune_n_jobs: int = 1,
    path_out: Path | None = None,
    out_data_batcher: bool = True,
    log_mode: Literal["console", "file", "both"] | None = "both",
) -> None

Caller

s3lst_ds.downscaling.tune.tune.tune

tune(config: TuneConfig) -> TuneOut

Wrangle the data of timestamps of interest, batch it into cross-validation and test datasets, perform an optimized cross-validated hyperparameter tuning of a multi-timestamp downscaling model and subsequently train and test it. Return or write the results to files.

Batching is done with respect to the issued timestamps, and, therefore, the whole data associated with a timestamp (a scene) is fully contained within a single batch. The resulting batch datasets are: - "test" dataset: timestamps for which there is Landsat data. - "cross_val" dataset: random split of the remaining timestamps into config.n_cross_val_folds folds, stratified with respect to a categorical variable config.var_cross_val_strat (if issued).

The optimized cross-validated hyperparameter tuning is done using optuna by trying suggested hyperparameter values within the search space defined in the issued config.params_tune_getter. The estimator of the downscaler (the downscaler before de-transformation and residual correction) is cross-validated for each suggested hyperparameter combination and the best one is selected based on the issued scoring metric config.best_scorer. Cross-validation is done by training the downscaler estimator on all cross-validation folds except one and scoring it on the latter, rotating the scoring fold until all are considered. The overall cross-validation score is computed from the arithmetic mean of the scores of each iteration.

Training of the tuned downscaler is done using the whole cross-validation data.

Testing is done by scoring the trained downscaler on the test dataset (for both Sentinel-3 and Landsat data).

WARNING: Note that for the sake of efficiency, during tuning, solely tuner-specific multiprocessing (set through config.tune_n_jobs parameter) is considered . Base model-specific multi-processing (set through the respective n_jobs parameter) is subsequently considered. Note that for the case of the MLPRegressor, config.tune_n_jobs is forcefully set to 1, regardless of the issued value, as the MLPRegressor, by default, always use all available processors - and, therefore, a value of config.tune_n_jobs greater than 1 would impair the process.

WARNING: Note that the units of the computed cross-validation score in the hyperparameter tuning are based on the ones of the transformed target (whose transform corresponds to the one set in the downscaler). For example, if the transform corresponds to "standardize" and best_scorer is set to "rmse", the computed score corresponds to the RMSE of the standardized target, which is unitless.

Parameters:

Name	Type	Description	Default
config	TuneConfig	Configurations for wrangling, batching, tuning, training, testing and writing.	required

Returns:

Type

Description

dict

Dictionary containing: - downscaler: Downscaler or Path The tuned downscaler (if config.path_out is not issued) or a path to the respective Joblib file. - params: dict[str, Any] The tuned hyperparameters of the downscaler (fixed hyperparameters are disregarded here). - score: dict[str, dict[str, dict[str, dict[str, float]]]] or Path Cross-validation, training and test scores either as a dictionary (if config.path_out is not issued) or as a path to the respective JSON file. The scores are keyed by batch ("cross_val", "train" or "test"), grid ("coarse" or "fine"), ground truth dataset ("sentinel" or "landsat") and metric ("r2", "rmse", etc.). Scores using Landsat ground truth data are solely computed in testing and if such data exists for the timestamps of interest (the Landsat data must be included in the issued config.data_batcher or in config.data_batcher_path_landsat if no data batcher is issued). - data_batcher: DataBatcher or Path The data batcher (if config.path_out is not issued) or a path to the respective Joblib file. The data batcher also contains the batched data and may be useful for debugging or for reusing it without need for reprocessing the original one. This only takes effect if parameter config.out_data_batcher is True.

Source code in src/s3lst_ds/downscaling/tune/tune.py

def tune(
    config: TuneConfig,
) -> TuneOut:
    """
    Wrangle the data of timestamps of interest, batch it into cross-validation and test
    datasets, perform an optimized cross-validated hyperparameter tuning of a
    multi-timestamp downscaling model and subsequently train and test it. Return or
    write the results to files.

    Batching is done with respect to the issued `timestamps`, and, therefore, the whole
    data associated with a timestamp (a scene) is fully contained within a single batch.
    The resulting batch datasets are:
        - `"test"` dataset: timestamps for which there is Landsat data.
        - `"cross_val"` dataset: random split of the remaining timestamps into
        `config.n_cross_val_folds` folds, stratified with respect to a categorical
        variable `config.var_cross_val_strat` (if issued).

    The optimized cross-validated hyperparameter tuning is done using
    [`optuna`](https://optuna.readthedocs.io/en/stable/) by trying suggested
    hyperparameter values within the search space defined in the issued
    `config.params_tune_getter`. The estimator of the downscaler (the downscaler before
    de-transformation and residual correction) is cross-validated for each suggested
    hyperparameter combination and the best one is selected based on the issued scoring
    metric `config.best_scorer`. Cross-validation is done by training the downscaler
    estimator on all cross-validation folds except one and scoring it on the latter,
    rotating the scoring fold until all are considered. The overall cross-validation
    score is computed from the arithmetic mean of the scores of each iteration.

    Training of the tuned downscaler is done using the whole cross-validation data.

    Testing is done by scoring the trained downscaler on the test dataset (for both
    Sentinel-3 and Landsat data).

    WARNING: Note that for the sake of efficiency, during tuning, solely tuner-specific
    multiprocessing (set through `config.tune_n_jobs` parameter) is considered . Base
    model-specific multi-processing (set through the respective `n_jobs` parameter) is
    subsequently considered. Note that for the case of the MLPRegressor,
    `config.tune_n_jobs` is forcefully set to `1`, regardless of the issued value, as
    the MLPRegressor, by default, always use all available processors - and, therefore,
    a value of `config.tune_n_jobs` greater than `1` would impair the process.

    WARNING: Note that the units of the computed cross-validation score in the
    hyperparameter tuning are based on the ones of the transformed target (whose
    transform corresponds to the one set in the downscaler). For example, if the
    transform corresponds to `"standardize"` and `best_scorer` is set to `"rmse"`, the
    computed score corresponds to the RMSE of the standardized target, which is
    unitless.

    Parameters
    ----------

    config: TuneConfig
        Configurations for wrangling, batching, tuning, training, testing and writing.

    Returns
    -------
    dict
        Dictionary containing:
        - downscaler: Downscaler or Path
            The tuned downscaler (if `config.path_out` is not issued) or a path to the
            respective Joblib file.
        - params: dict[str, Any]
            The tuned hyperparameters of the downscaler (fixed hyperparameters are
            disregarded here).
        - score: dict[str, dict[str, dict[str, dict[str, float]]]] or Path
            Cross-validation, training and test scores either as a dictionary (if
            `config.path_out` is not issued) or as a path to the respective JSON file.
            The scores are keyed by batch ("cross_val", "train" or "test"), grid
            ("coarse" or "fine"), ground truth dataset ("sentinel" or "landsat") and
            metric ("r2", "rmse", etc.). Scores using Landsat ground truth data are
            solely computed in testing and if such data exists for the timestamps of
            interest (the Landsat data must be included in the issued
            `config.data_batcher` or in `config.data_batcher_path_landsat` if no data
            batcher is issued).
        - data_batcher: DataBatcher or Path
            The data batcher (if `config.path_out` is not issued) or a path to the
            respective Joblib file. The data batcher also contains the batched data and
            may be useful for debugging or for reusing it without need for reprocessing
            the original one. This only takes effect if parameter
            `config.out_data_batcher` is `True`.
    """

    # ---> Handle logging

    # Create logger
    logger = RichLogger(
        name="tune",
        level=logging.INFO,
        file_path=(
            Path(config.path_out) / "tune.log" if config.path_out is not None else None
        ),
        file_mode="w",
        log_mode=config.log_mode,
    )

    # Redirect optuna logs to the logger
    optuna_logger = optuna.logging.get_logger("optuna")
    optuna_logger.handlers.clear()
    for handler in logger.base_logger.handlers:
        optuna_logger.addHandler(handler)

    # Print status message
    logger.console.print()
    logger.info("[bold]Tuning multi-timestamp downscaler[/bold]")

    # ---> Create output directory
    if config.path_out is not None:
        try:
            Path(config.path_out).mkdir(parents=True, exist_ok=True)

        except Exception as e:  # noqa: BLE001
            logger.error(
                "[bold red]Error creating the output directory."
                + f"\nError message: {e}"
                + "\nRun will stop.[/bold red]",
            )
            raise WritingError(
                "Error creating the output directory." + f"\nError message: {e}"
            )

    # ---> Get downscaler if it is issued

    # Load downscaler from file if a path is issued
    if isinstance(config.downscaler, Path):
        logger.console.print()
        logger.info("The downscaler will now be loaded from file.")
        try:
            with logger.console.status(
                f"{'':7}Loading downscaler from file[yellow]...[/yellow]",
                spinner="dots",
                spinner_style="bold blue",
            ):
                downscaler = joblib.load(config.downscaler)

        except Exception as e:  # noqa: BLE001
            logger.error(
                "[bold red]Error loading the downscaler from file."
                + f"\nError message: {e}"
                + "\nRun will stop.[/bold red]",
            )
            raise ReadingError(
                "Error loading the downscaler from file." + f"\nError message: {e}"
            )

        if not isinstance(downscaler, Downscaler):
            logger.error(
                "[bold red]The loaded downscaler is not a (multi-timestamp) Downscaler"
                " object." + "\nRun will stop.[/bold red]",
            )
            raise TypeError(
                "The loaded downscaler is not a (multi-timestamp) Downscaler object."
            )
        logger.info("[bold green]Downscaler loaded from file.[/bold green]")

    # Set downscaler if provided directly
    elif isinstance(config.downscaler, Downscaler):
        downscaler = config.downscaler

    # ---> Get data batcher if it is issued
    # Load data batcher from file if a path is issued
    if isinstance(config.data_batcher, Path):
        logger.console.print()
        logger.info("The data batcher will now be loaded from file.")
        try:
            with logger.console.status(
                f"{'':7}Loading data batcher from file[yellow]...[/yellow]",
                spinner="dots",
                spinner_style="bold blue",
            ):
                data_batcher = joblib.load(config.data_batcher)

        except Exception as e:  # noqa: BLE001
            logger.error(
                "[bold red]Error loading the data batcher from file."
                + f"\nError message: {e}"
                + "\nRun will stop.[/bold red]",
            )
            raise ReadingError(
                "Error loading the data batcher from file." + f"\nError message: {e}"
            )

        logger.info("[bold green]Data batcher loaded from file.[/bold green]")

    # Set data batcher if provided directly
    elif isinstance(config.data_batcher, DataBatcher):
        data_batcher = config.data_batcher

    # ---> Parse parameters

    # Parse downscaler predictors
    downscaler_X = (
        downscaler.cols_X if config.downscaler is not None else config.downscaler_X
    )

    # Parse downscaler transform
    downscaler_transform = (
        downscaler.transform  # type: ignore
        if config.downscaler is not None
        else config.downscaler_transform
    )

    # Parse path to Landsat data
    # NOTE: Landsat data will be solely used in coarse and fine test-scoring if it was
    # issued or already contained in an issued data batcher.
    data_batcher_data_wrangler_path_landsat = (
        data_batcher.data_wrangler.path_landsat
        if config.data_batcher is not None
        else config.data_batcher_data_wrangler_path_landsat
    )

    # Parse data wrangling variables
    if config.data_batcher is not None:
        data_batcher_data_wrangler_data_vars = data_batcher.data_wrangler.data_vars
    else:
        data_batcher_data_wrangler_vars = (
            config.data_batcher_data_wrangler_vars
            if config.data_batcher_data_wrangler_vars is not None
            else (
                downscaler.cols_X
                if config.downscaler is not None
                else config.downscaler_X
                if config.downscaler_X is not None
                else None
            )
        )
        data_batcher_data_wrangler_data_vars = DataVars().subset_X(
            data_batcher_data_wrangler_vars  # type: ignore
        )

    # Parse timestamps
    timestamps = (
        [
            pd.Timestamp(timestamp)
            if not isinstance(timestamp, pd.Timestamp)
            else timestamp
            for timestamp in config.timestamps
        ]
        if config.data_batcher is None and config.timestamps is not None
        else (
            data_batcher.data_wrangler.timestamps
            if config.data_batcher is not None
            else [
                pd.Timestamp(data_batcher_data_wrangler_path_sentinel3_folder.name)
                for data_batcher_data_wrangler_path_sentinel3_folder in config.data_batcher_data_wrangler_path_sentinel3.iterdir()  # type: ignore
                if data_batcher_data_wrangler_path_sentinel3_folder.is_dir()
            ]
        )
    )

    # Parse indicators for outputting variables
    out = {
        "downscaler": True,
        "params": True,
        "score": True,
        "data_batcher": config.out_data_batcher,
    }

    # Parse output paths
    path_out = {
        object_alias: (
            (
                config.path_out
                / (
                    object_alias
                    + (
                        ".joblib"
                        if object_alias not in ["params", "score"]
                        else ".json"
                    )
                )
            )
            if config.path_out is not None and out[object_alias] is True
            else None
        )
        for object_alias in out
    }

    # ---> Update parameters of the downscaler and data wrangler if they had been issued

    # Set logger
    if config.data_batcher is not None:
        data_batcher.data_wrangler.logger = logger
    if config.downscaler is not None:
        downscaler.logger = logger

    # Update masking variables and maximum number of workers of the downscaler if it had
    # been issued
    if config.downscaler is not None:
        downscaler.cols_mask = config.downscaler_masks
        downscaler.max_workers = config.downscaler_max_workers

    # Update transform (with the one of the downscaler, if it is different from the one
    # of the data wrangler) and maximum number of workers of the data wrangler if it had
    # been issued
    if config.data_batcher is not None:
        if data_batcher.data_wrangler.transform != downscaler_transform:  # type: ignore
            data_batcher.data_wrangler.transform = downscaler_transform  # type: ignore
        data_batcher.data_wrangler.max_workers = (
            config.data_batcher_data_wrangler_max_workers
        )

    # ---> Wrangle and batch the data if a data batcher was not issued
    if config.data_batcher is None:
        logger.console.print()
        logger.info("The data will now be wrangled.")
        try:
            # Define a data wrangler
            data_wrangler = DataWrangler(
                data_vars=data_batcher_data_wrangler_data_vars,
                path_sentinel3=config.data_batcher_data_wrangler_path_sentinel3,  # type: ignore
                path_spatial_pred=config.data_batcher_data_wrangler_path_spatial_pred,  # type: ignore
                aoi=config.data_batcher_data_wrangler_aoi,
                path_landsat=data_batcher_data_wrangler_path_landsat,
                timestamps=timestamps,
                transform=downscaler_transform,  # type: ignore
                max_workers=config.data_batcher_data_wrangler_max_workers,
                logger=logger,
            )

        except Exception as e:  # noqa: BLE001
            logger.error(
                "[bold red]Error wrangling the data."
                + f"\nError message: {e}"
                + "\nRun will stop.[/bold red]",
            )
            raise DataWranglingError(
                "Error wrangling the data." + f"\nError message: {e}"
            )

        logger.info("[bold green]Data wrangled.[/bold green]")

        logger.console.print()
        logger.info("The data will now be batched.")
        try:
            with logger.console.status(
                f"{'':7}Batching the data[yellow]...[/yellow]",
                spinner="dots",
                spinner_style="bold blue",
            ):
                # Define a data batcher
                data_batcher = DataBatcher(
                    data_wrangler=data_wrangler,
                    n_cross_val_folds=config.data_batcher_n_cross_val_folds,
                    var_cross_val_strat=config.data_batcher_var_cross_val_strat,
                    rnd_seed=config.data_batcher_rnd_seed,
                )

        except Exception as e:  # noqa: BLE001
            logger.error(
                "[bold red]Error batching the data."
                + f"\nError message: {e}"
                + "\nRun will stop.[/bold red]",
            )
            raise DataBatchingError(
                "Error batching the data." + f"\nError message: {e}"
            )

        logger.info("[bold green]Data batched.[/bold green]")

    # ---> Tune the downscaler
    logger.console.print()
    logger.info("The downscaler will now be tuned.")
    try:
        # Define a downscaler if it had not been issued
        if config.downscaler is None:
            downscaler = Downscaler(
                base_model=config.downscaler_base_model,
                cols_X=downscaler_X,
                cols_mask=config.downscaler_masks,
                scale=config.downscaler_scale,
                encode=config.downscaler_encode,
                lasso_sel=config.downscaler_lasso_sel,
                lasso_alpha=config.downscaler_lasso_alpha,
                max_workers=config.downscaler_max_workers,
                transform=downscaler_transform,  # type: ignore
                logger=logger,
            )

        # Set base model processors to 1 during tuning to not impair the process (as
        # multiple processes may already be used by the tuner itself)
        if "base_model__n_jobs" in downscaler.get_params():
            base_model__n_jobs = downscaler.base_model.n_jobs  # type: ignore
            downscaler.set_params(base_model__n_jobs=1)

        # Define optimization task
        # NOTE:
        # https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.create_study.html#optuna.study.create_study
        study = optuna.create_study(
            study_name="hparam_tuning",
            # Optimization direction
            direction=(
                "minimize"
                if get_scorer(config.best_scorer).greater_is_better is False
                else "maximize"
            ),
            # Method for suggesting new hyperparameter values
            # NOTE: TPESampler (Tree-structured Parzen Estimator) uses past results
            # to guide future trials (see
            # https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.TPESampler.html#optuna-samplers-tpesampler).
            sampler=optuna.samplers.TPESampler(seed=config.tune_rnd_seed),  # type: ignore
        )

        # Define progress bar and callback for optuna's optimizer
        logger.info("Performing tuning trials...")
        pbar = tqdm(
            # Prefix for the progressbar
            bar_format=f"{'':9}" + "{l_bar}{bar}{r_bar}",
            desc=f"{'':8}",
            total=config.tune_n_trials,  # type: ignore
            unit="trial",
            position=0,
            leave=True,  # Keep progress on the screen after completion.
            options={"console": logger.console},
        )

        def pbar_callback(
            study: optuna.study.Study, trial: optuna.trial.FrozenTrial
        ) -> None:
            """
            Callback function to update the progress bar after each trial.

            For more details about `optuna`'s callback functions, read [the
            documentation](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.Study.html#optuna.study.Study.optimize).
            """
            pbar.update()

        # Perform optimization task
        # NOTE:
        # https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.Study.html#optuna.study.Study.optimize
        study.optimize(
            func=get_objective(
                estimator=downscaler.estimator,
                data_batcher=data_batcher,
                params_tune_getter=config.params_tune_getter,
                sample_weight=config.sample_weight_fit,
                best_scorer=config.best_scorer,
            ),
            # Number of trials to perform
            n_trials=config.tune_n_trials,
            # Number of multiple processes
            # NOTE: https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/004_distributed.html#multi-thread-optimization
            # NOTE:  Note that for the case of the MLPRegressor, `tune_n_jobs` is
            # forcefully set to `1`, regardless of the issued value, as the
            # MLPRegressor, by default, always use all available processors - and,
            # therefore, a value of `tune_n_jobs` greater than `1` would impair the
            # process.
            n_jobs=(
                config.tune_n_jobs
                if not isinstance(downscaler.base_model, MLPRegressor)
                else 1
            ),
            # List of callback functions to be invoked at the end of each trial
            callbacks=[pbar_callback],
            show_progress_bar=False,
        )

        # Close progress bar
        pbar.close()

        # Get best trial number
        best_trial = study.best_trial.number

        # Get best score
        best_score = study.best_value

        # Get and set best hyperparameters
        best_params = config.params_tune_getter(study.best_trial)
        downscaler.set_params(**best_params)

        # Set base model number of processors to the issued one (since tuning is now
        # done and does not require more processors)
        if "base_model__n_jobs" in downscaler.get_params():
            downscaler.set_params(base_model__n_jobs=base_model__n_jobs)

    except Exception as e:  # noqa: BLE001
        logger.error(
            "[bold red]Error tuning the downscaler."
            + f"\nError message: {e}"
            + "\nRun will stop.[/bold red]",
        )
        raise TuningError("Error tuning the downscaler." + f"\nError message: {e}")

    logger.info(
        "[bold green]Downscaler tuned having as best hyperparameters"
        f"\n{best_params}"
        f"\nwhich were found at trial {best_trial} with cross-validation"
        f" {config.best_scorer} of {best_score:g}."
        "[/bold green]"
    )

    # ---> Retrain with the whole cross-validation data

    # Train the downscaler with the training data
    downscaler = train(
        downscaler=downscaler,
        data_batcher=data_batcher,
        sample_weight=config.sample_weight_fit,
        logger=logger,
    )

    # ---> Score the downscaler with respect to the training and test data
    score = {"cross_val": {"coarse": {"sentinel": {config.best_scorer: best_score}}}}
    score = score | score_train_test(
        downscaler=downscaler,
        data_batcher=data_batcher,
        sample_weight=config.sample_weight_fit,
        scorers=config.scorers,
        logger=logger,
    )

    # ---> Combine all the results in a dictionary

    object = {
        "downscaler": downscaler,
        "params": best_params,
        "score": score,
        "data_batcher": data_batcher,
    }

    # ---> Write downscaler, scores and data batcher to file if wanted
    for object_alias in object:  # noqa: PLC0206
        if path_out[object_alias] is not None:
            logger.console.print()
            logger.info(
                f"The {object_alias.replace('_', ' ')} will now be written to file."
            )
            try:
                with logger.console.status(
                    f"{'':7}Writing {object_alias.replace('_', ' ')} to file"
                    "[yellow]...[/yellow]",
                    spinner="dots",
                    spinner_style="bold blue",
                ):
                    if object_alias in ["params", "score"]:
                        pd.Series(object[object_alias]).to_json(
                            path_out[object_alias]  # type: ignore
                        )
                    else:
                        object[object_alias].save(path_out[object_alias])
            except Exception as e:  # noqa: BLE001
                logger.error(
                    f"[bold red]Error writing the {object_alias.replace('_', ' ')}"
                    " to file."
                    f"\nError message: {e}"
                    "\nRun will stop.[/bold red]",
                )
                raise WritingError(
                    f"Error writing the {object_alias.replace('_', ' ')} to file."
                    + f"\nError message: {e}"
                )

            logger.info(
                f"[bold green]{object_alias.replace('_', ' ').capitalize()} written"
                " to file.[/bold green]"
            )

    # ---> Show table with best hyperparameters
    table_hparams = Table(title="Best hyperparameters")
    table_hparams.add_column("Hyperparameter", justify="left")
    table_hparams.add_column("Value", justify="right")
    for param, value in best_params.items():
        table_hparams.add_row(
            param, f"{value:.5g}" if not isinstance(value, str) else str(value)
        )
    logger.console.print()
    logger.info(
        "The downscaler attained the following best hyperparameters:"
        "\n\n"
        + f"{
            get_rich_text_from_renderable(
                console=logger.console,
                renderable=table_hparams,
            )
        }"
    )

    # ---> Show table with scores if they were computed
    if score is not None:
        table_score = Table(title="Metrics")
        table_score.add_column("Batch", justify="left")
        table_score.add_column("Grid", justify="left")
        table_score.add_column("Ground truth", justify="left")
        table_score.add_column("Metric", justify="left")
        table_score.add_column("Value", justify="right")
        for batch in score:
            for i, grid in enumerate(score[batch].keys()):
                for j, ground_truth in enumerate(score[batch][grid].keys()):
                    for k, scorer in enumerate(score[batch][grid][ground_truth].keys()):
                        table_score.add_row(
                            batch if i == 0 and j == 0 and k == 0 else None,
                            grid if j == 0 and k == 0 else None,
                            ground_truth if k == 0 else None,
                            scorer,
                            f"{score[batch][grid][ground_truth][scorer]:.5g}",
                        )

        logger.console.print()
        logger.info(
            "The downscaler attained the following metrics:"
            "\n\n"
            + f"{
                get_rich_text_from_renderable(
                    console=logger.console,
                    renderable=table_score,
                )
            }"
        )

    # ---> Return the results
    logger.console.print()
    return TuneOut(
        **{
            object_alias: (
                object if path_out[object_alias] is None else path_out[object_alias]
            )
            for object_alias, object in object.items()
            if out[object_alias] is True
        }  # type: ignore
    )