bluemath_tk.downloaders package

Subpackages

• bluemath_tk.downloaders.copernicus package
  • Submodules
  • bluemath_tk.downloaders.copernicus.copernicus_downloader module
    • CopernicusDownloader
      • CopernicusDownloader.product
      • CopernicusDownloader.product_config
      • CopernicusDownloader.client
      • CopernicusDownloader.client
      • CopernicusDownloader.download_data()
      • CopernicusDownloader.download_data_era5()
      • CopernicusDownloader.list_datasets()
      • CopernicusDownloader.list_variables()
      • CopernicusDownloader.num_workers
      • CopernicusDownloader.product
      • CopernicusDownloader.product_config
      • CopernicusDownloader.products_configs
      • CopernicusDownloader.show_markdown_table()
  • Module contents

Module contents

Project: BlueMath_tk Sub-Module: downloaders Author: GeoOcean Research Group, Universidad de Cantabria Creation Date: 9 August 2024 Repository: https://github.com/GeoOcean/BlueMath_tk.git Status: Under development (Working)

class bluemath_tk.downloaders.BaseDownloader(base_path_to_download: str, debug: bool = True, check: bool = False)

Bases: BlueMathModel

Abstract class for BlueMath downloaders.

base_path_to_download

The base path to download the data.

Type:

str

debug

If True, the logger will be set to DEBUG level. Default is True.

Type:

bool, optional

check

If True, just file checking is required. Default is False.

Type:

bool, optional

download_data(*args, \*\*kwargs)

Downloads the data. This method must be implemented in the child class.

Notes

• This class is an abstract class and should not be instantiated.

• The download_data method must be implemented in the child class.

property base_path_to_download: str

property check: bool

property debug: bool

abstractmethod download_data(*args, **kwargs) → None

class bluemath_tk.downloaders.CopernicusDownloader(product: str, base_path_to_download: str, token: str = None, debug: bool = True, check: bool = False)

Bases: BaseDownloader

This is the main class to download data from the Copernicus Climate Data Store.

product

The product to download data from. Currently only ERA5 is supported.

Type:

str

product_config

The configuration for the product to download data from.

Type:

dict

client

The client to interact with the Copernicus Climate Data Store API.

Type:

cdsapi.Client

Notes

• The configuration for the products is stored in the products_configs attribute.

• The configuration for the ERA5 product is stored in the ERA5 folder.

property client: Client

download_data(*args, **kwargs) → str

Downloads the data for the product.

Parameters:

• *args – The arguments to pass to the download function.

• **kwargs – The keyword arguments to pass to the download function.

Returns:

The message with the fully downloaded files and the not fully downloaded files.

Return type:

str

Raises:

ValueError – If the product is not supported.

download_data_era5(variables: List[str], years: List[str], months: List[str], day: List[str] = None, time: List[str] = None, product_type: str = 'reanalysis', data_format: str = 'netcdf_legacy', download_format: str = 'unarchived', force: bool = False) → str

Downloads the data for the ERA5 product.

Parameters:

• variables (List[str]) – The list of variables to download.

• years (List[str]) – The list of years to download.

• months (List[str]) – The list of months to download.

• day (List[str], optional) – The list of days to download. Defaults to None.

• time (List[str], optional) – The list of times to download. Defaults to None.

• product_type (str, optional) – The product type to download. Defaults to “reanalysis”.

• data_format (str, optional) – The data format to download. Defaults to “netcdf_legacy”. This format is maintained to legacy to allow Thredds server to read the files.

• download_format (str, optional) – The download format to use. Defaults to “unarchived”.

• force (bool, optional) – Whether to force the download. Defaults to False.

Returns:

The message with the fully downloaded files and the not fully downloaded files. Error files are also included.

Return type:

str

Raises:

ValueError – If the variables, years, months, day, time, product_type, data_format, or download_format are not valid or if the force is not a boolean.

Notes

• The variables, years, months, day, and time must be lists of strings.

• The product_type, data_format, and download_format must be strings.

• The force must be a boolean.

• Leave variables empty to download all variables.

• Leave day empty to download all days.

• Leave time empty to download all times.

list_datasets() → List[str]

Lists the datasets available for the product.

Returns:

The list of datasets available for the product.

Return type:

List[str]

list_variables(type: str = None) → List[str]

Lists the variables available for the product. Filtering by type if provided.

Parameters:

type (str, optional) – The type of variables to list. Default is None.

Returns:

The list of variables available for the product.

Return type:

List[str]

property product: str

property product_config: dict

products_configs = {'ERA5': {'datasets': {'reanalysis-era5-pressure-levels': {'description': 'ERA5 Pressure Levels', 'mandatory_fields': ['product_type', 'variable', 'year', 'month', 'pressure_level', 'data_format', 'download_format'], 'optional_fields': ['day', 'time', 'area'], 'template': {'data_format': 'netcdf', 'day': ['01'], 'download_format': 'unarchived', 'month': ['01'], 'pressure_level': ['500'], 'product_type': ['reanalysis'], 'time': ['00:00'], 'variable': ['geopotential'], 'year': ['2019']}, 'types': ['pressure'], 'url': 'https://cds.climate.copernicus.eu/datasets/reanalysis-era5-pressure-levels?tab=overview'}, 'reanalysis-era5-single-levels': {'description': 'ERA5 Single Levels', 'mandatory_fields': ['product_type', 'variable', 'year', 'month', 'data_format', 'download_format'], 'optional_fields': ['day', 'time', 'area'], 'template': {'data_format': 'netcdf', 'day': ['01'], 'download_format': 'unarchived', 'month': ['01'], 'product_type': ['reanalysis'], 'time': ['00:00'], 'variable': ['significant_wave_height_of_first_swell_partition'], 'year': ['2019']}, 'types': ['atmosphere', 'ocean'], 'url': 'https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=form'}}, 'variables': {'dwww': {'cds_name': 'wave_spectral_directional_width_for_wind_waves', 'dataset': 'reanalysis-era5-single-levels', 'long_name': 'Wave spectral directional width for wind waves', 'nc_name': 'dwww', 'type': 'ocean', 'units': 'degrees'}, 'mdww': {'cds_name': 'mean_direction_of_wind_waves', 'dataset': 'reanalysis-era5-single-levels', 'long_name': 'Mean direction of wind waves', 'nc_name': 'mdww', 'type': 'ocean', 'units': 'degrees'}, 'mpww': {'cds_name': 'mean_period_of_wind_waves', 'dataset': 'reanalysis-era5-single-levels', 'long_name': 'Mean period of wind waves', 'nc_name': 'mpww', 'type': 'ocean', 'units': 's'}, 'mwd': {'cds_name': 'mean_wave_direction', 'dataset': 'reanalysis-era5-single-levels', 'long_name': 'Mean wave direction', 'nc_name': 'mwd', 'type': 'ocean', 'units': 'degrees'}, 'mwp': {'cds_name': 'mean_wave_period', 'dataset': 'reanalysis-era5-single-levels', 'long_name': 'Mean wave period', 'nc_name': 'mwp', 'type': 'ocean', 'units': 's'}, 'p140121': {'cds_name': 'significant_wave_height_of_first_swell_partition', 'dataset': 'reanalysis-era5-single-levels', 'long_name': 'Significant wave height of first swell partition', 'nc_name': 'p140121', 'type': 'ocean', 'units': 'm'}, 'p140122': {'cds_name': 'mean_wave_direction_of_first_swell_partition', 'dataset': 'reanalysis-era5-single-levels', 'long_name': 'Mean wave direction of first swell partition', 'nc_name': 'p140122', 'type': 'ocean', 'units': 'degrees'}, 'p140123': {'cds_name': 'mean_wave_period_of_first_swell_partition', 'dataset': 'reanalysis-era5-single-levels', 'long_name': 'Mean wave period of first swell partition', 'nc_name': 'p140123', 'type': 'ocean', 'units': 's'}, 'p140124': {'cds_name': 'significant_wave_height_of_second_swell_partition', 'dataset': 'reanalysis-era5-single-levels', 'long_name': 'Significant wave height of second swell partition', 'nc_name': 'p140124', 'type': 'ocean', 'units': 'm'}, 'p140125': {'cds_name': 'mean_wave_direction_of_second_swell_partition', 'dataset': 'reanalysis-era5-single-levels', 'long_name': 'Mean wave direction of second swell partition', 'nc_name': 'p140125', 'type': 'ocean', 'units': 'degrees'}, 'p140126': {'cds_name': 'mean_wave_period_of_second_swell_partition', 'dataset': 'reanalysis-era5-single-levels', 'long_name': 'Mean wave period of second swell partition', 'nc_name': 'p140126', 'type': 'ocean', 'units': 's'}, 'p140127': {'cds_name': 'significant_wave_height_of_third_swell_partition', 'dataset': 'reanalysis-era5-single-levels', 'long_name': 'Significant wave height of third swell partition', 'nc_name': 'p140127', 'type': 'ocean', 'units': 'm'}, 'p140128': {'cds_name': 'mean_wave_direction_of_third_swell_partition', 'dataset': 'reanalysis-era5-single-levels', 'long_name': 'Mean wave direction of third swell partition', 'nc_name': 'p140128', 'type': 'ocean', 'units': 'degrees'}, 'p140129': {'cds_name': 'mean_wave_period_of_third_swell_partition', 'dataset': 'reanalysis-era5-single-levels', 'long_name': 'Mean wave period of third swell partition', 'nc_name': 'p140129', 'type': 'ocean', 'units': 's'}, 'pp1d': {'cds_name': 'peak_wave_period', 'dataset': 'reanalysis-era5-single-levels', 'long_name': 'Peak wave period', 'nc_name': 'pp1d', 'type': 'ocean', 'units': 's'}, 'shww': {'cds_name': 'significant_height_of_wind_waves', 'dataset': 'reanalysis-era5-single-levels', 'long_name': 'Significant height of wind waves', 'nc_name': 'shww', 'type': 'ocean', 'units': 'm'}, 'swh': {'cds_name': 'significant_height_of_combined_wind_waves_and_swell', 'dataset': 'reanalysis-era5-single-levels', 'long_name': 'Significant height of combined wind waves and swell', 'nc_name': 'swh', 'type': 'ocean', 'units': 'm'}}}}

show_markdown_table() → None

Create a Markdown table from the configuration dictionary and print it.