RSGISLib HTTP Tools

Check URLs

rsgislib.tools.httptools.check_url_exists(url: str) → bool

A function which checks whether a url exists on a remote server (i.e., does not return a 404 or similar error code).

Parameters:

url – the URL on the remote server.

Returns:

boolean, true is url exists

Access HTTP APIs

rsgislib.tools.httptools.check_http_response(response: Response, url: str) → bool

Check the HTTP response and raise an exception with appropriate error message if request was not successful.

Parameters:

  • response –

  • url –

Returns:

rsgislib.tools.httptools.send_http_json_request(url: str, data: Dict = None, api_key: str = None, convert_to_json: bool = True, header_data: Dict = None) → Dict

A function which sends a http request with a json data packet. If an error occurs an exception will be raised.

Parameters:

  • url – The URL for the request to be sent.

  • data – dictionary of data which can be converted to str using json.dumps.

  • api_key – if provided then the api-key will be provided via the http header.

  • convert_to_json –

  • header_data – a dict of header information.

Returns:

A dict of data returned from the server.

Download Data

rsgislib.tools.httptools.download_file_http(input_url: str, out_file_path: str, username: str = None, password: str = None, no_except: bool = True)

A function which downloads a file from an URL and saves it to a local file. Note, the file is downloaded with ‘.incomplete’ extension before being renamed to the output file path.

Parameters:

  • input_url – The URL to be downlaoded

  • out_file_path – output file path (including the file name)

  • username – optionally provided a username (Default: None)

  • password – optionally provided a password (Default: None)

  • no_except – If True (Default) then expections are not outputted.

rsgislib.tools.httptools.wget_download_file(input_url: str, out_file_path: str, username: str = None, password: str = None, try_number: int = 10, time_out: int = 60, input_url_md5: str = None) -> (<class 'bool'>, <class 'str'>)

A function which downloads a file from a url using the wget command line tool. If a username or password are provided then both must be provided.

Parameters:

  • input_url_md5 –

  • input_url – string with the URL to be downloaded.

  • out_file_path – output file name and path.

  • username – username for the download, if required. Default is None meaning it will be ignored.

  • password – password for the download, if required. Default is None meaning it will be ignored.

  • try_number – number of attempts at the download. Default is 10.

  • time_out – number of seconds to time out Default is 60.

Returns:

boolean specifying whether the file had been successfully downloaded and a string with user feedback (e.g., error message)

Database Tools

rsgislib.tools.httptools.create_file_listings_db(db_json: str, file_urls: dict[str, str])

A function which builds a JSON database using the pysondb module with a .

Parameters:

  • db_json – The file path for the databases JSON file.

  • file_urls – a dictionary of URLs using the filename as the keys

rsgislib.tools.httptools.download_http_files_use_lst_db(db_json: str, out_dir_path: str, http_user: str = None, http_pass: str = None, use_wget: bool = False, wget_time_out: int = 60, check_file_exists: bool = False)

A function which uses the pysondb JSON database to download all the files recording whether files have been downloaded successful and the output path for the file.

Parameters:

  • db_json – file path for the JSON db file.

  • out_dir_path – the output path where data should be downloaded to.

  • http_user – the username, if required, for the ftp server.

  • http_pass – the password, if required, for the ftp server.

  • use_wget – boolean specifying whether to use wget to download the files. (Default: False).

  • wget_time_out – number of seconds to time out when using wget. (Default: 60)

  • check_file_exists – check if the output file already exists and only download if not present.