RSGISLib FTP Tools

Transverse Tree

rsgislib.tools.ftptools.get_ftp_file_listings(ftp_url: str, ftp_path: str, ftp_user: str, ftp_pass: str, ftp_timeout: int = 300, try_n_times: int = 5, mlsd_not_avail: bool = False)

Traverse the FTP server directory structure to create a list of all the files (full paths).

Note, it is not good practise to hard code passwords into you scripts. It would be recommended that you get the password from an environmental variable or config file rather than hard code the password.

Parameters:

  • ftp_url – the url for the ftp server

  • ftp_path – the path on the ftp server where to start the transverse of the tree.

  • ftp_user – The username for the ftp server

  • ftp_pass – the password for the ftp server

  • ftp_timeout – in seconds (None and system default will be used; system defaults are usual above 300 seconds)

  • try_n_times – if server connection fails try again (sleeping for 5 secs in between) n times for failing.

  • mlsd_not_avail – Option to support older FTP servers which don’t support the mlsd function. Default is False but if you get errors then suggest that you try setting this to True and trying again.

Returns:

directory by directory and simple list of files as tuple

rsgislib.tools.ftptools.create_file_listings_db(db_json: str, ftp_url: str, ftp_path: str, ftp_user: str, ftp_pass: str, ftp_timeout: int = 300, try_n_times: int = 5, mlsd_not_avail: bool = False)

A function which transverses the directory structure on an ftp server and saves the result to a JSON database using the pysondb module.

Note, it is not good practise to hard code passwords into you scripts. It would be recommended that you get the password from an environmental variable or config file rather than hard code the password.

Parameters:

  • db_json – The file path for the databases JSON file.

  • ftp_url – the url for the ftp server

  • ftp_path – the path on the ftp server where to start the transverse of the tree.

  • ftp_user – The username for the ftp server

  • ftp_pass – the password for the ftp server

  • ftp_timeout – in seconds (None and system default will be used; system defaults are usual above 300 seconds)

  • try_n_times – if server connection fails try again (sleeping for 5 secs in between) n times for failing.

  • mlsd_not_avail – Option to support older FTP servers which don’t support the mlsd function. Default is False but if you get errors then suggest that you try setting this to True and trying again.

Filter Files

rsgislib.tools.ftptools.filter_file_listings_db(db_json: str, filter_vals: List[str], out_db_file: str, filter_path=False)

A function which filters the database to find particular files of interest. A list of strings is provided and comparison will be done using the python ‘in’ operator and at least one of the filter string has to be within the file path/name for it to match.

Parameters:

  • db_json – file path for the JSON db file.

  • filter_vals – A list of strings which will be used to select files.

  • out_db_file – output file path for JSON db file.

  • filter_path – boolean, if true then filter will be applied to full path, if False then just the file name (default; False)

Download Data

rsgislib.tools.ftptools.download_ftp_file(ftp_url: str, remote_file: str, local_file: str, ftp_timeout: int = 300, ftp_user: str = None, ftp_pass: str = None, print_info: bool = True) → bool

A function to download a file from an FTP server.

Parameters:

  • ftp_url – The remote URL for the ftp server.

  • remote_file – the remote path on the ftp server for the file to be downloaded

  • local_file – the local path to where the file should be downloaded to.

  • ftp_timeout – the timeout for the download. Default: 300 seconds.

  • ftp_user – the username, if required, for the ftp server.

  • ftp_pass – the password, if required, for the ftp server.

  • print_info – bool for whether info should be printed to the console (default: True)

Returns:

boolean as to whether the file was successfully downloaded or not.

rsgislib.tools.ftptools.download_curl_ftp_file(ftp_url: str, remote_file: str, local_file: str, ftp_timeout: int = 300, ftp_user: str = None, ftp_pass: str = None, print_info: bool = True) → bool

A function to download a file from an FTP server using curl.

Parameters:

  • ftp_url – The remote URL for the ftp server.

  • remote_file – the remote path on the ftp server for the file to be downloaded

  • local_file – the local path to where the file should be downloaded to.

  • ftp_timeout – the timeout for the download. Default: 300 seconds.

  • ftp_user – the username, if required, for the ftp server.

  • ftp_pass – the password, if required, for the ftp server.

  • print_info – bool for whether info should be printed to the console (default: True)

Returns:

boolean as to whether the file was successfully downloaded or not.

rsgislib.tools.ftptools.download_files_use_lst_db(db_json: str, out_dir_path: str, ftp_timeout: int = 300, ftp_user: str = None, ftp_pass: str = None, create_dir_struct: bool = False, use_curl: bool = False)

A function which uses the pysondb JSON database to download all the files recording whether files have been downloaded successful and the output path for the file.

Parameters:

  • db_json – file path for the JSON db file.

  • out_dir_path – the output path where data should be downloaded to.

  • ftp_timeout – the timeout for the download. Default: 300 seconds.

  • ftp_user – the username, if required, for the ftp server.

  • ftp_pass – the password, if required, for the ftp server.

  • create_dir_struct – boolean specifying whether the folder structure on the ftp server should be maintained within the out_dir_path (True) or ignored and all the individual files just downloaded into the output directory as a flat structure (False; Default).

  • use_curl – boolean specifying whether to use CURL to download the files. (Default: False).

Upload Data

rsgislib.tools.ftptools.upload_ftp_file(local_file: str, ftp_url: str, remote_path: str, ftp_timeout: int = 300, ftp_user: str = None, ftp_pass: str = None, print_info: bool = True) → bool

A function for uploading a file to an ftp server.

Parameters:

  • local_file – the path to the local file to be uploaded.

  • ftp_url – the url for the ftp server for the upload.

  • remote_path – the path on the ftp server for the file to be uploaded to. this should not include the file name which will be taken from the local_file.

  • ftp_timeout – the timeout for the download. Default: 300 seconds.

  • ftp_user – the username, if required, for the ftp server.

  • ftp_pass – the password, if required, for the ftp server.

  • print_info – bool for whether info should be printed to the console (default: True)

Returns:

boolean as to whether the upload was successfully completed or otherwise.