Documentation

6.8. HTTP(S) file download

The http-pull location allows downloading content from a remote HTTP or HTTPS server. The download is stored in a file.

Note

The http-pull location is designed to be used only as the source for a transfer. Get in touch if you need to push / upload files to a remote HTTP server.

In the simplest form, this allows to perform an HTTP GET request on any URL and store the result into a file.

The location can also pull the content of a remote URL, using a POST` or a PUT request. The request can be made with an empty request body, or with a configured content body. The request can also be made using a set of configurable headers.

For more complex scenarios, SFTPPlus can make the request to an URL, with the request body based on an arbitrary local file. The local file acts as the "seed" for the destination file.

This location is designed for remote HTTP servers that don't provide a way to get the list of available files, and don't have any means to get the attribute of a file, before a download request. This is why the source_path and source_filter configuration options are ignored when used for an http-pull source location.

For standard HTTP based file transfer protocol, SFTPPlus provides dedicated locations, such as the WebDav/SharePoint or the Azure Files location.

For authenticating on the remote HTTP server, the HTTP Basic Auth method is used. Get in touch if you need a different authentication method.

Unlike a web browser, to protect the HTTPS connection you will have to explicitly configure the list of trusted CA and the location of the CRLs.

6.8.1. name

Default value:

Empty

Optional:

No

From version:

2.8.0

Values:
  • Any text.

Description:

Human-readable short text used to identify this location.

6.8.2. description

Default value:

Empty

Optional:

Yes

From version:

2.8.0

Values:
  • Any text.

Description:

Human-readable text that describes the purpose of this location.

6.8.3. type

Default value:

''

Optional:

No

From version:

2.6.0

Values:
  • filesystem - Local file system.

  • sftp - SFTP protocol v3 over SSH v2.

  • ftp - FTP protocol without any encryption.

  • ftpse - Explicit FTPS protocol.

  • ftpsi - Implicit FTPS protocol.

  • webdavs - WebDAV over HTTPS.

  • as2 - AS2 over HTTP or HTTPS

  • azure-file - Azure File Service.

  • smb - SMB / Windows Share

Description:

This option specifies the type of the location. Each type has a set of specific configuration options

6.8.4. idle_connection_timeout

Default value:

300

Optional:

Yes

From version:

3.0.0

Values:
  • Number of seconds

  • 0

Description:

This controls the automatic disconnection from the remote server after the location has not received any file transfer operation requests for the configured number of seconds.

Keep-alive command requests are not counted as file transfer operations. The connection gets automatically disconnected if keep-alive is the only command requested in the configured interval.

Disconnected locations automatically reconnect when a new file transfer operation request is made. For example, when a new file needs to be transferred to the remote server.

If the remote peer closes the connection before the configured timeout, the connection is left closed. It gets automatically reconnected when a new file transfer operation is requested.

Set to 0 to always keep the connection active, by forcing re-connection when the remote server closes the connection.

Note

The idle_connection_timeout is the maximum number of seconds before closing an idle connection to the server. If the remote server decides that the connection is idle and closes the connection, SFTPPlus doesn't try to "challenge" the server, leaving the connection closed. The connection is automatically reopened next time a file needs to be transferred.

6.8.5. idle_connection_keepalive_interval

Default value:

0

Optional:

Yes

From version:

3.0.0

Values:
  • Number of seconds

Description:

Send a keep-alive command every N seconds to avoid having the connection disconnected by the other peer due to inactivity.

Set to 0 to disable keep-alive commands.

The keep-alive command does not reset the idle connection timeout,

6.8.6. connection_retry_count

Default value:

12

Optional:

Yes

From version:

3.9.0

Values:
  • Number of retries

Description:

Number of times to retry connection to the location, when the initial connection fails.

Set to 0 to not retry.

6.8.7. connection_retry_interval

Default value:

300

Optional:

Yes

From version:

3.9.0

Values:
  • Number of seconds

Description:

Number of seconds to wait between connection attempts.

Set to 0 to retry right away without any delay.

6.8.8. url

Default value:

Empty

Optional:

No

Values:
  • text

From version:

4.23.0

Description:

Full URL address of the HTTP server resource or page from where the file content is pulled.

Or if you need a custom port number use:

url = https://files.example.com/path/to/resource

When this is defined as an http:// URL, the TLS/SSL configuration options are ignored.

6.8.9. username

Default value:

Empty

Optional:

Yes

From version:

4.23.0

Values:
  • Text.

Description:

Username used to authenticate to the remote server. Leave it empty if the remote server has no authentication.

6.8.10. password

Default value:

Empty

Optional:

Yes

From version:

4.23.0

Values:
  • Plain text password.

  • Empty.

Description:

This option specifies the password used to connect to the HTTP server.

It is defined in plain text format and sent over the network protected by the HTTPS connection.

6.8.11. request_method

Default value:

GET

Optional:

Yes

Values:
  • GET

  • POST

  • PUT

From version:

4.23.0

Description:

The HTTP method to use when making the request on the remote server.

You can configure it to values like POST or PUT. The value is case-insensitive.

When left empty, it will default to GET.

6.8.12. request_body

Default value:

Empty

Optional:

Yes

Values:
  • Any text

  • seed-file

From version:

4.23.0

Description:

Content used for the body of the HTTP request sent to the remote server.

Leave empty to send a request with an empty body.

When configured to seed-file it will make the request with having the body the content of a local file detected in the source_path configuration of each transfer.

This configuration is ignored when request_method is configured with the GET option.

6.8.13. request_headers

Default value:

Empty

Optional:

Yes

Values:
  • HEADER_NAME: HEADER VALUE

  • Multiple headers, one header per line.

From version:

4.23.0

Description:

The list of headers used for the HTTP request to send to the remote server.

Leave empty to send a request without extra headers.

Each header is defined using NAME: VALUE format. Multiple headers can be defined, one header per line.

In the example below, 2 extra headers are sent:

request_headers =
    X-Stream-Type: reports
    Content-Type: application/octet-stream

The requests will always contain the following standard headers, required for basic HTTP operations:

  • Host

  • User-Agent`

  • Content-Length

  • Authorization (if authentication is enabled)

6.8.14. response_code

Default value:

2*

Optional:

Yes

Values:
  • text

  • matching expression

From version:

4.23.0

Description:

The expected HTTP response code for a successful request.

This is used to detect a failed request and to trigger the retry mechanism.

It can be configured to a fixed value like 200 or 201. It can also be configured to a matching expressing like 2* or 200 | 201, for which in the first case it will match any code starting with 2 or for the second case, it will accept 200 or 201.

When the remote server returns a different response code, it is considered a failure and the transfer is retried, based on the transfer configuration.

When left empty, it will consider any response a success.

6.8.15. response_file_name

Default value:

http-pull.file

Optional:

Yes

Values:
  • fixed text

From version:

4.23.0

Description:

The name of the local file used to store the body of the response, when this is used as the source location.

This option is ignored when the location is used as the destination location.

It can be a fixed name or a template used to generate dynamic file names.

For example, when configured below, it will store the content of the url as a local file named stock-results.xml:

response_file_name = stock-results.xml

6.8.16. ssl_certificate_authority

Default value:

set-on-first-connection

Optional:

Yes

Values:
  • PEM content of a CA chain (Since 3.40.0)

  • PEM content of a pinned public key (Since 5.1.0)

  • Absolute path to a local file

  • set-on-first-connection (Since 5.0.0)

  • pin-public-key (Since 5.1.0)

  • ${MOZILLA_CA_ROOTS} (Since 5.0.0)

  • ${LETS_ENCRYPT_X3_CA}

  • ${MICROSOFT_IT_CA}

  • ${GO_DADDY_G2_G1}

  • disable-identity-security (Since 5.0.0)

From version:

1.6.0

Description:

This is used to validate the identity of the remote server.

Remote server identity can only be validated when the remote address or URL is configured using a fully-qualified domain name. IP-based validation would always fail as this is not a method accepted by Certificate Authorities (CAs).

Configured certificates need to be in PEM format.

Multiple Certificate Authorities can be configured, one after the other. They can be multiple root CAs or intermediate CAs.

Warning

When disable-identity-security is set, the identity of the remote server is not validated. All remote servers are accepted without validating their TLS/SSL certificates. Communication is encrypted and data is protected in transit. This can result in an encrypted connection to an unknown server.

When set-on-first-connection is used, the Certificate Authority of the remote server is configured automatically. The set-on-first-connection configuration value is automatically replaced by the actual CA chain of the remove server on the very first connection. For all subsequent connections, the server identity is validated against the automatically configured CA chain.

When pin-public-key is set, SFTPPlus accepts server certificates that have the same public key as the one discovered during the first connection to this server. This is used to implement certificate and public key pinning. SFTPPlus only pins the public key. This can be used for self-signed server certificates.

You can configure the client to validate the server's identity based on a fixed list of public keys. In this way, you can implement public key pinning. When public key validation is used, the public key infrastructure (PKI) certificate policies are not enforced. For example, the server certificate is accepted even if it's expired or issued for a different hostname.

This can be defined as an absolute path on the local filesystem to a file containing the certificates of the Certificate Authorities used to validate the remote peer.

A series of bundle CAs are distributed with SFTPPlus. They can be configured together and mixed with other CA certificates. The bundle CAs are available under the following names:

  • ${MOZILLA_CA_ROOTS} - All the root certification authorities accepted by the Mozilla's CA Certificate Program

  • ${LETS_ENCRYPT_X3_CA} - For Let's Encrypt X3 certificate authority.

  • ${MICROSOFT_IT_CA} - For all Microsoft IT CA certificates, used by SharePoint Online and other services provided by Microsoft.

  • ${GO_DADDY_G2_G1} - For all GoDaddy Certificate Bundles, G2 With Cross to G1.

Only servers using certificates signed by one of the configured Certificate Authorities are allowed to communicate with this client.

Leave it empty to disable checking the identity of the remote server.

6.8.17. ssl_domains

Default value:

Empty

Optional:

Yes

Values:
  • Fully qualified domain name (FQDN)

  • Comma separated list of fully qualified domain names

  • Empty

From version:

3.42.0 This configuration option defines the domain for which SFTPPlus will request certificates from the Let's Encrypt server.

The same domain can be shared by multiple services.

The domain name is handled as a case-insensitive lower case value.

You can generate a certificate with multiple domain names (Subject Alternative Name - SAN), by defining a comma-separated list of domain names. The first name from the list is used as the common name of the certificate, while the remaining names are used for the SAN extension.

For this option to be used, you need to define a lets-encrypt resource.

6.8.18. ssl_certificate

Default value:

Empty

Optional:

Yes

Values:
  • Absolute path on the local filesystem.

  • Certificate in PEM text format (Since 3.40.0).

  • Certificate in PKCS12 / PXF binary format (Since 4.0.0).

  • Empty

From version:

1.6.0

Description:

This can be defined as an absolute path on the local filesystem to the file containing the SSL certificate or chain of certificates used by the component.

File content should be encoded in the Privacy-Enhanced Mail (PEM) or PKCS12 / PFX formats.

File extension should be .p12 or .pfx for the file to be recognized as a PCKS-12 certificate. The password for the PCKS12 / PFX certificate should be set in the ssl_key_password configuration option.

Note

The path should not be longer than 256 characters.

You can also define the content of the certificate as text in PEM format. In this case the configuration will look as in the following example. It's important to start each line with at least one space character and keep the number of leading spaces constant:

ssl_certificate = -----BEGIN CERTIFICATE-----
    MIICaDCCAdGgAwIBAgIBDjANBgkqhkiG9w0BAQUFADBGMQswCQYDVQQGEwJHQjEP
    ...
    MORE CERTIFICATE DATA
    ...
    JZQaMjV9XxNTFOlNUTWswff3uE677wSVDPSuNkxo2FLRcGfPUxAQGsgL5Ts=
    -----END CERTIFICATE-----

When the value contains both the certificate and the key, the configuration will look as in the following example:

ssl_certificate = -----BEGIN RSA PRIVATE KEY-----
    MIICXgIBAAKBgQDOoZUYd8KMYbre5zZIwR+V6dO2+cCYVS46BHbRbqt7gczkoIWh
    ...
    MORE KEY DATA
    ...
    Wh+QF3UArO8r8RYv3HRcnBjrGh+yEK93wIifVNGgy63FIQ==
    -----END RSA PRIVATE KEY-----
    -----BEGIN CERTIFICATE-----
    MIICaDCCAdGgAwIBAgIBDjANBgkqhkiG9w0BAQUFADBGMQswCQYDVQQGEwJHQjEP
    ...
    MORE CERTIFICATE DATA
    ...
    JZQaMjV9XxNTFOlNUTWswff3uE677wSVDPSuNkxo2FLRcGfPUxAQGsgL5Ts=
    -----END CERTIFICATE-----

This certificate is sent to the remote peer during the SSL/TLS handshake process.

The certificate file can contain both the certificate and the private key, in which case you don't need to set the path to the private key. Only supported for PEM encoding.

The certificate file can contain the full chain of certificates. The targeted certificate should be first in the file, followed by the chained certificates. It will advertise the certificate chain in the same order as listed in the file. Only supported for PEM encoding. (Since 3.22.0)

For server-side components using TLS/SSL secure communication, this configuration option is required. If no value is defined here, the global ssl_certificate value is used.

For the client-side component using TLS/SSL, you can disable sending the certificate as part of the handshake, by leaving this configuration option empty.

6.8.19. ssl_key

Default value:

Empty

Optional:

Yes

Values:
  • Absolute path on the local filesystem.

  • Key as PEM text format (Since 3.40.0).

  • Empty

From version:

1.6.0

Description:

This can be defined as an absolute path on the local filesystem to the X.509 private key file used by this component.

File content should be encoded in the Privacy-Enhanced Mail (PEM) format.

Note

The path should not be longer than 256 characters.

When the value is defined as PEM text, the configuration will look as in the following example:

ssl_key = -----BEGIN RSA PRIVATE KEY-----
    MIICXgIBAAKBgQDOoZUYd8KMYbre5zZIwR+V6dO2+cCYVS46BHbRbqt7gczkoIWh
    ...
    MORE KEY DATA
    ...
    Wh+QF3UArO8r8RYv3HRcnBjrGh+yEK93wIifVNGgy63FIQ==
    -----END RSA PRIVATE KEY-----

If ssl_certificate is not defined, any value defined for this ssl_key configuration is ignored and the global ssl_key value is used.

If the value defined in ssl_certificate option already contains the private key, this option can be omitted by leaving it empty.

6.8.20. ssl_key_password

Default value:

Empty

Optional:

Yes

Values:
  • Password as plain text.

  • Empty

From version:

1.7.19

Description:

This is used to define the password of the private key, when the private X.509 key is stored as an encrypted file.

Leave it empty to not use a password for the private key file.

6.8.21. ssl_certificate_revocation_list

Default value:

Empty

Optional:

Yes

Values:
  • Comma separated list of CRL paths or HTTP URLs.

  • crl-distribution-points

  • ${MICROSOFT_IT_CRL}

  • Empty

From version:

1.6.0

Description:

It defines the locations from where one or more CRLs will be loaded.

Multiple CRLs are defined as a comma separated list.

It supports local files with absolute paths, in either of the following formats:

  • file:///unix/absolute/test-ca.crl

  • file://c:\\windows\\absolute\\test-ca.crl

Retrieving the CRL over HTTP is also supported. The HTTP request is done using non-persistent HTTP/1.1 connections. The URL will look as follows:

  • http://example.com/some.crl

CRL distribution points (CDP) are supported by using the crl-distribution-points configuration value.

When CRL distribution points are configured, the local certificate defined at ssl_certificate needs to have the CDP extension. The CDP advertised in the local certificate is loaded at startup in order to validate the configuration.

The distribution points configuration is mutually exclusive with local file or HTTP url configurations. When the certificate revocation list is configured to use CDP, all other configured CRL location are ignored.

Warning

HTTP redirection is not yet supported for CRL URLs. You have to configure the exact URL for the CRL.

Leave it empty to disable certificate revocation checks.

The certificate revocation list can only be used when the component is configured with CA certificates stored in a single file in PEM format.

When multiple or chained CA certificates are configured the CRL is only checked for the peer's certificate and not for the CA certificate or for an intermediate CA.

Warning

CDP publishing Delta CRL are not supported.

Note

If the certificate defines multiple HTTP-based distribution points in the CDP extension, only the first HTTP URI is used. All non HTTP or the other HTTP URIs are ignored.

The CRL file should be stored in PEM or DER format.

Note

This option is ignored if ssl_certificate_authority is not enabled.

6.8.22. ssl_certificate_revocation_list_refresh

Default value:

0

Optional:

Yes

Values:
  • Number of seconds

  • 0

From version:

2.8.0

Description:

This defined the number of seconds after which a configured CRL is reloaded by this component.

When set to 0, the CRL file is initially loaded at startup and then loaded again after the Next Update field advertised in the CRL.

If the Next Publish extension is present in the CRL and this option is set to 0, the CRL will be loaded again at the date and time specified in the Next Publish extension.

If the CRL does not advertise the Next Update field you will have to configure a number of seconds after which the CRL should be reloaded, otherwise you will get a configuration error.

For example, a value of 86400 means that the CRL will be re-read after one day.

For more details about the CRL reloading see the documentation for CRL reloading rules

Note

This option is ignored if ssl_certificate_authority is not enabled.

6.8.23. ssl_cipher_list

Default value:

secure

Optional:

Yes

Values:
  • List of SSL/TLS ciphers in OpenSSL format.

  • secure

From version:

1.7.4

Description:

This defined the list of ciphers accepted by this component while communicating over the network.

The special keyword secure contains all the algorithms that we currently consider secure.

Connections are closed if the remote peer has no common cipher in its list of configured ciphers.

When left empty, it will default to the secure configuration.

More information about the accepted values can be found at the cryptography guide

The format for this value is the same as the one used for defining the OpenSSL cipher list. More information can be found on the OpenSSL site.

6.8.24. ssl_allowed_methods

Default value:

secure

Optional:

Yes

Values:
  • secure

  • all

  • tlsv1.0

  • tlsv1.1

  • tlsv1.2

  • tlsv1.3

From version:

1.7.4

Description:

This defines the comma-separated list of SSL and TLS methods that are accepted by this component during the secure communication handshake.

Set this to secure to allow only the TLS methods that are currently considered secure. For now, this is TLS 1.2 and TLS 1.3 but this might be changed in the future. Any other configured value is ignored.

Set this to all to allow any supported SSL or TLS method. Any other configured value is ignored.

Currently, the following methods are officially supported:

  • tlsv1 or tlsv1.0, which is TLS 1.0.

  • tlsv1.1, which is TLS 1.1.

  • tlsv1.2, which is TLS 1.2.

  • tlsv1.3, which is TLS 1.3.

Note

SSLv3 is still supported, but highly discouraged, due to the SSLv3 POODLE vulnerability. In the case that you need to interact with an old SSL implementation that only supports SSLv3, it is highly recommended to force the usage of the non-CBC cipher RC4-SHA by configuring as:

[services/681f5f5d-0502-4ebb-90d5-5d5c549fac6b]
ssl_cipher_list = RC4-SHA

Support for SSLv3 will be removed in future versions.

SSLv2 is no longer supported since it is not secure.

In version 2.8.0, the following new methods were added: tlsv1.0 (alias for tlsv1), tlsv1.1 and tlsv1.2

Support for tlsv1.3 was added in version 3.47.0.

Prior to version 4.17.0, this was configured as a space separated value.