11.1.1. Introduction¶

Transfers are the pro-active component of SFTPPlus. While a file transfer service only acts in response to requests from remote clients, transfers actively check local or remote paths and initiate transfers based on changes in those folders.

Transfers are executed by transferring files between a source location and a destination location.

11.1.2. Source Location¶

Source location changes are detected by regularly checking the source folder for changes.

When a change is observed in the monitored source location, the monitor emits one of the following event types:

• File modified

• File/folder created

• File/folder deleted

• File/folder moved

Here is a list of file system changes that are ignored:

• Folder modified (since they are emitted whenever a member is created / removed / moved inside that folder)

• Permissions / Security / Attribute changes

For transfers of type Local filesystem only, created, moved, or removed folders will only emit an informational log event, without triggering the prior transfer command. The same behaviour is present when a file is removed.

For transfers of type other than Local filesystem only, folder events and file removed events are ignored, and no informational event is emitted.

Source events are observed only inside the configured path, and the server will ignore events outside the configured path. While in most cases this is the desired result, it affects 'moved' events. Moved events are only observed inside the location path. When you move a file from a path outside the monitored path, a create event is emitted. When you move a file to a path outside the monitored path, a delete event is emitted.

A transfer can monitor only direct ancestors of a folder, for non-recursive monitors, or all ancestors for recursive ones.

When the source location is not available (for example, its connection failed after all retries) the transfer will become stalled and no files will be transferred.

11.1.3. Source File Transfer¶

Once changes are observed in the source, SFTPPlus will start doing the actual transfer of the file's content.

Multiple files might be created at the source at the same time. All of them are observed as soon as they are stable.

To limit the load to the remote destination, SFTPPlus will only transfer one file at a time. Multiple files are not transferred in parallel at the same time.

All the files detected in the source folder of a transfer are placed into a queue. As soon as the current file is transferred, a new file is picked from the queue.

To thwart erroneous behaviour, SFTPPlus will only keep files in the queue up to 100 000 files. If more files are created and not transferred, the transfer will fail and stop.

11.1.4. Recursive Source Transfer¶

A recursive transfer is one in which you configure the base path for a transfer and SFTPPlus will traverse the whole path and sub-path and transfer files from there.

You can configure a transfer to check the source path for any file which is found in the source path itself or in any of the sub-paths.

The source path can be on a local filesystem or remote location such as a SFTP or FTPS server.

See below for an example:

The files will be transferred to the destination using a folder structure relative to the source path. On destination, the parent folder of each file is automatically created. The full source path hierarchy is not preserved.

For a recursive source transfer, there is a performance issue to consider. For each sub-folder, SFTPPlus will need to make a separate request (local or remote) to get the content and attributes of each member of that folder. When you have 1000 sub-folders, SFTPPlus will make at least 1000 requests.

In contrast, if you set up a non-recursive file transfer scenario, the files will be transferred only from a single folder and will make a single request.

See below for a simplified example:

In the above scenario, only two sub-folders have transfer rules associated with them. This means that SFTPPlus will only monitor filesystem changes within those two sub-folders, and not with the main home folder. Transfers are triggered based on the rules relevant to the two folders. Other benefits to this method beyond performance include; greater control for the administrator for their managed file transfer setup, more detailed logging of events associated with the transfer and monitored folders, and better troubleshooting capacity if the issues are traced back to the transfer rules themselves.

In addition, specific transfer rules can be configured based on custom conditions to trigger the transfer. The conditions could depend on factors such as the schedule, the file type (if not all files), the directory itself, directory actions that trigger the transfer rule and more.

11.1.5. Transferring files generated by slow producers¶

To ensure that your files are always correctly processed by SFTPPlus or other managed file transfer products, ensure that files:

• Do not have the final content.

• Do not have the final name.

• Are not placed in the final location.

When using the product to transfer files that are generated dynamically by other software, it can be very difficult to configure a stable delay that works in all cases.

One option is to set up very long stable intervals, but this comes with the drawback of delaying the processing of all the files.

We have encountered situations where the generated files were written to with interruptions -- significant delays between writes. Since both the file size and last modified timestamp remain unchanged, SFTPPlus considers the file stable after the stable interval passes, thus attempting to transfer it.

Because it is still kept open by the process generating it, you end up either transferring an incomplete file or failing to transfer it if the other process has opened it exclusively.

As a workaround and suggested approach for these cases, we recommend using either a different (temporary) folder for generating the files or using an extension filter for ignoring them. Both are detailed below.

The move/rename operation is atomic on Unix-like systems.

Based on our experience, this operation is quick on all modern operating systems even if for large file sizes.

When the final location or the files with final filenames have the final content, you can set up a short stable delay interval, which makes SFTPPlus act upon the final files as fast as possible.

11.1.6. Delete Source Directories¶

For recursive transfers, you can configure SFTPPlus to automatically delete the parent source directory, when that directory becomes empty.

Below is an example in which, after successfully transferring a file from path E:\\Reports\\2020-12-28\\sales.pdf, the file is removed right away while the parent is removed after 1 hour, if there are no other files in the same source path:

[transfers/15d46e5a-25f6-11e9-87b3-0b5ae5eda581]
source_path = E:\Reports\
delete_source_on_success = True

delete_souorce_parent_delay = 3600

Any other file transferred from the same parent source path will reset the delay after which the delete operation is executed.

11.1.7. Destination Location¶

The destination's locations are accessed only when a file which needs to be transferred is detected on the source location.

When the destination location is not available, the changes detected on the source location are still observed but no transfer is attempted. As soon as the destination location is available again, new changes observed on the destination will be processed.

The transfer will automatically start or resume the destination location when needed. It will not try to start the destination location when the location has failed.

11.1.8. Stop / Cancel a transferred file¶

When a transfer is stopped, any pending file is cancelled.

The current data transfer is stopped and no other actions are executed for that transfer.

11.1.9. Transferring files with a temporary name¶

A transfer can be configured to send / upload / push the file's content using a temporary name, renaming to its initial name once all the data was received by the server.

This is done using the destination_temporary_suffix configuration option.

In the example below, the content of the file E:ReportsINV-2019-08-23.pdf is sent to the remote server as a file /inbox/INV-2019-08-23.pdf.tmp. Once the file was received on the server, it is renamed from /inbox/INV-2019-08-23.pdf.tmp to /inbox/INV-2019-08-23.pdf:

[transfers/15d46e5a-25f6-11e9-87b3-0b5ae5eda581]
source_path = E:\Reports\
destination_path = /inbox

destination_temporary_suffix = .tmp

When destination_temporary_suffix is configured, any file transfer to the destination will be done using a temporary name.

In the case in which you only want a selected set of files transferred using temporary names or want to transfer files using a temporary prefix, you can leave destination_temporary_suffix empty and set destination_path_actions with the temporary-suffix or temporary-prefix actions. See the dedicated documentation for the temporary-suffix action.

11.1.10. Transferring to legacy / restricted remote servers¶

In order to transfer a file, SFTPPlus performs a set of extra operations in addition to the actual request to upload the file's content. For example, it can check if a file already exists at the destination in order to prevent on overwrite.

When dealing with legacy servers, some of these extra operations might not be supported. You can also have to deal with a remote server which restricts operations other than the actual file transfer itself.

SFTPPlus allows sending a file using only the basic file upload operations provided by the transfer protocol.

Below is a list of configuration options which trigger extra requests:

• Set the remote location with idle_connection_keepalive_interval = 0. In this way, SFTPPlus does not trigger idle folder listing in order to force keeping the connection opened.

• Disable destination overwriting rule with overwrite_rule = disable. In this way, SFTPPlus does not perform an extra request to check if a remote file already exists.

11.1.11. Configuring the Destination File Name and Path¶

The destination_path_actions option can be defined for a transfer and will allow using a destination path and file name different than the source path and file name. In this way, you can define transfers to dynamic destination paths.

This is configured using a set of rules which are applied based on expression matching the source path.

To highlight misconfiguration or omission in configuration, when a transfer has a destination_path_actions configured and it tries to transfer files not matching any of the configured rules, SFTPPlus will stop the whole transfer with a failure and will not process any further files.

The base destination path, as defined in destination_path is not affected by the file path and file name transformation.

A transfer can have multiple rules defined for the same path and the actions will be applied in series. The result from one action will be passed to the next one. Having the following configuration:

[transfers/313a9679-bc15-42e9-80ef-e2305959a3d1]
enabled: yes
name: copy-to-remote

recursive: yes
source_path: c:\out\sales
source_filter: *
delete_source_on_success: Yes

destination_uuid: 5f304286-8d56-41f9-ba57-420a2303e90c
destination_path: /Outbox/

destination_path_actions:
  *.pdf, only-filename
  *.pdf, lowercase

the files will be transferred as follows:

c:\out\sales\Report.PDF          -> /Outbox/report.pdf
c:\out\sales\18\Jan\Outcome.PDF  -> /Outbox/outcome.pdf

In the following sections, you will find detailed information for each of the available actions.

recursive: no
source_path: c:\out\sales
destination_path: /Outbox/
destination_path_actions:
  m/(std-|prv-)(.*)/, transform, final_{group.2}

c:\out\sales\std-Report.PDF      -> /Outbox/final_Report.PDF
c:\out\sales\prv-Outcome.txt     -> /Outbox/final_Outcome.txt

source_path: c:\out\sales
destination_path: /Outbox/
destination_path_actions:
  *.pdf, transform, {path.directory_path}/Preview-{path.file_name}
  *.txt, transform, {path.directory_path}/{path.file_base}-Estimation{path.file_extension}

c:\out\sales\Report.PDF          -> /Outbox/Preview-Report.PDF
c:\out\sales\Report.txt          -> /Outbox/Report-Estimation.txt
c:\out\sales\EMEA\Income.PDF  -> /Outbox/EMEA/Preview-Income.PDF
c:\out\sales\EMEA\Income.txt  -> /Outbox/EMEA/Income-Estimation.txt

source_path: c:\out\sales
destination_path: /Outbox/
destination_path_actions:
  *.pdf, transform, {now.iso_8601_compact}-{group.0}
  *.txt, transform, {path.directory_path}/{path.file_base}-{now.iso_8601_compact}{path.file_extension}

c:\out\sales\Report.PDF          -> /Outbox/20210121T244503-Report.PDF
c:\out\sales\Report.txt          -> /Outbox/Report-20210121T244503.txt
c:\out\sales\EMEA\Income.PDF  -> /Outbox/EMEA/20210121T244503-Income.PDF
c:\out\sales\EMEA\Income.txt  -> /Outbox/EMEA/Income-20210121T244503.txt

source_path: c:\out\sales
destination_path: /Out/
destination_path_actions:
  *.pdf, transform, /Staging/{path.source}
  *.txt, transform, final-{path.file_name}

c:\out\sales\Report.PDF          -> /Staging/Report.PDF
c:\out\sales\Report.txt          -> /Out/final-Report.txt
c:\out\sales\EMEA\Income.PDF  -> /Staging/EMEA/Income.PDF
c:\out\sales\EMEA\Income.txt  -> /Out/EMEA/Income-20210121T244503.txt

[transfers/313a9679-bc15-42e9-80ef-e2305959a3d1]
enabled: yes
name: copy-to-remote

recursive: yes
source_path: c:\out\sales
source_filter: *
delete_source_on_success: Yes

destination_uuid: 5f304286-8d56-41f9-ba57-420a2303e90c
destination_path: /Outbox/

destination_path_actions:
  *.pdf, only-filename
  *.txt, no-action

c:\out\sales\Report.PDF          -> /Outbox/Report.PDF
c:\out\sales\Report.txt          -> /Outbox/Report.txt
c:\out\sales\18\Jan\Outcome.PDF  -> /Outbox/Outcome.PDF
c:\out\sales\18\Jan\Outcome.txt  -> /Outbox/18/Jan/Outcome.txt

[transfers/313a9679-bc15-42e9-80ef-e2305959a3d1]
enabled: yes
name: push-to-remote

source_path: c:\out\sales
source_filter: *
delete_source_on_success: Yes

destination_uuid: 5f304286-8d56-41f9-ba57-420a2303e90c
destination_path: /Outbox/

destination_path_actions:
  *.pdf, temporary-suffix, .tmp

[transfers/313a9679-bc15-42e9-80ef-e2305959a3d1]
enabled: yes
name: push-to-remote

source_path: c:\out\sales
source_filter: *
delete_source_on_success: Yes

destination_uuid: 5f304286-8d56-41f9-ba57-420a2303e90c
destination_path: /Outbox/

destination_path_actions:
  *.pdf, temporary-prefix, TEMP_

[transfers/313a9679-bc15-42e9-80ef-e2305959a3d1]
enabled: yes
name: copy-to-remote

recursive: yes
source_path: c:\out\sales
source_filter: *
delete_source_on_success: Yes

destination_uuid: 5f304286-8d56-41f9-ba57-420a2303e90c
destination_path: /Outbox/

destination_path_actions:
  *.pdf, replace-separator, --
  *.txt, replace-separator, _

c:\out\sales\Report.PDF          -> /Outbox/Report.PDF
c:\out\sales\Report.txt          -> /Outbox/Report.txt
c:\out\sales\report.csv          -> Fail as it does not match any rule.

c:\out\sales\18\Jan\Report.PDF   -> /Outbox/18--Jan--Report.PDF
c:\out\sales\18\Jan\Report.txt   -> /Outbox/18_Jan_Report.txt
c:\out\sales\18\Jan\report.csv   -> Fail as it does not match any rule.

[transfers/313a9679-bc15-42e9-80ef-e2305959a3d1]
enabled: yes
name: copy-to-remote

recursive: yes
source_path: /out/sales
source_filter: *
delete_source_on_success: Yes

destination_uuid: 5f304286-8d56-41f9-ba57-420a2303e90c
destination_path: /Outbox/

destination_path_actions:
  *.pdf, lowercase
  *, no-action

/out/sales/Report.PDF          -> /Outbox/report.pdf
/out/sales/Report.txt          -> /Outbox/Report.txt

/out/sales/Jan/Report.PDF      -> /Outbox/jan/report.pdf
/out/sales/Jan/Report.txt      -> /Outbox/Jan/Report.txt

[transfers/313a9679-bc15-42e9-80ef-e2305959a3d1]
enabled: yes
name: copy-to-remote

recursive: yes
source_path: /out/sales
source_filter: *
delete_source_on_success: Yes

destination_uuid: 5f304286-8d56-41f9-ba57-420a2303e90c
destination_path: /Outbox/

destination_path_actions:
  *, fixed-path, //'PDEB.OU11.DRSAFG(+1)'

/out/sales/Report.PDF          -> //'PDEB.OU11.DRSAFG(+1)'
/out/sales/Report.txt          -> //'PDEB.OU11.DRSAFG(+1)'

/out/sales/Jan/Report.PDF      -> //'PDEB.OU11.DRSAFG(+1)'
/out/sales/Jan/Report.txt      -> //'PDEB.OU11.DRSAFG(+1)'

11.1.12. Fallback destination path¶

When using the destination_path_actions to define a transfer with dynamic destination path, some transfer might fail as the computed destination path doesn't exist.

The destination_fallback_path configuration can be used as a catchall method to ensure the remote files are transferred to the destination location.

For example, if you want to pull reports from a server, and automatically route them into directories using destination_path_actions, and one of the route targets does not exist, you can route them to a fallback folder.

A transfer can have multiple actions defined for the same path, and the actions will be applied in series. The result from one action will be passed to the next one.

 [transfers/313a9679-bc15-42e9-80ef-e2305959a3d1]
 enabled: yes
 name: pull-partner-reports

 recursive: no
 source_uuid: 5f304286-8d56-41f9-ba57-420a2303e90c
 source_path: /remote/partner/

 destination_path: D:\reports\acme-inc\others
 destination_fallback_path: D:\reports\acme-inc\unknown

 destination_path_actions:
   m/.*report_(.*).pdf/, transfer, D:\reports\acme-inc\{group.1}\{path.file_name}
   *, no-action

Assuming that the following directories exist on the local filesystem, which is the destination:

D:\reports\acme-inc\sales\    -> receives sales reports
D:\reports\acme-inc\stocks\   -> receives stocks reports
D:\reports\acme-inc\unknown\  -> catchall for other report files
D:\reports\acme-inc\others\   -> catchall for any non-report file.

the files will be transferred as follows:

/remote/partner/report_sales_Jan.pdf    -> D:\reports\acme-inc\sales\report_sales_Jan.pdf
/remote/partner/report_stocks_Jan.pdf   -> D:\reports\acme-inc\stocks\report_stocks_Jan.pdf
/remote/partner/report_toys_Jan.pdf     -> D:\reports\acme-inc\unknown\report_toys_Jan.pdf
/remote/partner/returns_Jan.pdf         -> D:\reports\acme-inc\others\returns_Jan.pdf

Note that report_toys_Jan.pdf is not copied into the others directory. It's copied into the unknown directory. It has the pattern of a report file, but there is no dedicated report folder named toys on the destination to receive it.

11.1.13. Source content conversion on the fly¶

When performing a transfer, SFTPPlus can convert the content of the source file and write the transformed content to the destination while keeping the source file unchanged.

The content conversion is done on the fly / on the wire, without changing the content of the source file.

Below is an example for configuring a transfer to upload a source file with UTF-16 encoding by writing the content on the remote location using ASCII encoding:

[transfers/313a9679-bc15-42e9-80ef-e2305959a3d1]
enabled: yes
name: copy-utf16-remote-ascii

recursive: no
source_path: /out/sales
source_filter: *
delete_source_on_success: Yes

destination_uuid: 5f304286-8d56-41f9-ba57-420a2303e90c
destination_path: /Outbox/

destination_content_actions:
  *.txt, encoding, utf-16-to-ascii
  *, no-action

With the above configuration, a file named /out/sales/Report.txt that is stored on the source disk as UTF-16 is transferred to /Outbox/Report.txt as ASCII content, while the source file is kept as UTF-16.

A file named /out/sales/Report.pdf is transferred to the destination, having the exact content as the source file:

/out/sales/Report.pdf          -> /Outbox/Report.pdf with unmodified content.
/out/sales/Report.txt          -> /Outbox/Report.txt with ASCII encoding.

For the encoding, utf-16-to-ascii action, if the source file has no BOM, UTF-16LE is assumed as the default value. Get in touch with us if you need to handle UTF-16BE as the default encoding.

11.1.14. Transfer States¶

A transfer can be in one of the following states:

• Stopped - the transfer is not active and is not configured to be active at any time.

• Started - the transfer is active and processing files from the source.

• suspended - the transfer was started, but it is outside of its activity schedule.

• Source failed - the source location is not available and files at the source are not processed. Processing will be started as soon as the source location is available, either by fixing the errors and manually starting it or by an automated location reconnection process.

• Destination failed - The destination location is not available. Processing will resume as soon as the destination location is available, either by fixing the errors or manually starting it.

• Failed - a critical error occurred while executing the transfer and the transfer was stopped. Manual intervention is required to restart the transfer.

11.1.15. Location States¶

A location can be in one of the following states:

• Stopped - no transfer is using this location and it was not manually started.

• Started - the location is connected to the remote location.

• Disconnected - the location is not connected to the remote location, but it will auto-reconnect when required by a transfer.

• Failed - the location failed to connect to the remote location. Manual intervention is required to restart the location.

11.1.16. Fault-tolerant transfers¶

SFTPPlus will do its best to perform the requested transfers and under certain circumstances will retry on transfer failures.

From the configuration, you can define the number of times to retry and the time interval to wait after retries.

Once all retries are exhausted, the transfer will fail and SFTPPlus will no longer try to process/transfer that file.

Files that were previously failed to be transferred, can be processed/transferred again by restarting the transfer.

11.1.17. Fault-tolerant source monitoring¶

When a transfer is started, SFTPPlus will check for the required permissions to monitor the source path for changes, the configured path exists and this path is valid.

If the source path was available at startup, but after some time it is no longer available and it fails to get the content of the source path, SFTPPlus will retry the after changes_poll_interval seconds.

This is done for a number of time configured by the retry_count option. An event is emitted on each failure.

Once the source path is available again, the transfer is resumed and new or changed files will be transferred.

The most common errors found on all supported protocols are:

• Source path not found.

• Source path without reading or listing permissions.

• Connection closed while the source was checked.

11.1.18. Resumed transfers¶

Resuming unfinished transfers, either upload or download, is supported across available file transfer protocols as follows:

11.1.19. Hard Links and Symbolic Links¶

Hard links and symbolic links are observed as normal files.

When both the link and original are monitored, changes to the link will trigger notifications for the original file as well as any other hard links pointing to it.

When only the link is monitored, changes to the source will also trigger notifications.

11.1.20. Resolving duplicate / successive events¶

When an event occurs at the source location, it is not signaled right away. A buffer time interval is used to remove duplicate events and emit the event only once the file or folder is stable.

For example, when a large file is copied, the operating system will emit a stream of events after each intermediary change. The monitor will queue these changes and merge them into a single event.

The time frame in which duplicate events are resolved is 1 second. After an event occurs, it waits 1 second before emitting the event. If no event occurs for the same source, the event is emitted. Otherwise, the event is merged and the time interval is extended by another second in which it can be further merged with another event, or emitted.

Here is the list of events that are emitted. Path is used as a generic reference to a file or a folder:

• Existing - for paths that are already present when the process is started.

• Modify - for a path that has a changed size or modify time.

• Delete - for a path that is no longer in the monitored path. Either removed from there or moved to a path that is not monitored.

• Create - for a path that has appeared in the monitored path. Either created there or moved from a path that is not monitored.

• Moved - for a path that was renamed / moved inside the monitored path.

Here are the rules based on which events are resolved:

• Existing + Modify = Existing

• Existing + Delete = Ignore

• Modify + Modify = Modify (and wait for next event)

• Modify + Delete = Delete (without delay)

• Modify + Move = Move + Modify (modify is last)

• Create + Modify = Create (and wait for next event)

• Create + Delete = Discard (file was created for a very short time)

• Create + Move = Create (at the moved path)

• Delete.File + Create.File = Modify (the file was completely replaced)

• Delete.Folder + Create.File = Create.File

• Moved + Created = Moved + Created (with delay)

• Moved + Modified = Moved + Modified (with delay)

• Moved + Deleted = Deleted for the source path

11.1.21. Executing commands using pre/post transfer executables or scripts¶

SFTPPlus allows configuring a transfer to execute at certain transfer steps:

• Execute before processing a source file

• Execute after the success of a file transfer

• Execute after the failure of a file transfer

The configuration from this example, will copy (indicated by delete_source_on_success: No) files from a source (indicated by source_path and source_uuid) to a destination (indicated by destination_path and destination_uuid). A timestamp-always is set in the overwrite_rule in order to always copy the source file with an amended timestamp and prevent accidental overwriting on the destination. It will poll (via changes_poll_interval) for changes to the folder every 5 seconds. After 10 seconds of no changes to a file (indicated by stable_interval), it will consider the file stable and ready to be processed.

A script (saved as a local file at C:\acme\sftpplus\transfer-jobs\after-success.bat) is executed once a file that has been successfully transferred is considered stable.

The script is configured as:

[transfers/7db823d8-05f8-4481-be98-b87a826ded28]
execute_after_success = C:\acme\sftpplus\transfer-jobs\after-success.bat

There are additional processing actions within the custom script. Processing actions include:

• Creating a zipped archive from a target source.

• Affixing a date to the archive.

• Moving the archive to another destination.

An example of the above configuration:

[transfers/7db823d8-05f8-4481-be98-b87a826ded28]
enabled = Yes
name = inbox-transfer
description = Transfer Inbox files.
source_uuid = local-file-system-source-uuid
source_path = C:\acme\sftpplus\accounting\inbox
delete_source_on_success: No
destination_uuid = local-file-system-destination-uuid
destination_path = C:\acme\sftpplus\accounting\inbox-final
execute_after_success = C:\acme\sftpplus\transfer-jobs\after-success.bat
stable_interval = 10
changes_poll_interval = 5
overwrite_rule = timestamp-always
batch_interval = 0
retry_count = 0
retry_wait = 0

11.1.22. Executing pre-/post- transfer destination commands¶

SFTPPlus allows configuring a transfer to execute a list of commands via the following options:

• Execute after success on destination

• Execute before on destination

• Execute before success on destination.

These commands need to be available in the SFTPPlus client-shell implementation.

To check what commands are available, please start the client-shell and run:

> help

To find out more about a particular command, run:

> help COMMAND_NAME

These commands are configured as part of the Transfer section in the Local Manager GUI and it can be added in the configuration text file like below:

[transfers/99feb83f-a99d-42f1-a86c-1562c3281bb6]
name = transfer_sample
execute_on_destination_after_success = rename /path1 /path2; delete /path3
execute_on_destination_before = client-shell-COMMAND /file/path
execute_on_destination_before =
rename /path-to/  /path2
delete /path3

Transfers can execute either a single command or multiple commands.

Multiple commands are separated by a semicolon (;) or on a separate line.

11.1.23. Working with the source monitor¶

[transfers/b904e6a6-c29b-4ccf-8abd-edcae4d3324f]
changes_poll_interval = 3600
stable_interval = 60

11.1.24. Transfer scheduling¶

A transfer can be configured to be active all the time or be active based on a simple scheduler.

The scheduler available in SFTPPlus allows you to define transfer start and transfer stop events in a 24-hour interval or weekly interval. The current scheduler does not know the exact day of the month or year. It only knows about the current time of the day and the current day of the week.

To create a scheduled transfer, you will have to define the start and stop actions at different hours across a day or week.

For example, to have the transfer started daily at 10:00, and stopped at 14:00, you can use the short notation:

[transfers/6b775eec-f98b-4527-811f-fd8c5c71a028]
schedule = 10:00-start, 14:00-stop

or the weekly longer notation:

[transfers/6b775eec-f98b-4527-811f-fd8c5c71a028]
schedule = Monday-10:00-start, Monday-14:00-stop
           Tuesday-10:00-start, Tuesday-14:00-stop
           Wednesday-10:00-start, Wednesday-14:00-stop
           Thursday-10:00-start, Thursday-14:00-stop
           Friday-10:00-start, Friday-14:00-stop
           Saturday-10:00-start, Saturday-14:00-stop
           Sunday-10:00-start, Sunday-14:00-stop

To have the transfer started daily at 10:00, stopped at 14:00, restarted at 17:00, and stopped at 18:00, use:

[transfers/6b775eec-f98b-4527-811f-fd8c5c71a028]
schedule = 10:00-start, 14:00-stop, 17:00-start, 18:00-stop

You can also have the scheduler extended over two consecutive days. To have the transfer started daily at 23:00 and then stopped the next day at 01:00, use:

[transfers/6b775eec-f98b-4527-811f-fd8c5c71a028]
schedule = 23:00-start, 01:00-stop

For defining actions on different days throughout the week, you can add the name of the day before the time value. To have a transfer started at 10:00, stopped at 14:00 every Monday, Wednesday, and Friday, use:

[transfers/6b775eec-f98b-4527-811f-fd8c5c71a028]
schedule = Monday-10:00-start, Monday-14:00-stop
           Wednesday-10:00-start, Wednesday-14:00-stop
           Friday-10:00-start, Friday-14:00-stop

You can also have start and stop events spanning different days. To have a transfer that is active from Monday to Wednesday and then again active from Friday to Saturday, use:

[transfers/6b775eec-f98b-4527-811f-fd8c5c71a028]
schedule = Monday-10:00-start, Wednesday-18:00-stop
           Friday-10:00-start, Saturday-18:00-stop

At this stage, the transfer's changes_poll_interval is not aligned with the schedule. We are currently working on improving this to align the interval with the schedule.

For example, the configuration below will schedule to check files every 2 hours (or 7200 seconds) between 8AM and 6PM:

[transfers/6b775eec-f98b-4527-811f-fd8c5c71a028]
schedule = 08:00-start, 18:00-stop
changes_poll_interval = 7200

If the transfer was started at 13:35, it will do a check at 13:35 instead of what you might expect to be done at 14:00 The next check will be scheduled for 15:35.

Please get in touch if you want a transfer with complex scheduling rules.

11.1.25. Batch transfer operation¶

The default transfer operation for SFTPPlus is to transfer a file as soon as it is detected and considered stable. Each file will be processed as an independent transfer. The pre/post command hooks and events are executed as a separate process for each file.

[transfers/a4747959-25de-470a-b76e-afe08071a70d]
enabled: yes
name: Move report set.
description: Move all report files once the report is finalized.
   A report.desc file is generated by the external reporting process
   to signal that the process is complete.

recursive: Yes
source_uuid: c4b74ed0-b952-4edc-be24-7d8e1e6d8068
source_path: c:\jobs\reports
source_filter: m/*.(pdf|desc)/
delete_source_on_success: Yes

destination_uuid: 92eb50d4-bfc2-4b15-a369-f0d74dc68dac
destination_path: /dropbox/

batch_interval: 0
batch_marker_path: *done.desc

c:\jobs\reports\ACC-123\in.pdf
c:\jobs\reports\ACC-123\out.pdf
c:\jobs\reports\ACC-123\response\report.pdf
c:\jobs\reports\MON-542\office.pdf
c:\jobs\reports\MON-542\storage.pdf
c:\jobs\reports\MON-542\response\report.pdf
c:\jobs\reports\MON-542\done.desc

c:\jobs\reports\ACC-123\in.pdf               - Not transferred
c:\jobs\reports\ACC-123\out.pdf              - Not transferred
c:\jobs\reports\ACC-123\response\report.pdf  - Not transferred
c:\jobs\reports\MON-542\office.pdf           - Transferred
c:\jobs\reports\MON-542\storage.pdf          - Transferred
c:\jobs\reports\MON-542\response\report.pdf  - Transferred
c:\jobs\reports\MON-542\done.desc            - Transferred

c:\jobs\reports\ACC-123\in.pdf
c:\jobs\reports\ACC-123\out.pdf
c:\jobs\reports\ACC-123\done.desc            - Marker file
c:\jobs\reports\MON-542\storage.pd
c:\jobs\reports\MON-542\done.desc            - Marker file

c:\jobs\reports\MON-542\done.desc        - Marker file
c:\jobs\reports\MON-542\storage.pdf      - Triggered by MON-542\done.desc
c:\jobs\reports\MON-542\resp\done.desc   - Marker file
c:\jobs\reports\MON-542\resp\report.pdf  - Triggered by MON-542\done.desc
                                           and MON-542\resp\done.desc

[transfers/a4747959-25de-470a-b76e-afe08071a70d]
enabled: yes
name: Move report set.
recursive: No
source_uuid: c4b74ed0-b952-4edc-be24-7d8e1e6d8068
source_path: c:\jobs\reports
source_filter: *.pdf | *.marker
delete_source_on_success: Yes
destination_uuid: 92eb50d4-bfc2-4b15-a369-f0d74dc68dac
destination_path: /dropbox/

batch_interval: 0
batch_marker_path:
  *expenses.done
  *orders.done

c:\jobs\reports\June-expenses.pdf
c:\jobs\reports\June-expenses.done
c:\jobs\reports\June-orders.pdf

c:\jobs\reports\June-expenses.pdf
c:\jobs\reports\June-expenses.done
c:\jobs\reports\June-orders.pdf
c:\jobs\reports\June-orders.done

11.1.26. Pushing files to mainframe systems¶

The IBM z/OS (OS/390) mainframe operating system provides 2 main methods to access files:

• MVS record-oriented data access methods (PDS PDS/E) backed by sequential or direct access storage devices (DASD).

• UNIX System Services (USS) using regular Unix paths file /parent/file.txt.

The UNIX System Services provide a standard POSIX path which can be handled similarly to the way Linux of macOS file access works.

SFTPPlus supports uploading files to a record-oriented dataset and setting the options for the upload.

Dovetail Co:Z SFTP for z/OS or Tectia SFTP Server for z/OS provide various extensions to facilitate interoperability for MVS datasets.

Below is an example for setting the record format and the logical record length for pushing a file to a server using Tectia SFTP. Each file is appended to the PDEB.OU11.DRSAFG data set. The destination path is set to the UNIX root (/), but it is ignored, as we configure the transfer for a fixed path. File transfer advice strings (FTADV) is used to specify z/OS specific options:

destination_uuid: 44cef305-4f01-43b4-9894-6394e91d50db
destination_path: /
destination_path_actions:
    *, fixed-path, "/FTADV:RECFM=FB,LRECL=1024/___//'PDEB.OU11.DRSAFG(+1)'"

Here is an example for a similar transfer to a Co:Z SFTP server:

destination_uuid: 44cef305-4f01-43b4-9894-6394e91d50db
destination_path: /
execute_on_destination_before: ls /+lrecl=1024,recfm=fb
destination_path_actions:
    *, fixed-path, //PDEB.OU11.DRSAFG(+1)