FTP Connector

Usage

The FTP connector supports the FTP, FTPS, and SFTP transfer protocols.

Connector types

There are the following configuration options for FTP connectors:

Input Connector

Fetches files from an FTP server.
Output Connector

Writes files to a directory on an FTP server.

Module Variables

When executing an FTP connector, the following module variables are set:

Alias

Name of the alias.
isAliasSelected

If set to true, the alias is used instead of user name and password.
ReadFileDate

Last change date for the input file in the format dd.mm.yyyy hh:mm:ss.
ReadFileName

Name of the input file including the extension.
ReadFileSize

Size of the input file in bytes.
ReadFileWildcardname

Outputs the content of the wildcard used when configuring the FTP connector in the file name.
WriteFileDate

Only for the FTP output connector. Date of generation of the output file in the format dd.MM.yyyy HH:mm:ss.
WriteFileName

Name of the output file.
WriteFileWildcardname

Only for the FTP output connector if timestamps or wildcards are used in the file names to be generated. Outputs the content of the wildcard used when configuring the FTP output connector in the file name.

Refer to Workflow Variables and Mappings.

Dialog Descriptions

Dialog FTP Connector Properties

The properties dialog is for querying the access data for the FTP server.

Different control elements are displayed depending on the selected protocol.

Basic configuration

Protocol: Select one of the following protocols:
- FTP (File Transfer Protocol)
  
  This exchange is not secure.
  
  Between two hosts connected by TCP/IP, a connection is established via two ports:
  - One port is used to exchange commands.
  - The other port is used for exchanging unencrypted data.
- FTPS (FTP via SSL)
  
  Extension of FTP.
  - The control port is secured by SSL.
  - Data exchange is not secured by default. The FTP Connector enhances the FTPS protocol so that data transfer can also be secured using SSL.
- SFTP (SSH FTP)
  
  Adds authentication of users and servers to the encrypted data exchange.
  SFTP is easier to use through NAT gateways than FTPS.
Server name

Domain name of an FTP server, for example, ftp.example.com or name of the proxy server (<proxyserver>).
Port
- FTP and FTPS
  
  Default is port 21.
- SFTP
  
  Default is port 22.
  
  If the connection test fails, ask your administrator which port should be used to reach the FTP server.
  
  You use the Default button to restore the default value following port changes.
Connection mode (only for FTP/FTPS)
- Active
  
  The client opens a random port and informs the server of the port number and its own IP address using the PORT command.
- Passive
  
  The client sends a PASV command. The server opens a port and passes it to the client along with the IP address.
- Auto
  
  In almost all cases, Auto is the correct setting. Select Active or Passive only if your administrator has configured the FTP server accordingly.
  
  Enable automatic substitution of the remote host IP (only for FTP or FTPS and if the connection mode is actually Passive - either in Passive mode or if it is resolved to be Passive while using Auto)
  
  Select this option to enable the automatic substitution of the remote host IP for passive mode connections. This checkbox is selected by default.
  
  You can also control this option using the module property IsAutoSubstitution in the variable mapping.
Transfer type (only for FTP/FTPS)

Specify whether the files to be transferred contain binary or ASCII data.
Locale

Server locale. The server locale defines the choice of the character set and date specification, amongst others.

Encoding

Specify the encoding of the control channel for the communication connection between the connector and the FTP server.

The control channel is used to send commands, including file names, as character strings. In order to make sure that special characters (for example umlauts) are correctly transmitted, defining the encoding is essential since it can differ from server to server.

Enable network speed limit (only for FTP/FTPS)

Select this option to limit the network bandwidth. This checkbox is deselected by default.

Speed (KB/s)

If Enable network speed limit checkbox is selected configure the speed by specifying the required value (whole multiples of 256 KB/s).
Reuse SFTP connections

Select this option to cache SFTP sessions during workflow execution time. This may improve the processing time of an SFTP connector.

Pool size

If the Reuse SFTP connections checkbox is selected, configure the number of SFTP sessions to be cached. The higher the number the more sessions get cached. It requires JVM memory to be allocated for storing the session in the cache.

Limitations

The SFTP session cache has certain limitations:
- The cache key consists of:
  - INUBIT workflow execution processID
  - SFTP server hostname
  - SFTP server username
- Each session is only cached (process ID) during a workflow execution and is removed from the cache when the workflow execution is complete.
- A session can be reused across all workflows that are part of the same workflow execution (processID).
- A session cannot be reused within the same or different workflows of another workflow execution.
  
  By default, an asynchronous Workflow Connector generates a new processID during the transition.
  
  Potential error
  
  As a session reuse is not possible across different workflow executions, each execution will create it’s own SFTP session. When caching is enabled, the stored sessions have an extended duration compared to an uncached execution. This can cause problems with the corresponding FTP server, since some servers limit the amount of parallel sessions opened (e.g. with same authentication).
  
  If the error message SSH_MSG_DISCONNECT: 2 Too many users appears, it indicates that the FTP server limits the amount of multiple parallel sessions. Switch to an uncached execution and try again.

Proxy settings (for input and output connectors only when SFTP protocol is chosen)

Enable Proxy Settings

Activate this checkbox to enter proxy settings.
Server name

Host name or IP address of the proxy server
Port

Port to access the proxy server
Authentication required

Activate this checkbox if an authentication is required to connect to the proxy server.
Login name

Username to connect to the proxy server
Password

Password of the user to connect to the proxy server

Authentication

If you have a separate area for the FTP server with a required login, enter the login data into the relevant fields.

When using an FTP proxy, specify your data using the following syntax:

Login: <ftpuser>@<ftpserver>
Password: <ftppassword>

To access the server anonymously, click Anonymous login. This sets the username to anonymous and the password to anonymous@example.com.

For authentication, you can also use credentials managed by the Credentials Manager. Refer to Using the Credentials Manager for Authentication.

SSL configuration (for FTPS)

Support older SSL versions (from SSL 3.0 onwards)

When activated, SSL versions as of SSL 3.0 are supported.
Use implicit FTPS

This term describes an older FTPS version that is no longer contained in the current specification. Ask the administrator of the FTPS server whether implicit FTPS is available. This type of FTPS runs on port 990 by default.

For compatibility reasons, the protection buffer size (PBSZ) parameter with value 0 is always sent when FTPS is selected.

The following fields are valid both for the current version of FTPS and for implicit FTPS.
Security mechanism

Select a protocol for data transfer:
- SSL or TLS (Secure Sockets Layer/Transport Layer Security)
  
  TLS 1.1 and 1.2 are supported. The best possible protocol is used.
  
  For security reasons, TLS 1.0 or lower is not supported.
- TLS_C
  
  TLS incl. compression of encrypted data.
Secured data transfer

By default, only transfer in the control channel is encrypted when using FTPS. The FTP Connector also enables the encryption of data being transferred in the data channel.

When activated, data is encrypted before exchange.
SSL configuration

For authenticating servers and/or clients. Refer to Dialog SSL Configuration.

SSH configuration (for SFTP)

Enable additional SFTP log output

Select this option if you want to see the key exchange information in the trace log.
Disable SHA-1

Select this option if you are using older services that do not support SHA-1 encryption. This avoids errors during the execution of the connector, thus preventing a workflow termination.
Validate DNSSEC

Select this option if you want to validate the SFTP server with a DNSSEC-enabled domain.
Server authentication
- If not activated, the FTP Connector authenticates to the SFTP server of your business partner using the credentials specified at the FTP Connector. The identity of the remote SFTP server is not verified. Hence, this option is not recommended.
- If activated, the identity of the given SFTP server of your business partner is verified. For this purpose, you must add the SFTP server and the public key to a known_hosts file, upload this file into the FTP Connector, and/or enter the path to this file using the following options.
  
  If you have both uploaded the known_hosts file and specified a path to the known_hosts file, the file specified with the path takes priority.
  - Upload known_hosts file
    
    Use the button to add the known_hosts file containing trusted hosts. The file is stored with the FTP Connector and is directly available each time a connection to the SFTP server is established.
  - Path
    
    Path to the known_hosts file.
    
    Advantages: The file remains in the file system. Changes to the known_hosts file are taken into account by the FTP Connector module for each execution.
Client authentication

When activated, the FTP Connector must authenticate itself. To add the private key or keystore, click Upload private key. The key must be available in OpenSSH or in SECSH format.

In order to load the keystore, you must possibly enter a password.

Configuring SFTP Cipher Algorithms

Parameter

Description

SFTP cipher algorithms

When using SFTP connections you can adjust the used cipher algorithms within the Workbench. This configuration is applicable for a single SFTP connection. If a connection fails due to a missing or unknown algorithm, you can add this missing information in the Workbench.

Click the administration guide 176 0 ] button to open the Property table with all existing JSch keys and values.

Do not delete existing values as it may cause problems while making SFTP connection or execution.

For information on adding new keys and identifying the key refer to Configuring SFTP Cipher Algorithms.

Connection test

Test connection

For testing whether the connection can be successfully established using your configuration.

Changing file permissions (Linux only)

The FTP Connector has a hidden Module Property SSHumask, which enables to influence the permissions of files used in your Technical Workflow. In order to use this function, first you have to create the Module Property SSHumask, refer to Adding Module Properties.

Then create a new mapping in the dialog Variable mapping of the FTP Connector, refer to Dialog variable mapping.

Choose Module Property SSHumask as Target in the dialog Variable mapping. The source, for example, can be a Variable of the type STRING or a Static Value, both must consist of 3 numbers, for example, 133.

Dialog Executing FTP Commands

You use the dialog to specify one or more commands that are to be executed before or after data transfer. For example, you can create directories or delete files before data transfer takes place.

The commands must be written in uppercase. If you use multiple commands, each command has its own line.

The commands must comply with standard syntax, refer to https://en.wikipedia.org/wiki/List_of_FTP_commands for FTP/ FTPS.

Possible SFTP pre/post commands are: mkdir, cd, rename.

Do not use the following commands: delete, mkdir, get, and put. These commands generate errors.

Switching to another directory:

When you log in to an FTP server, your home directory is displayed. This is normally a directory such as /ftproot/group/username. As a user, you can only see your username directory. Some FTP servers are configured in such a way that absolute path specifications are required when you switch to a directory below the current directory in the hierarchy. To switch to a subdirectory such as /ftproot/group/username/test then e.g. the following command is required:

ftp>CWD /ftproot/group/username/test

Dialog Input File Names

(Input Connector)

This dialog offers the following options:

Mode

File mode

For each execution, a file in the specified directory is read (as long as its name corresponds to the pattern entered in the File > Name field). The file that is read depends on your specifications for Read order.

Directory mode

All files in the specified directory are read (as long as their file names correspond to the pattern entered in the File > Name field. In directory mode, an IBISDirectory XML output message is always generated. This means that the Technical Workflow starts in each case.

In directory mode, the IBISDirectory tag contains the absolute path for all SFTP server types. For servers that throw an error, the absolute path will be empty. Customers who face this issue have to modify their existing workflows in such a way that the absolute path is provided by a workflow variable. Customers are not affected if they use SFTP server types that do not cause this error.

Directory

Name

Absolute path to the directory from which the FTP Connector is to transfer files.
With the icon , you can open the file chooser dialog. The file path of the selected file is adopted to the directory field, the file name to the file field. If you want to transfer files directly from the home directory of your FTP server, leave this field blank.
The file chooser dialog is supported for:

Protocols: FTP

Operating systems: Windows 7; Linux
Force directory deletion after reading

If you select this option, all files, sub folders, empty sub folders, and the directory itself defined above are being deleted after reading.
Delete empty subdirectories after reading (for SFTP protocol only)

Option is disabled if Force directory deletion after reading is checked.

If you select this option, an empty directory is deleted after reading. If all files have been read and deleted from a given directory and the directory becomes empty, the directory is deleted.

File

With optional wildcard

If you select this option, you can specify the complete file name or use a wildcard (*) in the file name. Example: If you specify *.xml, all XML files in the specified directory are transferred.
With regular expression

If selected, you can use regular expressions to describe a pattern that applies to file name character strings of files to be read. This provides a powerful tool for selecting files.

Regular expressions must be delimited by forward slashes (/).

Example: /(\d\d)\.(\d\d)\.(\d\d\d\d)/

Refer to Regular Expressions.

Name

File name. Can contain a wildcard or regular expression.
Use wildcard content of the preceding File or FTP Connector

This option is active only if you select the With optional wildcard checkbox. If you select this option, the wildcard content generated by a previously executed File or FTP Connector is used. Example:

Test*.xml was specified in the File name field in the configuration of a previously executed File Connector. The File Connector forwarded a file with the name Test5.xml as the output message. In this case, the wildcard content is the digit 5. If you also specify a wildcard in the current connector, such as Output*.xml, the wildcard is replaced by 5 and the output of the current connector is Output5.xml.

Rename files after reading

If you want files to be renamed following data transfer, select this checkbox. You can change the filename by adding a prefix or suffix:

Prefix: empty textfield by default

Suffix: textfield with .Ok by default

All files are downloaded and renamed on the server one by one. If a file fails to be read or renamed, processing is stopped immediately. Files that have already been renamed will remain on the server.

You can either select Rename files after reading or Delete files after reading. Selecting both is not possible.

Delete files after reading

If you want files to be deleted following data transfer, select this checkbox.
Throw error if delete fails

If an error occurs, module execution and the workflow are continued. If this option is activated and an error occurs, an error entry for the workflow with this module is displayed in the Queue Manager. If an error output is configured, it runs. Otherwise, the execution of the module is terminated.
Maximum number of scheduled executions per call

You use this specification to define the maximum number of times the workflow is to be started per triggering event. The triggering event is the reaching of a particular time point (if the scheduler is activated). The workflow is then started as many times as necessary until either

there is no more data to be fetched/sent or
the number defined in Maximum number of executions per call has been reached

By restricting the maximum number of executions, you control the system load on the server.

In Test mode, this specification is ignored and the workflow is only started once.

The option is only active when the scheduler is activated.

Read order

Defines the read order and sorting when reading files.

File check

Abort workflow if file does not exist:
- In test mode:
  
  If access to the specified file fails, the workflow execution is terminated with an error.
- Scheduled input FTP Connector:
  
  According to the interval specified, the FTP Connector checks if a file is available. The workflow is only started if a file exists.
- Input FTP Connector in the middle of a workflow:
  
  If the file does not exist, the execution is continued with an empty message. No entry is made in the log.
Create XML output message with status information on whether the file exists or not

The FTP Connector specifies an XML file that contains a single <FileExists> attribute. This value is false if the specified file does not exist and true if the specified file exists.
Create XML output message with status information that the file does not exist; otherwise, abort workflow

If the input file does not exist, the FTP Connector outputs an XML file with the element FileExists="false". In all other error cases, the workflow execution terminates with an error.

Directory configuration

(Only if directory mode is selected)

File number limit

Maximum number of files that can be read.
Limit total size

Maximum total size of files to be read.
Inclusive Data

If selected, the entire content of the file is read. If not selected, only the file metadata are read, for example, name, file size, and date.
Include directories

If selected, in addition to the files in the specified directory the names of the first-level subdirectories below the specified directory are read and added to the output file. If not selected, only the files are read from the specified directory and added to the output file.
With subdirectories (selectable only if Include directories is enabled)

If selected, all files in the specified directory and in all subdirectories are read, also the names of all subdirectories are read, and all the data are added to the output file. If not selected, the Include directories option settings take effect.

Dialog Output File Name

(Output Connector)

This dialog offers the following options:

Input Format

While uploading multiple files to the FTP server, the input format can be set.

Data

The input messages consist of data that should be written into an FTP server without being changed.
IBISDirectory-XML

Select this option to upload multiple files in the IBISDirectory format. Using this option, the files can also be stored in a chosen directory.

IBISDirectory-XML input format

Directory
- Name
  
  Specify the directory on the FTP server that the FTP Connector has to use when writing files. With the icon, you can open the folder chooser dialog. The path of the selected folder is adopted to the directory field.
  The file chooser dialog is supported for:
  
  Protocols: FTP
  
  Operating systems: Windows 7; Linux
- Create non-existent directories
  
  If activated, missing directories on the FTP server are generated.

Write mode

Overwrite existing file:

If a file with the same name exists, it is overwritten.

Append data to file

If you select this option, the output message is appended to the existing file each time the FTP Connector is executed. This can be useful if you want to log cumulative log data in a file. This mechanism cannot be used if wildcards are used in file names. Access rights cannot be changed.

This mechanism cannot be applied if you use wildcards in the file name. Not all FTP servers support this mechanism. If the server in question does not support this mechanism, an exception is generated.

Skip if file already exist

Skips writing into an existing file but creates a new one if no such file exists.

In the watchpoint result file, the ftpConnector.skippedFiles workflow variable contains the names of the already existing files.

Module output configuration
- Set module input message as output message
  
  If this option is activated, the input message is appended to the output message.
Permission (for OpenSSH only)

If this checkbox is activated, the file permissions are stored. Otherwise, the permissions are not set during execution. In this section, you can set the access rights for the file(s) to be uploaded. By default, for a new module, only the owner has write permissions. Group and others have only read permissions.

Permissions cannot be set if the Add data to file option is set.

When having imported modules from an older INUBIT version, check whether the permissions are set correctly.

To set/deprive permissions, check/uncheck the checkboxes for read ®, write (W), and/or execute (X) permission for the Owner, the Group, and Others. Additionally, the permissions are displayed as octal values.

Data input format

File

Name

Name of the file the Output Connector writes.

You can use an asterisk (*) in the file name to insert a counter at the position in question in order to prevent files from being overwritten. Example: /Outputdata/output*.xml.

If you do so, files with the names output211.xml, output212.xml, output213.xml, and so on are generated on the FTP server.

You can have a date, timestamp, or process ID added to the file name:

Date

Definition without date format

file_%TimeStamp%%.xml generates file_06082008-151047.xml with
file = filename
12122002= Date formatted as ddMMyyyy
151047= time formatted as hhmmss
Definition with date format hh:

File_%TimeStamp%hh%.xml generates File_11.xml
Definition with date format ddMM:

File_%TimeStamp%ddMM%.xml generates File_2309.xml

Timestamp

The timestamp additionally contains a specification in milliseconds:
file_%TimeStamp%ddMMyyyy-HHmmss-SSS%.xml results in file_12122002-151304-027.xml

Process ID
file_PID_%ProcessId%.xml

Use wildcard content of the preceding File or FTP Connector

If you select this option, the wildcard content generated by a previously executed File or FTP Connector is used.
Use the file name of the previously executed File Connector

If you select this option, the file name used by a previously executed File Connector is used. Example: Text.xml was specified in the Directory and file name field in the configuration of a previously executed File Connector. The File Connector forwards a file called Test.xml as the output message. This file name is also used here. If you specify an additional file name in the Name field, it is ignored.

Create file with .tmp extension and rename after transfer is completed (for SFTP only)

If this option is activated, the file to be written on the SFTP Server initially gets the .tmp extension while it is transferred. Once the transfer is completed, the file is renamed to the configured file name.
Write mode
- Overwrite existing file:
  
  If a file with the same name exists, it is overwritten.

Append data to file: If you select this option, the output message is appended to the existing file each time the FTP

Connector is executed. This can be useful if you want to log cumulative log data in a file. This mechanism cannot be used if wildcards are used in file names. Access rights cannot be changed.

Module output configuration
- Set module input message as output message
  
  If this option is activated, the input message is appended to the output message.
Permission (for OpenSSH only)

If this checkbox is activated, the file permissions are stored. Otherwise, the permissions are not set during execution.

In this section, you can set the access rights for the file(s) to be uploaded. By default, for a new module, only the owner has write permissions. Group and others have only read permissions.

Permissions cannot be set if the Add data to file option is set.

When having imported modules from an older INUBIT version, check whether the permissions are set correctly.

To set/deprive permissions, check/uncheck the checkboxes for read (R), write (W), and/or execute (X) permission for the Owner, the Group, and Others.

Additionally, the permissions are displayed as octal values.