File Connector

Usage

File connectors enable communication of the INUBIT Process Engine with the file system.

Connector types

The following configuration options are available:

  • File Input Connector

    Passes files and directories from the file system of the INUBIT Process Engine to a workflow for processing.

  • File Input Listener Connector

    Automatically passes files and directories from the file system of the INUBIT Process Engine to a workflow for processing if given files or files in the given directory are added or updated. When the workflow is activated, imported, or published for the first time in the file mode, one file is read. The file that comes first in alphanumerical order is used.

existing files are also read.

+

If a File Input Listener Connector observes a frequently modified directory (for example containing log files), performance issues might occur.

  • File Output Connector

    Writes the result of a workflow to a file in a directory of the INUBIT Process Engine. If the result is a zip archive, the File Output decompresses it before storing it.

    Due to a Tomcat 9.0.0.M1 security update for Linux, others will have no read, write, and execute permission, group will have no access to write and execute, and owner will have no access to execute the files and directories created.

Module Variables

The following module variables are set, when executing a File Connector. You can read these variables at the succeeding modules of the workflow:

  • ReadFileDate

    Last change date for the input file in the format dd.mm.yyyy hh:mm:ss.

  • ReadFileName

    Name of the input file including the extension.

  • ReadFileSize

    Size of the input file in bytes.

  • ReadFileWildcardname

    Outputs the content of the wildcard used in the file name when configuring the file connector, if the Use wildcard content of the preceding File or FTP connector option is selected.

  • WriteFileDate

    Only relevant for an output connector if Data is selected as the input format. Creation date of the output file in the format dd.MM.yyyy HH:mm:ss

  • WriteFileName

    Only relevant for an output connector if Data is selected as the input format. Name of the output file.

  • WriteFileWildcardname

    Only relevant for an output connector if Data is selected as the input format and e.g. timestamps or wildcards are used in the file names to be generated. Specifies the content of the wildcard used in the file name when configuring the file output connector. To use the WriteFileWildcardName variable you have to map it to the `ReadFileWildcardName `variable.

For information on other variables and their use refer to Workflow Variables and Mappings.

Dialog Descriptions

Dialog File(s) to Read

(Input Connector, Input Listener Connector)

This dialog offers the following options:

Mode

Specify the mode in which the file connector should work:

  • File mode

    For each execution, a file in the specified directory is read (as long as its name corresponds to the pattern entered in the File > Name field). The file that is read depends on your specifications for Read order.

In File mode, only one file per workflow execution is read, even when using wildcards, for example: *.txt

To read more than one file per workflow execution, use the Directory mode.

If you want to process all files in a directory successively, the Delete files after reading option must be activated. Otherwise, the first file is used repeatedly as a starting point.

  • Directory mode

    All files in the specified directory are read (as long as their file names correspond to the pattern entered in the File > Name field). In directory mode, an XML output message is always generated. This means that the Technical Workflow starts in any case.

    This makes it possible, e.g., to react to the current state of the file with a Demultiplexer or something similar located after the file input connector, refer to Demultiplexer.

Example:

module guide 1028 2

Directory

  • Name

    Path to the directory from which files are to be read. The path can be entered as absolute or relative path to the directory <inubit-installdir>/inubit/server/process_engine/bin.

File

  • Use wildcard

    If selected, you can use a wildcard (asterisk/*) or enter the complete file name.

  • Use regular expression

    If selected, Perl-compliant regular expressions can be used to create character string patterns to select the files to read.

    Regular expressions must be delimited by forward slashes (/). Example: /(\d\d)\.(\d\d)\.(\d\d\d\d)/

  • Name

    File name. Optional wildcard or regular expression.

  • Invert

    If activated, the pattern specified for the file name is inverted. As a result, all files whose name does not correspond to the pattern are read.

  • Ignore case sensitivity (only available if the Use wildcard option is activated)

    If activated, case-sensitivity will be ignored for file names including their extensions.

  • Use wildcard content of the preceding File or FTP connector (only available if the Use wildcard option is activated)

    Example: Test*.xml was specified in the File name field in the configuration of a previously executed file connector. A file with the name Test5.xml was passed by the file connector as an output message. In this case, the wildcard content is the digit 5. If you have also specified a wildcard in the current connector (for example, Output*.xml), the digit 5 is inserted into the wildcard position. The output of the connector configured here is therefore Output5.xml.

  • Delete files after reading

    If activated, files are deleted after they have been read. Activate this option to process all files, one after the other, in a directory.

    You can also activate the Throw error if, delete fails option so that an error entry is displayed in the Queue Manager for the workflow with this module if a file cannot be deleted. If an error output is configured, it is run. Otherwise, module execution is terminated.

  • Wait for file clearance/Check interval in seconds

    (Only in file mode.)

    Activate this option to ensure that the file input connector does not read any files while other applications are writing to them. If this option is selected, the file input connector checks the timestamp and sizes of the files to be read at configurable intervals and compares them. The files are only read if the specifications have not changed.

    The following options can only be used when the scheduler is active.

  • Maximum number of executions per scheduled call (not available in listener mode)

    You use this specification to define the maximum number of times the workflow is to be started per triggering event. The triggering event is the attainment of predefined point in time (if the scheduler is activated). The workflow is then started as many times as necessary until either

    • there is no more data to be fetched/sent or

    • the number defined in Maximum number of executions per call has been reached

      By restricting the maximum number of executions, you control the system load on the server. In Test mode, this specification is ignored, and the workflow is only started once.

  • Continue execution on failure of previous calls

    If the value of Max. number of executions per scheduled call is higher than 1, this option can be selected to continue the specified number of calls even after a previous failure.

Read order

Only in

  • File mode

  • Directory mode (when data transfer type is IBISDirectory-XML)

If more than one file is transmitted, you define which files are to be transmitted first and in which order.

Directory configuration

(If directory mode is selected)

  • Include directories

    If selected, the file connector also reads the directory itself as well as the files in the specified directory.

    • with subdirectories (not available in listener mode)

      If selected, the file connector also reads all subdirectories of the specified directory recursively.

  • Delete empty directories after reading/Delete files after reading

    Deletes empty directories once they have been read; this is a reasonable combination with the Delete files after reading option.

  • Force directory deletion after reading

    Activate this checkbox to delete the source directory even if it is not empty. If the directory exists, it will be deleted. Otherwise, the deletion of the directory will be aborted.

  • File number limit

    Maximum number of files that can be read.

  • Limit total size

    Maximum total size of files to be read. Once the limit is reached, the file connector will cease to read any input files.

File check

(If file mode is selected)

  • Abort workflow if file does not exist:

    • In test mode:

      The execution of the workflow is terminated with an error if access to the specified file fails.

    • Scheduled input File Connector:

      According to the specified interval, the File Connector checks if a file is available. The workflow is only started if a file exists.

    • Input File Connector in the middle of a workflow:

      If the file does not exist, the execution is continued with an empty message. No entry is made in the log.

  • Create XML output message with status information that the file exists and is empty; otherwise abort workflow

    Checks whether a relevant file with size=0 exists. If yes, an XML file with the element <FileExists> and the value true is output. Otherwise, the workflow execution is terminated with an error. The input file is not output.

  • Create XML output message with status information on whether the file exists or not

    Checks whether a relevant file is available. An XML file is output with the element <FileExists> and the value false if the specified file does not exist; if it exists, the value of the <FileExists> element is true. The input file is not output.

  • Create XML output message with status information that the file does not exist, otherwise abort workflow

    Checks whether a relevant file exists. If not, an XML file with the element <FileExists> and the value false is output. Otherwise, the workflow execution is terminated with an error. The input file is not output.

Data transfer

(If Directory mode is selected; in File mode, only for Abort workflow if file does not exist.)

  • Type

    Defines how input messages should be passed to the workflow:

    • Data

      The input messages are forwarded in unchanged form. Select this option if the input messages are, e.g., in XML format.

    • IBISDirectory-Xml

      Defines that the input messages are to be passed in IBISDirectory XML format. This involves generating an XML wrapper element for each input message.

    • Zip

      The input files are compressed in a Zip file.

  • Encoding (only for Type IBISDirectory XML and Zip)

    Defines the encoding of the character set.

XML configuration

(Only if Data transfer > Type = IBISDirectory XML)

  • Inclusive data

    If selected, the entire file is read. If not selected, only the file metadata are read, e.g., name, file size, and date. If binary data shall be read, the Encode data base64 option below must be activated.

  • Encode data base64 (only available if the Inclusive data option is activated)

    Activate this option if binary data shall be read. This kind of data must be encoded before it can be included in the XML properly. As long as this option is not activated, a corresponding warning is displayed.

    • Compress data (only available if Encode data base64 is activated)

      Activate this option to compress binary data read. Compressing will save storage and reduce the size of the XML output.

      The data is compressed as a Zip file.

  • Store absolute path into XML

    If selected, the absolute path of the file is stored in a separate path attribute.

Dialog File to Write

(Output Connector)

You use the dialog to specify the format of the input messages and define how the messages are to be written to the file system.

Input format

Specify the format of the input messages:

  • Data

    The input messages consist of data that should be written into a file without being changed.

  • IBISDirectory-Xml

    The input messages come along in IBISDirectory-Xml format and are stored in a chosen directory.

  • Zip

    The input files are compressed in a Zip file. If selected the File Connector expects a zip archive as input message. The File Connector decompresses this archive and stores the decompressed files in the given directory.

Encoding (for Zip input format only)

The selection made here defines the encoding of file and directory names.

Directory

  • Name (Mandatory)

    Absolute path to the directory that the file connector is to use when writing files.

    If no directory name is specified, an error message appears when executing the module.

In addition, you can add a date, timestamp, process ID to the file name to be generated.

  • Date

    The date specification also has a timestamp. If you specify the file name in the format file_%TimeStamp%%.xml, the output file name can be, for example, file_12122012-151047.xml. file is the file name, followed by a two-digit day specification, two-digit month specification, four-digit year specification (ddMMyyyy). The number following the hyphen is the time in two-digit hour specification, two-digit minute specification, and two-digit second specification (HHmmss).

  • Timestamp

    The timestamp is created exactly like the date stamp (see above). A milliseconds specification is also used. If you specify the file name as file_%TimeStamp%ddMMyyyy-HHmmss-SSS%.xml, the following file name for example is used as the name of the output file: file_12122012-151304-027.xml. The final, three-digit specification represents milliseconds.

  • Process ID

    The process ID is the unique number created for a workflow execution. To pass the process ID of the workflow execution with the output file, enter the file name in the following format: file_PID_%ProcessId%.xml. As a result, e.g., the following file name is used as the name of the output file: file_PID_18.xml.

    In conjunction with "Generate non-existent directories", you can create a day- or month-based archive structure for messages.

  • Create non-existing directories

    Generates all non-existent directories from the path specification.

  • Rename existing directory to

    If this checkbox is activated, you can enter a new name the existing directory configured in the Name field shall be renamed to. If the source directory does not exist in the file system, the workflow will be aborted. If the destination directory exists, the source directory will not be renamed.

File (for Data input format only)

  • Name

    You can also use wildcards (*) and add a timestamp and process ID in file names (refer to option Directory > Name).

  • Use wildcard content of the preceding File or FTP connector

    Example: A Test*.xml file was specified in the Directory and file name field in the configuration of a file connector that was executed previously. A file with the name Test5.xml was passed by the file connector as an output message. In this case, the wildcard content is the digit 5. If you have also specified a wildcard in the current connector (for example, Output*.xml), the digit 5 is entered in place of the wildcard. The output of the connector configured here is therefore Output5.xml.

  • Use the file name of the previously executed file connector

    Example: A Test.xml file was specified in the Directory and file name field in the configuration of a previously executed file connector. The file connector forwards a file called Test.xml as the output message. This file name is then also used here.

  • Safe writing

    This option makes sure that the application waiting for the file fetches the file only after its completion. If selected, the data is written initially to a temporary file. When this process is completed, the temporary file is renamed according to the given file name. Creating the temporary file can take longer for larger files.

Write mode (for both input formats Data and IBISDirectory-Xml only)

  • Overwrite existing file

    An existing file with the same name is overwritten.

  • Append data to file

    Each time the workflow is executed the file to be read is appended to an existing file should one exist. This can be useful if you want to record cumulative log data in a file. This option cannot be used if wildcards are specified in the file name.

  • Skip if file already exist (for IBISDirectory-Xml input format only)

    Skips writing into an existing file but creates a new one if no such file exists. In the watchpoint result file, the fileConnector.skippedFiles workflow variable contains the names of the already existing files.

Module output configuration

  • Set input message additionally as output message

    If this option is selected, the input file is passed to the subsequent module. Depending on the size of the input files, this option can affect the performance of your system.

    Refer to Module Variables.

Permission (only for Linux)

The selections made here define the read, write, and execute permissions for the owner, the group, and others.

If you do not select anything, the default permissions from the operating system will be applied.