File Connector
Usage
File connectors enable communication of the INUBIT Process Engine with the file system.
Connector types
The following configuration options are available:
-
File Input Connector
Passes files and directories from the file system of the INUBIT Process Engine to a workflow for processing.
-
File Input Listener Connector
Automatically passes files and directories from the file system of the INUBIT Process Engine to a workflow for processing if given files or files in the given directory are added or updated. When the workflow is activated, imported, or published for the first time in the file mode, one file is read. The file that comes first in alphanumerical order is used.
existing files are also read.
+
If a File Input Listener Connector observes a frequently modified directory (for example containing log files), performance issues might occur. |
-
File Output Connector
Writes the result of a workflow to a file in a directory of the INUBIT Process Engine. If the result is a zip archive, the File Output decompresses it before storing it.
Due to a Tomcat 9.0.0.M1 security update for Linux, others will have no read, write, and execute permission, group will have no access to write and execute, and owner will have no access to execute the files and directories created.
Module Variables
The following module variables are set, when executing a File Connector. You can read these variables at the succeeding modules of the workflow:
-
ReadFileDate
Last change date for the input file in the format
dd.mm.yyyy hh:mm:ss
. -
ReadFileName
Name of the input file including the extension.
-
ReadFileSize
Size of the input file in bytes.
-
ReadFileWildcardname
Outputs the content of the wildcard used in the file name when configuring the file connector, if the Use wildcard content of the preceding File or FTP connector option is selected.
-
WriteFileDate
Only relevant for an output connector if Data is selected as the input format. Creation date of the output file in the format
dd.MM.yyyy HH:mm:ss
-
WriteFileName
Only relevant for an output connector if Data is selected as the input format. Name of the output file.
-
WriteFileWildcardname
Only relevant for an output connector if Data is selected as the input format and e.g. timestamps or wildcards are used in the file names to be generated. Specifies the content of the wildcard used in the file name when configuring the file output connector. To use the
WriteFileWildcardName
variable you have to map it to the `ReadFileWildcardName `variable.
For information on other variables and their use refer to Workflow Variables and Mappings.
Dialog Descriptions
Dialog File(s) to Read
(Input Connector, Input Listener Connector)
This dialog offers the following options:
Mode
Specify the mode in which the file connector should work:
-
File mode
For each execution, a file in the specified directory is read (as long as its name corresponds to the pattern entered in the File > Name field). The file that is read depends on your specifications for Read order.
In File mode, only one file per workflow execution is read, even when using wildcards, for example: To read more than one file per workflow execution, use the Directory mode. |
If you want to process all files in a directory successively, the Delete files after reading option must be activated. Otherwise, the first file is used repeatedly as a starting point. |
-
Directory mode
All files in the specified directory are read (as long as their file names correspond to the pattern entered in the File > Name field). In directory mode, an XML output message is always generated. This means that the Technical Workflow starts in any case.
This makes it possible, e.g., to react to the current state of the file with a Demultiplexer or something similar located after the file input connector, refer to Demultiplexer.
Example:
Directory
-
Name
Path to the directory from which files are to be read. The path can be entered as absolute or relative path to the directory
<inubit-installdir>/inubit/server/process_engine/bin
.
File
-
Use wildcard
If selected, you can use a wildcard (asterisk/*) or enter the complete file name.
-
Use regular expression
If selected, Perl-compliant regular expressions can be used to create character string patterns to select the files to read.
Regular expressions must be delimited by forward slashes (/). Example:
/(\d\d)\.(\d\d)\.(\d\d\d\d)/
Refer to Regular Expressions.
-
Name
File name. Optional wildcard or regular expression.
Refer to Regular Expressions.
-
Invert
If activated, the pattern specified for the file name is inverted. As a result, all files whose name does not correspond to the pattern are read.
-
Ignore case sensitivity (only available if the Use wildcard option is activated)
If activated, case-sensitivity will be ignored for file names including their extensions.
-
Use wildcard content of the preceding File or FTP connector (only available if the Use wildcard option is activated)
Example:
Test*.xml
was specified in the File name field in the configuration of a previously executed file connector. A file with the nameTest5.xml
was passed by the file connector as an output message. In this case, the wildcard content is the digit5
. If you have also specified a wildcard in the current connector (for example,Output*.xml
), the digit 5 is inserted into the wildcard position. The output of the connector configured here is thereforeOutput5.xml
. -
Delete files after reading
If activated, files are deleted after they have been read. Activate this option to process all files, one after the other, in a directory.
You can also activate the Throw error if, delete fails option so that an error entry is displayed in the Queue Manager for the workflow with this module if a file cannot be deleted. If an error output is configured, it is run. Otherwise, module execution is terminated.
-
Wait for file clearance/Check interval in seconds
(Only in file mode.)
Activate this option to ensure that the file input connector does not read any files while other applications are writing to them. If this option is selected, the file input connector checks the timestamp and sizes of the files to be read at configurable intervals and compares them. The files are only read if the specifications have not changed.
The following options can only be used when the scheduler is active.
-
Maximum number of executions per scheduled call (not available in listener mode)
You use this specification to define the maximum number of times the workflow is to be started per triggering event. The triggering event is the attainment of predefined point in time (if the scheduler is activated). The workflow is then started as many times as necessary until either
-
there is no more data to be fetched/sent or
-
the number defined in Maximum number of executions per call has been reached
By restricting the maximum number of executions, you control the system load on the server. In Test mode, this specification is ignored, and the workflow is only started once.
-
-
Continue execution on failure of previous calls
If the value of Max. number of executions per scheduled call is higher than 1, this option can be selected to continue the specified number of calls even after a previous failure.
Read order
Only in
|
If more than one file is transmitted, you define which files are to be transmitted first and in which order.
Directory configuration
(If directory mode is selected)
-
Include directories
If selected, the file connector also reads the directory itself as well as the files in the specified directory.
-
with subdirectories (not available in listener mode)
If selected, the file connector also reads all subdirectories of the specified directory recursively.
-
-
Delete empty directories after reading/Delete files after reading
Deletes empty directories once they have been read; this is a reasonable combination with the Delete files after reading option.
-
Force directory deletion after reading
Activate this checkbox to delete the source directory even if it is not empty. If the directory exists, it will be deleted. Otherwise, the deletion of the directory will be aborted.
-
File number limit
Maximum number of files that can be read.
-
Limit total size
Maximum total size of files to be read. Once the limit is reached, the file connector will cease to read any input files.
File check
(If file mode is selected)
-
Abort workflow if file does not exist:
-
In test mode:
The execution of the workflow is terminated with an error if access to the specified file fails.
-
Scheduled input File Connector:
According to the specified interval, the File Connector checks if a file is available. The workflow is only started if a file exists.
-
Input File Connector in the middle of a workflow:
If the file does not exist, the execution is continued with an empty message. No entry is made in the log.
-
-
Create XML output message with status information that the file exists and is empty; otherwise abort workflow
Checks whether a relevant file with size=0 exists. If yes, an XML file with the element
<FileExists>
and the valuetrue
is output. Otherwise, the workflow execution is terminated with an error. The input file is not output. -
Create XML output message with status information on whether the file exists or not
Checks whether a relevant file is available. An XML file is output with the element
<FileExists>
and the valuefalse
if the specified file does not exist; if it exists, the value of the<FileExists>
element istrue
. The input file is not output. -
Create XML output message with status information that the file does not exist, otherwise abort workflow
Checks whether a relevant file exists. If not, an XML file with the element
<FileExists>
and the valuefalse
is output. Otherwise, the workflow execution is terminated with an error. The input file is not output.
Data transfer
(If Directory mode is selected; in File mode, only for Abort workflow if file does not exist.)
-
Type
Defines how input messages should be passed to the workflow:
-
Data
The input messages are forwarded in unchanged form. Select this option if the input messages are, e.g., in XML format.
-
IBISDirectory-Xml
Defines that the input messages are to be passed in IBISDirectory XML format. This involves generating an XML wrapper element for each input message.
-
Zip
The input files are compressed in a Zip file.
-
-
Encoding (only for Type IBISDirectory XML and Zip)
Defines the encoding of the character set.
XML configuration
(Only if Data transfer > Type = IBISDirectory XML)
-
Inclusive data
If selected, the entire file is read. If not selected, only the file metadata are read, e.g., name, file size, and date. If binary data shall be read, the Encode data base64 option below must be activated.
-
Encode data base64 (only available if the Inclusive data option is activated)
Activate this option if binary data shall be read. This kind of data must be encoded before it can be included in the XML properly. As long as this option is not activated, a corresponding warning is displayed.
-
Compress data (only available if Encode data base64 is activated)
Activate this option to compress binary data read. Compressing will save storage and reduce the size of the XML output.
The data is compressed as a Zip file.
-
-
Store absolute path into XML
If selected, the absolute path of the file is stored in a separate
path
attribute.
Dialog File to Write
(Output Connector)
You use the dialog to specify the format of the input messages and define how the messages are to be written to the file system.
Input format
Specify the format of the input messages:
-
Data
The input messages consist of data that should be written into a file without being changed.
-
IBISDirectory-Xml
The input messages come along in IBISDirectory-Xml format and are stored in a chosen directory.
-
Zip
The input files are compressed in a Zip file. If selected the File Connector expects a zip archive as input message. The File Connector decompresses this archive and stores the decompressed files in the given directory.
Encoding (for Zip input format only)
The selection made here defines the encoding of file and directory names.
Directory
-
Name (Mandatory)
Absolute path to the directory that the file connector is to use when writing files.
If no directory name is specified, an error message appears when executing the module.
In addition, you can add a date, timestamp, process ID to the file name to be generated.
-
Date
The date specification also has a timestamp. If you specify the file name in the format
file_%TimeStamp%%.xml
, the output file name can be, for example,file_12122012-151047.xml
.file
is the file name, followed by a two-digit day specification, two-digit month specification, four-digit year specification (ddMMyyyy
). The number following the hyphen is the time in two-digit hour specification, two-digit minute specification, and two-digit second specification (HHmmss
). -
Timestamp
The timestamp is created exactly like the date stamp (see above). A milliseconds specification is also used. If you specify the file name as
file_%TimeStamp%ddMMyyyy-HHmmss-SSS%.xml
, the following file name for example is used as the name of the output file:file_12122012-151304-027.xml.
The final, three-digit specification represents milliseconds. -
Process ID
The process ID is the unique number created for a workflow execution. To pass the process ID of the workflow execution with the output file, enter the file name in the following format:
file_PID_%ProcessId%.xml.
As a result, e.g., the following file name is used as the name of the output file:file_PID_18.xml.
In conjunction with "Generate non-existent directories", you can create a day- or month-based archive structure for messages.
-
Create non-existing directories
Generates all non-existent directories from the path specification.
-
Rename existing directory to
If this checkbox is activated, you can enter a new name the existing directory configured in the Name field shall be renamed to. If the source directory does not exist in the file system, the workflow will be aborted. If the destination directory exists, the source directory will not be renamed.
File (for Data input format only)
-
Name
You can also use wildcards (
*
) and add a timestamp and process ID in file names (refer to option Directory > Name). -
Use wildcard content of the preceding File or FTP connector
Example: A
Test*.xml
file was specified in the Directory and file name field in the configuration of a file connector that was executed previously. A file with the nameTest5.xml
was passed by the file connector as an output message. In this case, the wildcard content is the digit5
. If you have also specified a wildcard in the current connector (for example,Output*.xml
), the digit5
is entered in place of the wildcard. The output of the connector configured here is thereforeOutput5.xml
. -
Use the file name of the previously executed file connector
Example: A
Test.xml
file was specified in the Directory and file name field in the configuration of a previously executed file connector. The file connector forwards a file calledTest.xml
as the output message. This file name is then also used here. -
Safe writing
This option makes sure that the application waiting for the file fetches the file only after its completion. If selected, the data is written initially to a temporary file. When this process is completed, the temporary file is renamed according to the given file name. Creating the temporary file can take longer for larger files.
Write mode (for both input formats Data and IBISDirectory-Xml only)
-
Overwrite existing file
An existing file with the same name is overwritten.
-
Append data to file
Each time the workflow is executed the file to be read is appended to an existing file should one exist. This can be useful if you want to record cumulative log data in a file. This option cannot be used if wildcards are specified in the file name.
-
Skip if file already exist (for IBISDirectory-Xml input format only)
Skips writing into an existing file but creates a new one if no such file exists. In the watchpoint result file, the
fileConnector.skippedFiles
workflow variable contains the names of the already existing files.
Module output configuration
-
Set input message additionally as output message
If this option is selected, the input file is passed to the subsequent module. Depending on the size of the input files, this option can affect the performance of your system.
Refer to Module Variables.
Permission (only for Linux)
The selections made here define the read, write, and execute permissions for the owner, the group, and others.
If you do not select anything, the default permissions from the operating system will be applied. |