Replication

This service replicates data from an RDBMS to OpenSearch.

Impact on System Performance

Replication has a direct impact on system performance. In particular, the number of threads (Replication → Settings → Replication_Threads) and the interval (replicationInterval) play a role here. The more jobs that run in a short period of time, the higher the CPU utilization. This is especially true when large amounts of data are being replicated.

Although replicationBlockSize has a positive effect on replication duration, it also directly influences Apache Karaf’s memory requirements.

Preparing for Data Record Timeliness

Before setting up replication, a few prerequisites should be met. Replication requires a column containing exclusively increasing timestamps for each table (both Log and Childlog), as this timestamp is used to subsequently identify newer or updated records. If different processes, servers, etc., log data to the database in parallel—and thus potentially with a time lag (even if only by milliseconds)—it may happen that individual records are not displayed in BPC because they were committed with a timestamp older than the most recent replicated one.

It is therefore recommended:

to log the database server’s time rather than the logging process’s server time, since different process servers may have slightly different times
to ensure that this column is actually updated with every INSERT or UPDATE

Both can be easily achieved using an additional (hidden) column with a default value and an associated trigger. This has the advantage that nothing changes at all for the logging process)

Adding a technical TIMESTAMP column for replication

Oracle

For PM-Log and Childlog (without a prefix) in Oracle, the procedure for creating them looks like this, for example:

--LOG:
--Add a column (with 6 decimal places of precision); set the default value for inserts to the current time in UTC (this eliminates the need for adjustments in the logging process)
ALTER TABLE LOG ADD (DB_UPDATE_TS TIMESTAMP DEFAULT SYSTIMESTAMP AT TIME ZONE 'UTC' NOT NULL);

--“Redistribute” existing log entries; otherwise, replication won't handle it well:
--Either put them all in one block (!not recommended for tables with more than 1 million entries!):
UPDATE LOG SET DB_UPDATE_TS = TIMESTAMP;
Commit;

  --or for large datasets and live systems that also need to perform operations on the tables, as an anonymous PL/SQL block with minimal undo tablespace requirements:
  --In this case, an index should also be created on the timestamp column to avoid full table scans!
    declare begin
      FOR counter IN 0 .. 3650 LOOP
        --dbms_output.put_line(to_char(to_date('2010-01-01', 'YYYY-MM-DD') + counter, 'YYYY-MM-DD') || ' - ' || to_char(to_date('2010-01-01', 'YYYY-MM-DD') + 1 + counter, 'YYYY-MM-DD'));
        --LOG:
        update log set DB_UPDATE_TS = TIMESTAMP where TIMESTAMP between to_date('2010-01-01', 'YYYY-MM-DD') + counter and to_date('2010-01-01', 'YYYY-MM-DD') + 1 + counter;
        commit;
        --CHILDLOG:
        update childlog set DB_UPDATE_TS = TIMESTAMP where TIMESTAMP between to_date('2010-01-01', 'YYYY-MM-DD') + counter and to_date('2010-01-01', 'YYYY-MM-DD') + 1 + counter;
        commit;
      END LOOP;
    end;

--Create a trigger that resets the column with every update (this eliminates the need for adjustments to the logging process)
CREATE OR REPLACE
TRIGGER LOG_DB_UPDATE_TS
BEFORE UPDATE ON LOG
REFERENCING NEW AS NEW OLD AS OLD
FOR EACH ROW
DECLARE
BEGIN
:NEW.DB_UPDATE_TS := SYSTIMESTAMP AT TIME ZONE 'UTC';
END;
/

--Create Index:
CREATE INDEX IDX_LOG_DBLU ON LOG(DB_UPDATE_TS) COMPUTE STATISTICS;


--Here's the whole thing again for Childlog:
ALTER TABLE CHILDLOG ADD (DB_UPDATE_TS TIMESTAMP DEFAULT SYSTIMESTAMP AT TIME ZONE 'UTC' NOT NULL);

--See the PL/SQL block above, if applicable!
UPDATE CHILDLOG SET DB_UPDATE_TS = TIMESTAMP;
commit;

CREATE OR REPLACE
TRIGGER CHILDLOG_DB_UPDATE_TS
BEFORE UPDATE ON CHILDLOG
REFERENCING NEW AS NEW OLD AS OLD
FOR EACH ROW
DECLARE
BEGIN
:NEW.DB_UPDATE_TS := SYSTIMESTAMP AT TIME ZONE 'UTC';
END;
/

CREATE INDEX IDX_CHILDLOG_DBLU ON CHILDLOG(DB_UPDATE_TS) COMPUTE STATISTICS;
--Finished

MSSQL

TIMESTAMP should be set to UTC; otherwise, a blind spot will occur between 2 and 3 a.m. during the daylight saving time change! The code does not yet account for this—you may want to use SYSUTCDATETIME instead of SYSDATETIME —but be sure to check the behavior in BPC.

The trigger for Childlog is missing. However, this is usually not needed, since entries are always only added and never updated.

/* Add columns. Use DATETIME2 for higher precision than DATETIME. Since MSSQL does not support millisecond precision, use the SYSDATETIME() function instead of current_timestamp; this function provides nanosecond precision. */
ALTER TABLE [LOG] ADD DB_UPDATE_TS DATETIME2 DEFAULT SYSDATETIME() NOT NULL;
GO

ALTER TABLE [CHILDLOG] ADD DB_UPDATE_TS DATETIME2 DEFAULT SYSDATETIME() NOT NULL;
GO

/* Initialize columns */
BEGIN
    UPDATE [LOG] SET DB_UPDATE_TS = [timestamp];
END
GO

BEGIN
    UPDATE [CHILDLOG] SET DB_UPDATE_TS = [timestamp];
END
GO

/* Trigger */
CREATE TRIGGER LOG_DB_UPDATE_TS
  ON [LOG]
  AFTER UPDATE
  AS
BEGIN
    IF NOT UPDATE(DB_UPDATE_TS)
    BEGIN
        UPDATE t
            SET t.DB_UPDATE_TS = SYSDATETIME()
            FROM [LOG] AS t
            INNER JOIN inserted AS i
            ON t.PROCESSID = i.PROCESSID;
    END
END
GO

/* Indices */
CREATE INDEX IDX_LOG_DBLU ON LOG (DB_UPDATE_TS);
CREATE INDEX IDX_CHILDLOG_DBLU ON CHILDLOG (DB_UPDATE_TS);

MySQL

and for MySQL, it’s very simple:

/* Adds a sufficiently precise timestamp managed by the database to the table */
ALTER TABLE LOG ADD DB_UPDATE_TS TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6)
ALTER TABLE CHILDLOG ADD DB_UPDATE_TS TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6)

PostgreSQL

and for PostgreSQL, also via triggers:

ALTER TABLE log ADD db_update_ts timestamp NOT null DEFAULT (timezone('UTC', now()));
ALTER TABLE childlog ADD db_update_ts timestamp NOT null DEFAULT (timezone('UTC', now()));

CREATE OR REPLACE FUNCTION update_db_update_ts_column()
RETURNS TRIGGER AS $$
BEGIN
    NEW.db_update_ts = timezone('UTC', now());
    RETURN NEW;
END;
$$ language 'plpgsql';

CREATE TRIGGER update_log_db_update_ts BEFORE INSERT OR UPDATE ON log FOR EACH ROW EXECUTE PROCEDURE update_db_update_ts_column();
CREATE TRIGGER update_childlog_db_update_ts BEFORE INSERT OR UPDATE ON childlog FOR EACH ROW EXECUTE PROCEDURE update_db_update_ts_column();

CREATE INDEX idx_log_dblu ON log (db_update_ts);
CREATE INDEX idx_childlog_dblu ON childlog (db_update_ts);

Configuration

Data Sources

Data sources are connections to individual databases. The individual replication jobs then use/reference these. These are set up under Backend Connections as type "data_source" and referenced by their component ID.

Interface

A dedicated interface is available for setting up individual replication jobs under Settings → Replication → Components → Editor. Through this interface, entries can be created, deleted, duplicated, and also enabled or disabled individually.

Configuration Parameters of the Replication Module

The various Parameters and their associated functions are described below. These can be found under BPC Administration → Replication → General.

Module

General module settings

Name (ID) Description

Name (`ID`)	Description
Module Without GUI (`noGui`)	Indicator for the front end that there is no user interface for loading this module. Should not be changed.
Icon (`module_iconCls`)	Individually selectable icon that is displayed before the title. If no icon is to be displayed, e.g. "none" can be entered. If the field is empty, a standard icon is selected.

Module Without GUI
(noGui)

Indicator for the front end that there is no user interface for loading this module. Should not be changed.

Icon
(module_iconCls)

Individually selectable icon that is displayed before the title.
If no icon is to be displayed, e.g. "none" can be entered. If the field is empty, a standard icon is selected.

Basic Data

Settings for the basic data

Name (ID) Description

Name (`ID`)	Description
Threads (`replicatorThreadCount`)	Number of threads used for the simultaneous execution of replication jobs. Attention! This has an influence on the CPU and especially RAM consumption of the system.
Templates (`templates`)	Templates with various settings. These can be loaded into replication jobs. This is useful if very similar replication settings are frequently used.

Threads
(replicatorThreadCount)

Number of threads used for the simultaneous execution of replication jobs.
Attention! This has an influence on the CPU and especially RAM consumption of the system.

Templates
(templates)

Templates with various settings. These can be loaded into replication jobs. This is useful if very similar replication settings are frequently used.

Logging

Settings for logging in replication

Name (ID) Description

Name (`ID`)	Description
Enable (`replicationJobsLogEnabled`)	Determines whether the replication job runs should be logged in the special OpenSearch index 'bpc-replicationjobs-log'. The data can be viewed via the Replication Jobs Monitor. Logging can also be (de)activated for each individual job.
Interval (`replicationJobsLogBulkWritePeriodInSeconds`)	Interval (in seconds) at which collected log data is written for replication. The data can be viewed via the Replication Jobs Monitor.
Cleanup Interval (`replicationJobsLogCleanupPeriodInMinutes`)	Interval (in minutes) at which old log data is deleted. Which data is deleted is controlled by the retention period.
Retention period (`replicationJobsLogDeleteEntriesOlderThan`)	Specifies how long log data should be retained. The time of deletion is controlled via the cleanup interval.

Enable
(replicationJobsLogEnabled)

Determines whether the replication job runs should be logged in the special OpenSearch index 'bpc-replicationjobs-log'. The data can be viewed via the Replication Jobs Monitor.
Logging can also be (de)activated for each individual job.

Interval
(replicationJobsLogBulkWritePeriodInSeconds)

Interval (in seconds) at which collected log data is written for replication. The data can be viewed via the Replication Jobs Monitor.

Cleanup Interval
(replicationJobsLogCleanupPeriodInMinutes)

Interval (in minutes) at which old log data is deleted. Which data is deleted is controlled by the retention period.

Retention period
(replicationJobsLogDeleteEntriesOlderThan)

Specifies how long log data should be retained. The time of deletion is controlled via the cleanup interval.

Tail Sync logging

Settings for logging of tail sync runs

Name (ID) Description

Name (`ID`)	Description
Enable (`replicationJobsTailsyncLogEnabled`)	Determines whether the tail sync runs should be logged in the special OpenSearch index 'bpc-tailsync-log'. The data can be viewed via the Tailsync Monitor. Logging can also be (de)activated for each individual replication job.
Cleanup Interval (`replicationJobsTailsyncLogCleanupPeriodInMinutes`)	Interval (in minutes) at which old tail sync log data is deleted. Which data is deleted is controlled by the retention period.
Retention period (`replicationJobsTailsyncLogDeleteEntriesOlderThan`)	Specifies how long tail sync log data should be retained. The time of deletion is controlled via the cleanup interval.

Enable
(replicationJobsTailsyncLogEnabled)

Determines whether the tail sync runs should be logged in the special OpenSearch index 'bpc-tailsync-log'. The data can be viewed via the Tailsync Monitor.
Logging can also be (de)activated for each individual replication job.

Cleanup Interval
(replicationJobsTailsyncLogCleanupPeriodInMinutes)

Interval (in minutes) at which old tail sync log data is deleted. Which data is deleted is controlled by the retention period.

Retention period
(replicationJobsTailsyncLogDeleteEntriesOlderThan)

Specifies how long tail sync log data should be retained. The time of deletion is controlled via the cleanup interval.

Configuration Parameters of a Replication Job

The various parameters and their associated functions are described below. These can be found under BPC Administration → Replication → Components. It is recommended to use the specialized interface: BPC Administration → Replication → Editor

Module

General module settings

Name (ID) Description

Name (`ID`)	Description
Icon (`module_iconCls`)	Individually selectable icon that is displayed before the title. If no icon is to be displayed, e.g. "none" can be entered. If the field is empty, a standard icon is selected.

Icon
(module_iconCls)

Individually selectable icon that is displayed before the title.
If no icon is to be displayed, e.g. "none" can be entered. If the field is empty, a standard icon is selected.

Source

Settings for the source system

Name (ID) Description

Name (`ID`)	Description
Database Connection (`rdmsDataSourceName`)	Database connection for accessing the data. This must first be created via Backend Connection of type Data Source.
SELECT (CTE) (`sourceCommonTableExpressionQuery`)	If this value is set, this query is used as the source instead of the table name. May only contain the SELECT statement of a Common Table Expression (CTE). The WITH $sourceTable$ AS ( $sourceCommonTableExpressionQuery$ ) $bpcQuery$; is generated so that it matches the queries placed on it by the BPC. If this option is used, then 'Source_Table' (sourceTable) is used as the name of the CTE. Incidentally, this can match the name of an existing database table.
Table (`sourceTable`)	Name of the table or view in the source database. If a SELECT (CTE) is specified, this is used as the name in the CTE. Must not be empty!
Time Zone (`sourceTimeZone`)	If the date columns do not contain time zone information, they are interpreted with the specified time zone. This does not apply to the time column. There is a separate time zone setting for this.
Primary Key (`idColumns`)	Columns for the creation of a unique key in OpenSearch. An incorrect configuration leads to data records being overwritten.
Last Update (`lastUpdateColumn`)	The column must contain the time of the last change to the data record. It is crucial for the replication function that the correct time is entered here for each change to the data record. It is advisable to have this set via a DB trigger. For the performance of the replication and to reduce the load on the source database, a sorted index should always be present in this column.
Last Update Time Zone (`lastUpdateColumnTimeZone`)	If the last update column does not contain time zone information, it is interpreted with the specified time zone. Does not affect other date columns. There is a separate time zone configuration for these.
Timeout (`sourceQueryTimeoutInSeconds`)	Defines how long the JDBC driver waits for a response from the DB. Specified in seconds.

Database Connection
(rdmsDataSourceName)

Database connection for accessing the data. This must first be created via Backend Connection of type Data Source.

SELECT (CTE)
(sourceCommonTableExpressionQuery)

If this value is set, this query is used as the source instead of the table name. May only contain the SELECT statement of a Common Table Expression (CTE). The WITH $sourceTable$ AS ( $sourceCommonTableExpressionQuery$ ) $bpcQuery$; is generated so that it matches the queries placed on it by the BPC. If this option is used, then 'Source_Table' (sourceTable) is used as the name of the CTE. Incidentally, this can match the name of an existing database table.

Table
(sourceTable)

Name of the table or view in the source database. If a SELECT (CTE) is specified, this is used as the name in the CTE.
Must not be empty!

Time Zone
(sourceTimeZone)

If the date columns do not contain time zone information, they are interpreted with the specified time zone.
This does not apply to the time column. There is a separate time zone setting for this.

Primary Key
(idColumns)

Columns for the creation of a unique key in OpenSearch. An incorrect configuration leads to data records being overwritten.

Last Update
(lastUpdateColumn)

The column must contain the time of the last change to the data record.
It is crucial for the replication function that the correct time is entered here for each change to the data record. It is advisable to have this set via a DB trigger.
For the performance of the replication and to reduce the load on the source database, a sorted index should always be present in this column.

Last Update Time Zone
(lastUpdateColumnTimeZone)

If the last update column does not contain time zone information, it is interpreted with the specified time zone.
Does not affect other date columns. There is a separate time zone configuration for these.

Timeout
(sourceQueryTimeoutInSeconds)

Defines how long the JDBC driver waits for a response from the DB. Specified in seconds.

Target

Settings for the target system

Name (ID) Description

Name (`ID`)	Description
Index (`targetIndex`)	Name of the index in which the data is to be stored. This is created automatically if required.
Index Creation Settings (`targetIndexCreationSettings`)	Settings that differ from the default. The value is set as its "settings" value when creating/generating an OpenSearch index. It is also used during reindexing, as this creates a new index. If this field is empty, then the index creation settings of the Core Services are used. If it is set, then only these are used. The index creation settings of the Core Service must then be used as a basis via copy&paste.
Field Mappings (`targetIndexMappings`)	Optional setting for the individual fields (also known as mapping). This can be used, for example, to define the data type of the field.
Dynamic Field Templates (`targetIndexDynamicTemplates`)	Settings that differ from the standard. The value is set as "dynamic_templates" when creating/generating an OpenSearch index in field mappings ("mappings"). Is also used during reindexing, as this creates a new index. If this field is empty, then the dynamic field templates of the Core Services are used.
Adapt Field Names (`targetIndexCaseSensitivityOfFields`)	The capitalization of the fields created in OpenSearch can be changed here. The field names are formed from the column names of the database.

Index
(targetIndex)

Name of the index in which the data is to be stored. This is created automatically if required.

Index Creation Settings
(targetIndexCreationSettings)

Settings that differ from the default. The value is set as its "settings" value when creating/generating an OpenSearch index. It is also used during reindexing, as this creates a new index.
If this field is empty, then the index creation settings of the Core Services are used.
If it is set, then only these are used. The index creation settings of the Core Service must then be used as a basis via copy&paste.

Field Mappings
(targetIndexMappings)

Optional setting for the individual fields (also known as mapping). This can be used, for example, to define the data type of the field.

Dynamic Field Templates
(targetIndexDynamicTemplates)

Settings that differ from the standard. The value is set as "dynamic_templates" when creating/generating an OpenSearch index in field mappings ("mappings"). Is also used during reindexing, as this creates a new index.
If this field is empty, then the dynamic field templates of the Core Services are used.

Adapt Field Names
(targetIndexCaseSensitivityOfFields)

The capitalization of the fields created in OpenSearch can be changed here. The field names are formed from the column names of the database.

Advanced Settings

Advanced settings

Name (ID) Description

Name (`ID`)	Description
Enable (`replicationEnabled`)	Activates the execution of this replication job.
Start Delay (`replicationDelay`)	Delay of this replication (in seconds) after the initial start of replication execution. Can be used to speed up the start of the BPC or to initially prioritize other replication jobs.
Interval (`replicationInterval`)	Interval in seconds at which this replication job should be executed. This interval is not guaranteed if there are not enough threads available and too many replication jobs are running (possibly for too long).
Start Date (`replicationStartDate`)	Only data that is newer than this point in time is replicated. Date format "yyyy-MM-dd' 'HH:mm:ss.SSS".
Day Range per Run (`replicationBlockDayRange`)	If data from the past is replicated, this controls the number of days that should be processed for each job run. A high value can lead to the source database and OpenSearch being heavily loaded. This may also block other replication jobs.
Maximum Records (`replicationBlockSize`)	Maximum number of data records that can be loaded at once from the source database and written to OpenSearch. This value directly influences the consumption of memory in Karaf, as the memory required to store all data records in the maximum size in the memory is reserved. However, a large value has a positive effect on the speed of replication.
Replicate Binary Files (`replicationSyncFiles`)	Replicates columns of type BLOB. Attention! This can significantly affect the memory requirements and performance of OpenSearch.
Decompress Binary Files (`replicationUnzipSyncedFiles`)	If binary data is replicated and the option is activated, the system checks whether the data was compressed with GZip and decompresses it before saving.
Restart Replication Where Left Off (`restartReplicationWhereLeftOff`)	If this option is active, the replication job is continued at the point in the data where it last stopped when it is restarted or reactivated. Otherwise, replication begins at the configured start time.
Adjust Date Limit (`adjustUpperDateLimitInSeconds`)	Adjustment of the upper time limit (in seconds). Influences the upper date limit when selecting data by adding this value to the current time. Changing the value can lead to data records being replicated unnecessarily several times or changes only being replicated with a certain delay.
Data Management Organization ID (`vamOrganizationId`)	If set, special features are used for the replication of data management data. Specifically, only the data of the entered organizationId (as in data management configuration) is replicated
Enable (`replicationLoggingEnabled`)	Activates logging for this replication job. In addition, logging must be generally active on the replication module.

Enable
(replicationEnabled)

Activates the execution of this replication job.

Start Delay
(replicationDelay)

Delay of this replication (in seconds) after the initial start of replication execution. Can be used to speed up the start of the BPC or to initially prioritize other replication jobs.

Interval
(replicationInterval)

Interval in seconds at which this replication job should be executed. This interval is not guaranteed if there are not enough threads available and too many replication jobs are running (possibly for too long).

Start Date
(replicationStartDate)

Only data that is newer than this point in time is replicated.
Date format "yyyy-MM-dd' 'HH:mm:ss.SSS".

Day Range per Run
(replicationBlockDayRange)

If data from the past is replicated, this controls the number of days that should be processed for each job run. A high value can lead to the source database and OpenSearch being heavily loaded. This may also block other replication jobs.

Maximum Records
(replicationBlockSize)

Maximum number of data records that can be loaded at once from the source database and written to OpenSearch. This value directly influences the consumption of memory in Karaf, as the memory required to store all data records in the maximum size in the memory is reserved. However, a large value has a positive effect on the speed of replication.

Replicate Binary Files
(replicationSyncFiles)

Replicates columns of type BLOB.
Attention! This can significantly affect the memory requirements and performance of OpenSearch.

Decompress Binary Files
(replicationUnzipSyncedFiles)

If binary data is replicated and the option is activated, the system checks whether the data was compressed with GZip and decompresses it before saving.

Restart Replication Where Left Off
(restartReplicationWhereLeftOff)

If this option is active, the replication job is continued at the point in the data where it last stopped when it is restarted or reactivated. Otherwise, replication begins at the configured start time.

Adjust Date Limit
(adjustUpperDateLimitInSeconds)

Adjustment of the upper time limit (in seconds). Influences the upper date limit when selecting data by adding this value to the current time. Changing the value can lead to data records being replicated unnecessarily several times or changes only being replicated with a certain delay.

Data Management Organization ID
(vamOrganizationId)

If set, special features are used for the replication of data management data. Specifically, only the data of the entered organizationId (as in data management configuration) is replicated

Enable
(replicationLoggingEnabled)

Activates logging for this replication job.
In addition, logging must be generally active on the replication module.

Shadow Copy

A shadow copy copies all documents (created after the defined 'replicationStartDate') of the OpenSearch index (see 'targetIndex') into a new index at the specified time. In the end, the alias is switched to the new index and the previous index is deleted.

Name (ID) Description

Name (`ID`)	Description
Enable (`replicationShadowCopyEnabled`)	Activate creation of shadow copies.
Cron Pattern (`replicationShadowCopyCronPattern`)	A Cron-like pattern (using Quartz Scheduler syntax) to define when the shadow copies are to be created. (See <a target="_blank" href="https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html">Quartz CronTrigger Documentation</a>)
Number of Copies (`replicationShadowCopyKeepCopiesCount`)	Number of shadow copies to be kept.

Enable
(replicationShadowCopyEnabled)

Activate creation of shadow copies.

Cron Pattern
(replicationShadowCopyCronPattern)

A Cron-like pattern (using Quartz Scheduler syntax) to define when the shadow copies are to be created. (See <a target="_blank" href="https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html">Quartz CronTrigger Documentation</a>)

Number of Copies
(replicationShadowCopyKeepCopiesCount)

Number of shadow copies to be kept.

Tail Sync

The 'Tail Sync' works on the current index (see 'targetIndex') and synchronizes older data (new/changed/deleted database records) with OpenSearch so that they match the database again.

Name (ID) Description

Name (`ID`)	Description
Enable (`replicationTailSyncEnabled`)	Activates the Tail Sync function. This deletes old data records or subsequently synchronizes data records between the source and target. This is necessary if data records are deleted in the source. New or changed data records should be recognized via regular replication if the setting is correct. If the tail sync is not enabled it can still be started manually and the other tail sync settings will be used.
Cron Pattern (`replicationTailSyncCronPattern`)	A Cron-like pattern (using Quartz Scheduler syntax) to define when the tail sync should be started. (See <a target="_blank" href="https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html">Quartz CronTrigger Documentation</a>)
Maximum Records (`replicationTailSyncBlockSize`)	Maximum number of data records that can be loaded at once from the source database and written to OpenSearch. This value directly influences the consumption of memory in Karaf, as the memory required to store all data records in the maximum size in the memory is reserved. However, a large value has a positive effect on the speed of replication.
Start Date (`replicationTailSyncRelativeStartDate`)	The relative start date specified. Synchronization starts from this date. Data before the replication start date will continue to be deleted. This date is only used if it is after the regular start date and before the end date. Only use in special cases!
End Date (`replicationTailSyncRelativeEndDate`)	The relative end date specified. Synchronization takes place up to this point in time. The end should not overlap with the regular replication (interval + buffer). Syntax: <code>n seconds\|minutes\|hours\|days\|weeks\|months\|years ago</code>
Day Range per Run (`replicationTailSyncBlockDayRange`)	This controls the number of days that should be processed in each run. A high value can lead to the source database and OpenSearch being heavily loaded.
Delete Old Records (`replicationTailSyncRelativeDeleteOlderThanDate`)	The relative deletion date to be specified. All documents that are older are deleted. If this option is not set, it will not be executed. Syntax: <code>n seconds\|minutes\|hours\|days\|weeks\|months\|years ago</code>
Enable Tail Sync logging (`replicationTailSyncLoggingEnabled`)	If enabled, Tail Sync runs are logged in the `bpc-tailsync-log` index.

Enable
(replicationTailSyncEnabled)

Activates the Tail Sync function. This deletes old data records or subsequently synchronizes data records between the source and target. This is necessary if data records are deleted in the source. New or changed data records should be recognized via regular replication if the setting is correct. If the tail sync is not enabled it can still be started manually and the other tail sync settings will be used.

Cron Pattern
(replicationTailSyncCronPattern)

A Cron-like pattern (using Quartz Scheduler syntax) to define when the tail sync should be started. (See <a target="_blank" href="https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html">Quartz CronTrigger Documentation</a>)

Maximum Records
(replicationTailSyncBlockSize)

Start Date
(replicationTailSyncRelativeStartDate)

The relative start date specified. Synchronization starts from this date. Data before the replication start date will continue to be deleted. This date is only used if it is after the regular start date and before the end date.
Only use in special cases!

End Date
(replicationTailSyncRelativeEndDate)

Day Range per Run
(replicationTailSyncBlockDayRange)

This controls the number of days that should be processed in each run. A high value can lead to the source database and OpenSearch being heavily loaded.

Delete Old Records
(replicationTailSyncRelativeDeleteOlderThanDate)

Enable Tail Sync logging
(replicationTailSyncLoggingEnabled)

If enabled, Tail Sync runs are logged in the `bpc-tailsync-log` index.

Consistency Check

After replication runs, a simple consistency check is performed. The number of documents in the source and target are compared.

Name (ID) Description

Name (`ID`)	Description
Consistency Check Frequency (`replicationConsistencyCheckFrequency`)	The frequency with which the consistency check is performed. The number of data records in the source database is compared with the number in OpenSearch. The check can severely affect the performance of the replication and should be switched off if historical data is replicated. 0 = no check; 3 = check every 3rd replication

Consistency Check Frequency
(replicationConsistencyCheckFrequency)

The frequency with which the consistency check is performed. The number of data records in the source database is compared with the number in OpenSearch. The check can severely affect the performance of the replication and should be switched off if historical data is replicated.
0 = no check; 3 = check every 3rd replication

Lookup Joins

These can be used to enrich replicated documents with additional data. For example, if the data to be replicated only contains a partner ID, the partner's name and other info can be added for monitoring.

Name (ID) Description

Name (`ID`)	Description
Lookup Joins (`join`)	Can be used to automatically enrich logging documents with additional data, e.g. if the data to be logged only contains an ID and further associated data can be loaded via an existing index.

Lookup Joins
(join)

Can be used to automatically enrich logging documents with additional data, e.g. if the data to be logged only contains an ID and further associated data can be loaded via an existing index.

Basic Settings

Setting (Key) Data Type Description

Setting (Key)	Data Type	Description
Replication_Enabled (`replicationEnabled`)	boolean	Enable/disable the replication job
Replication_StartDate (`replicationStartDate`)	String	Replicate records that are newer than this date. Date format "yyyy-MM-dd' 'HH:mm:ss.SSS" or as a relative value in the format: `1 second\|minute\|hour\|day\|week\|month\|year ago n seconds\|minutes\|hours\|days\|weeks\|months\|years ago` Examples `1970-01-01 00:00:00.000 1 week ago 6 months ago 42 days ago`

Replication_Enabled
(replicationEnabled)

boolean

Enable/disable the replication job

Replication_StartDate
(replicationStartDate)

String

Replicate records that are newer than this date.
Date format "yyyy-MM-dd' 'HH:mm:ss.SSS"

as a relative value in the format:

1 second|minute|hour|day|week|month|year ago
n seconds|minutes|hours|days|weeks|months|years ago

Examples

1970-01-01 00:00:00.000
1 week ago
6 months ago
42 days ago

Source

Setting (Key) Data Type Description

Source_DataSource
(rdmsDataSourceName)

String

Data source to be used (ID of the Backend Connections of type data_source)

Source_Table
(sourceTable)

String

Table name of the source or name of the CTE if the optional 'Source_CommonTableExpressionQuery' is used.

Source_Timezone
(sourceTimeZone)

String

Time zone of the date fields used in the source database table (internally uses TimeZone.getTimeZone). Applies only to the actual data and not to the 'lastUpdateColumn' column.

Examples

UTC
GMT+1
GMT-8
America/Los_Angeles
Europe/Berlin
Etc/GMT
CET

Source_IdColumns
(idColumns)

String

Columns used to form a unique key in OpenSearch. For example: "PROCESSID,CHILDID"

Source_LastUpdateColumn
(lastUpdateColumn)

String

This column is used to determine the age of the record

It is strongly recommended to create an index on these columns in the source database.

Source_LastUpdateColumnTimezone
(lastUpdateColumnTimeZone)

String

Time zone used in the source database table for the data in the lastUpdateColumn column.

This setting is only used in conjunction with [adjustUpperDateLimitInSeconds].

Examples

UTC
GMT+1
GMT-8
America/Los_Angeles
Europe/Berlin
Etc/GMT
CET

Source_QueryTimeoutInSeconds
(sourceQueryTimeoutInSeconds)

Integer

Specifies how long the JDBC driver waits for a response from the database. See also JDBC java.sql.Statement.setQueryTimeout()

Source_CommonTableExpressionQuery
(sourceCommonTableExpressionQuery)

String

Can be used instead of database views. Must contain only the SELECT statement of a Common Table Expression (CTE). The WITH $sourceTable$ AS ( $sourceCommonTableExpressionQuery$ ) $bpcQuery$; is generated so that it matches the queries issued by the BPC. If this option is used, then 'Source_Table' (sourceTable) is used as the name of the CTE. Incidentally, this can match the name of an existing database table.

Target

Setting (Key) Data Type Description

Setting (Key)	Data Type	Description
Target_Index (`targetIndex`)	String	Target index in OpenSearch. If it does not yet exist, the index is created automatically.
Target_CaseSensitivityOfFields (`targetIndexCaseSensitivityOfFields`)	String	Specifies how the fields should be created in OpenSearch (case sensitivity). `asSource` = created exactly as returned by the database `lowerCase` = column names are converted to lowercase `upperCase` = column names are converted to uppercase
Target_IndexCreationSettings (`targetIndexCreationSettings`)	JSON	Assign different settings to the target index during creation. If this field is empty, the Core Services settings → Core_IndexCreationSettings are used as usual. If it is set, only these settings are used. In this case, the settings from Core_IndexCreationSettings must be used as a basis via copy and paste. Example with "Index Sorting" added `{ "number_of_shards": "5", "number_of_replicas": "1", "index": { "sort.field": "LASTUPDATE", "sort.order": "desc" }, "analysis": { "normalizer": { "lowercaseNormalizer": { "filter": [ "lowercase" ], "char_filter": [], "type": "custom" } } } }` If "Index Sorting" is to be used, OpenSearch mappings must also be created immediately for the specified sorting fields (LASTUPDATE in the example) (see Target_IndexMappings).
Target_IndexMappings (`targetIndexMappings`)	JSON	Assign a mapping to the target index upon creation. This should only be necessary in specific cases. The mapping for the “Index Sorting” example above. `{ "properties": { "LASTUPDATE": { "type": "date" } } }`
Target_IndexDynamicTemplates (`targetIndexDynamicTemplates`)	JSON	Assign a custom mapping to the target index. If this is set, the global setting (see Core Settings → Core_IndexDynamicTemplates) is not used. The Elasticsearch documentation (link provided until the OpenSearch documentation is on par) contains more information about the capabilities of dynamic templates. In the following example, all fields that OpenSearch recognizes as text fields (strings) are assigned a mapping (“alle_textfelder”) in which the content is not analyzed (this saves storage space, and the data can still be displayed). Plus one exception (“special_case”): For all text fields with the name suffix ‘name’, our default mapping is used. Example: `[ { "spezialfall": { "match_mapping_type": "string", "match": "name", "mapping": { "type": "text", "fields": { "lowercase": { "normalizer": "lowercaseNormalizer", "type": "keyword" }, "raw": { "type": "keyword" } } } } }, { "alle_textfelder": { "match_mapping_type": "string", "match": "", "mapping": { "type": "keyword", "analyzer": false } } } ]` To specify the OpenSearch type for a database field. Occasionally, OpenSearch may make a mistake with the mapping and use an inappropriate type. Specific example: The Oracle column named ‘ZAHL’ with the data type ‘NUMBER(10,2)’ is created in the OpenSearch mapping as type ‘long’ instead of ‘float’. This can be corrected using the example below. Example `[ { "ZAHL_long_als_float": { "match_mapping_type": "long", "match": "ZAHL", "mapping": { "type": "float" } } } ]`

Target_Index
(targetIndex)

String

Target index in OpenSearch.
If it does not yet exist, the index is created automatically.

Target_CaseSensitivityOfFields
(targetIndexCaseSensitivityOfFields)

String

Specifies how the fields should be created in OpenSearch (case sensitivity).

asSource = created exactly as returned by the database
lowerCase = column names are converted to lowercase
upperCase = column names are converted to uppercase

Target_IndexCreationSettings
(targetIndexCreationSettings)

JSON

Assign different settings to the target index during creation.

If this field is empty, the Core Services settings → Core_IndexCreationSettings are used as usual.
If it is set, only these settings are used. In this case, the settings from Core_IndexCreationSettings must be used as a basis via copy and paste.

Example with "Index Sorting" added

{
   "number_of_shards": "5",
   "number_of_replicas": "1",
   "index": {
      "sort.field": "LASTUPDATE",
      "sort.order": "desc"
   },
   "analysis": {
      "normalizer": {
         "lowercaseNormalizer": {
            "filter": [
               "lowercase"
            ],
            "char_filter": [],
            "type": "custom"
         }
      }
   }
}

If "Index Sorting" is to be used, OpenSearch mappings must also be created immediately for the specified sorting fields (LASTUPDATE in the example) (see Target_IndexMappings).

Target_IndexMappings
(targetIndexMappings)

JSON

Assign a mapping to the target index upon creation. This should only be necessary in specific cases.

The mapping for the “Index Sorting” example above.

{
   "properties": {
      "LASTUPDATE": {
         "type": "date"
      }
   }
}

Target_IndexDynamicTemplates
(targetIndexDynamicTemplates)

JSON

Assign a custom mapping to the target index. If this is set, the global setting (see Core Settings → Core_IndexDynamicTemplates) is not used.

The Elasticsearch documentation (link provided until the OpenSearch documentation is on par) contains more information about the capabilities of dynamic templates.

In the following example, all fields that OpenSearch recognizes as text fields (strings) are assigned a mapping (“alle_textfelder”) in which the content is not analyzed (this saves storage space, and the data can still be displayed). Plus one exception (“special_case”): For all text fields with the name suffix ‘name’, our default mapping is used.

Example:

[
  {
    "spezialfall": {
      "match_mapping_type": "string",
      "match": "*name",
      "mapping": {
        "type": "text",
        "fields": {
          "lowercase": {
            "normalizer": "lowercaseNormalizer",
            "type": "keyword"
          },
          "raw": {
            "type": "keyword"
          }
        }
      }
    }
  },
  {
    "alle_textfelder": {
      "match_mapping_type": "string",
      "match": "*",
      "mapping": {
        "type": "keyword",
        "analyzer": false
      }
    }
  }
]

To specify the OpenSearch type for a database field. Occasionally, OpenSearch may make a mistake with the mapping and use an inappropriate type. Specific example: The Oracle column named ‘ZAHL’ with the data type ‘NUMBER(10,2)’ is created in the OpenSearch mapping as type ‘long’ instead of ‘float’. This can be corrected using the example below.

Example

[
  {
    "ZAHL_long_als_float": {
      "match_mapping_type": "long",
      "match": "ZAHL",
      "mapping": {
        "type": "float"
      }
    }
  }
]

Advanced Settings / Advanced

Setting (Key) Data Type Description

Replication_RestartWhereLeftOff
(restartReplicationWhereLeftOff)

boolean

When the server starts or a job is modified, the replication jobs are restarted and begin replicating from the beginning again (see replicationStartDate). If this setting is set to 'true', replication resumes from the date of the most recent record that was replicated. These timestamps are stored as metadata in the OpenSearch index.

Replication_Delay
(replicationDelay)

Integer

Delay in seconds after which replication starts

Replication_Interval
(replicationInterval)

Integer

Interval in seconds for replication

Replication_BlockDayRange
(replicationBlockDayRange)

Integer

Number of days (block) to be processed during each job run. Do not set this value too high, as this will cause an increased load on the source database and will not allow OpenSearch/Lucene any time to catch its breath.

Example: 10 days—the job has just reached March 10, 2015; then the records from March 10, 2015, through March 20, 2015, will be replicated, and in the next run, those from March 20, 2015, through March 30, 2015.

Replication_BlockSize
(replicationBlockSize)

Integer

Block size for transferring data from the database to OpenSearch. Number of database records that the JDBC driver reads as a block and keeps in memory.

It currently appears that a larger block size is better. See also: Replication Duration. But please don’t overdo it, as this will otherwise result in OutOfMemory exceptions. A block size of 2500 is already too high for some database tables.

Replication_SyncFiles
(replicationSyncFiles)

boolean

Synchronization of BLOBs

Please note that retroactive activation is currently not possible.

Replication_UnzipSyncedFiles
(replicationUnzipSyncedFiles)

boolean

Automatically unpacks the contents of synchronized files.

Only applies if replicationSyncFiles is enabled.

Replication_AdjustUpperDateLimitInSeconds
(adjustUpperDateLimitInSeconds)

Integer

Workaround for a database issue where record updates—initiated by a database trigger—are written too late and therefore cannot be taken into account during the replication run (we’re talking about 1–3 seconds here).

This option selects records with an update timestamp minus the specified value. By default (0), this option is disabled, and records are selected as before using a timestamp in the future (replicationBlockDayRange).

Example: 3 seconds. Records are selected from the timestamp of the most recently replicated record up to the current database timestamp minus 3 seconds. Of course, the most recent data will not always be available in BPC, but hopefully it will be in a consistent state with the database.

Replication_VamOrganizationId
(vamOrganizationId)

String

experimental

If not empty, special features for replicating Data Management data are applied. Specifically, only the data associated with the entered organizationId (as specified in the Data Management configuration) is replicated.

Replication_LoggingEnabled
(replicationLoggingEnabled)

boolean

Here, logging of replication runs can be enabled or disabled for each replication job.

Snapshots

During a snapshot, all documents (created after the specified 'replicationStartDate') from the OpenSearch index (see 'targetIndex') are copied to a new index at the specified time. Finally, the alias is redirected to the new index, and the previous index is deleted. This can be performed, for example, once a week to restore a lean index.

Additional note: Documents marked as deleted in OpenSearch (either manually deleted or deleted by Tail Sync) are, of course, not included in the new index. Caution: The associated replication run is suspended until the shadow copy has been created.

Setting (Key) Data Type Description

Setting (Key)	Data Type	Description
ShadowCopy_Enabled (`replicationShadowCopyEnabled`)	boolean	Enable the creation of shadow copies.
ShadowCopy_CronPattern (`replicationShadowCopyCronPattern`)	String	Cron-like pattern (following Quartz scheduler syntax) to specify when the shadow copies should be created. Examples `0 15 16 ? * SUN = Jeden Sonntag um 16:15 Uhr 0 /30 * * ? = Alle 30 Minuten 30 59 11 ? * 1,2,3,4,5 = Am Montag, Dienstag, Mittwoch, Donnerstag und Freitag um 11:59:30 Uhr` Additional examples and documentation can be found on the Quartz scheduler website.
ShadowCopy_KeepCopiesCount (`replicationShadowCopyKeepCopiesCount`)	Integer	Number of shadow copies to be retained. `0` = No shadow copies are retained `3` = 3 shadow copies are retained (these are in the 'Close' status to conserve resources)

ShadowCopy_Enabled
(replicationShadowCopyEnabled)

boolean

Enable the creation of shadow copies.

ShadowCopy_CronPattern
(replicationShadowCopyCronPattern)

String

Cron-like pattern (following Quartz scheduler syntax) to specify when the shadow copies should be created.

Examples

0 15 16 ? * SUN = Jeden Sonntag um 16:15 Uhr
0 */30 * * * ? = Alle 30 Minuten
30 59 11 ? * 1,2,3,4,5 = Am Montag, Dienstag, Mittwoch, Donnerstag und Freitag um 11:59:30 Uhr

Additional examples and documentation can be found on the Quartz scheduler website.

ShadowCopy_KeepCopiesCount
(replicationShadowCopyKeepCopiesCount)

Integer

Number of shadow copies to be retained.

0 = No shadow copies are retained

3 = 3 shadow copies are retained (these are in the 'Close' status to conserve resources)

Tail Sync

The 'Tail Sync' operates on the current index (see 'targetIndex') and synchronizes older data (new/modified/deleted database records) with OpenSearch so that it matches the database again. This can be performed, for example, once every night. Documents older than the replication start date (see 'replicationStartDate') are first deleted from the OpenSearch index. It then goes through the database in blocks (default: 10-day increments) and synchronizes it with OpenSearch. In the process, new or modified database records are added to the OpenSearch index, and documents that no longer exist in the database are deleted from OpenSearch. Tail Sync works only with the specified 'idColumns' and 'lastUpdateColumn' fields (Hint: a suitable DB index works wonders!). The complete data is read only for new or modified database records.

As an alternative or in addition to a scheduled Tail Sync, it can also be started manually. For this purpose, there is a button at Replication → Jobs for each replication job that starts a Tail Sync. For manual starts, the settings replicationTailSyncEnabled and replicationTailSyncCronPattern have no effect; all other settings are taken into account.

The sync does not run up to the current date (see [replicationTailSyncRelativeEndDate]), as these records continue to be processed by our normal replication.

Setting (Key) Data Type Description

TailSync_Enabled
(replicationTailSyncEnabled)

boolean

Enable 'Tail Sync'

TailSync_CronPattern
(replicationTailSyncCronPattern)

String

Cron-like pattern (following Quartz scheduler syntax) to specify when Tail Sync should start.

Examples

0 5 2 * * ? = Jede Nacht um 2:05 Uhr
0 35 21 ? * Sun = Jeden Sonntag um 21:35 Uhr
30 5 3 ? * 1,2,3,4,5 = Am Montag, Dienstag, Mittwoch, Donnerstag und Freitag um 3:05:30 Uhr

Additional examples and documentation can be found on the Quartz scheduler website.

TailSync_BlockSize
(replicationTailSyncBlockSize)

Integer

Block size of the database driver. Note the information regarding the replicationBlockSize option.

TailSync_RelativeStartDate
(replicationTailSyncRelativeStartDate)

String

The relative start date. Synchronization should begin from this point onward. Data that predates the replication start date will continue to be deleted. This date is only used if it falls after the regular start date and before the end date.

1 second|minute|hour|day|week|month|year ago
n seconds|minutes|hours|days|weeks|months|years ago

This option should only be used in special cases. Inconsistencies may occur if records in the database that predate this relative start date are deleted. These records will then remain in the OpenSearch index even though they no longer exist in the database.

TailSync_RelativeEndDate
(replicationTailSyncRelativeEndDate)

String

The relative end date. Synchronization should be performed up to this point and no further.

1 second|minute|hour|day|week|month|year ago
n seconds|minutes|hours|days|weeks|months|years ago

TailSync_BlockDayRange
(replicationTailSyncBlockDayRange)

Integer

Number of days (block) during which the data should be processed. See also replicationBlockDayRange.

TailSync_RelativeDeleteOlderThanDate
(replicationTailSyncRelativeDeleteOlderThanDate)

String

The relative deletion date to be specified. All documents older than this will be deleted.

If this option is not set, all documents older than the replication start date will be deleted (see replicationStartDate).

TailSync_LoggingEnabled
(replicationTailSyncLoggingEnabled)

boolean

The relative deletion date to be specified. Specifies whether Tail Sync runs in the bpc-tailsync-log index should be logged for this replication job.

Additionally, the global setting replicationJobsTailsyncLogEnabled must be enabled.

Consistency Check

A simple consistency check is performed after the replication runs. This compares the number of documents in the source and the destination. Specifically, this comparison covers the period between 'replicationStartDate' and the latest date present in the destination (OpenSearch).

With large amounts of data (tens of millions of records), the system may experience increased load, and the initial replication may be significantly slowed down. Enable the consistency check only if you are certain that the data will be fully replicated.

Setting (Key) Data Type Description

Setting (Key)	Data Type	Description
ConsistencyCheck_Frequency (`replicationConsistencyCheckFrequency`)	Integer	The frequency at which the consistency check is performed. Examples: `0` = never performed `1` = performed after every replication run `10` = performed after every 10th replication run

ConsistencyCheck_Frequency
(replicationConsistencyCheckFrequency)

Integer

The frequency at which the consistency check is performed.

Examples:

0 = never performed
1 = performed after every replication run
10 = performed after every 10th replication run

Lookup Joins

Can be used to enrich documents to be replicated with additional data. For example, if the data to be replicated contains only a partner ID and the partner’s name, etc., is still required in the monitor.

OpenSearch uses denormalization, which means this data is included in the document and is available just like all other data (high-performance search, aggregation, etc.). Prerequisite: The lookup “tables” must be available as separate indexes and can be imported, for example, via an additional replication from a database table.

The lookup data can be updated manually (BPC Settings → Overview → Status → Replication → Jobs → Job → Synchronize Lookup Joins) or automatically using our OpenSearch BPC plugin (os-BPC-plugin).

Multiple lookup tables can be referenced; the possible values for an ENTRY are described below. Structure: "join": [ { ENTRY }, { ENTRY }, … ]

For the value comparison to work, the column types in the database tables must be identical. For string fields, “*.raw” should be used as the lookupKeyField; for numeric fields, however, use the field name without the “.raw” suffix

Lookup Joins Configuration Example

[
    {
        "keyField": "PARTNER",
        "lookupIndex": "lookup-partner",
        "lookupKeyField": "ID.raw",
        "resultFieldsPrefix": "partner_",
        "resultFieldsExcluded": [ "ID", "LASTUPDATE" ]
    },
    {
        "keyField": "MESSAGETYPE",
        "lookupIndex": "lookup-messagetype",
        "lookupKeyField": "ID.raw",
        "resultFieldsPrefix": "messagetype_",
        "resultFieldsExcluded": [ "ID", "LASTUPDATE" ]
    }
]

Field	Data Type	Description
keyField	String	The key field in the table to be replicated. The data is retrieved from the lookup index based on the value of this field. Example: PARTNER_ID This contains the partner’s ID, and the specified lookup index contains the actual partner data to be retrieved. (Hint: DB Foreign Key)
keyFieldValuesSeparator	String	If the keyField contains multiple values separated by a delimiter, this delimiter can be specified here. The lookup join is then performed on the individual values. Example: The 'lookupIndex' contains the following data: ID = 20, LONGNAME = Gram ID = 30, LONGNAME = Kilogram ID = 42, LONGNAME = Zentimeter If the keyFieldValuesSeparator is '%%', the resultFieldsPrefix is 'MENGENEINHEIT_', and the passed keyValue is '42%%20%%30', then the following field is created: MENGENHEIT_LONGNAME = Zentimeter%%Gramm%%Kilogramm
lookupIndex	String	The OpenSearch index containing the lookup data. Example: lookup-partner In this example, the partner’s detailed data.
lookupKeyField	String	Example: ID.raw This field searches for the value of the 'keyField' (see above). (Hint: DB Primary Key)
resultFieldsPrefix	String	The fields to be retrieved from the lookup index must be prefixed with a unique prefix to avoid conflicts with existing fields. Example: partner_
resultFieldsIncluded	String Array	If not all fields from the lookup index are to be included, the fields to be included can be specified here. Example: [ "FIRST_NAME", "LAST_NAME" ]
resultFieldsExcluded	String Array	If almost all fields are to be included, the fields to be excluded can be specified here. Example: [ "ID", "UPDATED" ]

Field

Data Type

Description

keyField

String

The key field in the table to be replicated. The data is retrieved from the lookup index based on the value of this field.

Example: PARTNER_ID
This contains the partner’s ID, and the specified lookup index contains the actual partner data to be retrieved. (Hint: DB Foreign Key)

keyFieldValuesSeparator

String

If the keyField contains multiple values separated by a delimiter, this delimiter can be specified here. The lookup join is then performed on the individual values.

Example:

The 'lookupIndex' contains the following data:

ID = 20, LONGNAME = Gram
ID = 30, LONGNAME = Kilogram
ID = 42, LONGNAME = Zentimeter

If the keyFieldValuesSeparator is '%%', the resultFieldsPrefix is 'MENGENEINHEIT_', and the passed keyValue is '42%%20%%30', then the following field is created:

MENGENHEIT_LONGNAME = Zentimeter%%Gramm%%Kilogramm

lookupIndex

String

The OpenSearch index containing the lookup data.

Example: lookup-partner
In this example, the partner’s detailed data.

lookupKeyField

String

Example: ID.raw
This field searches for the value of the 'keyField' (see above). (Hint: DB Primary Key)

resultFieldsPrefix

String

The fields to be retrieved from the lookup index must be prefixed with a unique prefix to avoid conflicts with existing fields.

Example: partner_

resultFieldsIncluded

String Array

If not all fields from the lookup index are to be included, the fields to be included can be specified here.

Example: [ "FIRST_NAME", "LAST_NAME" ]

resultFieldsExcluded

String Array

If almost all fields are to be included, the fields to be excluded can be specified here.

Example: [ "ID", "UPDATED" ]

Logging Replications

Replication job runs can be logged and displayed, for example, via the automatically created “Replication Jobs Monitor.” Similarly, tail sync runs can also be logged and viewed in the “Tail Sync Logs Monitor.” These monitors are created at startup if they do not already exist. They cannot be permanently deleted. Likewise, the indexes bpc-replicationjobs-log and bpc-tailsync-log are automatically created when a run needs to be logged.

Logging can be enabled or disabled globally as well as for each replication job (see the corresponding settings above). By default, logging is enabled globally for both replications and tail syncs, but disabled in the individual replication jobs. Therefore, logging is not enabled by default.

The log entries for replication runs are written in bulk to the aforementioned index after some time. This has little impact on the system’s overall performance. However, a very large number of entries can be generated in a very short time (millions in a few hours), so you might want to run the cleanup often enough and keep an eye on the number of documents it contains: Settings → Core Services → Indexes

Since tail syncs occur much less frequently, the bpc-tailsync-log index should not grow too quickly during tail sync logging; nevertheless, it is regularly purged of old entries (see configuration above).

Keywords: