Die BPC Version 4.1 wird nicht mehr gewartet.

Sollten Sie diese BPC Version nutzen, empfehlen wir Ihnen eine Migration auf eine aktuelle Version. Die Dokumentation zur neusten BPC Version finden Sie hier. Sollten Sie Fragen haben, wenden Sie sich bitte an unseren Support.

Replication

This service replicates data from an RDBMS to OpenSearch.

Automatic Migration of Replication Settings

When BPC version >= 3.1 is started for the first time, any existing replication settings are automatically migrated to the new “Replication” module. Once all replication jobs have been successfully migrated, please—to avoid confusion and misunderstandings—delete the two old replication settings (Core Services → Settings) “Replication_Jobs” and “Replication_Threads.” These are marked there with orange bars.

Impact on System Performance

Replication has a direct impact on system performance. In particular, the number of threads (Replication → Settings → Replication_Threads) and the interval (replicationInterval) play a role. The more jobs that run in a short period of time, the higher the CPU utilization. This is especially true when large amounts of data are being replicated.

While the replicationBlockSize has a positive effect on replication duration, it also directly affects Apache Karaf’s memory requirements.

Preparing for Data Record Timeliness

Before setting up replication, a few prerequisites should be met. Replication requires a column containing exclusively increasing timestamps for each table (both Log and Childlog), as this timestamp is used to subsequently identify newer or updated records. If different processes, servers, etc., log data to the database in parallel—and thus potentially with a time lag (even if only by milliseconds)—it may happen that individual records are not displayed in BPC because they were committed with a timestamp older than the most recent replicated one.

It is therefore recommended:

  1. to log the database server’s time rather than the logging process’s server time, since different process servers may have slightly different times

  2. to ensure that this column is actually updated with every INSERT or UPDATE

Both can be easily achieved using an additional (hidden) column with a default value and an associated trigger. This has the advantage that nothing changes at all for the logging process)

Adding a technical TIMESTAMP column for replication

Oracle

For PM-Log and Childlog (without a prefix) in Oracle, the procedure for creating them looks like this, for example:

--LOG:
--Spalte (mit 6Nachkommastellen Präzission) ergänzen, Defaultvalue für Inserts auf die aktuelle Uhrzeit in UTC setzen (damit ist keine Anpassung im Loggingprozess nötig)
ALTER TABLE LOG ADD (DB_UPDATE_TS TIMESTAMP DEFAULT SYSTIMESTAMP AT TIME ZONE 'UTC' NOT NULL);

--Bestehende Logeinsträge wieder "verteilen", sonst kommt die Replikation nicht gut klar:
--Entweder alle in einem Block (!besser nicht mit Tabellen > 1mio Einträge!):
UPDATE LOG SET DB_UPDATE_TS = TIMESTAMP;
Commit;

  --oder bei großen Datenmengen und Livesystemen, die auch noch was mit den Tabellen machen wollen, als anonymer PL/SQL-Block mit wenig Undo-Tablespacebedarf:
  --hierfür sollte zusätzlich ein Index auf der Timestamp-Spalte liegen um FullTableScans zu vermeiden!
    declare begin
      FOR counter IN 0 .. 3650 LOOP
        --dbms_output.put_line(to_char(to_date('2010-01-01', 'YYYY-MM-DD') + counter, 'YYYY-MM-DD') || ' - ' || to_char(to_date('2010-01-01', 'YYYY-MM-DD') + 1 + counter, 'YYYY-MM-DD'));
        --LOG:
        update log set DB_UPDATE_TS = TIMESTAMP where TIMESTAMP between to_date('2010-01-01', 'YYYY-MM-DD') + counter and to_date('2010-01-01', 'YYYY-MM-DD') + 1 + counter;
        commit;
        --CHILDLOG:
        update childlog set DB_UPDATE_TS = TIMESTAMP where TIMESTAMP between to_date('2010-01-01', 'YYYY-MM-DD') + counter and to_date('2010-01-01', 'YYYY-MM-DD') + 1 + counter;
        commit;
      END LOOP;
    end;

--Einen Trigger anlegen, der die Spalte bei jedem Update neu setzt (damit ist keine Anpassung im Loggingprozess nötig)
CREATE OR REPLACE
TRIGGER LOG_DB_UPDATE_TS
BEFORE UPDATE ON LOG
REFERENCING NEW AS NEW OLD AS OLD
FOR EACH ROW
DECLARE
BEGIN
:NEW.DB_UPDATE_TS := SYSTIMESTAMP AT TIME ZONE 'UTC';
END;
/

--Index erstellen:
CREATE INDEX IDX_LOG_DBLU ON LOG(DB_UPDATE_TS) COMPUTE STATISTICS;


--Das ganze Geraffel noch mal für Childlog:
ALTER TABLE CHILDLOG ADD (DB_UPDATE_TS TIMESTAMP DEFAULT SYSTIMESTAMP AT TIME ZONE 'UTC' NOT NULL);

--Siehe ggf. PLSQL-Block oben!
UPDATE CHILDLOG SET DB_UPDATE_TS = TIMESTAMP;
commit;

CREATE OR REPLACE
TRIGGER CHILDLOG_DB_UPDATE_TS
BEFORE UPDATE ON CHILDLOG
REFERENCING NEW AS NEW OLD AS OLD
FOR EACH ROW
DECLARE
BEGIN
:NEW.DB_UPDATE_TS := SYSTIMESTAMP AT TIME ZONE 'UTC';
END;
/

CREATE INDEX IDX_CHILDLOG_DBLU ON CHILDLOG(DB_UPDATE_TS) COMPUTE STATISTICS;
--Fertig

MSSQL

TIMESTAMP should be set to UTC; otherwise, a blind spot will occur between 2 and 3 a.m. during the daylight saving time change! The code does not yet account for this—you may want to use SYSUTCDATETIME instead of SYSDATETIME —but be sure to check the behavior in BPC.

The trigger for Childlog is missing. However, this is usually not needed, since entries are only ever added and never updated.

 
/* Spalten hinzufügen. DATETIME2 für bessere Auflösung als DATETIME. Offensichtlich ist MSSQL nicht millisekundengenau, daher statt current_timestamp die Funktion SYSDATETIME() verwenden, diese ist nanosekundengenau. */
ALTER TABLE [LOG] ADD DB_UPDATE_TS DATETIME2 DEFAULT SYSDATETIME() NOT NULL;
GO

ALTER TABLE [CHILDLOG] ADD DB_UPDATE_TS DATETIME2 DEFAULT SYSDATETIME() NOT NULL;
GO

/* Spalten initial füllen */
BEGIN
    UPDATE [LOG] SET DB_UPDATE_TS = [timestamp];
END
GO

BEGIN
    UPDATE [CHILDLOG] SET DB_UPDATE_TS = [timestamp];
END
GO

/* Trigger */
CREATE TRIGGER LOG_DB_UPDATE_TS
  ON [LOG]
  AFTER UPDATE
  AS
BEGIN
    IF NOT UPDATE(DB_UPDATE_TS)
    BEGIN
        UPDATE t
            SET t.DB_UPDATE_TS = SYSDATETIME()
            FROM [LOG] AS t
            INNER JOIN inserted AS i
            ON t.PROCESSID = i.PROCESSID;
    END
END
GO

/* Indizes */
CREATE INDEX IDX_LOG_DBLU ON LOG (DB_UPDATE_TS);
CREATE INDEX IDX_CHILDLOG_DBLU ON CHILDLOG (DB_UPDATE_TS);

MySQL

and for MySQL, it’s very simple:

# Fügt der Tabelle einen von der DB verwalteten ausreichend genauen Timestamp hinzu
ALTER TABLE LOG ADD DB_UPDATE_TS TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6)
ALTER TABLE CHILDLOG ADD DB_UPDATE_TS TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6)

PostgreSQL

and for PostgreSQL, also via triggers:

ALTER TABLE log ADD db_update_ts timestamp NOT null DEFAULT (timezone('UTC', now()));
ALTER TABLE childlog ADD db_update_ts timestamp NOT null DEFAULT (timezone('UTC', now()));

CREATE OR REPLACE FUNCTION update_db_update_ts_column()
RETURNS TRIGGER AS $$
BEGIN
    NEW.db_update_ts = timezone('UTC', now());
    RETURN NEW;
END;
$$ language 'plpgsql';

CREATE TRIGGER update_log_db_update_ts BEFORE INSERT OR UPDATE ON log FOR EACH ROW EXECUTE PROCEDURE update_db_update_ts_column();
CREATE TRIGGER update_childlog_db_update_ts BEFORE INSERT OR UPDATE ON childlog FOR EACH ROW EXECUTE PROCEDURE update_db_update_ts_column();

CREATE INDEX idx_log_dblu ON log (db_update_ts);
CREATE INDEX idx_childlog_dblu ON childlog (db_update_ts);

Configuration

Data Sources

Data sources are connections to individual databases. The individual replication jobs then use/reference these. They are set up under Backend Connections as type "data_source" and referenced by their component ID.

Interface

A dedicated interface is available for setting up individual replication jobs under Settings → Replication → Components → Editor. Through the interface, entries can be created, deleted, duplicated, and also enabled or disabled individually.

Configuration Parameters of the Replication Module

The various parameters and their associated functions are described below. These can be found under BPC Administration → Replication → General.

Setting (Key) Data Type Description

Replication_Threads
(replicatorThreadCount)

int

Number of threads used to execute replication jobs simultaneously.

Replication_Templates
(templates)

JSON

To provide templates that can then be selected for individual replication jobs (“Load Template”). To use these to populate fields such as “Index Settings,” “Index Mappings,” and “Dynamic Templates.”

Logs_Enabled
(replicationJobsLogEnabled)

boolean

Determines whether the runs of the replication jobs (if enabled there) should be logged in the special OpenSearch index 'BPC-replicationjobs-log' or not.

Logs_BulkWritePeriodInSeconds
(replicationJobsLogBulkWritePeriodInSeconds)

int

Specified in seconds. Logging these entries after every replication run would negatively impact the system’s overall performance. Therefore, the log entries are collected and written in bulk after this period.

Logs_CleanupPeriodInMinutes
(replicationJobsLogCleanupPeriodInMinutes)

int

Specified in minutes. Interval at which the 'BPC-replicationjobs-log' index should be cleaned up.

Logs_DeleteEntriesOlderThan
(replicationJobsLogDeleteEntriesOlderThan)

text

Specifies how long documents should be retained in the 'BPC-replicationjobs-log' index. Used during cleanup. Example: 3 days ago

Configuration Parameters of a Replication Job

The various Parameters and their associated functions are described below. These can be found under BPC Administration → Replication → Components. It is recommended to use the specialized interface: BPC Administration → Replication → Editor

Basic Settings

Setting (Key) Data Type Description

Replication_Enabled
(replicationEnabled)

boolean

Enable/disable the replication job

Replication_StartDate
(replicationStartDate)

String

Replicate records that are newer than this date.
Date format "yyyy-MM-dd' 'HH:mm:ss.SSS"

or

as a relative value in the format:

1 second|minute|hour|day|week|month|year ago
n seconds|minutes|hours|days|weeks|months|years ago
Examples
1970-01-01 00:00:00.000
1 week ago
6 months ago
42 days ago

Source

Setting (Key) Data type Description

Source_DataSource
(rdmsDataSourceName)

String

Data source to be used (ID of Backend Connections of type data_source)

Source_Table
(sourceTable)

String

Table name of the source

Source_Timezone
(sourceTimeZone)

String

Time zone of the date fields used in the source database table (internally uses TimeZone.getTimeZone). Applies only to the actual data and not to the 'lastUpdateColumn' column.

Examples
UTC
GMT+1
GMT-8
America/Los_Angeles
Europe/Berlin
Etc/GMT
CET

Source_IdColumns
(idColumns)

String

Columns used to form a unique key in OpenSearch. For example: "PROCESSID,CHILDID"

Source_LastUpdateColumn
(lastUpdateColumn)

String

This column is used to determine the age of the record

It is strongly recommended to create an index on these columns in the source database.

Source_LastUpdateColumnTimezone
(lastUpdateColumnTimeZone)

String

Time zone used in the source database table for the data in the lastUpdateColumn column.

This setting is only used in conjunction with [adjustUpperDateLimitInSeconds].
Examples
UTC
GMT+1
GMT-8
America/Los_Angeles
Europe/Berlin
Etc/GMT
CET

Source_QueryTimeoutInSeconds
(sourceQueryTimeoutInSeconds)

Integer

Specifies how long the JDBC driver waits for a response from the database. See also JDBC java.sql.Statement.setQueryTimeout()

Target

Setting (Key) Data Type Description

Target_Index
(targetIndex)

String

Target index in OpenSearch.
If it does not yet exist, the index is created automatically.

Target_CaseSensitivityOfFields
(targetIndexCaseSensitivityOfFields)

String

Specifies how the fields should be created in OpenSearch (case-sensitive or case-insensitive).

  • asSource = Created exactly as returned by the database

  • lowerCase = Column names are converted to lowercase

  • upperCase = Column names are converted to uppercase

Target_IndexCreationSettings
(targetIndexCreationSettings)

JSON

Assign different settings to the target index during creation.

  • If this field is empty, the Core Services settings → Core_IndexCreationSettings are used as usual.

  • If it is set, only these settings are used. In this case, the settings from Core_IndexCreationSettings must be copied and pasted as a basis.

Example with "Index Sorting" added
{
   "number_of_shards": "5",
   "number_of_replicas": "1",
   "index": {
      "sort.field": "LASTUPDATE",
      "sort.order": "desc"
   },
   "analysis": {
      "normalizer": {
         "lowercaseNormalizer": {
            "filter": [
               "lowercase"
            ],
            "char_filter": [],
            "type": "custom"
         }
      }
   }
}

If "Index Sorting" is to be used, OpenSearch mappings must also be created immediately for the specified sorting fields (LASTUPDATE in the example) (see Target_IndexMappings).

Target_IndexMappings
(targetIndexMappings)

JSON

Assign a mapping to the target index upon creation. This should only be necessary in certain cases.

The mapping for the “Index Sorting” example above.
{
   "properties": {
      "LASTUPDATE": {
         "type": "date"
      }
   }
}

Target_IndexDynamicTemplates
(targetIndexDynamicTemplates)

JSON

Assign a custom mapping to the target index. If this is set, the global setting (see Core Settings → Core_IndexDynamicTemplates) is not used.

The Elasticsearch documentation (link provided until the OpenSearch documentation is equivalent) contains more information about the capabilities of dynamic templates.

In the following example, all fields that OpenSearch recognizes as text fields (strings) are assigned a mapping (“alle_textfelder”) in which the content is not analyzed (this saves storage space, and the data can still be displayed). Plus one exception (“spezialfall”): For all text fields with the name suffix 'name', our default mapping is used.

Example
[
  {
    "spezialfall": {
      "match_mapping_type": "string",
      "match": "*name",
      "mapping": {
        "type": "text",
        "fields": {
          "lowercase": {
            "normalizer": "lowercaseNormalizer",
            "type": "keyword"
          },
          "raw": {
            "type": "keyword"
          }
        }
      }
    }
  },
  {
    "alle_textfelder": {
      "match_mapping_type": "string",
      "match": "*",
      "mapping": {
        "type": "keyword",
        "analyzer": false
      }
    }
  }
]

To specify the OpenSearch type for a database field. Occasionally, OpenSearch may make a mistake with the mapping and use an inappropriate type. Specific example: The Oracle column named ‘ZAHL’ with the data type ‘NUMBER(10,2)’ is created in the OpenSearch mapping as type ‘long’ instead of ‘float’. This can be corrected using the example below.

Example
[
  {
    "ZAHL_long_als_float": {
      "match_mapping_type": "long",
      "match": "ZAHL",
      "mapping": {
        "type": "float"
      }
    }
  }
]

Advanced Settings / Advanced

Setting (Key) Data Type Description

Replication_RestartWhereLeftOff
(restartReplicationWhereLeftOff)

boolean

When the server starts or a job is modified, the replication jobs are restarted and begin replicating from the beginning again (see replicationStartDate). If this setting is set to 'true', replication resumes from the date of the most recent record that was replicated. These timestamps are stored as metadata in the OpenSearch index.

Replication_Delay
(replicationDelay)

Integer

Delay in seconds after which replication starts

Replication_Interval
(replicationInterval)

Integer

Interval in seconds for replication

Replication_BlockDayRange
(replicationBlockDayRange)

Integer

Number of days (block) to be processed during each job run. Do not set this value too high, as this will cause an increased load on the source database and prevent OpenSearch/Lucene from getting a breather.

Example: 10 days. The job has just reached March 10, 2015; then the records from March 10, 2015, through March 20, 2015, will be replicated, and in the next run, those from March 20, 2015, through March 30, 2015.

Replication_BlockSize
(replicationBlockSize)

Integer

Block size for the transfer from the database to OpenSearch. Number of database records that the JDBC driver reads as a block and keeps in memory.

It currently appears that a larger block size is better. See also: Replication Duration. But please don’t overdo it, as this can lead to OutOfMemory exceptions. A block size of 2500 is already too high for some database tables.

Replication_SyncFiles
(replicationSyncFiles)

boolean

Synchronization of BLOBs

Subsequent activation is currently not possible.

Replication_UnzipSyncedFiles
(replicationUnzipSyncedFiles)

boolean

Automatically unpacks the contents of synchronized files.

Only applies if replicationSyncFiles is enabled.

Replication_AdjustUpperDateLimitInSeconds
(adjustUpperDateLimitInSeconds)

Integer

Workaround for a database issue where record updates—initiated by a database trigger—are written too late and therefore cannot be taken into account during the replication run (we’re talking about 1–3 seconds here).

With this option, records are selected based on an update timestamp minus the specified value. By default (0), this option is disabled, and records are selected as before using a timestamp in the future (replicationBlockDayRange). Since we are using the database’s timestamp function here, this option is currently only supported for Oracle, MS SQL, and MySQL.

Example: 3 seconds. Records are selected from the timestamp of the last replicated record up to the current database timestamp minus 3 seconds. Of course, this means the most recent data may not always be available in BPC, but hopefully it will be in a consistent state with the database.

Replication_VamOrganizationId
(vamOrganizationId)

String

experimental

If not empty, special features for replicating Data Management data are used. Specifically, only the data associated with the entered organizationId (as specified in the Data Management configuration) is replicated.

Replication_LoggingEnabled
(replicationLoggingEnabled)

boolean

Here, logging of replication runs can be enabled or disabled for each replication job.

Snapshots

With a snapshot, all documents (created after the specified 'replicationStartDate') in the OpenSearch index (see 'targetIndex') are copied to a new index at the specified time. Finally, the alias is redirected to the new index, and the previous index is deleted. This can be performed, for example, once a week to maintain a lean index.

Additional note: Documents marked as deleted in OpenSearch (either manually deleted or deleted by Tail Sync) are, of course, not included in the new index. Caution: The associated replication run is suspended until the shadow copy has been created.

Setting (Key) Data Type Description

ShadowCopy_Enabled
(replicationShadowCopyEnabled)

boolean

Enable the creation of shadow copies.

ShadowCopy_CronPattern
(replicationShadowCopyCronPattern)

String

Cron-like pattern (following Quartz scheduler syntax) to specify when the shadow copies should be created.

Examples
0 15 16 ? * SUN = Jeden Sonntag um 16:15 Uhr
0 */30 * * * ? = Alle 30 Minuten
30 59 11 ? * 1,2,3,4,5 = Am Montag, Dienstag, Mittwoch, Donnerstag und Freitag um 11:59:30 Uhr

Additional examples and documentation can be found on the Quartz scheduler website.

ShadowCopy_KeepCopiesCount
(replicationShadowCopyKeepCopiesCount)

Integer

Number of shadow copies to be retained.

0 = No shadow copies are retained

3 = 3 shadow copies are retained (these are in the 'Close' status to conserve resources)

Tail Sync

The 'Tail Sync' operates on the current index (see 'targetIndex') and synchronizes older data (new/modified/deleted database records) with OpenSearch so that it matches the database again. This can be performed, for example, once every night. Documents older than the replication start date (see 'replicationStartDate') are first deleted from the OpenSearch index. It then goes through the database in blocks (default: 10-day increments) and synchronizes it with OpenSearch. In the process, new or modified database records are added to the OpenSearch index, and documents that no longer exist in the database are deleted from OpenSearch. Tail Sync works only with the specified 'idColumns' and 'lastUpdateColumn' fields (Hint: a suitable DB index works wonders!). The complete data is only read when there are new or modified database records.

The sync does not run up to the current date (see [replicationTailSyncRelativeEndDate]), since these records continue to be processed by our normal replication.
Setting (Key) Data Type Description

TailSync_Enabled
(replicationTailSyncEnabled)

boolean

Enable 'Tail Sync'

TailSync_CronPattern
(replicationTailSyncCronPattern)

String

Cron-like pattern (following Quartz scheduler syntax) to specify when Tail Sync should start.

Examples
0 5 2 * * ? = Jede Nacht um 2:05 Uhr
0 35 21 ? * Sun = Jeden Sonntag um 21:35 Uhr
30 5 3 ? * 1,2,3,4,5 = Am Montag, Dienstag, Mittwoch, Donnerstag und Freitag um 3:05:30 Uhr

Additional examples and documentation can be found on the Quartz scheduler website.

TailSync_BlockSize
(replicationTailSyncBlockSize)

Integer

Block size of the database driver. Note the information regarding the replicationBlockSize option.

TailSync_RelativeStartDate
(replicationTailSyncRelativeStartDate)

String

The relative start date. Synchronization should begin from this point onward. Data that predates the replication start date will continue to be deleted. This date is only used if it falls after the regular start date and before the end date.

1 second|minute|hour|day|week|month|year ago
n seconds|minutes|hours|days|weeks|months|years ago

This option should only be used in special cases. Inconsistencies may occur if records in the database that predate this relative start date are deleted. These records will then remain in the OpenSearch index even though they no longer exist in the database.

TailSync_RelativeEndDate
(replicationTailSyncRelativeEndDate)

String

The relative end date. Synchronization should be performed up to this point and no further.

1 second|minute|hour|day|week|month|year ago
n seconds|minutes|hours|days|weeks|months|years ago

TailSync_BlockDayRange
(replicationTailSyncBlockDayRange)

Integer

Number of days (block) during which the data should be processed. See also replicationBlockDayRange.

TailSync_RelativeDeleteOlderThanDate
(replicationTailSyncRelativeDeleteOlderThanDate)

String

The relative deletion date to be specified. All documents older than this will be deleted. If this option is not set, it will not be executed.

Consistency Check

A simple consistency check is performed after the replication runs. This compares the number of documents in the source and the destination. Specifically, this comparison covers the period between 'replicationStartDate' and the latest date present in the destination (OpenSearch).

With large amounts of data (tens of millions of records), the system may experience increased load, and the initial replication may be significantly slowed down. Enable the consistency check only if you are certain that the data will be fully replicated.
Setting (Key) Data Type Description

ConsistencyCheck_Frequency
(replicationConsistencyCheckFrequency)

Integer

The frequency at which the consistency check is performed.

Examples:

  • 0 = never performed

  • 1 = performed after every replication run

  • 10 = performed after every 10th replication run

Lookup Joins

Can be used to enrich documents to be replicated with additional data. For example, if the data to be replicated contains only a partner ID and the partner’s name, etc., is still required in the monitor.

OpenSearch uses denormalization, which means this data is included in the document and is available just like all other data (high-performance search, aggregation, etc.). Prerequisite: The lookup “tables” must be available as separate indexes and can be imported, for example, via an additional replication from a database table.

The lookup data can be updated manually (BPC Settings → Overview → Status → Replication → Jobs → Job → Synchronize Lookup Joins) or automatically using our OpenSearch BPC plugin (os-BPC-plugin).

Multiple lookup tables can be referenced; the possible values for an ENTRY are described below. Structure: "join": [ { ENTRY }, { ENTRY }, …​ ]

For the value comparison to work, the column types in the database tables must be identical. If the fields are strings, "*.raw" should be used as the lookupKeyField; for numeric fields, however, use the field name without the ".raw"
Lookup Joins Configuration Example
[
    {
        "keyField": "PARTNER",
        "lookupIndex": "lookup-partner",
        "lookupKeyField": "ID.raw",
        "resultFieldsPrefix": "partner_",
        "resultFieldsExcluded": [ "ID", "LASTUPDATE" ]
    },
    {
        "keyField": "MESSAGETYPE",
        "lookupIndex": "lookup-messagetype",
        "lookupKeyField": "ID.raw",
        "resultFieldsPrefix": "messagetype_",
        "resultFieldsExcluded": [ "ID", "LASTUPDATE" ]
    }
]
Field Data Type Description

keyField

String

The key field in the table to be replicated. The data is retrieved from the lookup index based on the value of this field.

Example: PARTNER_ID
This field contains the partner’s ID, and the specified lookup index contains the actual partner data to be retrieved. (Hint: DB Foreign Key)

lookupIndex

String

The OpenSearch index containing the lookup data.

Example: lookup-partner
In this example, the partner’s detailed data.

lookupKeyField

String

Example: ID.raw
This field searches for the value of the 'keyField' (see above). (Hint: DB Primary Key)

resultFieldsPrefix

String

The fields to be retrieved from the lookup index must be prefixed with a unique prefix to avoid conflicts with existing fields.

Example: partner_

resultFieldsIncluded

String Array

If not all fields from the lookup index are to be included, the fields to be included can be specified here.

Example: [ "FIRST_NAME", "LAST_NAME" ]

resultFieldsExcluded

String Array

If you want to include almost all fields, you can specify the fields to be excluded here.

Example: [ "ID", "UPDATED" ]

Replication Jobs Logging

Replication job runs can be logged and displayed, for example, via the automatically created “Replication Jobs Monitor.” This monitor is created at startup if it does not already exist. Therefore, it cannot be permanently deleted. Similarly, the ‘BPC-replicationjobs-log’ index is automatically created whenever a run needs to be logged.

Logging can be enabled or disabled globally or on a per-replication-job basis (see the corresponding settings above). By default, logging is enabled globally and disabled for replication jobs. So nothing is logged initially!

The log entries are written in bulk to the aforementioned index after some time. This has little impact on the system’s overall performance. However, a very large number of entries can be generated in a very short time (millions in a few hours), so you might want to run the cleanup often enough and keep an eye on the number of documents it contains: Settings → Core Services → Indexes

Keywords:
Press Esc to exit full screen
Modal image placeholder
Press Esc to exit full screen