OpenSearch

OpenSearch is one of the central system components of the BPC.

The manufacturer’s documentation can be found at https://opensearch.org/docs/latest/

Configuration

RAM

The amount of allocated RAM is set via the environment variable OPENSEARCH_JAVA_OPTS. This environment variable is also used when employing the central configuration via bpc.env.sh. Alternatively, the <OPENSEARCH-CONFIG-DIR>/jvm.options can be modified within the installation. See also https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/

Please also refer to the memory examples for the available hardware under System Requirements.

Resource Limits

On UNIX-based operating systems, there are various resource limits for processes. For OpenSearch to work properly, these limits usually need to be increased.

To do this, edit the file /etc/security/limits.conf and add the following line. Replace BPCUSER with the username of the user who starts the OpenSearch process.

BPCUSER  -  nofile  65535

Swapping

Swapping should be avoided on the system whenever possible, as it significantly limits performance.

The setting for swapping depends on your operating system.

Moving the Configuration Directory

By default, the OpenSearch configuration files are located in the directory opensearch/config.

The configuration directory can be set to any directory by setting the environment variable ` OPENSEARCH_PATH_CONF ` via the central configuration file (bpc.env.sh or bpc.env.cmd). Relative paths are relative to the OpenSearch installation directory.

It is recommended to move the configuration directory out of the OpenSearch directory, e.g., to <BPC-INSTALL-DIR>/opensearch_config. This ensures that the configuration files are not overwritten during an OpenSearch update.

When installing via a bundle file, the directory is automatically moved to <BPC-INSTALL-DIR>/opensearch_config.

When moving the directory manually (relocating the configuration directory), ensure that any existing symlinks (virtimo/ssl) are preserved or recreated.

In this documentation, we use the identifier <OPENSEARCH-CONFIG-DIR> for the configuration directory.

Shards and Replicas of Indexes

These are the two settings for an index that repeatedly cause problems and confusion. This is a complex topic, and there is no 100% solution for how the settings number_of_shards and number_of_replicas should be configured for each index.

As a refresher on what replicas and shards are:

Replica

These are the “backups” of shards in case a node in the cluster fails.
The number of replicas can be set to 0. However, this only makes sense for a single-node or development system.
In an OpenSearch cluster with >= 2 nodes, this value should be set to at least 1. This ensures that a backup is available on another node in the cluster.

Shard

A shard contains the documents of an index.
An index can consist of one or more shards.
A shard can hold approximately 50 GB of data and approximately 200 million documents.
Larger shards may be slower during search queries and take longer to recover in the event of a failure.
Each shard requires heap memory.
Each shard uses one thread.

Oversharding

Since each shard requires heap memory, the maximum memory allocated to OpenSearch is a critical factor in preventing performance issues. This is referred to as oversharding. There is a rule of thumb (source) for calculating the number of shards up to which performance remains good: Summe(Max Heap der Nodes in GB) * 20

Maximum Number of Shards

The default maximum number of shards that can be created in the cluster is set to 1,000. If this value is exceeded by creating additional indexes, an error like this action would add [5] total shards, but this cluster currently has [998]/[1000] maximum shards open; will occur.

Possible solutions:

If sufficient heap memory is available on the nodes in the cluster, the limit can be increased as follows. Rule of thumb example: 3 nodes with 32 GB of heap memory each * 20 = 1,920 shards
```
curl -X PUT -H "Content-Type: application/json" 'http://localhost:9200/_cluster/settings' -d '
{
  "persistent": {
    "cluster.max_shards_per_node": 1920
  }
}'
```
While this limit can also be increased with less memory, doing so comes at the expense of performance and should only be done temporarily.
Delete empty, unused, and closed indexes
Check the shards of existing indexes and, if necessary, reduce them by running a reindex

Authentication and Authorization

The BPC comes with security features for OpenSearch already enabled out of the box, and authentication and authorization via mTLS are already set up (see also Secure Connection (TLS/HTTPS)). This already provides comprehensive access protection.

You can also find more information on this in the official documentation.

Direct OpenSearch Access

Due to access restrictions (see Authentication and Authorization), there are several things to keep in mind when accessing OpenSearch directly, for example via curl.

There are essentially three ways to access OpenSearch. Access via a certificate is recommended, as no further configuration changes are required in this case.

All of the following examples assume that TLS (see Secure Connection (TLS/HTTPS)) is enabled. If this is not the case, you must replace https:// with http://.

Note that, by default, OpenSearch is only accessible via the local network interface. If you want to access OpenSearch from outside the network, you must adjust the corresponding configuration.

Access with a Client Certificate (mTLS)

This is the default authentication method for OpenSearch. Instructions for configuring this are described at Secure Connection (TLS/HTTPS).

You need the correct client certificate for authentication. This certificate is stored in a Java keystore (JKS) in the BPC. Depending on the access method, you’ll need the client certificate in a different format.

Extracting the Client Certificate from the Java Keystore

To use it with tools like CURL, you can extract it and save it in PEM format, for example.

The following examples assume that you are using the virtimo_keystore.jks. If you are using a different keystore, different passwords, or aliases, you must adapt the examples accordingly. The examples refer to the default configuration described at Secure Connection (TLS/HTTPS).

Extracting the client certificate using the shell

In the following example, the certificate is first extracted in PKCS12 format using the programs keytool (part of Java) and OpenSSL, and then converted to PEM format.

Extracting the Client Certificate and Saving It as a opensearch_bpc.pem

keytool -importkeystore \
  -srckeystore virtimo_keystore.jks \
  -srcstoretype JKS \
  -srcalias opensearch_bpc \
  -destkeystore opensearch_bpc.p12 \
  -deststoretype PKCS12 \
  -srcstorepass virtimo \
  -deststorepass virtimo

openssl pkcs12 \
  -in opensearch_bpc.p12 \
  -nocerts \
  -nodes \
  -out opensearch_bpc-key.pem \
  -password pass:virtimo

openssl pkcs12 \
  -in opensearch_bpc.p12 \
  -nokeys \
  -out opensearch_bpc-cert.pem \
  -password pass:virtimo

cat opensearch_bpc-key.pem opensearch_bpc-cert.pem > opensearch_bpc.pem

rm opensearch_bpc.p12
rm opensearch_bpc-key.pem
rm opensearch_bpc-cert.pem

Extracting the Client Certificate Using KeyStore Explorer

KeyStore Explorer makes it easy to extract certificates from Java keystores.

Open the keystore virtimo_keystore.jks
Right-click on the 'opensearch_bpc' entry
Export → Export Key Pair
In the export dialog: Format = PEM, Password = NONE → Export

CURL example with a certificate in PEM format

The following example expects the client certificate to be in the file opensearch_bpc.pem.

Example CURL call

curl --insecure --cert opensearch_bpc.pem 'https://localhost:9200'

Access with Basic Auth and internal users

It is recommended that you do not use this access method. Doing so significantly lowers the security level. If you still wish to use this method, it is recommended that you change the passwords of internal users (see Change the password for internal OpenSearch users) or replace those users.

For this access method, you must ensure that certificate-based authentication is only optional. To do this, the parameter ` plugins.security.ssl.http.clientauth_mode ` in OpenSearch (opensearch.yml) must be set to ` OPTIONAL ` (see also the OpenSearch documentation on TLS client authentication).

You should not set ` plugins.security.ssl.http.clientauth_mode ` to ` NONE `. This will prevent Karaf from connecting to OpenSearch.

OpenSearch is initially configured with several demo users (see also the OpenSearch documentation), such as admin with the password admin. As long as plugins.security.ssl.http.clientauth_mode is set to REQUIRED, these cannot be used.

It is also possible to switch Karaf to use this authentication method. However, this is not recommended.

Change the password for internal OpenSearch users

It is strongly recommended that you change the passwords of internal users and remove unused users.

Here is an example of how to change the password for the user admin.

We are in the OpenSearch directory
```
cd opensearch
```

Make the required OpenSearch Security Plugin tools executable

chmod +x ./plugins/opensearch-security/tools/hash.sh
chmod +x ./plugins/opensearch-security/tools/securityadmin.sh

Generate the hash of the new password

./plugins/opensearch-security/tools/hash.sh -p sehr_starkes_admin_pw

**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
$2y$12$5VMnjOnHEoZucsTzxFmdZu3qbddy0tJIGmHPcPep8phE965Dq1ZBK

Replace the hash of the admin user in <OPENSEARCH-CONFIG-DIR>/opensearch-security/internal_users.yml

admin:
  hash: "$2y$12$5VMnjOnHEoZucsTzxFmdZu3qbddy0tJIGmHPcPep8phE965Dq1ZBK"
  reserved: true
  backend_roles:
    - "admin"
  description: "Demo admin user"

In <OPENSEARCH-CONFIG-DIR>/opensearch.yml, temporarily add the bpc user as an admin user (last line with CN=BPC, …).

plugins.security.authcz.admin_dn:
  - "CN=opensearch_admin,OU=Dev,O=Virtimo AG,L=Berlin,C=DE"
  - "CN=bpc,OU=client,O=Virtimo AG,L=Berlin,C=DE" # Add this line

Restart OpenSearch

Update the internal users in OpenSearch

./plugins/opensearch-security/tools/securityadmin.sh \
  --ignore-clustername \
  --type internalusers \
  --file <OPENSEARCH-CONFIG-DIR>/opensearch-security/internal_users.yml \
  --keystore <OPENSEARCH-CONFIG-DIR>/virtimo/ssl/virtimo_keystore.jks \
  --keystore-alias opensearch_bpc \
  --keystore-password virtimo \
  --truststore <OPENSEARCH-CONFIG-DIR>/virtimo/ssl/virtimo_truststore.jks

Test whether the new admin password was set correctly

curl --insecure -u admin:sehr_starkes_admin_pw 'https://localhost:9200'

Revoke admin privileges from the user bpc. To do this, remove the previously added line from <OPENSEARCH-CONFIG-DIR>/opensearch.yml.
```
plugins.security.authcz.admin_dn:
  - "CN=opensearch_admin,OU=Dev,O=Virtimo AG,L=Berlin,C=DE"
```
Restart OpenSearch

Access with the security feature disabled

It is not recommended to disable access protection for OpenSearch.

You can disable the security feature via the configuration (opensearch.yml) using the following line:

Excerpt from opensearch.yml with the security feature disabled

plugins.security.disabled: true

After OpenSearch has been restarted, you can access OpenSearch without authentication.

Example of a CURL request without authentication

curl 'http://localhost:9200'

To ensure that BPC continues to function, you must adjust the schema in the de.virtimo.bpc.core.opensearch.hosts variable. To do this, change the variable in the central configuration file:

export DE_VIRTIMO_BPC_CORE_DE_VIRTIMO_BPC_CORE_OPENSEARCH_HOSTS="http://localhost:$DE_VIRTIMO_BPC_CORE_DE_VIRTIMO_BPC_CORE_OPENSEARCH_PORT"

Keywords: