OpenSearch

OpenSearch is one of the central system components of the BPC.

The manufacturer’s documentation can be found at https://opensearch.org/docs/latest/

Configuration

RAM

The amount of allocated RAM is set via the environment variable OPENSEARCH_JAVA_OPTS. This environment variable is also used when using the central configuration via bpc.env.sh central configuration via bpc.env.sh Alternatively, the OPENSEARCH_CONFIG_VERZEICHNIS/jvm.options can also be changed within the installation. See also https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/

Please also refer to the RAM examples for the available hardware under System requirements

Resource limits

There are various resource limits for processes on UNIX-based operating systems. In order for OpenSearch to work properly, these usually need to be extended.

To do this, edit the file /etc/security/limits.conf and add the following line. In doing so, BPCUSER must be changed to the user name that starts the OpenSearch process.

BPCUSER - nofile 65535

Swapping

Swapping should be avoided on the system if possible, as this severely restricts performance.

The setting for swapping depends on your operating system.

Swapping the configuration directory

By default, the OpenSearch configuration files are located in the directory opensearch/config.

The configuration directory can be set to any directory by setting the environment variable OPENSEARCH_PATH_CONF via the central configuration file (bpc.env.sh or bpc.env.cmd). Relative paths refer to the OpenSearch installation directory.

It is recommended to move the configuration directory out of the OpenSearch directory, e.g. to BPC_VERZEICHNIS/opensearch_config. This prevents the configuration files from being overwritten during an OpenSearch update.

When installing via a bundle file, the directory is automatically moved to BPC_VERZEICHNIS/opensearch_config.

When moving the configuration directory manually, make sure that any symlinks (virtimo/ssl) are retained or reset.

In this documentation, we use the identifier OPENSEARCH_CONFIG_VERZEICHNIS for the configuration directory.

Shards and replicas of indices

These are the two settings of an index that always cause problems and confusion. This is a complex topic and there is no 100% solution as to how the settings number_of_shards and number_of_replicas should be set for each index.

To refresh what replica and shard are:

Replica

These are the "backup copies" of shards if a node in the cluster should fail.
The number of replicas can be set to 0. However, this only makes sense for a single node or development system.
In an OpenSearch cluster with >= 2 nodes, this value should be set to at least 1. So that there is a backup copy on another node in the cluster.

Shard

A shard contains the documents of an index.
An index can consist of 1 or more shards.
A shard can hold approx. 50GB of data and approx. 200 million documents.
Larger shards can be slower for search queries and take longer to recover in the event of an error.
Each shard requires heap memory.
Each shard uses a thread Each shard uses one thread.

Oversharding

Since each shard requires heap memory, the maximum memory made available to OpenSearch is an important point so that performance problems do not occur. This is then called oversharding. There is a rule of thumbhttps://www.elastic.co/de/blog/managing-and-troubleshooting-elasticsearch-memory[(source]) for calculating the number of shards up to which performance is still good: Summe(Max Heap der Nodes in GB) * 20

Maximum number of shards

The maximum number of shards that can be created in the cluster is set to 1000 by default. If this value is exceeded by creating further indices, an error such as this action would add [5] total shards, but this cluster currently has [998]/[1000] maximum shards open; occurs.

Possible solutions:

If there is sufficient heap memory available on the nodes in the cluster, the limit can be increased as follows. Rule of thumb Example: 3 nodes each with 32 GB heap memory * 20 = 1920 shards
```
curl -X PUT -H "Content-Type: application/json" 'http://localhost:9200/_cluster/settings' -d '
{
  "persistent": {
    "cluster.max_shards_per_node": 1920
  }
}'
```
Kann zwar auch bei weniger Speicher erhöht werden, dann aber zulasten der Performance und sollte nur temporär erhöht werden.
Delete empty, unused and closed indices
Check the shards of the existing indices and reduce them by a reindex if necessary.

Authentication and authorization

The BPC is delivered with the security feature for OpenSearch already activated and authentication and authorization set up via mTLS (see also Secure connection (TLS/HTTPS)). This already offers comprehensive access protection.

Further information on this can also be found in the official documentation.

Direct OpenSearch access

Due to the access restrictions (see Authentication and authorization), a few things must be taken into account for direct access to OpenSearch, for example via curl.

There are essentially three ways to access OpenSearch. Access via certificate is recommended, as in this case no further configuration changes need to be made.

All of the following examples assume that TLS (see Secure connection (TLS/HTTPS)) is activated. If this is not the case, you must replace https:// with http://.

Please note that OpenSearch can only be accessed via the local network interface in the delivery state. If you want to access OpenSearch from outside, you must change the corresponding configuration corresponding configuration

Access with client certificate (mTLS)

This is the standard authentication for OpenSearch. How to configure this is described at Secure connection (TLS/HTTPS).

You need the correct client certificate for authentication. This is stored in the BPC in a Java Keystore (JKS). Depending on the access type, you will need the client certificate in a different format.

Extract client certificate from the Java keystore

For use with tools such as CURL, you can extract this and save it in PEM format, for example.

The following examples assume that you are using virtimo_keystore.jks. If you use a different keystore, different passwords or aliases, you must adapt the examples accordingly. The examples refer to the delivery status described in Secure connection (TLS/HTTPS).

Extraction of the client certificate with the shell

In the following example, the certificate is first extracted in PKCS12 format using the programs keytool (part of Java) and OpenSSL and then converted to PEM format.

Extraction of the client certificate and storage as opensearch_bpc.pem

keytool -importkeystore \
  -srckeystore virtimo_keystore.jks \
  -srcstoretype JKS \
  -srcalias opensearch_bpc \
  -destkeystore opensearch_bpc.p12 \
  -deststoretype PKCS12 \
  -srcstorepass virtimo \
  -deststorepass virtimo

openssl pkcs12 \
  -in opensearch_bpc.p12 \
  -nocerts \
  -nodes \
  -out opensearch_bpc-key.pem \
  -password pass:virtimo

openssl pkcs12 \
  -in opensearch_bpc.p12 \
  -nokeys \
  -out opensearch_bpc-cert.pem \
  -password pass:virtimo

cat opensearch_bpc-key.pem opensearch_bpc-cert.pem > opensearch_bpc.pem

rm opensearch_bpc.p12
rm opensearch_bpc-key.pem
rm opensearch_bpc-cert.pem

Extraktion des Client-Zertifikats mit dem KeyStore Explorer

Mit dem KeyStore Explorer ist es einfach möglich Zertifikate aus Java Keystores zu extrahieren.

Den Keystore virtimo_keystore.jks öffnen
Kontextmenü auf dem Eintrag 'opensearch_bpc'
Exportieren → Schlüsselpaar exportieren
Im Export Dialog: Format = PEM, Passwort = KEINES → Export

CURL Beispiel mit Zertifikat im PEM Format

Das folgende Beispiel erwartet das Client-Zertifikat in der Datei opensearch_bpc.pem.

Exemplarischer Aufruf per CURL

curl --insecure --cert opensearch_bpc.pem 'https://localhost:9200'

Zugriff mit Basic Auth und internen Benutzer

Es wird empfohlen, diese Zugriffsmethode nicht zu nutzen. Damit wird das Sicherheitsniveau signifikant gesenkt. Sollten Sie dies dennoch nutzen wollen, wird empfohlen die Passwörter der internen Benutzer zu ändern (siehe Passwort der internen OpenSearch Benutzer ändern) oder die Benutzer zu ersetzen.

Für diesen Zugriff müssen Sie sicherstellen, dass die Authentifizierung per Zertifikat nur optional ist. Dafür muss in OpenSearch (opensearch.yml) der Parameter plugins.security.ssl.http.clientauth_mode auf OPTIONAL stehen (siehe auch OpenSearch Dokumentation TLS Client authentication).

Sie sollten plugins.security.ssl.http.clientauth_mode nicht auf NONE stellen. Dadurch kann Karaf keine Verbindung mehr zu OpenSearch herstellen.

OpenSearch ist initial bereits mit mehreren Demo-Benutzern konfiguriert (siehe auch OpenSearch Dokumentation), wie z.B. admin mit dem Passwort admin. Solange plugins.security.ssl.http.clientauth_mode auf REQUIRED gesetzt ist, können diese nicht verwendet werden.

Es ist auch möglich, den Karaf Zugriff auf diese Authentifizierungsmethode umzustellen. Davon wird jedoch abgeraten.

Siehe auch BPC Konfigurationsdatei.

Beispielkonfiguration karaf/etc/de.virtimo.bpc.core.cfg

de.virtimo.bpc.core.opensearch.username = admin
de.virtimo.bpc.core.opensearch.password = admin

Wurden die Einstellungen angepasst und OpenSearch neu gestartet, so kann der Zugriff mittels basic auth und einem internen Benutzer ausgeführt werden.

Exemplarischer Aufruf per CURL mit Demo User admin

curl --insecure -u admin:admin 'https://localhost:9200'

Passwort der internen OpenSearch Benutzer ändern

Es wird dringend empfohlen, die Passwörter der internen Benutzer zu ändern und nicht benutzte Benutzer zu entfernen.

Hier ein Beispiel, wie Sie das Password vom Benutzer admin ändern.

Wir befinden uns im OpenSearch Verzeichnis
```
cd opensearch
```

Benötigte OpenSearch Security Plugin Tools ausführbar machen

chmod +x ./plugins/opensearch-security/tools/hash.sh
chmod +x ./plugins/opensearch-security/tools/securityadmin.sh

Hash des neuen Passwortes generieren

./plugins/opensearch-security/tools/hash.sh -p sehr_starkes_admin_pw

**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
$2y$12$5VMnjOnHEoZucsTzxFmdZu3qbddy0tJIGmHPcPep8phE965Dq1ZBK

Replace the hash of the admin user in OPENSEARCH_CONFIG_VERZEICHNIS/opensearch-security/internal_users.yml

admin:
  hash: "$2y$12$5VMnjOnHEoZucsTzxFmdZu3qbddy0tJIGmHPcPep8phE965Dq1ZBK"
  reserved: true
  backend_roles:
    - "admin"
  description: "Demo admin user"

Temporarily add the bpc user as admin user in OPENSEARCH_CONFIG_VERZEICHNIS/opensearch.yml (last line with CN=bpc, …).

plugins.security.authcz.admin_dn:
  - "CN=opensearch_admin,OU=Dev,O=Virtimo AG,L=Berlin,C=DE"
  - "CN=bpc,OU=client,O=Virtimo AG,L=Berlin,C=DE" # diese Zeile hinzufügen

Restart OpenSearch

Update the internal users in OpenSearch

./plugins/opensearch-security/tools/securityadmin.sh \
  --ignore-clustername \
  --type internalusers \
  --file OPENSEARCH_CONFIG_VERZEICHNIS/opensearch-security/internal_users.yml \
  --keystore config/virtimo/ssl/virtimo_keystore.jks \
  --keystore-alias opensearch_bpc \
  --keystore-password virtimo \
  --truststore OPENSEARCH_CONFIG_VERZEICHNIS/virtimo/ssl/virtimo_truststore.jks

Test whether the new admin password has been set correctly

curl --insecure -u admin:sehr_starkes_admin_pw 'https://localhost:9200'

Remove the admin rights from the user bpc. To do this, remove the previously added line in OPENSEARCH_CONFIG_VERZEICHNIS/opensearch.yml.
```
plugins.security.authcz.admin_dn:
  - "CN=opensearch_admin,OU=Dev,O=Virtimo AG,L=Berlin,C=DE"
```
Restart OpenSearch

Access with deactivated security feature

It is not recommended to deactivate the access protection for OpenSearch.

You can deactivate the security feature via the configuration (opensearch.yml) with the following line:

Section of opensearch.yml with deactivated security feature

plugins.security.disabled: true

After OpenSearch has been restarted, you can access OpenSearch without authentication.

Example of a CURL call without authentication

curl --insecure 'https://localhost:9200'

It is also possible to switch Karaf access to this authentication method. However, this is not recommended.