MinIO as S3 backup for OpenSearch

In the following, we will discuss how MinIO can be used as an S3 backup replacement for on-promise installations. It is a good alternative to the shared file systems that would have to be used for backups in the OpenSearch cluster. See BPC Cluster

Installation

The installation is shown here using the example of Docker. Further installation options can be found in the MinIO documentation.

First, a directory must be created in which MinIO can store its data. /opt/minio/data is used here as an example. This path is not a must and can be located anywhere on the hard disk.

mkdir -p /opt/minio/data

Now start MinIO via Docker:

docker run -d \
  --restart=unless-stopped \
  -p 9000:9000 \
  -p 9001:9001 \
  --name minio \
  -v /opt/minio/data:/data \
  -e "MINIO_ROOT_USER=rootroot" \
  -e "MINIO_ROOT_PASSWORD=virtimos" \
  quay.io/minio/minio server /data --console-address ":9001"

Create S3 bucket and define access to it

Open the MinIO admin website in the browser: http://127.0.0.1:9000

User: rootroot
Password: virtimos

Create a bucket under Administrator → Buckets:

Bucket name: opensearch
Leave the features deactivated

Create a new 'Access Key' under User → Access Keys:

Exemplary Access Key: ZlDIa08zIdmoPaKCnXa4
Exemplary Secret Key: HNUyLdeupJWJ33JlEDUZ9zzF4QjYQfzSfe7wZfCo
leave the remaining options untouched

Prepare OpenSearch

To do this, change to the OpenSearch installation directory.

Now install the OpenSearch S3 plugin:

bin/opensearch-plugin install --batch repository-s3

The 'Access Key' and 'Secret Key' from above must be stored in the OpenSearch keystore for access to MinIO:

echo 'ZlDIa08zIdmoPaKCnXa4' | bin/opensearch-keystore add --stdin s3.client.default.access_key
echo 'HNUyLdeupJWJ33JlEDUZ9zzF4QjYQfzSfe7wZfCo' | bin/opensearch-keystore add --stdin s3.client.default.secret_key

Expand the OpenSearch configuration file with a text editor:

config/opensearch.yml

# Anpassung zur Verwendung des S3 kompatiblen MinIO
s3.client.default.protocol: http
s3.client.default.endpoint: localhost:9000
s3.client.default.region: eu-central-1
s3.client.default.path_style_access: true

Set the following environment variables before starting OpenSearch:

export AWS_EC2_METADATA_DISABLED=true
export AWS_REGION=eu-central-1

Now OpenSearch can be started.

Take over existing backups (optional)

This step must be done before switching to the S3 backup repository. The S3 bucket created above must not contain any files.

By default, the snapshots/backups of the OpenSearch indices are stored in the file system. They can be found in the following directory: <BPC_INSTALLATIONS_ORDNER>/opensearch_data/bpc_backup

All files and directories from this directory can simply be copied to the S3 bucket. As we are using MinIO here, its command line tool mc is also used. However, any other S3 tool can be used to copy the files and directories.

Install the Minio CLI tool mc:

macOS

brew install minio/stable/mc

Give the command line tool mc access to the MinIO installed above:

mc alias set local http://127.0.0.1:9000 rootroot virtimos

Now copy the content from the <BPC_INSTALLATIONS_ORDNER>/opensearch_data/bpc_backup directory into the S3 bucket 'opensearch' created above:

mc cp --recursive <BPC_INSTALLATIONS_ORDNER>/opensearch_data/bpc_backup/* local/opensearch

Configure BPC

Set the S3 bucket as the BPC backup repository.

BPC Administration → Core Services → General → Core_BackupRepository

{
    "type": "s3",
    "settings": {
        "bucket": "opensearch",
        "protocol": "http",
        "endpoint": "localhost:9000",
        "path_style_access": true
    }
}

After saving, the OpenSearch snapshots/backups are stored in the S3 bucket.

Keywords: