MinIO as an S3 Backup for OpenSearch

The following section discusses how MinIO can be used as an S3 backup alternative for on-premises installations. It is a good alternative to the shared file systems that would otherwise be required for backups in an OpenSearch cluster. See BPC Cluster

Installation

The installation is demonstrated here using Docker as an example. Additional installation options can be found in the MinIO documentation.

First, you must create a directory where MinIO can store its data. Here, /opt/minio/data is used as an example. This path is not mandatory and can be located anywhere on the hard drive.

mkdir -p /opt/minio/data

Now start MinIO via Docker:

docker run -d \
  --restart=unless-stopped \
  -p 9000:9000 \
  -p 9001:9001 \
  --name minio \
  -v /opt/minio/data:/data \
  -e "MINIO_ROOT_USER=rootroot" \
  -e "MINIO_ROOT_PASSWORD=virtimos" \
  quay.io/minio/minio server /data --console-address ":9001"

Create an S3 bucket and configure access to it

Open the MinIO admin website in your browser: http://127.0.0.1:9000

User: rootroot
Password: virtimos

Under Administrator → Buckets, create a bucket:

Bucket Name: opensearch
Leave the features disabled

Under User → Access Keys, create a new 'Access Key':

Example Access Key: ZlDIa08zIdmoPaKCnXa4
Sample Secret Key: HNUyLdeupJWJ33JlEDUZ9zzF4QjYQfzSfe7wZfCo
Leave the remaining options unchanged

Prepare OpenSearch

To do this, navigate to the OpenSearch installation directory.

Now install the OpenSearch S3 plugin:

bin/opensearch-plugin install --batch repository-s3

The 'Access Key' and 'Secret Key' from above must be stored in the OpenSearch keystore to access MinIO:

echo 'ZlDIa08zIdmoPaKCnXa4' | bin/opensearch-keystore add --stdin s3.client.default.access_key
echo 'HNUyLdeupJWJ33JlEDUZ9zzF4QjYQfzSfe7wZfCo' | bin/opensearch-keystore add --stdin s3.client.default.secret_key

Edit the OpenSearch configuration file with a text editor:

config/opensearch.yml

# Adaptation for use with the S3-compatible MinIO
s3.client.default.protocol: http
s3.client.default.endpoint: localhost:9000
s3.client.default.region: eu-central-1
s3.client.default.path_style_access: true

Before starting OpenSearch, set the following environment variables:

export AWS_EC2_METADATA_DISABLED=true
export AWS_REGION=eu-central-1

OpenSearch can now be started.

Import existing backups (optional)

This step must be performed before switching to the S3 backup repository. Alternatively, the S3 bucket created above must not contain any files.

By default, the snapshots/backups of the OpenSearch indices are stored in the file system. They can be found in the following directory: <BPC_INSTALLATIONS_ORDNER>/opensearch_data/bpc_backup

All files and directories from this directory can simply be copied to the S3 bucket. Since we are using MinIO here, we’ll also use its command-line tool mc. However, any other S3 tool can be used to copy the files and directories.

Install the MinIO CLI tool mc:

macOS

brew install minio/stable/mc

Grant the command-line tool mc access to the MinIO installed above:

mc alias set local http://127.0.0.1:9000 rootroot virtimos

Now copy the contents from the <BPC_INSTALLATIONS_ORDNER>/opensearch_data/bpc_backup directory to the 'opensearch' S3 bucket created above:

mc cp --recursive <BPC_INSTALLATIONS_ORDNER>/opensearch_data/bpc_backup/* local/opensearch

Configure BPC

Set the S3 bucket as the BPC backup repository.

BPC Administration → Core Services → General → Core_BackupRepository

{
    "type": "s3",
    "settings": {
        "bucket": "opensearch",
        "protocol": "http",
        "endpoint": "localhost:9000",
        "path_style_access": true
    }
}

After saving, the OpenSearch snapshots/backups will be stored in the S3 bucket.

Keywords: