Creating indices

If OpenSearch indices are to be used in your own bundles to persist data (no settings), it is advisable to have them created via the BPC core.

This offers the following advantages

some boilerplate code is avoided
simple definition of the index settings and mappings via a JSON file
if the index does not exist, it is created automatically
the indices are created with an alias according to BPC specifications
can be called from the BPC Reindex functionality

Procedure in brief

create a managed_indices.json in the resources directory of your own backend bundle
make its content known to the OpenSearchService provided by the BPC core

Directory structure

Structure of the JSON file

Create a managed_indices.json in the resources directory of your own backend bundle. This can be used for several indices to be created and must have the following structure.

manages_indices.json

{
  "managedIndices": [
    {
      "name": "irgend-ein-index-name-1",
      "create_on_start": true|false,
      "hidden": true|false,
      "settings": { ... },
      "mappings": { ... }
    },
    {
      "name": "irgend-ein-index-name-2",
      "create_on_start": true|false,
      "hidden": true|false,
      "settings": { ... },
      "mappings": { ... }
    },
    ...
}

name - the name of the index or this then becomes the alias of the index.
create_on_start - Flag whether the index should be created immediately at the start of the bundle (registration). If not set, then it is the same as if the value was set to true.
hidden - Flag whether the index should be displayed in the deployment dialog and in combobox selection options. If not set, then it is the same as if the value was set to false.
settings - the settings of an OpenSearch index. Only set these if you know what you are doing. Please do not set number_of_shards and number_of_replicas here, otherwise these settings can no longer be adjusted via other mechanisms. If they are not set, the values specified in the core setting Core_IndexCreationSettings are used when creating.
mappings - the mappings of an OpenSearch index.

Simple example

for an index with the name 'example-queries'.

manages_indices.json

{
  "managedIndices": [
    {
      "name": "example-queries",
      "mappings": {
        "properties": {
          "queryName": {
            "type": "keyword"
          },
          "query": {
            "type": "keyword"
          },
          "chartConfig": {
            "type": "text",
            "index": false
          }
        }
      }
    }
  ]
}

Registration

In the following ExampleBaseModule.java example, OpenSearchService is retrieved from the BPC core and in the prepareManagedIndices(OpenSearchService es) method, the managed_indices.json of the bundle is loaded from OpenSearchService, registered and indices are created. Take the opportunity to look at the methods provided by OpenSearchService.

ExampleBaseModule.java

package com.company.example;

import de.virtimo.bpc.api.ModuleManager;
import de.virtimo.bpc.api.exception.OpenSearchRelatedException;
import de.virtimo.bpc.api.service.OpenSearchService;
import de.virtimo.bpc.module.AbstractInstantiableModule;
import org.osgi.framework.Bundle;
import org.osgi.framework.BundleContext;
import org.osgi.framework.ServiceReference;
import org.osgi.util.tracker.ServiceTracker;

import java.util.logging.Level;
import java.util.logging.Logger;

public abstract class ExampleBaseModule extends AbstractInstantiableModule {
    private static final Logger LOG = Logger.getLogger(ExampleBaseModule.class.getName());

    private ServiceTracker<OpenSearchService, OpenSearchService> openSearchServiceTracker;
    private OpenSearchService openSearchService;

    public ExampleBaseModule(ModuleManager moduleManager) {
        super(moduleManager);
    }

    @Override
    public void setModuleBundle(Bundle moduleBundle) {
        super.setModuleBundle(moduleBundle);

        if (openSearchServiceTracker == null) {
            openSearchServiceTracker = new OpenSearchServiceTracker(moduleBundle.getBundleContext());
            openSearchServiceTracker.open();
        }
    }

    @Override
    public void destroy() {
        super.destroy();

        if (openSearchServiceTracker != null) {
            openSearchServiceTracker.close();
            openSearchServiceTracker = null;
        }
    }

    public synchronized OpenSearchService getOpenSearchService() {
        LOG.info("getOpenSearchService");
        if (openSearchService == null) {
            throw new IllegalStateException("OpenSearchService is not available at the moment.");
        }
        return openSearchService;
    }

    private class OpenSearchServiceTracker extends ServiceTracker<OpenSearchService, OpenSearchService> {
        public OpenSearchServiceTracker(BundleContext context) {
            super(context, OpenSearchService.class, null);
        }

        @Override
        public OpenSearchService addingService(ServiceReference<OpenSearchService> reference) {
            openSearchService = super.addingService(reference);
            try {
                prepareManagedIndices(openSearchService);
            } catch (OpenSearchRelatedException ex) {
                LOG.log(Level.SEVERE, "Failed to create the managed indices.", ex);
            }
            return openSearchService;
        }

        @Override
        public void removedService(ServiceReference<OpenSearchService> reference, OpenSearchService service) {
            openSearchService = null;
            super.removedService(reference, service);
        }
    }

    private synchronized void prepareManagedIndices(OpenSearchService oss) throws OpenSearchRelatedException {
        LOG.info("prepareManagedIndices oss=" + oss);
        oss.prepareManagedIndices(getModuleBundle());
    }
}

Schreiben von Dokumenten

Das BPC geht davon aus, dass Indices folgenden Aufbau beim Namen haben <aliasname>_<timestamp>. Und daß ein Alias für diesen Index gesetzt ist. Beim lesen/schreiben/löschen der Dokumente wird immer mit dem Alias und nie mit dem physischen Indexnamen gearbeitet.

Beim Schreiben von Dokumenten legt OpenSearch automatisch einen Index an (wenn nicht vorhanden) und schätzt die Feldtypen. Das ist problematisch, wenn zum Beispiel zur Laufzeit ein Index gelöscht wird. Beim nächsten Schreiben eines Dokumentes würde dann der Index falsch angelegt werden.

Möglichkeit 1

Um das zu umgehen, die Methoden index, delete, search, searchByQuery, etc. vom OpenSearchService verwenden anstatt der vom RestHighLevelClient.

Die Methoden haben die gleiche Signatur. Allerdings werden die tückischen RuntimeExceptions (unchecked exception) von OpenSearch abgefangen und durch OpenSearchRelatedException (checked exception) ersetzt. Beim index/bulk-Aufruf wird dann, wenn nicht vorhanden, der Index anhand der Daten aus der managed_indices.json angelegt.

Beispiel RestHighLevelClient (managed_indices.json wird nicht verwendet)

IndexRequest myIndexRequest = ...;
RestHighLevelClient restHighLevelClient = openSearchService.getClient();
restHighLevelClient.index(myIndexRequest, RequestOptions.DEFAULT);

Beispiel OpenSearchService (managed_indices.json wird verwendet)

IndexRequest myIndexRequest = ...;
openSearchService.index(myIndexRequest, RequestOptions.DEFAULT);

Möglichkeit 2

Eine Hilfsmethode einführen, die vor jeder Schreib-Aktion ausgeführt wird. Folgend eine Beispielmethode für einen Index mit dem Alias 'books'. Der OpenSearchService hält den Status von Indices vor und kann dann bei nicht Existenz den Index per managed_indices.json anlegen.

private static final String BOOKS_INDEX_ALIAS_NAME = "books";

private void createBooksIndexIfMissing() throws SystemException {
    try {
        BpcIndexState booksIndexState = oss.getIndexState(BOOKS_INDEX_ALIAS_NAME);
        booksIndexState.prepareUsing(new BpcIndexCreateCallable() {
            @Override
            public String createIndex(OpenSearchService oss) throws OpenSearchRelatedException, ServiceNotFoundException, ModuleNotFoundException {
                return oss.getManagedIndicesHandler().createManagedIndex(BOOKS_INDEX_ALIAS_NAME);
            }
        });
    } catch (Exception ex) {
        throw new SystemException(CoreErrorCode.UNEXPECTED, "Failed to prepare the books index.", ex);
    }
}

Keywords: