Creating Indexes

If you plan to use OpenSearch indexes in your own bundles to persist data (not settings), it is recommended to have them created via the BPC Core.

This offers the following advantages

It eliminates a significant amount of boilerplate code
Easy configuration of index settings and mappings via a JSON file
If the index does not exist, it is automatically created
The indexes are created with an alias according to BPC specifications
Can be called from the BPC reindex functionality

Procedure at a Glance

Create a ` managed_indices.json ` file in the ` resources ` directory of your own backend bundle
Specify its contents in the ` OpenSearchService ` file provided by the BPC Core

Directory Structure

Structure of the JSON file

Create a ` managed_indices.json ` file in the ` resources ` directory of your own backend bundle. This file can be used for multiple indices to be created and must have the following structure.

manages_indices.json

{
  "managedIndices": [
    {
      "name": "irgend-ein-index-name-1",
      "create_on_start": true|false,
      "hidden": true|false,
      "settings": { ... },
      "mappings": { ... }
    },
    {
      "name": "irgend-ein-index-name-2",
      "create_on_start": true|false,
      "hidden": true|false,
      "settings": { ... },
      "mappings": { ... }
    },
    ...
}

name - the name of the index; this will then become the alias of the index.
create_on_start - Flag indicating whether the index should be created immediately upon bundle startup (registration). If not set, it behaves as if the value were set to true.
hidden - Flag indicating whether the index should be displayed in the deployment dialog and in combo box options. If not set, it behaves as if the value were set to false.
settings - The settings for an OpenSearch index. Set these only if you know what you are doing. Please do not set number_of_shards and number_of_replicas here; otherwise, these settings can no longer be adjusted via other mechanisms. If they are not set, the values specified in the core setting Core_IndexCreationSettings will be used when the index is created.
mappings - the mappings of an OpenSearch index.

Simple example

for an index named 'example-queries'.

manages_indices.json

{
  "managedIndices": [
    {
      "name": "example-queries",
      "mappings": {
        "properties": {
          "queryName": {
            "type": "keyword"
          },
          "query": {
            "type": "keyword"
          },
          "chartConfig": {
            "type": "text",
            "index": false
          }
        }
      }
    }
  ]
}

Registration

In the following ExampleBaseModule.java example, the OpenSearchService is retrieved from the BPC Core, and in the prepareManagedIndices(OpenSearchService es) method, the OpenSearchService loads the bundle’s managed_indices.json, registers it, and creates the indices. While you’re at it, take a look at the methods provided by OpenSearchService.

ExampleBaseModule.java

package com.company.example;

import de.virtimo.bpc.api.ModuleManager;
import de.virtimo.bpc.api.exception.OpenSearchRelatedException;
import de.virtimo.bpc.api.service.OpenSearchService;
import de.virtimo.bpc.module.AbstractInstantiableModule;
import org.osgi.framework.Bundle;
import org.osgi.framework.BundleContext;
import org.osgi.framework.ServiceReference;
import org.osgi.util.tracker.ServiceTracker;

import java.util.logging.Level;
import java.util.logging.Logger;

public abstract class ExampleBaseModule extends AbstractInstantiableModule {
    private static final Logger LOG = Logger.getLogger(ExampleBaseModule.class.getName());

    private ServiceTracker<OpenSearchService, OpenSearchService> openSearchServiceTracker;
    private OpenSearchService openSearchService;

    public ExampleBaseModule(ModuleManager moduleManager) {
        super(moduleManager);
    }

    @Override
    public void setModuleBundle(Bundle moduleBundle) {
        super.setModuleBundle(moduleBundle);

        if (openSearchServiceTracker == null) {
            openSearchServiceTracker = new OpenSearchServiceTracker(moduleBundle.getBundleContext());
            openSearchServiceTracker.open();
        }
    }

    @Override
    public void destroy() {
        super.destroy();

        if (openSearchServiceTracker != null) {
            openSearchServiceTracker.close();
            openSearchServiceTracker = null;
        }
    }

    public synchronized OpenSearchService getOpenSearchService() {
        LOG.info("getOpenSearchService");
        if (openSearchService == null) {
            throw new IllegalStateException("OpenSearchService is not available at the moment.");
        }
        return openSearchService;
    }

    private class OpenSearchServiceTracker extends ServiceTracker<OpenSearchService, OpenSearchService> {
        public OpenSearchServiceTracker(BundleContext context) {
            super(context, OpenSearchService.class, null);
        }

        @Override
        public OpenSearchService addingService(ServiceReference<OpenSearchService> reference) {
            openSearchService = super.addingService(reference);
            try {
                prepareManagedIndices(openSearchService);
            } catch (OpenSearchRelatedException ex) {
                LOG.log(Level.SEVERE, "Failed to create the managed indices.", ex);
            }
            return openSearchService;
        }

        @Override
        public void removedService(ServiceReference<OpenSearchService> reference, OpenSearchService service) {
            openSearchService = null;
            super.removedService(reference, service);
        }
    }

    private synchronized void prepareManagedIndices(OpenSearchService oss) throws OpenSearchRelatedException {
        LOG.info("prepareManagedIndices oss=" + oss);
        oss.prepareManagedIndices(getModuleBundle());
    }
}

Writing documents

The BPC assumes that indexes have the following naming structure: <aliasname>_<timestamp>. And that an alias is set for this index. When reading, writing, or deleting documents, the system always uses the alias and never the physical index name.

When writing documents, OpenSearch automatically creates an index (if it does not exist) and estimates the field types. This is problematic if, for example, an index is deleted at runtime. The next time a document is written, the index would then be created incorrectly.

Option 1

To work around this, use the methods index, delete, search, searchByQuery, etc., from OpenSearchService instead of those from RestHighLevelClient.

The methods have the same signature. However, the tricky RuntimeExceptions (unchecked exceptions) from OpenSearch are caught and replaced with OpenSearchRelatedException (checked exceptions). When calling index/bulk, if the index does not exist, it is created based on the data from managed_indices.json.

Example: RestHighLevelClient (managed_indices.json is not used)

IndexRequest myIndexRequest = ...;
RestHighLevelClient restHighLevelClient = openSearchService.getClient();
restHighLevelClient.index(myIndexRequest, RequestOptions.DEFAULT);

Example OpenSearchService (managed_indices.json is used)

IndexRequest myIndexRequest = ...;
openSearchService.index(myIndexRequest, RequestOptions.DEFAULT);

Option 2

Introduce a helper method that is executed before every write operation. Below is an example method for an index with the alias 'books'. The ` OpenSearchService ` tracks the status of indices and can then create the index using ` managed_indices.json ` if it does not exist.

private static final String BOOKS_INDEX_ALIAS_NAME = "books";

private void createBooksIndexIfMissing() throws SystemException {
    try {
        BpcIndexState booksIndexState = oss.getIndexState(BOOKS_INDEX_ALIAS_NAME);
        booksIndexState.prepareUsing(new BpcIndexCreateCallable() {
            @Override
            public String createIndex(OpenSearchService oss) throws OpenSearchRelatedException, ServiceNotFoundException, ModuleNotFoundException {
                return oss.getManagedIndicesHandler().createManagedIndex(BOOKS_INDEX_ALIAS_NAME);
            }
        });
    } catch (Exception ex) {
        throw new SystemException(CoreErrorCode.UNEXPECTED, "Failed to prepare the books index.", ex);
    }
}

Keywords: