Managed OpenSearch specifics
Managed OpenSearch default visualization
Each Managed OpenSearch plan comes with a default visualization included in the OpenSearch Dashboards.
The Dashboard interface is available as a second connection string from the service details - see the Connection Data in the Portal. Use this URL to connect to the visualization tool and start making queries and reports on your data.
OpenSearch replication factor
The Managed OpenSearch plans have different replication factors:
- Hobbyist and Startup plans are single node based, so no replication factor is supported.
- For Business and Premium plans, the replication factor can be adjusted to your application requirements from 1 replica to N-1 replicas where N is the total number of nodes in your cluster.
The Managed OpenSearch database service will automatically adjust the replication factor when the following conditions occur:
- If
number_of_replicasis too large for the current cluster size, it is automatically lowered to the maximum possible value (the number of nodes on the cluster minus 1). - If
number_of_replicasis 0 on a multi-node cluster, it is automatically increased to 1. - If
number_of_replicasis between 1 and the maximum value, it is not adjusted.
When the replication factor (number_of_replicas value) is greater than the size
of the cluster, number_of_replicas is automatically lowered because it is not
possible to replicate index shards to more nodes than there are on the
cluster.
Specific options when creating or updating an OpenSearch service
You can find all the specific options for OpenSearch by using the CLI help:
exo dbaas create --help-opensearch
--opensearch-dashboard-enabled Enable or disable OpenSearch Dashboards (default: true)
--opensearch-dashboard-max-old-space-size Memory limit in MiB for OpenSearch Dashboards. Note: The memory reserved by OpenSearch Dashboards is not available for OpenSearch. (default: 128)
--opensearch-dashboard-request-timeout Timeout in milliseconds for requests made by OpenSearch Dashboards towards OpenSearch (default: 30000)
--opensearch-fork-from-service Service name
--opensearch-index-patterns JSON Array of index patterns (https://openapi-v2.exoscale.com/#operation-get-dbaas-service-opensearch-200-index-patterns)
--opensearch-index-template-mapping-nested-objects-limit The maximum number of nested cli-flag objects that a single document can contain across all nested types. Default is 10000.
--opensearch-index-template-number-of-replicas The number of replicas each primary shard has.
--opensearch-index-template-number-of-shards The number of primary shards that an index should have.
--opensearch-ip-filter Allow incoming connections from CIDR address block
--opensearch-keep-index-refresh-interval index.refresh_interval is reset to default value for every index to be sure that indices are always visible to search. Set to true disable this.
--opensearch-max-index-count Maximum number of indexes to keep before deleting the oldest one
--opensearch-recovery-backup-name Name of a backup to recover from for services that support backup names
--opensearch-settings OpenSearch-specific settings (JSON)
--opensearch-version OpenSearch major version
Restricting connections from the internet
By default, Exoscale DBaaS are not accessible from the whole internet. Data does not transmit in clear over the network as it is SSL encrypted by default and authenticated.
To allow incoming connections to your database service, you can add a filter allowing:
- just one IP address,
- a network range,
- or a combination of IP address and network range
To do this, update your service or create it with the IP filter, which is a comma separated list of CIDRs:
exo dbaas update -z de-muc-1 test-opensearch --opensearch-ip-filter=1.2.3.4/24,5.6.7.8/32
OpenSearch vs Elasticsearch
OpenSearch is an open source fork original Elasticsearch project. When Elastic, the company behind the original Elasticsearch moved to a restrictive license, the changes meant that Exoscale is not able to offer Elasticsearch as a service. The community - including Exoscale DBaaS partner Aiven - joined forces to create and maintain OpenSearch based on the last open source licensed releases of both Elasticsearch and Kibana (v7.10.2) which are now OpenSearch and OpenSearch Dashboards.
Managed OpenSearch additional configuration
You can list all settings available for the database service by using the command:
exo dbaas type show opensearch --settings opensearch
┼───────────────────────────────────────────────────────┼────────────────┼────────────────────────────────────────────────────┼
│ KEY │ TYPE │ DESCRIPTION │
┼───────────────────────────────────────────────────────┼────────────────┼────────────────────────────────────────────────────┼
│ thread_pool_analyze_size │ integer │ Size for the thread pool. See documentation for │
│ │ │ exact details. Do note this may have maximum value │
│ │ │ depending on CPU count - value is automatically │
│ │ │ lowered if set to higher than maximum value. │
│ │ │ * Minimum: 1 / Maximum: 128 │
│ search_max_buckets │ [integer null] │ Maximum number of aggregation buckets allowed in a │
│ │ │ single response. OpenSearch default value is used │
│ │ │ when this is not defined. │
│ │ │ * Minimum: 1 / Maximum: 20000 │
│ │ │ * Example: 10000 │
│ email_sender_username │ [string] │ │
│ thread_pool_force_merge_size │ integer │ Size for the thread pool. See documentation for │
│ │ │ exact details. Do note this may have maximum value │
│ │ │ depending on CPU count - value is automatically │
│ │ │ lowered if set to higher than maximum value. │
│ │ │ * Minimum: 1 / Maximum: 128 │
│ email_sender_password │ [string] │ Sender email password for Opensearch alerts to │
│ │ │ authenticate with SMTP server │
│ │ │ * Example: very-secure-mail-password │
│ http_max_content_length │ integer │ Maximum content length for HTTP requests to the │
│ │ │ OpenSearch HTTP API, in bytes. │
│ │ │ * Minimum: 1 / Maximum: 2.147483647e+09 │
│ thread_pool_search_size │ integer │ Size for the thread pool. See documentation for │
│ │ │ exact details. Do note this may have maximum value │
│ │ │ depending on CPU count - value is automatically │
│ │ │ lowered if set to higher than maximum value. │
│ │ │ * Minimum: 1 / Maximum: 128 │
│ indices_recovery_max_bytes_per_sec │ integer │ Limits total inbound and outbound recovery traffic │
│ │ │ for each node. Applies to both peer recoveries as │
│ │ │ well as snapshot recoveries (i.e., restores from a │
│ │ │ snapshot). Defaults to 40mb │
│ │ │ * Minimum: 40 / Maximum: 400 │
│ email_sender_name │ [string] │ This should be identical to the Sender name │
│ │ │ defined in Opensearch dashboards │
│ │ │ * Example: alert-sender │
│ thread_pool_analyze_queue_size │ integer │ Size for the thread pool queue. See documentation │
│ │ │ for exact details. │
│ │ │ * Minimum: 10 / Maximum: 2000 │
│ thread_pool_get_queue_size │ integer │ Size for the thread pool queue. See documentation │
│ │ │ for exact details. │
│ │ │ * Minimum: 10 / Maximum: 2000 │
│ indices_queries_cache_size │ integer │ Percentage value. Default is 10%. Maximum amount │
│ │ │ of heap used for query cache. This is an expert │
│ │ │ setting. Too low value will decrease query │
│ │ │ performance and increase performance for other │
│ │ │ operations; too high value will cause issues with │
│ │ │ other OpenSearch functionality. │
│ │ │ * Minimum: 3 / Maximum: 40 │
│ http_max_initial_line_length │ integer │ The max length of an HTTP URL, in bytes │
│ │ │ * Minimum: 1024 / Maximum: 65536 │
│ │ │ * Example: 4096 │
│ script_max_compilations_rate │ string │ Script compilation circuit breaker limits the │
│ │ │ number of inline script compilations within a │
│ │ │ period of time. Default is use-context │
│ │ │ * Example: 75/5m │
│ cluster_routing_allocation_node_concurrent_recoveries │ integer │ How many concurrent incoming/outgoing shard │
│ │ │ recoveries (normally replicas) are allowed to │
│ │ │ happen on a node. Defaults to 2. │
│ │ │ * Minimum: 2 / Maximum: 16 │
│ thread_pool_search_queue_size │ integer │ Size for the thread pool queue. See documentation │
│ │ │ for exact details. │
│ │ │ * Minimum: 10 / Maximum: 2000 │
│ thread_pool_get_size │ integer │ Size for the thread pool. See documentation for │
│ │ │ exact details. Do note this may have maximum value │
│ │ │ depending on CPU count - value is automatically │
│ │ │ lowered if set to higher than maximum value. │
│ │ │ * Minimum: 1 / Maximum: 128 │
│ thread_pool_write_queue_size │ integer │ Size for the thread pool queue. See documentation │
│ │ │ for exact details. │
│ │ │ * Minimum: 10 / Maximum: 2000 │
│ action_auto_create_index_enabled │ boolean │ Explicitly allow or block automatic creation of │
│ │ │ indices. Defaults to true │
│ │ │ * Example: false │
│ indices_query_bool_max_clause_count │ integer │ Maximum number of clauses Lucene BooleanQuery can │
│ │ │ have. The default value (1024) is relatively high, │
│ │ │ and increasing it may cause performance issues. │
│ │ │ Investigate other approaches first before │
│ │ │ increasing this value. │
│ │ │ * Minimum: 64 / Maximum: 4096 │
│ cluster_max_shards_per_node │ integer │ Controls the number of shards allowed in the │
│ │ │ cluster per data node │
│ │ │ * Minimum: 100 / Maximum: 10000 │
│ │ │ * Example: 1000 │
│ override_main_response_version │ boolean │ Compatibility mode sets OpenSearch to report its │
│ │ │ version as 7.10 so clients continue to work. │
│ │ │ Default is false │
│ │ │ * Example: true │
│ thread_pool_search_throttled_queue_size │ integer │ Size for the thread pool queue. See documentation │
│ │ │ for exact details. │
│ │ │ * Minimum: 10 / Maximum: 2000 │
│ thread_pool_search_throttled_size │ integer │ Size for the thread pool. See documentation for │
│ │ │ exact details. Do note this may have maximum value │
│ │ │ depending on CPU count - value is automatically │
│ │ │ lowered if set to higher than maximum value. │
│ │ │ * Minimum: 1 / Maximum: 128 │
│ reindex_remote_whitelist │ [array null] │ Whitelisted addresses for reindexing. Changing │
│ │ │ this value will cause all OpenSearch instances to │
│ │ │ restart. │
│ http_max_header_size │ integer │ The max size of allowed headers, in bytes │
│ │ │ * Minimum: 1024 / Maximum: 262144 │
│ │ │ * Example: 8192 │
│ indices_recovery_max_concurrent_file_chunks │ integer │ Number of file chunks sent in parallel for each │
│ │ │ recovery. Defaults to 2. │
│ │ │ * Minimum: 2 / Maximum: 5 │
│ indices_fielddata_cache_size │ [integer null] │ Relative amount. Maximum amount of heap memory │
│ │ │ used for field data cache. This is an expert │
│ │ │ setting; decreasing the value too much will │
│ │ │ increase overhead of loading field data; too much │
│ │ │ memory used for field data cache will decrease │
│ │ │ amount of heap available for other operations. │
│ │ │ * Minimum: 3 / Maximum: 100 │
│ │ │ * Default: <nil> │
│ action_destructive_requires_name │ [boolean null] │ │
│ indices_memory_index_buffer_size │ integer │ Percentage value. Default is 10%. Total amount of │
│ │ │ heap used for indexing buffer, before writing │
│ │ │ segments to disk. This is an expert setting. Too │
│ │ │ low value will slow down indexing; too high value │
│ │ │ will increase indexing performance but causes │
│ │ │ performance issues for query performance. │
│ │ │ * Minimum: 3 / Maximum: 40 │
│ thread_pool_write_size │ integer │ Size for the thread pool. See documentation for │
│ │ │ exact details. Do note this may have maximum value │
│ │ │ depending on CPU count - value is automatically │
│ │ │ lowered if set to higher than maximum value. │
│ │ │ * Minimum: 1 / Maximum: 128 │
┼───────────────────────────────────────────────────────┼────────────────┼────────────────────────────────────────────────────┼
You can also update the settings of your database service with the following command:
exo dbaas update --zone de-fra-1 target-opensearch-service-name --opensearch-settings '{"http_max_header_size":262144}'
Note
The parameter of --opensearch-settings has to be in JSON format.
OpenSearch access control lists (ACLs)
To set up access control lists (ACLs) for OpenSearch content, you need to:
- Create a new user in the
Userstab - Add a new ACL related to that specific user by creating rules within the
ACLtab - click on
Submit
The new rules will take effect once you toggle the Enable ACL option on the right of the ACL tab.
Note
-
Rules are defined separately for each user as pattern/permission combinations.
-
The pattern defines the indices that the permission applies to. Patterns are glob-style, where * (an asterisk) matches any number of characters and ? (a question mark) matches any character.
-
Enabling ACLs does not restrict access to OpenSearch Dashboards itself, but all requests done by OpenSearch Dashboards are checked against the current user’s ACLs.
-
In practice, for OpenSearch Dashboards to function properly, you must grant the user admin-level access to the _msearch interface (permission: admin, pattern: _msearch) or switch on the ExtendedAcl option. Knowing that _msearch, _mget and so on are “top-level” API endpoints of OpenSearch. Only rules where the pattern starts with _ are considered for top-level API access.
-
You can switch on the
Enable Extended ACLoption for the service to enforce index rules in a limited fashion for requests that only use the _mget, _msearch and _bulk APIs. When Extended ACL is enabled, service users can access these APIs as long as all operations only target indices that they have appropriate permissions for.