How to build a resilient managed OpenSearch Service

You can send logs to OpenSearch via Fluentd or Logstash; both are open-source data collectors, and Fluentd offers better performance with Kubernetes nodes. Other tools are also available.

Sending logs to different OpenSearch services (targets) is possible and especially useful for production environments, which will allow you to deploy services on different zones for more security.

Prerequisites

• An Ubuntu instance
• Two OpenSearch services

Configure the Log sending

First, install Fluentd.

Open /etc/rsyslogd.conf file and append the following line:

*.* @127.0.0.1:5140

This tells rsyslogd to forward logs to port 5140 (to which Fluentd will be listening).

Now restart the rsyslogd service.

Configure Fluentd to listen to syslog messages and forward them to our two OpenSearch services:

First open the fluent.conf file.

/etc/fluent.conf

Add the following configuration:

<source>
@type syslog
port 5140
tag system
</source>

<match **>
@type opensearch
host my-first-opensearch-exoscale-e3c9e78d-a5c6-4727-9c23-90c0a4c23c7c.aivencloud.com
port 21699
scheme https
ssl_verify false
index_name fluentd
user avnadmin
password xxxxx
</match>

<match **>
@type opensearch
host my-second-opensearch-exoscale-e3c9e78d-a5c6-4727-9c23-90c0a4c23c7c.aivencloud.com
port 21699
scheme https
ssl_verify false
index_name fluentd
user avnadmin
password xxxx
</match>

Run Fluentd:

sudo fluentd -c /etc/fluent.conf

When the setup of your instance is completed, you can connect to your OpenSearch dashboard with the dashboard URI, user and password.

Create an index pattern in your OpenSearch Service. See our webinar on OpenSearch and GDPR-compliant use of DBaaS at around the 39-minute mark for a walkthrough.