How to build a resilient managed OpenSearch Service
You can send logs to OpenSearch via Fluentd or Logstash; both are open-source data collectors, and Fluentd offers better performance with Kubernetes nodes. Other tools are also available.
Sending logs to different OpenSearch services (targets) is possible and especially useful for production environments, which will allow you to deploy services on different zones for more security.
The behavior of the application performance management (or APM) needs to be verified in the long term, but nothing prevents scaling resources vertically if necessary.
- An Ubuntu instance
- Two OpenSearch services
Configure the Log sending
First, install Fluentd.
/etc/rsyslogd.conf file and append the following line:
This tells rsyslogd to forward logs to port 5140 (to which Fluentd will be listening).
Now restart the rsyslogd service.
Configure Fluentd to listen to syslog messages and forward them to our two OpenSearch services:
First open the
Add the following configuration:
<source> @type syslog port 5140 tag system </source> <match **> @type opensearch host my-first-opensearch-exoscale-e3c9e78d-a5c6-4727-9c23-90c0a4c23c7c.aivencloud.com port 21699 scheme https ssl_verify false index_name fluentd user avnadmin password xxxxx </match> <match **> @type opensearch host my-second-opensearch-exoscale-e3c9e78d-a5c6-4727-9c23-90c0a4c23c7c.aivencloud.com port 21699 scheme https ssl_verify false index_name fluentd user avnadmin password xxxx </match>
sudo fluentd -c /etc/fluent.conf
When the setup of your instance is completed, you can connect to your OpenSearch dashboard with the dashboard URI, user and password.
Create an index pattern in your OpenSearch Service. See our webinar on OpenSearch and GDPR-compliant use of DBaaS at around the 39-minute mark for a walkthrough.
You can also use Logstash, as we demonstrate in the webinar.