How to build a resilient managed OpenSearch Service
Sending logs to OpenSearch can be done via Fluentd or Logstash; both are open-source data collectors, and Fluentd offers better performance with Kubernetes nodes. Other tools are also available.
Sending to different OpenSearch services (targets) is possible and especially useful for production environments, which will allow you to deploy services on different zones for more security.
The behaviour of the Application performance management (APM) is to be verified in the long term; nothing prevents scaling resources vertically if necessary.
- Ubuntu instance
- Two OpenSearch services
Configuring the Log sending
/etc/rsyslogd.conffile and append the following line
Then restart the rsyslogd service:
→ This tells rsyslogd to forward logs to port 5140 (to which Fluentd will be listening).
- Configuring Fluentd to listen to syslog messages and forwarding it to our two OpenSearch services
<source> @type syslog port 5140 tag system </source> <match **> @type opensearch host my-first-opensearch-exoscale-e3c9e78d-a5c6-4727-9c23-90c0a4c23c7c.aivencloud.com port 21699 scheme https ssl_verify false index_name fluentd user avnadmin password xxxxx </match> <match **> @type opensearch host my-second-opensearch-exoscale-e3c9e78d-a5c6-4727-9c23-90c0a4c23c7c.aivencloud.com port 21699 scheme https ssl_verify false index_name fluentd user avnadmin password xxxx </match>
- Running Fluentd
sudo fluentd -c /etc/fluent.conf
When the setup of your instance is completed, you can connect to your OpenSearch dashboard (using the dashboard uri, user and password).
Last step, create an index pattern in your OpenSearch Service: please refer to our webinar min 39:00.
You can also use Logstash, as showed in the webinar.