Sending logs to OpenSearch can be done via Fluentd or Logstash; both are open-source data collectors, and Fluentd offers better performance with Kubernetes nodes. Other tools are also available.

Sending to different OpenSearch services (targets) is possible and especially useful for production environments, which will allow you to deploy services on different zones for more security.

Note

The behaviour of the Application performance management (APM) is to be verified in the long term; nothing prevents scaling resources vertically if necessary.

Prerequisites

  • Ubuntu instance
  • Two OpenSearch services

Configuring the Log sending

  • Install Fluentd

  • Open /etc/rsyslogd.conf file and append the following line

*.* @127.0.0.1:5140

Then restart the rsyslogd service:

→ This tells rsyslogd to forward logs to port 5140 (to which Fluentd will be listening).

  • Configuring Fluentd to listen to syslog messages and forwarding it to our two OpenSearch services
/etc/fluent.conf

<source>
@type syslog 
port 5140 
tag system
</source>

<match **>
@type opensearch
host my-first-opensearch-exoscale-e3c9e78d-a5c6-4727-9c23-90c0a4c23c7c.aivencloud.com
port 21699
scheme https
ssl_verify false
index_name fluentd
user avnadmin
password xxxxx
</match>

<match **>
@type opensearch
host my-second-opensearch-exoscale-e3c9e78d-a5c6-4727-9c23-90c0a4c23c7c.aivencloud.com
port 21699
scheme https
ssl_verify false
index_name fluentd
user avnadmin
password xxxx
</match>
  • Running Fluentd
sudo fluentd -c /etc/fluent.conf

  • When the setup of your instance is completed, you can connect to your OpenSearch dashboard (using the dashboard uri, user and password).

  • Last step, create an index pattern in your OpenSearch Service: please refer to our webinar min 39:00.

Note

You can also use Logstash, as showed in the webinar.