Managed ClickHouse® metrics available via Prometheus

Managed ClickHouse® services can expose their metrics on a Prometheus scrape endpoint. The endpoint is not enabled by default: you create a Prometheus external endpoint, attach your service to it, then scrape it.

Enable the Prometheus endpoint

1. Create a Prometheus external endpoint with the credentials your Prometheus server uses:

   exo dbaas external-endpoint create prometheus my-prometheus \
     --prometheus-basic-auth-username PROMETHEUS_USER \
     --prometheus-basic-auth-password PROMETHEUS_PASSWORD

2. Attach your Managed ClickHouse service to that endpoint, using the endpoint ID returned above (exo dbaas external-endpoint list shows it again):

   exo dbaas external-integration attach prometheus \
     --source-service-name my-clickhouse \
     --destination-endpoint-id ENDPOINT_ID

3. Read the scrape host and port from the service. They are published as prometheus-uri, and the port is not part of the service components list:

   exo x get-dbaas-service-clickhouse my-clickhouse -z ch-gva-2 -q '"prometheus-uri"'

Scrape the endpoint

The endpoint uses TLS with a certificate issued by the certificate authority of your organization’s DBaaS services, and it is protected by the basic authentication credentials of the external endpoint. Retrieve the CA certificate first:

exo dbaas ca-certificate -z ch-gva-2 > ca.pem

Use the host and port published in prometheus-uri as CLICKHOUSE_HOSTNAME and PROMETHEUS_PORT. The port was 9273 on the services used for this guide.

Then request the metrics:

curl --cacert ca.pem \
    --user 'PROMETHEUS_USER:PROMETHEUS_PASSWORD' \
    'https://CLICKHOUSE_HOSTNAME:PROMETHEUS_PORT/metrics'

The response is standard Prometheus exposition format, with one label set per service and node:

# HELP clickhouse_metrics_query Telegraf collected metric
# TYPE clickhouse_metrics_query untyped
clickhouse_metrics_query{cloud="exoscale-ch-gva-2",host="my-clickhouse-1",project="exoscale-ORGANIZATION_ID",service="my-clickhouse",service_type="clickhouse"} 6

Exposed metrics

The exposed metric set depends on the plan and the topology of your service: single-node services expose no Keeper or replication queue metrics, while multi-node services add them. The metrics come from the following families:

Retrieve the exact list for your own service with the curl command above, for example:

curl -s --cacert ca.pem --user 'PROMETHEUS_USER:PROMETHEUS_PASSWORD' \
    'https://CLICKHOUSE_HOSTNAME:PROMETHEUS_PORT/metrics' | grep '^# HELP' | awk '{print $3}' | sort -u

Related pages

• Managed ClickHouse® metrics available in Datadog
• Use External Integrations