# Exoscale Documentation — Kafka > Managed Kafka: fully managed Kafka clusters for real-time data streaming, with patching, monitoring and backups handled by Exoscale. --- url: https://community.exoscale.com/product/dbaas/service-specific/kafka/ --- # Kafka Apache Kafka is a distributed event streaming platform built for high-throughput, fault-tolerant, real-time data pipelines and messaging. - [Quick Start - Kafka](https://community.exoscale.com/product/dbaas/service-specific/kafka/quick-start/index.md) - Get started quickly with a guided setup to achieve your first successful use of the service, including the required steps and configuration. - [Overview - Kafka](https://community.exoscale.com/product/dbaas/service-specific/kafka/overview/index.md) - Terminology and key features at a glance, learn the core concepts, capabilities, and options of this service before you dive deeper. - [How-To - Kafka](https://community.exoscale.com/product/dbaas/service-specific/kafka/how-to/index.md) - Step-by-step guides to configure the service and use specific features, with clear instructions for common tasks and problem solving. - [Service Boundaries - Kafka](https://community.exoscale.com/product/dbaas/service-specific/kafka/service-boundaries/index.md) - Service boundaries: quotas, limits, and the guaranteed service levels (SLA) for this product, including key constraints to plan and operate reliably. - [Reference - Kafka](https://community.exoscale.com/product/dbaas/service-specific/kafka/reference/index.md) - Reference documentation for API, CLI, and tools, browse comprehensive details, parameters, commands, and integration notes. --- url: https://community.exoscale.com/product/dbaas/service-specific/kafka/quick-start/ --- # Quick Start - Kafka ## Specific Options for Kafka You can find all the specific options for Kafka by using the CLI help: ```bash exo dbaas create --help-kafka --kafka-connect-settings Kafka Connect configuration settings (JSON format) --kafka-enable-cert-auth enable certificate-based authentication method --kafka-enable-kafka-connect enable Kafka Connect --kafka-enable-kafka-rest enable Kafka REST --kafka-enable-sasl-auth enable SASL-based authentication method --kafka-enable-schema-registry enable Schema Registry --kafka-ip-filter allow incoming connections from CIDR address block --kafka-rest-settings Kafka REST configuration settings (JSON format) --kafka-schema-registry-settings Schema Registry configuration settings (JSON format) --kafka-settings Kafka configuration settings (JSON format) --kafka-version Kafka major version ``` ## Additional Configuration A large number of service specific configuration options are available: - Kafka Connect - Kafka REST - Kafka Registry - Kafka general The general settings for Managed Kafka are: ```bash exo dbaas type show kafka --settings kafka ``` Output: ```bash ┼────────────────────────────────────────────────────────────┼─────────┼──────────────────────────────────────────────────────┼ │ KEY │ TYPE │ DESCRIPTION │ ┼────────────────────────────────────────────────────────────┼─────────┼──────────────────────────────────────────────────────┼ │ log_segment_delete_delay_ms │ integer │ The amount of time to wait before deleting a file │ │ │ │ from the filesystem │ │ │ │ * Minimum: 0 / Maximum: 3.6e+06 │ │ │ │ * Example: 60000 │ │ log_roll_ms │ integer │ The maximum time before a new log segment is │ │ │ │ rolled out (in milliseconds). │ │ │ │ * Minimum: 1 / Maximum: 9.223372036854776e+18 │ │ log_cleaner_min_compaction_lag_ms │ integer │ The minimum time a message will remain uncompacted │ │ │ │ in the log. Only applicable for logs that are │ │ │ │ being compacted. │ │ │ │ * Minimum: 0 / Maximum: 9.223372036854776e+18 │ │ max_incremental_fetch_session_cache_slots │ integer │ The maximum number of incremental fetch sessions │ │ │ │ that the broker will maintain. │ │ │ │ * Minimum: 1000 / Maximum: 10000 │ │ │ │ * Example: 1000 │ │ group_min_session_timeout_ms │ integer │ The minimum allowed session timeout for registered │ │ │ │ consumers. Longer timeouts give consumers more │ │ │ │ time to process messages in between heartbeats at │ │ │ │ the cost of a longer time to detect failures. │ │ │ │ * Minimum: 0 / Maximum: 60000 │ │ │ │ * Example: 6000 │ │ offsets_retention_minutes │ integer │ Log retention window in minutes for offsets topic │ │ │ │ * Minimum: 1 / Maximum: 2.147483647e+09 │ │ │ │ * Example: 10080 │ │ log_flush_interval_ms │ integer │ The maximum time in ms that a message in any topic │ │ │ │ is kept in memory before flushed to disk. If not │ │ │ │ set, the value in log.flush.scheduler.interval.ms │ │ │ │ is used │ │ │ │ * Minimum: 0 / Maximum: 9.223372036854776e+18 │ │ log_retention_hours │ integer │ The number of hours to keep a log file before │ │ │ │ deleting it │ │ │ │ * Minimum: -1 / Maximum: 2.147483647e+09 │ │ log_retention_ms │ integer │ The number of milliseconds to keep a log file │ │ │ │ before deleting it (in milliseconds), If not set, │ │ │ │ the value in log.retention.minutes is used. If set │ │ │ │ to -1, no time limit is applied. │ │ │ │ * Minimum: -1 / Maximum: 9.223372036854776e+18 │ │ min_insync_replicas │ integer │ When a producer sets acks to 'all' (or '-1'), │ │ │ │ min.insync.replicas specifies the minimum number │ │ │ │ of replicas that must acknowledge a write for the │ │ │ │ write to be considered successful. │ │ │ │ * Minimum: 1 / Maximum: 7 │ │ │ │ * Example: 1 │ │ log_preallocate │ boolean │ Should pre allocate file when create new segment? │ │ │ │ * Example: false │ │ group_initial_rebalance_delay_ms │ integer │ The amount of time, in milliseconds, the group │ │ │ │ coordinator will wait for more consumers to join a │ │ │ │ new group before performing the first rebalance. A │ │ │ │ longer delay means potentially fewer rebalances, │ │ │ │ but increases the time until processing begins. │ │ │ │ The default value for this is 3 seconds. During │ │ │ │ development and testing it might be desirable to │ │ │ │ set this to 0 in order to not delay test execution │ │ │ │ time. │ │ │ │ * Minimum: 0 / Maximum: 300000 │ │ │ │ * Example: 3000 │ │ log_message_timestamp_difference_max_ms │ integer │ The maximum difference allowed between the │ │ │ │ timestamp when a broker receives a message and the │ │ │ │ timestamp specified in the message │ │ │ │ * Minimum: 0 / Maximum: 9.223372036854776e+18 │ │ log_index_size_max_bytes │ integer │ The maximum size in bytes of the offset index │ │ │ │ * Minimum: 1.048576e+06 / Maximum: 1.048576e+08 │ │ │ │ * Example: 1.048576e+07 │ │ message_max_bytes │ integer │ The maximum size of message that the server can │ │ │ │ receive. │ │ │ │ * Minimum: 0 / Maximum: 1.000012e+08 │ │ │ │ * Example: 1.048588e+06 │ │ log_roll_jitter_ms │ integer │ The maximum jitter to subtract from │ │ │ │ logRollTimeMillis (in milliseconds). If not set, │ │ │ │ the value in log.roll.jitter.hours is used │ │ │ │ * Minimum: 0 / Maximum: 9.223372036854776e+18 │ │ transaction_remove_expired_transaction_cleanup_interval_ms │ integer │ The interval at which to remove transactions that │ │ │ │ have expired due to transactional.id.expiration.ms │ │ │ │ passing (defaults to 3600000 (1 hour)). │ │ │ │ * Minimum: 600000 / Maximum: 3.6e+06 │ │ │ │ * Example: 3.6e+06 │ │ log_message_downconversion_enable │ boolean │ This configuration controls whether │ │ │ │ down-conversion of message formats is enabled to │ │ │ │ satisfy consume requests. │ │ │ │ * Example: true │ │ log_cleaner_max_compaction_lag_ms │ integer │ The maximum amount of time message will remain │ │ │ │ uncompacted. Only applicable for logs that are │ │ │ │ being compacted │ │ │ │ * Minimum: 30000 / Maximum: 9.223372036854776e+18 │ │ socket_request_max_bytes │ integer │ The maximum number of bytes in a socket request │ │ │ │ (defaults to 104857600). │ │ │ │ * Minimum: 1.048576e+07 / Maximum: 2.097152e+08 │ │ log_segment_bytes │ integer │ The maximum size of a single log file │ │ │ │ * Minimum: 1.048576e+07 / Maximum: 1.073741824e+09 │ │ replica_fetch_max_bytes │ integer │ The number of bytes of messages to attempt to │ │ │ │ fetch for each partition (defaults to 1048576). │ │ │ │ This is not an absolute maximum, if the first │ │ │ │ record batch in the first non-empty partition of │ │ │ │ the fetch is larger than this value, the record │ │ │ │ batch will still be returned to ensure that │ │ │ │ progress can be made. │ │ │ │ * Minimum: 1.048576e+06 / Maximum: 1.048576e+08 │ │ log_flush_interval_messages │ integer │ The number of messages accumulated on a log │ │ │ │ partition before messages are flushed to disk │ │ │ │ * Minimum: 1 / Maximum: 9.223372036854776e+18 │ │ │ │ * Example: 9.223372036854776e+18 │ │ max_connections_per_ip │ integer │ The maximum number of connections allowed from │ │ │ │ each ip address (defaults to 2147483647). │ │ │ │ * Minimum: 256 / Maximum: 2.147483647e+09 │ │ log_index_interval_bytes │ integer │ The interval with which Kafka adds an entry to the │ │ │ │ offset index │ │ │ │ * Minimum: 0 / Maximum: 1.048576e+08 │ │ │ │ * Example: 4096 │ │ transaction_state_log_segment_bytes │ integer │ The transaction topic segment bytes should be kept │ │ │ │ relatively small in order to facilitate faster log │ │ │ │ compaction and cache loads (defaults to 104857600 │ │ │ │ (100 mebibytes)). │ │ │ │ * Minimum: 1.048576e+06 / Maximum: 2.147483647e+09 │ │ │ │ * Example: 1.048576e+08 │ │ log_cleanup_policy │ string │ The default cleanup policy for segments beyond the │ │ │ │ retention window │ │ │ │ * Supported values: │ │ │ │ - delete │ │ │ │ - compact │ │ │ │ - compact,delete │ │ │ │ * Example: delete │ │ default_replication_factor │ integer │ Replication factor for autocreated topics │ │ │ │ * Minimum: 1 / Maximum: 10 │ │ producer_purgatory_purge_interval_requests │ integer │ The purge interval (in number of requests) of the │ │ │ │ producer request purgatory(defaults to 1000). │ │ │ │ * Minimum: 10 / Maximum: 10000 │ │ group_max_session_timeout_ms │ integer │ The maximum allowed session timeout for registered │ │ │ │ consumers. Longer timeouts give consumers more │ │ │ │ time to process messages in between heartbeats at │ │ │ │ the cost of a longer time to detect failures. │ │ │ │ * Minimum: 0 / Maximum: 1.8e+06 │ │ │ │ * Example: 1.8e+06 │ │ auto_create_topics_enable │ boolean │ Enable auto creation of topics │ │ │ │ * Example: true │ │ connections_max_idle_ms │ integer │ Idle connections timeout: the server socket │ │ │ │ processor threads close the connections that idle │ │ │ │ for longer than this. │ │ │ │ * Minimum: 1000 / Maximum: 3.6e+06 │ │ │ │ * Example: 540000 │ │ log_cleaner_min_cleanable_ratio │ number │ Controls log compactor frequency. Larger value │ │ │ │ means more frequent compactions but also more │ │ │ │ space wasted for logs. Consider setting │ │ │ │ log.cleaner.max.compaction.lag.ms to enforce │ │ │ │ compactions sooner, instead of setting a very high │ │ │ │ value for this option. │ │ │ │ * Minimum: 0.2 / Maximum: 0.9 │ │ │ │ * Example: 0.5 │ │ log_retention_bytes │ integer │ The maximum size of the log before deleting │ │ │ │ messages │ │ │ │ * Minimum: -1 / Maximum: 9.223372036854776e+18 │ │ compression_type │ string │ Specify the final compression type for a given │ │ │ │ topic. This configuration accepts the standard │ │ │ │ compression codecs ('gzip', 'snappy', 'lz4', │ │ │ │ 'zstd'). It additionally accepts 'uncompressed' │ │ │ │ which is equivalent to no compression; and │ │ │ │ 'producer' which means retain the original │ │ │ │ compression codec set by the producer. │ │ │ │ * Supported values: │ │ │ │ - gzip │ │ │ │ - snappy │ │ │ │ - lz4 │ │ │ │ - zstd │ │ │ │ - uncompressed │ │ │ │ - producer │ │ log_cleaner_delete_retention_ms │ integer │ How long are delete records retained? │ │ │ │ * Minimum: 0 / Maximum: 3.1556926e+11 │ │ │ │ * Example: 8.64e+07 │ │ num_partitions │ integer │ Number of partitions for autocreated topics │ │ │ │ * Minimum: 1 / Maximum: 1000 │ │ log_message_timestamp_type │ string │ Define whether the timestamp in the message is │ │ │ │ message create time or log append time. │ │ │ │ * Supported values: │ │ │ │ - CreateTime │ │ │ │ - LogAppendTime │ │ replica_fetch_response_max_bytes │ integer │ Maximum bytes expected for the entire fetch │ │ │ │ response (defaults to 10485760). Records are │ │ │ │ fetched in batches, and if the first record batch │ │ │ │ in the first non-empty partition of the fetch is │ │ │ │ larger than this value, the record batch will │ │ │ │ still be returned to ensure that progress can be │ │ │ │ made. As such, this is not an absolute maximum. │ │ │ │ * Minimum: 1.048576e+07 / Maximum: 1.048576e+09 │ ┼────────────────────────────────────────────────────────────┼─────────┼──────────────────────────────────────────────────────┼ ``` You can also update the settings of your database service: ```bash exo dbaas update --zone de-fra-1 target-kafka-service-name --kafka-settings '{"compression_type":"lz4"} ``` > [!NOTE] > The parameter of `--kafka-settings` has to be in JSON format. --- url: https://community.exoscale.com/product/dbaas/service-specific/kafka/overview/ --- # Overview - Kafka Exoscale **Managed Kafka** delivers fully managed Kafka clusters so you can stream data in real time without operating brokers yourself. Provision a cluster in minutes, scale as traffic grows, and let Exoscale handle patching, monitoring & backup. ## Terminology Service : a managed Kafka deployment (one or more brokers) Broker : a single Kafka server that stores topic partitions and serves client requests Topic : a named stream of records split into partitions Partition : an ordered, replicated log segment of a topic Plan : resource & redundancy tier such as `startup-2`, `business-9x-16` ACL : rule that authorises a principal on a topic, group or cluster resource Schema Registry / Kafka Connect / REST : optional components you can enable per service ## Features Managed lifecycle : automated provisioning, rolling upgrades and in-place re-sizing High availability : multi-broker clusters, automatic leader election, 99.99 % SLA on Business & Premium plans Strong security : mTLS certificate authentication by default; optional SASL/PLAIN for testing Observability built-in : metrics shipped to Exoscale Monitoring dashboards, logs streamable to Object Storage Add-ons on demand : toggle Kafka Connect, REST Proxy or Schema Registry with a single CLI flag API-first control : full feature set exposed via `exo dbaas`; Portal surfaces common day-to-day tasks --- url: https://community.exoscale.com/product/dbaas/service-specific/kafka/how-to/ --- # How-To - Kafka Step-by-step guides to configure the service and use specific features, with clear instructions for common tasks and problem solving. - [Connect to Kafka](https://community.exoscale.com/product/dbaas/service-specific/kafka/how-to/connect-instance/index.md) - Connect to Managed Kafka with SSL certificates or SASL, create topics, and produce or consume messages using your preferred client library. - [Cycle Kafka Credentials](https://community.exoscale.com/product/dbaas/service-specific/kafka/how-to/credential-cycling/index.md) - Cycle Kafka credentials by creating a new service user, updating client certificates and keys, and removing old access securely. - [Retrieve Kafka Certificates](https://community.exoscale.com/product/dbaas/service-specific/kafka/how-to/ca-retrieve/index.md) - Retrieve the Kafka CA certificate, client certificate, and access key with the Exoscale CLI to configure secure SSL client connections. - [Use SSL Authentication](https://community.exoscale.com/product/dbaas/service-specific/kafka/how-to/auth-ca/index.md) - Use SSL authentication with Exoscale Managed Kafka by configuring trusted certificates, keystores, and truststores for secure client access. --- url: https://community.exoscale.com/product/dbaas/service-specific/kafka/how-to/connect-instance/ --- # Connect to Kafka At this point, you can create a topic using your programming language of choice. Here is an example with [kafka-python](https://pypi.org/project/kafka-python/). Please note that if you use the kafka-python guide, you must use your servername:port to establish a successful connection, **not** the one in the following examples. ```python # topic-ssl.py # This script creates a topic named demo_topic from kafka.admin import KafkaAdminClient, NewTopic admin_client = KafkaAdminClient( bootstrap_servers="test-kafka-exoscale-a334664a-3015-4b86-8725-edf585d6cfb4.aivencloud.com:21701", client_id="test", security_protocol="SSL", ssl_cafile="ca.pem", ssl_certfile="test-kafka_cert.pem", ssl_keyfile="test-kafka_key.pem", ) topic_list = [] topic_list.append(NewTopic(name="demo-topic", num_partitions=1, replication_factor=1)) admin_client.create_topics(new_topics=topic_list, validate_only=False) #EOF ``` At this point, you can produce and consume messages. Here is another example with [kafka-python](https://pypi.org/project/kafka-python/): ## Producer ```python # producer-ssl.py # This script connects to Kafka and send a few messages from kafka import KafkaProducer producer = KafkaProducer( bootstrap_servers="test-kafka-exoscale-a334664a-3015-4b86-8725-edf585d6cfb4.aivencloud.com:21701", security_protocol="SSL", ssl_cafile="ca.pem", ssl_certfile="test-kafka_cert.pem", ssl_keyfile="test-kafka_key.pem", ) for i in range(1, 4): message = "message number {}".format(i) print("Sending: {}".format(message)) producer.send("demo-topic", message.encode("utf-8")) # Force sending of all messages producer.flush() #EOF ``` ## Consumer ```python # consumer-ssl.py # This script receives messages from a Kafka topic from kafka import KafkaConsumer consumer = KafkaConsumer( "demo-topic", auto_offset_reset="earliest", bootstrap_servers="test-kafka-exoscale-a334664a-3015-4b86-8725-edf585d6cfb4.aivencloud.com:21701", client_id="test", group_id="demo-group", security_protocol="SSL", ssl_cafile="ca.pem", ssl_certfile="test-kafka_cert.pem", ssl_keyfile="test-kafka_key.pem" ) # Call poll twice. First call will just assign partitions for our # consumer without actually returning anything for _ in range(2): raw_msgs = consumer.poll(timeout_ms=1000) for tp, msgs in raw_msgs.items(): for msg in msgs: print("Received: {}".format(msg.value)) # Commit offsets so we won't get the same messages again consumer.commit() #EOF ``` At this point, can produce and consume messages using your programming language of choice. Please note that if you haven't created a topic already, you must create one. Otherwise, the connection will not be successful. ## SASL Connection If you want to check for the SASL authentication status, you can query it specifically with: ```bash exo dbaas show test-kafka -z de-muc-1 --output-template '{{.Kafka.AuthenticationMethods.SASL}}' ``` > [!NOTE] > SASL is less secure. The SASL authentication method for Kafka clusters is less secure than SSL certificate authentication. However, it is easier to manipulate for tests and demonstration purposes. Use at your own risk. To enable SASL, set the update flag with: ```bash exo dbaas update -z de-muc-1 test-kafka --kafka-enable-sasl-auth true ``` Output: ```bash Updating Database Service "test-kafka"... ┼─────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼ │ DATABASE SERVICE │ │ ┼─────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼ │ Zone │ de-muc-1 │ │ Name │ test-kafka │ │ Type │ kafka │ │ Plan │ startup-2 │ │ Disk Size │ 90 GiB │ │ State │ running │ │ Creation Date │ 2021-11-01 15:11:37 +0000 UTC │ │ Update Date │ 2021-11-01 15:55:54 +0000 UTC │ │ Nodes │ 3 │ │ Node CPUs │ 2 │ │ Node Memory │ 2.0 GiB │ │ Termination Protected │ true │ │ Maintenance │ wednesday (18:48:43) │ │ URI │ test-kafka-exoscale-08b0165e-ef03-47ec-926f-f01163d557ed.aivencloud.com:21701 │ │ IP Filter │ 0.0.0.0/0 │ │ Authentication Methods │ certificate, SASL │ │ Kafka Connect Enabled │ false │ │ Kafka REST Enabled │ false │ │ Schema Registry Enabled │ false │ │ Components │ │ │ │ kafka test-kafka-exoscale-08b0165e-ef03-47ec-926f-f01163d557ed.aivencloud.com:21701 auth:certificate route:dynamic usage:primary │ │ │ kafka test-kafka-exoscale-08b0165e-ef03-47ec-926f-f01163d557ed.aivencloud.com:21712 auth:sasl route:dynamic usage:primary │ │ │ │ │ ACL │ │ │ │ default username:avnadmin topic:* permission:admin │ │ │ │ │ Users │ avnadmin (primary) │ ┼─────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼ ``` At this point, the service offers a different port for the SASL connection, whereas the endpoint remains the same. You need to retrieve the SASL port. From the example above, we now have SASL port 21712. The last thing we need is the username and password, which you can use the following full JSON output to retrieve: ```bash exo dbaas show test-kafka -z de-muc-1 -O json ``` --- url: https://community.exoscale.com/product/dbaas/service-specific/kafka/how-to/credential-cycling/ --- # Cycle Kafka Credentials In the context of Exoscale Managed Kafka, cycling credentials is a critical task to maintain secure access control and minimize potential security risks. This involves the process of creating a new service user and handling the associated SSL certificates and keys effectively. #### Steps to Cycle Credentials By cycling credentials regularly, Exoscale Managed Kafka users can maintain a robust security posture, mitigating risks associated with credential exposure or compromise. This process not only strengthens security but also aligns with best practices for managing secure access in a dynamic cloud environment. - **Create a New Service User** Begin by generating a new service user within the Exoscale platform. This user will be used to access Kafka resources with renewed credentials,ensuring any exposed or old credentials are replaced. As part of this creation process, a new SSL certificate and private key are issued to the new service user. These are crucial components forauthentication and must be stored securely. - **Retrieve Certificate and Key** Once the new service user is created, immediately download and store the SSL certificate and private key. Proper storage of these credentials isessential to prevent unauthorized access and to maintain uninterrupted secure communications with the Kafka cluster. If possible, use a secure secrets management tool to store these credentials, reducing the risk of exposure. - **Update Client Configuration** Replace the old credentials in your clients (producers/consumers) with the new certificate and key. Update the `keystore` and `truststore`configurations to reflect these changes. This step ensures that your clients can continue to connect to Exoscale Managed Kafka seamlessly and securely using the new certificates. - **Delete the Previous Service User** After successfully updating your clients with the new credentials, delete the old service user from the Exoscale platform. This step is crucialto prevent any potential misuse of outdated credentials. Deleting the service user formally invalidates the old credentials, ensuring that only the new credentials are valid for accessing the Kafka resources. --- url: https://community.exoscale.com/product/dbaas/service-specific/kafka/how-to/ca-retrieve/ --- # Retrieve Kafka Certificates To retrieve the CA for a newly created cluster, run the following command: ```bash exo dbaas ca-certificate > ca.pem ``` The file `ca.pem` now contains the CA for the Exoscale DBaaS. To retrieve the access certificate and key, we need to use the output templates of the CLI. ```bash exo dbaas show test-kafka -z de-muc-1 --output-template '{{.Kafka.ConnectionInfo.AccessCert}}' ``` Output: ```bash -----BEGIN CERTIFICATE----- MIIEPDCCAqSgAwIBAgIUf+hoGaT3XjwEbguv9LYqNiVUw54wDQYJKoZIhvcNAQEM BQAwOjE4MDYGA1UEAwwvYjJmODcwNmUtMDBmNi00M2MzLTkyMjktYTU5ZTYzODhm ZGQwIFByb2plY3QgQ0EwHhcNMjExMTAxMTUxMTM2WhcNMjQwMTMwMTUxMTM2WjA7 .... .... qI70kNpCdbiqVf9EXvlmEkGFbKkR5rGW+QvdGwXTLBs= -----END CERTIFICATE----- ``` For further handling, we will put the certificate and key into files. First for the certificate: ```bash exo dbaas show test-kafka -z de-muc-1 --output-template '{{.Kafka.ConnectionInfo.AccessCert}}' > test-kafka_cert.pem ``` And then for the key: ```bash exo dbaas show test-kafka -z de-muc-1 --output-template '{{.Kafka.ConnectionInfo.AccessKey}}' > test-kafka_key.pem ``` --- url: https://community.exoscale.com/product/dbaas/service-specific/kafka/how-to/auth-ca/ --- # Use SSL Authentication When connecting producers and consumers to a topic within Exoscale Managed Kafka, the sole supported connection mechanism is through SSL certificate validation. This robust security measure ensures that both data integrity and confidentiality are maintained throughout the transmission process. ## SSL Certificate Validation Exoscale Managed Kafka uses SSL/TLS for encryption and authentication, which requires each producer and consumer to present a valid SSL certificate before a connection can be established. This authentication method prevents unauthorized access and guarantees that only verified entities can interact with the Kafka cluster. - **Certificate Authority (CA)** Exoscale provides a trusted Certificate Authority that issues SSL certificates. Clients must use a certificate signed by this CA for successful authentication. This CA acts as a trusted third party, vouching for the identity of the participating entities. - **Client Configuration** Each client (producer or consumer) is required to maintain a __keystore__ containing the SSL certificate and private key. This keystore is crucial for establishing the client's identity during the SSL handshake process. Clients must also configure a __truststore__ that contains the CA certificate provided by Exoscale. The truststore is used to validate the server's certificate during the SSL handshake, ensuring that the client is communicating with an authentic Kafka broker. - **Connection Setup:** When a client attempts to connect to a Kafka broker, an SSL handshake occurs. During this handshake, the client presents its certificate, and the broker validates it against the CA. Simultaneously, the broker presents its certificate, which the client validates using its truststore. Once both parties' certificates are verified, a secure SSL connection is established, enabling the client to produce or consume messages securely. ## Benefits of SSL - **Security** By using SSL certificate validation, Exoscale Managed Kafka ensures a high level of security, protecting data from eavesdropping, tampering, and forgery. - **Trust** Leveraging a trusted Certificate Authority provides assurance that the entities interacting with the Kafka cluster are verified and trusted. - **Compliance** SSL certificate validation helps meet compliance requirements for data protection and privacy standards, making Exoscale Managed Kafka suitable for handling sensitive data. --- url: https://community.exoscale.com/product/dbaas/service-specific/kafka/service-boundaries/ --- # Service Boundaries - Kafka - [Limits and Quotas - Kafka](https://community.exoscale.com/product/dbaas/service-specific/kafka/service-boundaries/limits-and-quotas/index.md) - Managed Kafka Limitations and Quotas - [SLA - Kafka](https://community.exoscale.com/product/dbaas/service-specific/kafka/service-boundaries/sla/index.md) - Managed Kafka Service Level Agreement - [Pricing ↗︎](https://www.exoscale.com/pricing/#managed-apache-kafka) --- url: https://community.exoscale.com/product/dbaas/service-specific/kafka/service-boundaries/limits-and-quotas/ --- # Limits and Quotas - Kafka ## Limits Portal Feature Gap : advanced ACLs and component toggles are CLI / API only Single-Zone Clusters : brokers run in one zone; cross-zone replication must be configured externally No Custom Broker Plug-Ins : installing third-party JARs on the service is not supported Topic bBckups : automatic snapshot export is not available; rely on replication and downstream consumers SASL Security : provided for testing but less secure than certificate authentication Service Plans Restrictions : Access to certain plans (e.g., Business, Premium) requires activation by the support team. Requests can be made during service creation. ## Quotas | Usage | Quota | | -------------------------------------------------------- | ------------------------ | | DBaaS services overall | 10 | ## Availability Managed Kafka is available in all zones. --- url: https://community.exoscale.com/product/dbaas/service-specific/kafka/service-boundaries/sla/ --- # SLA - Kafka Refere to the overall [DBaaS SLA](https://community.exoscale.com/product/dbaas/service-boundaries/sla/index.md). --- url: https://community.exoscale.com/product/dbaas/service-specific/kafka/reference/ --- # Reference - Kafka - [API DBaaS](https://community.exoscale.com/reference/api/dbaas/dbaas-kafka/index.md) - All operations on DBaaS. (external) - [CLI DBaaS](https://community.exoscale.com/reference/cli/exo/dbaas/index.md) - All commands on DBaaS. (external)