Overview

Exoscale On-demand Inference is a managed AI service within the Concrete AI product suite that allows you to access and query AI models through flexible, pay-per-use endpoints on Exoscale’s sovereign GPU infrastructure.

Architecture

On-Demand Inference exposes public models through a shared, zone-scoped OpenAI-compatible endpoint (inference.<zone>.exoscale-cloud.com/v1). A router in front of the shared inference infrastructure authenticates your AI API key, checks that its scope permits the requested model, and forwards the request to the deployment serving that model.

The serving deployments run on GPU infrastructure operated by Exoscale; no compute resources are provisioned in your organization. Token usage is recorded per model and per AI API key, which is what per-token billing is based on.

On-demand Inference is designed for applications with variable traffic patterns or teams looking to integrate AI capabilities rapidly without ongoing idle hardware costs. It removes the need for upfront infrastructure provisioning, enabling you to instantly query models through an OpenAI-compatible API and pay strictly for what you consume, such as tokens generated, pages processed, or minutes of audio transcribed.

Terminology

Understanding the key concepts used in On-demand Inference will help you consume models effectively.

Public Model

A ready-to-use AI model available on Exoscale that can be queried immediately without uploading custom weights or managing underlying model deployments.

Model Category

A classification that groups models by their primary use case. A model can belong to one or more categories. For example, categories could include general-purpose, coding, embedding, or transcription.

Model Family

A group of related models that share the same architecture or model lineage. Each model belongs to exactly one family, while a family can contain multiple models or model versions. For example, model families could include Qwen, Mistral, or GLM.

Model Capability

A supported input, output, or function. A model can have one or more capabilities, such as text input, image input, text generation, embeddings, or tool calling.

The category, family and capability values above are illustrative examples only. They do not represent or guarantee the values currently available in the model catalog.

Endpoint

A shared, fully managed inference API endpoint that accepts requests for multiple public models and streams or returns the model output on demand.

AI API Key

A secure token used to authenticate requests sent to the inference endpoints, possibly restricted to a set of public models.

Model lifecycle

The On-demand Inference model catalog evolves as new models and model versions become available. Each model has a lifecycle status so that you can assess whether it is suitable for production and plan migrations when necessary.

New models and model versions are added as separate catalog entries. Adding a new version does not automatically deprecate an existing model, and requests are never automatically redirected to a replacement. When an Active model is deprecated, Exoscale publishes its EOL date at least 30 calendar days in advance and identifies a recommended replacement when one is available.

Before the EOL date, update your application to use the replacement model name and test for differences in capabilities, pricing, context limits, and API behavior. Preview models can be changed or retired with a shorter notice period. Notice periods can also be shortened when required by urgent legal, security, licensing, or platform-integrity issues.

Features

OpenAI-Compatible API

Use familiar OpenAI client libraries and request formats with minimal application changes.

Model Catalog

Discover available public models and inspect their category, family, capabilities, size, lifecycle status, pricing, and context limits.

Scoped AI API Keys

Grant an AI API key access to all public models or restrict it to selected models. Update its scope or revoke the key when access is no longer required.

Model Lifecycle Transparency

Review each model’s lifecycle status, EOL date, and recommended replacement, when available, before planning and testing a migration.

Usage Visibility

View usage and billing statistics across the service, including request and token consumption by model, AI API key, location, and billable token type (input, output, or cache read, where applicable).

Cached Tokens

A cache read occurs when a request reuses eligible prompt tokens already stored in the cache. These reused tokens are charged at the lower cache-read price instead of the standard input-token price, lowering the cost of repeated context during conversations. Cache-write support, including TTL configuration, is coming soon.

Reliability Commitments

Review the current SLA status before planning production workloads.

Shared Inference Infrastructure

Run inference on Exoscale’s shared, high-performance GPU clusters, eliminating hardware setup overhead and minimum usage commitments.

Transparent Pricing

Review current and planned rates in Tentative Pricing.

Automatic Scaling

Effortlessly handle incoming request bursts; the shared infrastructure automatically handles scaling capacity to maintain fast response times.

Transparency on Location

Review current model and location availability in Limits and Quotas.

Sovereign & Secure

All models processing takes place strictly within Swiss & European data centers fully aligned with GDPR and data sovereignty requirements.

Data privacy and handling

Inference Data Retention

Exoscale does not store inference request payloads or responses. It retains only metadata and information required for customer usage reporting.

Use of Customer Data

Exoscale does not use your prompts, inputs, or model responses to train, retrain, or fine-tune AI models.