--- title: "Overview" description: "Run AI models on demand with flexible, pay-per-token OpenAI-compatible endpoints." url: https://community.exoscale.com/product/concrete-ai/on-demand-inference/overview/ product: "On-Demand Inference" section: product last_updated: 2026-10-22 --- > For AI agents: the documentation index is at https://community.exoscale.com/llms.txt. Every page is available as markdown at `index.md` or with `Accept: text/markdown`. # Overview Exoscale On-demand Inference is a managed AI service within the Concrete AI product suite that allows you to access and query AI models through flexible, pay-per-use endpoints on Exoscale’s sovereign GPU infrastructure. > [!IMPORTANT] > **Private Preview** > > On-Demand Inference is currently available as a private preview. Review the > [service boundaries](https://community.exoscale.com/product/concrete-ai/on-demand-inference/service-boundaries/index.md) for current > limitations, availability, pricing, and SLA status. ## Architecture On-Demand Inference exposes public models through a shared, zone-scoped OpenAI-compatible endpoint (`inference..exoscale-cloud.com/v1`). A router in front of the shared inference infrastructure authenticates your AI API key, checks that its scope permits the requested model, and forwards the request to the deployment serving that model. The serving deployments run on GPU infrastructure operated by Exoscale; no compute resources are provisioned in your organization. Token usage is recorded per model and per AI API key, which is what per-token billing is based on. On-demand Inference is designed for applications with variable traffic patterns or teams looking to integrate AI capabilities rapidly without ongoing idle hardware costs. It removes the need for upfront infrastructure provisioning, enabling you to instantly query models through an OpenAI-compatible API and pay strictly for what you consume, such as tokens generated, pages processed, or minutes of audio transcribed. ## Terminology Understanding the key concepts used in On-demand Inference will help you consume models effectively. Public Model : A ready-to-use AI model available on Exoscale that can be queried immediately without uploading custom weights or managing underlying model deployments. Model Category : A classification that groups models by their primary use case. A model can belong to one or more categories. For example, categories could include general-purpose, coding, embedding, or transcription. Model Family : A group of related models that share the same architecture or model lineage. Each model belongs to exactly one family, while a family can contain multiple models or model versions. For example, model families could include Qwen, Mistral, or GLM. Model Capability : A supported input, output, or function. A model can have one or more capabilities, such as text input, image input, text generation, embeddings, or tool calling. The category, family and capability values above are illustrative examples only. They do not represent or guarantee the values currently available in the model catalog. Endpoint : A shared, fully managed inference API endpoint that accepts requests for multiple **public models** and streams or returns the model output on demand. AI API Key : A secure token used to authenticate requests sent to the inference endpoints, possibly restricted to a set of **public models**. ## Model lifecycle The On-demand Inference model catalog evolves as new models and model versions become available. Each model has a lifecycle status so that you can assess whether it is suitable for production and plan migrations when necessary. | Status | Description | What does it mean | | --- | --- | --- | | Preview | A model available for early evaluation and not intended for production use. | No model-specific availability commitment applies. Its capabilities, behavior, pricing, and availability can change. | | Active | A fully supported production model. | The On-demand Inference service SLA applies, and the model is available for new and existing use. | | Deprecated | An active model that is scheduled for retirement and has a published end-of-life (EOL) date. | The model remains operational until its EOL date. Test and migrate to the recommended replacement, when one is available. | | EOL | A retired model that has been removed from the service. | Requests that use the retired model name fail with a `404` error. | New models and model versions are added as separate catalog entries. Adding a new version does not automatically deprecate an existing model, and requests are never automatically redirected to a replacement. When an Active model is deprecated, Exoscale publishes its EOL date at least 30 calendar days in advance and identifies a recommended replacement when one is available. Before the EOL date, update your application to use the replacement model name and test for differences in capabilities, pricing, context limits, and API behavior. Preview models can be changed or retired with a shorter notice period. Notice periods can also be shortened when required by urgent legal, security, licensing, or platform-integrity issues. ## Features OpenAI-Compatible API : Use familiar OpenAI client libraries and request formats with minimal application changes. Model Catalog : Discover available public models and inspect their category, family, capabilities, size, lifecycle status, pricing, and context limits. Scoped AI API Keys : Grant an AI API key access to all public models or restrict it to selected models. Update its scope or revoke the key when access is no longer required. Model Lifecycle Transparency : Review each model's lifecycle status, EOL date, and recommended replacement, when available, before planning and testing a migration. Usage Visibility : View usage and billing statistics across the service, including request and token consumption by model, AI API key, location, and billable token type (input, output, or cache read, where applicable). Cached Tokens : A cache read occurs when a request reuses eligible prompt tokens already stored in the cache. These reused tokens are charged at the lower [cache-read price](https://community.exoscale.com/product/concrete-ai/on-demand-inference/service-boundaries/limits-and-quotas/index.md/index.md#tentative-pricing) instead of the standard input-token price, lowering the cost of repeated context during conversations. Cache-write support, including TTL configuration, is coming soon. Reliability Commitments : Review the current [SLA status](/product/concrete-ai/on-demand-inference/service-boundaries/sla/index.md) before planning production workloads. Shared Inference Infrastructure : Run inference on Exoscale’s shared, high-performance GPU clusters, eliminating hardware setup overhead and minimum usage commitments. Transparent Pricing : Review current and planned rates in [Tentative Pricing](https://community.exoscale.com/product/concrete-ai/on-demand-inference/service-boundaries/limits-and-quotas/index.md). Automatic Scaling : Effortlessly handle incoming request bursts; the shared infrastructure automatically handles scaling capacity to maintain fast response times. Transparency on Location : Review current model and location availability in [Limits and Quotas](/product/concrete-ai/on-demand-inference/service-boundaries/limits-and-quotas/index.md). Sovereign & Secure : All models processing takes place strictly within Swiss & European data centers fully aligned with GDPR and data sovereignty requirements. ## Data privacy and handling Inference Data Retention : Exoscale does not store inference request payloads or responses. It retains only metadata and information required for customer usage reporting. Use of Customer Data : Exoscale does not use your prompts, inputs, or model responses to train, retrain, or fine-tune AI models.