# Exoscale Documentation — Dedicated Inference > Exoscale Dedicated Inference documentation for AI model deployment, scaling, monitoring, performance, and reference guides. --- url: https://community.exoscale.com/product/concrete-ai/dedicated-inference/ --- # Dedicated Inference - [Quick Start - Dedicated Inference](https://community.exoscale.com/product/concrete-ai/dedicated-inference/quick-start/index.md) - Deploy your first Dedicated Inference model and call it through an OpenAI-compatible endpoint. - [Overview - Dedicated Inference](https://community.exoscale.com/product/concrete-ai/dedicated-inference/overview/index.md) - Terminology and key features at a glance, learn the core concepts, capabilities, and options of this service before you dive deeper. - [How-To - Dedicated Inference](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/index.md) - Practical guides for deploying, scaling, troubleshooting and optimizing Dedicated Inference endpoints. - [Service Boundaries - Dedicated Inference](https://community.exoscale.com/product/concrete-ai/dedicated-inference/service-boundaries/index.md) - Service boundaries: quotas, limits, and the guaranteed service levels (SLA) for this product, including key constraints to plan and operate reliably. - [Reference - Dedicated Inference](https://community.exoscale.com/product/concrete-ai/dedicated-inference/reference/index.md) - Reference documentation for API, CLI, and tools, browse comprehensive details, parameters, commands, and integration notes. - [Recipes - Dedicated Inference](https://community.exoscale.com/product/concrete-ai/dedicated-inference/recipes/index.md) - Informational, concise guides showing how to integrate with external tools and solve specific scenarios. --- url: https://community.exoscale.com/product/concrete-ai/dedicated-inference/quick-start/ --- # Quick Start - Dedicated Inference In this quick start guide, you will deploy your first AI model using Exoscale Dedicated Inference and expose it as a production-ready inference endpoint running on dedicated GPUs. This guide uses `mistralai/Ministral-3-8B-Reasoning-2512` on one RTX 6000 Pro GPU. The model is imported from Hugging Face, stored in Exoscale Object Storage, and served through an OpenAI-compatible endpoint. ## Prerequisites You need: * the Exoscale CLI (`exo`) [installed and configured](https://community.exoscale.com/reference/cli/exo/index.md) * an Exoscale IAM API key with access to the `compute` and `ai` services, for the CLI workflow * enough GPU quota ## First Steps via CLI ### Configure CLI Access Dedicated Inference is managed through the Exoscale CLI and API. Create a role with `compute` and `ai` access: ```bash exo iam role create ai-role --policy '{ "default-service-strategy": "deny", "services": { "compute": {"type": "allow", "rules": []}, "ai": {"type": "allow", "rules": []} } }' ``` Create an IAM API key for the role: ```bash exo iam api-key create ai-key ai-role ``` Add the key to your CLI configuration: ```bash exo config add ``` ### Check GPU Availability Check whether your organization can use the RTX 6000 Pro GPU type in `de-fra-1` (besides quota and authorization, the GPU End User Certificate should be signed for the RTX 6000 Pro): ```bash exo dedicated-inference deployment instance-type -z de-fra-1 ``` Look for `gpurtx6000pro` in the output. A GPU type is shown as authorized when your organization can use it and there is enough capacity in the zone. If it is not available, choose another zone or request access or quota through the Exoscale Portal. ### Create the Model Import the model in Exoscale: ```bash exo dedicated-inference model create mistralai/Ministral-3-8B-Reasoning-2512 -z de-fra-1 ``` This downloads the model files from Hugging Face and stores them in Exoscale Object Storage in `de-fra-1`. The deployment will use this stored copy later, so it does not need to fetch the model from Hugging Face at startup. Check the model state: ```bash exo dedicated-inference model list -z de-fra-1 ``` Wait until the model is `created`. ### Create the Deployment Create a deployment on one RTX 6000 Pro GPU: ```bash exo dedicated-inference deployment create ministral-3-8b-reasoning \ --model-name mistralai/Ministral-3-8B-Reasoning-2512 \ --gpu-type gpurtx6000pro \ --gpu-count 1 \ --replicas 1 \ --inference-engine-params '--enable-auto-tool-choice --tool-call-parser=mistral --reasoning-parser=mistral --max-model-len=128000 --language-model-only' \ -z de-fra-1 ``` This model supports a long context window. The example sets `--max-model-len=128000` to reduce memory use while keeping a large context. `--language-model-only` runs the model in text-only mode, which is useful when you do not need image input. The Mistral parser settings enable tool calling and reasoning output for this model family. Check the deployment state: ```bash exo dedicated-inference deployment show ministral-3-8b-reasoning -z de-fra-1 ``` Wait until the deployment is `ready`. ### Get the Endpoint and Create an AI API Key Show the deployment details: ```bash exo dedicated-inference deployment show ministral-3-8b-reasoning -z de-fra-1 ``` Copy the deployment URL. It should include the `/v1` API path. Then create an [AI API Key](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) scoped to this deployment: ```bash exo ai api-key create ministral-client \ --deployment \ --zone de-fra-1 ``` The command returns the plaintext key value once. Store it safely, then export the endpoint and key: ```bash export ENDPOINT_URL="https://.inference.de-fra-1.exoscale-cloud.com/v1" export API_KEY="" ``` For scopes, multiple deployments, rotation, and revocation, see [AI API Keys](https://community.exoscale.com/product/concrete-ai/quick-start/index.md). ### Send a Test Request Dedicated Inference endpoints are OpenAI-compatible. You can call the chat completions route with a standard OpenAI-style request body. ```bash curl -X POST "$ENDPOINT_URL/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $API_KEY" \ -d '{ "model": "mistralai/Ministral-3-8B-Reasoning-2512", "messages": [ { "role": "user", "content": "Write a short poem about Switzerland." } ], "max_tokens": 300 }' ``` If the request returns a completion, your deployment is running and serving requests. ### Clean Up Scale to zero when you want to stop GPU billing but keep the deployment and endpoint URL. [AI API Keys](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) are managed independently and can scope one or more deployments: ```bash exo dedicated-inference deployment scale ministral-3-8b-reasoning 0 -z de-fra-1 ``` Scale back up when needed: ```bash exo dedicated-inference deployment scale ministral-3-8b-reasoning 1 -z de-fra-1 ``` Delete the deployment when you no longer need the endpoint: ```bash exo dedicated-inference deployment delete ministral-3-8b-reasoning -z de-fra-1 ``` Delete the model if you no longer need the stored model files: ```bash exo dedicated-inference model delete -z de-fra-1 ``` Only delete the model when you are sure you will not need it again. If you delete it, you must import it again from Hugging Face before creating a new deployment. ## First Steps via Portal In the [Exoscale Portal](https://portal.exoscale.com/), select **Concrete AI** to create a model, deploy it, and prepare the credentials for inference requests. ### Create a Model Under Concrete AI, under **Models**, click **Add**. Select the target zone, add the Hugging Face model Name and Token (only needed for gated models). Select **Add** again and wait until the model status is **Created** before using it in a deployment. ### Create a Deployment Under **Deployments**, select **Add**. Add the Deployment Name, select the model and the Zone will be authomatically selected to match the zone where the model was deployed. Next, set the GPU type, GPU count, replica count, and any inference engine parametars. Select **Add** and then wait until its status is **Ready**. ### Get the Endpoint Open the ready deployment and copy its inference endpoint URL. ### Create an AI API Key Under **AI API Keys** click **Add**, add the Name of the API keys, select the inference end point and after clicking **Add** again the Secret will be shown. Copy the Secret and please be aware that this value is only available at the moment of creation and will not be retrievable after leaving the page. ## Next Steps * [Use CLI Commands](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/cli-commands/index.md) * [Monitor and Troubleshoot](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/monitor-troubleshoot/index.md) * [AI API Keys](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) --- url: https://community.exoscale.com/product/concrete-ai/dedicated-inference/overview/ --- # Overview - Dedicated Inference Exoscale Dedicated Inference is a managed AI service within the [Concrete AI](https://www.exoscale.com/ai-cloud-infrastructure/) product suite that allows you to deploy and run AI models as production-ready inference endpoints on Exoscale’s dedicated, sovereign GPU infrastructure. It removes the complexity of infrastructure setup, scaling, and GPU management, providing a simple and reliable way to expose AI models through an OpenAI-compatible API. Dedicated Inference is designed for teams that want to move from experimentation to production without building and operating their own AI platform. ## Terminology Understanding the key concepts used in Dedicated Inference will help you deploy models efficiently. Model : A model is an AI artifact (for example from Hugging Face) that is imported into Exoscale and stored securely in Exoscale Object Storage for deployment. Deployment : A deployment represents a running inference endpoint backed by dedicated GPU resources. Each deployment exposes an OpenAI-compatible API endpoint. Replica : A replica is a running copy of a deployment used to scale inference capacity and handle concurrent requests. GPU Count : Defines how many GPUs are assigned to a single model instance, enabling large models to run across multiple GPUs. Dedicated GPU : A physical GPU allocated exclusively to a deployment rather than shared with other customers. This helps keep performance more predictable because a deployment is not competing with unrelated workloads for the same GPU. ## Features Dedicated GPU Performance : Each deployment runs on dedicated NVIDIA GPUs, ensuring consistent performance without resource contention. Bring Your Own Model : Import public, gated, or private Hugging Face models, including models owned by your Hugging Face organization. OpenAI-Compatible API : Integrate deployed models directly into applications using a standard OpenAI-compatible API Flexible Scaling : Scale deployments by changing replicas. Scale to zero to stop GPU billing while keeping the endpoint URL. [AI API Keys](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) are managed independently. The endpoint will not answer requests while scaled to zero, but you can scale it back up later. Sovereign & Secure : All deployments run in European data centers with strong isolation between organizations, supporting GDPR and data sovereignty requirements. Transparent Pricing : Billing is based on per-second GPU usage and standard Object Storage costs, with no token-based pricing. --- url: https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/ --- # How-To - Dedicated Inference Find practical guides for deploying models, scaling inference endpoints, troubleshooting issues and optimizing performance or costs. - [Use CLI Commands](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/cli-commands/index.md) - Manage Dedicated Inference models and deployments with the Exoscale CLI. - [Import Gated Models](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/deploy-gated-model/index.md) - Import gated or private Hugging Face models with a read token. - [Trusted model providers](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/trusted-models/index.md) - Understand trusted model providers, remote code execution, and the process for requesting additional providers. - [Monitor and Troubleshoot](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/monitor-troubleshoot/index.md) - Check deployment state, inspect logs, and diagnose common Dedicated Inference issues. - [Optimize Deployment Costs](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/optimize-costs/index.md) - Control Dedicated Inference costs with right-sized GPUs, replica scaling, scale-to-zero, and cleanup. - [Optimize Performance](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/optimize-performance/index.md) - Improve inference latency and throughput with context tuning, quantization, KV-cache optimization, and speculative decoding. - [Run OCR](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/run-ocr/index.md) - Deploy LightOnOCR on Dedicated Inference and extract text from images with an OpenAI-compatible endpoint. --- url: https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/cli-commands/ --- # Use CLI Commands Use the Exoscale CLI (`exo`) to manage Dedicated Inference from your terminal. Dedicated Inference has two main resources: * A **model** is a model from Hugging Face that Exoscale downloads and stores for later use. Hugging Face is a public model registry where enterprises, AI labs and teams publish AI models. * A **deployment** runs one of those models on dedicated GPUs and exposes it through an endpoint. All Dedicated Inference commands are available under the `exo dedicated-inference` namespace: ```bash exo dedicated-inference --help ``` For the full command reference, see the [Dedicated Inference CLI documentation](https://community.exoscale.com/reference/cli/exo/dedicated-inference/index.md). ## Model Management Create a model before you create a deployment. This gives Exoscale a copy of the model files in the selected zone, so deployments don't need to fetch the model directly from Hugging Face every time. ### Create a Model Register a model from Hugging Face. ```bash exo dedicated-inference model create \ [--huggingface-token ] \ -z ``` Parameters: * ``: Hugging Face model identifier, for example `mistralai/Ministral-3-8B-Reasoning-2512` * `--huggingface-token`: Hugging Face access token. Required for gated models. * `-z, --zone`: Exoscale zone where the model is stored Example with a public model: ```bash exo dedicated-inference model create mistralai/Ministral-3-8B-Reasoning-2512 -z de-fra-1 ``` Example with a gated model: ```bash exo dedicated-inference model create \ --huggingface-token hf_YOUR_TOKEN \ -z de-fra-1 ``` A gated model is a Hugging Face model that requires authentication before you can download it. > [!NOTE] > Model creation can take several minutes. Larger models have more files to download and store. Models are stored in Exoscale Object Storage (SOS), and only models using the `safetensors` format are supported. ### List Models List models available in a zone. ```bash exo dedicated-inference model list -z ``` Example: ```bash exo dedicated-inference model list -z de-fra-1 ``` The output includes the model ID, model name, state, size, creation date and update date. ### Show Model Details Display details for a model. ```bash exo dedicated-inference model show -z ``` Example: ```bash exo dedicated-inference model show 12345678-1234-1234-1234-123456789abc -z de-fra-1 ``` Use the model ID when you want to avoid ambiguity, especially if you have several models with similar names. ### Delete a Model Delete a model from storage. ```bash exo dedicated-inference model delete -z ``` Example: ```bash exo dedicated-inference model delete 12345678-1234-1234-1234-123456789abc -z de-fra-1 ``` > [!WARNING] > Deleting a model is permanent. You can't delete a model while an active deployment is using it. ## Deployment Management A deployment runs a registered model on dedicated GPUs and has its own endpoint URL. Authenticate requests with an [AI API Key](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) scoped to the deployment. ### Create a Deployment Create an inference deployment from a registered model. ```bash exo dedicated-inference deployment create \ --model-name \ --gpu-type \ --gpu-count \ --replicas \ -z ``` Parameters: * ``: Name of the deployment * `--model-name`: Model name as registered from Hugging Face * `--model-id`: Exoscale model ID, as an alternative to `--model-name` * `--gpu-type`: GPU type, for example `gpua5000`, `gpu3` or `gpurtx6000pro` * `--gpu-count`: Number of GPUs per replica * `--replicas`: Number of replicas to create * `--inference-engine-params`: Optional vLLM settings passed to the inference engine, such as context length, memory settings, text-only mode or model-specific parsers * `-z, --zone`: Exoscale zone where the deployment is created Choose `gpu-count` based on model size and memory needs. A larger model, a longer context window or heavier inference settings may require more GPU memory. Use `replicas` when you need more parallel capacity for the same model. Basic deployment: ```bash exo dedicated-inference deployment create ministral-3-8b-reasoning \ --model-name mistralai/Ministral-3-8B-Reasoning-2512 \ --gpu-type gpurtx6000pro \ --gpu-count 1 \ --replicas 1 \ --inference-engine-params '--enable-auto-tool-choice --tool-call-parser=mistral --reasoning-parser=mistral --max-model-len=128000 --language-model-only' \ -z de-fra-1 ``` > [!NOTE] > Total GPU usage is `gpu-count × replicas`. The GPU count can't be changed after deployment creation. To use a different GPU count, create a new deployment. ### Configure Inference Engine Parameters Use `--inference-engine-params` to pass settings to the inference engine. These parameters matter. Many models won't run well with the default settings on every GPU type. Some may fail during startup if the context length, memory usage, multimodal processing or model-specific options don't fit the hardware you selected. Before creating a deployment, check whether the model needs specific inference engine parameters for your GPU type and GPU count. This is especially important for large models, long-context models, quantized models, multimodal models and models that require custom loading options. Examples: * `--max-model-len=128000` limits the maximum context length. Lower values use less memory and can prevent startup failures caused by insufficient GPU memory. * `--language-model-only` disables multimodal inputs and runs a multimodal model in text-only mode. Use it when you don't need image input. * `--gpu-memory-utilization=0.85` controls how much GPU memory the inference engine can use. * `--max-num-seqs=8` limits the number of concurrent sequences handled by one engine instance. * `--tool-call-parser=mistral` enables Mistral tool-call parsing for compatible models. * `--reasoning-parser=mistral` enables Mistral reasoning output parsing for compatible models. * `--speculative-config={JSON}` configures speculative decoding, which can improve generation speed when the model and decoding method are compatible. Dedicated Inference sets tensor parallelism from `--gpu-count`. For example, `--gpu-count 2` allocates two GPUs to each replica and sets tensor parallelism for the vLLM deployment. You don't need to pass `--tensor-parallel-size` in `--inference-engine-params`. To list supported parameters for the selected inference engine version: ```bash exo dedicated-inference deployment create --inference-engine-parameter-help ``` For more information, see: * [vLLM engine arguments](https://docs.vllm.ai/en/latest/serving/engine_args.html) * [vLLM recipes for popular open-weight models](https://docs.vllm.ai/projects/recipes/en/latest/index.html) The vLLM recipes are useful before deployment because they include model-specific runbooks. They often show which vLLM version is needed, which hardware has been tested and which parameters are required for popular open-weight models. Example for `mistralai/Ministral-3-8B-Reasoning-2512` in text-only mode on one RTX 6000 Pro GPU in `de-fra-1`: ```bash exo dedicated-inference deployment create ministral-3-8b-reasoning \ --model-name mistralai/Ministral-3-8B-Reasoning-2512 \ --gpu-type gpurtx6000pro \ --gpu-count 1 \ --replicas 1 \ --inference-engine-params '--enable-auto-tool-choice --tool-call-parser=mistral --reasoning-parser=mistral --max-model-len=128000 --language-model-only' \ -z de-fra-1 ``` This example runs `mistralai/Ministral-3-8B-Reasoning-2512` on one RTX 6000 Pro GPU in `de-fra-1`. The model has a 256K context window, but the example sets `--max-model-len=128000` to reduce memory use. The `--language-model-only` parameter disables multimodal inputs and runs the model in text-only mode, which is useful when you don't need image input. The Mistral parser settings enable tool calling and reasoning output for this model family. ### List Deployments List deployments in a zone. ```bash exo dedicated-inference deployment list -z ``` Example: ```bash exo dedicated-inference deployment list -z de-fra-1 ``` The output includes the deployment name, model, GPU type, GPU count, replica count, state and endpoint URL. ### Show Deployment Details Display details for a deployment. ```bash exo dedicated-inference deployment show -z ``` Example: ```bash exo dedicated-inference deployment show ministral-3-8b-reasoning -z de-fra-1 ``` Use this command to check whether the deployment is ready and to retrieve its endpoint URL, GPU configuration and replica count. ### Scale a Deployment Change the number of replicas for a deployment. ```bash exo dedicated-inference deployment scale -z ``` Parameters: * ``: Name of the deployment to scale * ``: New number of replicas. Use `0` to scale to zero. * `-z, --zone`: Exoscale zone Scale up: ```bash exo dedicated-inference deployment scale ministral-3-8b-reasoning 3 -z de-fra-1 ``` Scale down: ```bash exo dedicated-inference deployment scale ministral-3-8b-reasoning 1 -z de-fra-1 ``` Scale to zero (stop GPU billing): ```bash exo dedicated-inference deployment scale ministral-3-8b-reasoning 0 -z de-fra-1 ``` Scaling changes the number of running model replicas. It doesn't change the model, endpoint URL, GPU type, or GPU count per replica. [AI API Keys](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) are managed independently. > [!NOTE] > Scaling to zero stops GPU billing while keeping the deployment and endpoint URL. [AI API Keys](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) remain independently managed. Scaling back from zero can take several minutes because the model has to be loaded again. ### Create an AI API Key After the deployment is ready, create an [AI API Key](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) scoped to its deployment ID. The key value is returned once and is used in the `Authorization: Bearer ` header. ### View Deployment Logs Retrieve deployment logs for troubleshooting. ```bash exo dedicated-inference deployment logs -z [--tail ] ``` Example: ```bash exo dedicated-inference deployment logs ministral-3-8b-reasoning -z de-fra-1 --tail 100 ``` Logs are useful when a deployment doesn't become ready, when a model fails to load or when inference requests return errors. ### Delete a Deployment Delete a deployment and stop GPU billing. ```bash exo dedicated-inference deployment delete -z ``` Example: ```bash exo dedicated-inference deployment delete ministral-3-8b-reasoning -z de-fra-1 ``` > [!NOTE] > Deleting a deployment stops GPU billing and makes the endpoint inactive. The registered model remains available in storage, so you can create another deployment from it later. ## Check Available Instance Types List GPU types available to your organization in a zone. ```bash exo dedicated-inference deployment instance-type -z ``` Example: ```bash exo dedicated-inference deployment instance-type -z de-fra-1 ``` A GPU type is shown as authorized when your organization is allowed to use it and there is enough capacity in the selected zone. Check instance types before creating a deployment. It helps you avoid failed deployment attempts caused by missing permissions, quota limits or lack of available GPU capacity. If you lack a sufficient quota or the desired GPU type is not available in your preferred zone, open a support ticket so options for making that GPU available in that zone can be discussed and considered. ## Common Workflows ### Deploy a Model End to End This example deploys `mistralai/Ministral-3-8B-Reasoning-2512`, a reasoning model with a long context window and [Mistral-specific](https://recipes.vllm.ai/mistralai/Ministral-3-8B-Reasoning-2512?features=tool_calling%2Creasoning%2Cencoder_parallel) inference settings. ```bash # 1. Register the model exo dedicated-inference model create mistralai/Ministral-3-8B-Reasoning-2512 -z de-fra-1 # 2. Check model state exo dedicated-inference model list -z de-fra-1 # 3. Create a deployment exo dedicated-inference deployment create ministral-3-8b-reasoning \ --model-name mistralai/Ministral-3-8B-Reasoning-2512 \ --gpu-type gpurtx6000pro \ --gpu-count 1 \ --replicas 1 \ --inference-engine-params '--enable-auto-tool-choice --tool-call-parser=mistral --reasoning-parser=mistral --max-model-len=128000 --language-model-only' \ -z de-fra-1 # 4. Check deployment state and endpoint URL exo dedicated-inference deployment show ministral-3-8b-reasoning -z de-fra-1 # 5. Create an AI API Key scoped to the deployment ID exo ai api-key create ministral-client --deployment -z de-fra-1 ``` This model has a 256K context window. The example sets `--max-model-len=128000` to reduce memory usage on the selected GPU. It also passes text-only mode, tool-calling and reasoning parser settings used by this model family. ### Manage Costs ```bash # Scale down when demand is low exo dedicated-inference deployment scale ministral-3-8b-reasoning 1 -z de-fra-1 # Scale to zero to stop GPU billing while keeping the deployment exo dedicated-inference deployment scale ministral-3-8b-reasoning 0 -z de-fra-1 # Scale back up when needed exo dedicated-inference deployment scale ministral-3-8b-reasoning 1 -z de-fra-1 # Delete the deployment when it is no longer needed exo dedicated-inference deployment delete ministral-3-8b-reasoning -z de-fra-1 # Delete the model if it is no longer used exo dedicated-inference model delete -z de-fra-1 ``` Scaling to zero is usually the right choice for temporary pauses, such as nights, weekends, demos or test environments. Delete the deployment when you no longer need the endpoint. ## Troubleshooting For deployment states, logs, model loading issues, GPU memory errors and inference request failures, see [Monitor and troubleshoot deployments](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/monitor-troubleshoot/index.md). ## Zone Availability GPU availability depends on the zone and GPU type. See [GPU availability by zone](https://www.exoscale.com/gpu/#comparison-gpu) for the current GPU-by-zone matrix. ## Next Steps * [Monitor and Troubleshoot](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/monitor-troubleshoot/index.md) * [Optimize Performance](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/optimize-performance/index.md) * [Optimize Deployment Costs](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/optimize-costs/index.md) --- url: https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/deploy-gated-model/ --- # Import Gated Models Use this guide when a Hugging Face model can't be downloaded anonymously. This includes: * gated public models that require license acceptance * private models in your Hugging Face account * models owned by your Hugging Face organization (like internal fine-tuned models stored in a private Hugging Face repository) A gated model is a model whose files require approval and/or authentication before download. For Dedicated Inference, the import process is the same for gated, private, and organization-owned models: use a Hugging Face token from a user that can read the repository. Dedicated Inference needs the token when creating the model in Exoscale. After the model is created, Exoscale uses the stored copy of the model files. ## Before You Start You need: * access to the model repository on Hugging Face * a Hugging Face token with read access to the model repository on Hugging Face * the Exoscale CLI configured with Dedicated Inference permissions If you don't already have a Hugging Face token for the gated or private model repository, you may need to create a Hugging Face account. Sign up at [https://huggingface.co](https://huggingface.co). If the model requires license acceptance, accept the terms on Hugging Face before importing it into Dedicated Inference. ### Create a Hugging Face Token Refer to the [Hugging Face documentation](https://huggingface.co/docs/hub/en/security-tokens) for the most up-to-date steps. Use one of these: * a read token * a fine-grained token scoped to the model repository with read access Store the token in an environment variable: ```bash export HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" ``` ## Create the Model in Exoscale Run: ```bash exo dedicated-inference model create / \ --huggingface-token "$HF_TOKEN" \ -z de-fra-1 ``` `` is the Hugging Face user or organization that owns the model. The model ID must match the Hugging Face repository name exactly, including capitalization. During creation, Exoscale downloads the model files and stores them in the selected zone. Larger models take longer because there is more data to fetch and store. > [!NOTE] > Dedicated Inference supports model weights in the `safetensors` format. Model creation can fail if the repository doesn't include supported weights. ## Check the Import List models in the zone: ```bash exo dedicated-inference model list -z de-fra-1 ``` Wait until the model reaches `created`. If creation fails, check these first: * the model ID is exact * the token can read the repository * the Hugging Face user accepted the model terms * the model files use a supported format ## License Responsibility Exoscale downloads the model files you request. You remain responsible for following the model license and any terms set by the model owner. ## Next Steps * [Use CLI Commands](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/cli-commands/index.md) * [Monitor and Troubleshoot](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/monitor-troubleshoot/index.md) * [Optimize Deployment Costs](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/optimize-costs/index.md) --- url: https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/trusted-models/ --- # Trusted model providers Some Hugging Face models require custom code provided by the model author in order to run correctly. These models require the `trust-remote-code` option during deployment. For security and platform integrity reasons, Exoscale only allows remote code execution for a predefined list of trusted model providers. ## Why is `trust-remote-code` restricted? When enabled, `trust-remote-code` allows model-specific Python code from a Hugging Face repository to be executed as part of the model loading process. While many models use this mechanism legitimately, it also introduces additional security and operational considerations. To ensure a secure and reliable platform for all customers, Dedicated Inference only permits remote code execution for reviewed and approved model providers. ## Trusted Model Providers The following Hugging Face organizations are currently approved for use with `trust-remote-code`: - `baidu` - `deepseek-ai` - `huggingface` - `ibm-granite` - `meta-llama` - `minimaxai` - `mistralai` - `moonshotai` - `microsoft` - `nvidia` - `openai` - `poolside` - `qwen` - `sentence-transformers` - `stabilityai` - `unsloth` - `xai-org` - `xiaomimimo` - `zai-org` > This list may evolve over time as additional providers are reviewed and approved. ## Requesting Additional Providers If your model requires `trust-remote-code` and its provider is not currently approved, you can request a review by opening a support ticket in the Exoscale Portal. Please include: - The Hugging Face model URL - A brief description of your use case - Any known runtime requirements or custom dependencies Our team will review the request and determine whether the provider can be added to the trusted-provider allowlist. > Approval is not guaranteed and depends on security, licensing, operational, and compatibility considerations. ## Additional Runtime Requirements Some models require more than the `trust-remote-code` option. Certain OCR, multimodal, and research-focused models depend on additional Python packages, custom decoding logic, custom logits processors, or other runtime components that are not available in the standard Dedicated Inference environment. As a result, a model may remain unsupported even if its provider is approved for remote code execution. ## Troubleshooting If you receive an error similar to: ```text Parameter 'trust-remote-code' is not permitted for model '' ``` the model provider is not currently on the trusted-provider allowlist. To request a review of the provider, follow the process described in the [Requesting Additional Providers](#requesting-additional-providers) section. ## Next Steps * [Use CLI Commands](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/cli-commands/index.md) * [Monitor and Troubleshoot](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/monitor-troubleshoot/index.md) * [Import Gated Models](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/deploy-gated-model/index.md) --- url: https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/monitor-troubleshoot/ --- # Monitor and Troubleshoot Use this guide when a deployment doesn't become `ready`, requests fail, or you need logs for support. A deployment starts from model files stored in zone-local Object Storage. At startup, the files are copied into the runtime, loaded by vLLM, and exposed through the deployment URL. The deployment state shows the phase. Logs show the cause. Set these variables for the examples: ```bash export ZONE="de-fra-1" export DEPLOYMENT="ministral-3-8b-reasoning" ``` ## Check State List deployments: ```bash exo dedicated-inference deployment list -z "$ZONE" ``` Show one deployment: ```bash exo dedicated-inference deployment show "$DEPLOYMENT" -z "$ZONE" ``` Use `show` to check the state, endpoint URL, model, GPU type, GPU count, replicas, and state details. | State | Meaning | What to do | | --- | --- | --- | | `creating` | The request was accepted | Wait, then check again | | `preparing` | Runtime resources are being prepared | Check logs if it stays there | | `deploying` | vLLM is starting and loading the model | Check logs if it takes too long | | `ready` | The endpoint can accept requests | Test the client request | | `scaling` | Replicas are changing | Wait for scaling to finish | | `updating` | Deployment settings are being applied | Wait, then check state again | | `error` | The deployment failed | Check state details and logs | Large models can spend more time in `deploying`, `updating`, or `scaling` when runtime replicas start or restart. This is expected while model files are copied and loaded into GPU memory. ## View Logs Show recent logs first: ```bash exo dedicated-inference deployment logs "$DEPLOYMENT" -z "$ZONE" ``` Fetch more logs when needed: ```bash exo dedicated-inference deployment logs "$DEPLOYMENT" -z "$ZONE" --tail 500 ``` Look for: * `CUDA out of memory` * Python tracebacks * invalid vLLM parameters * model loading messages Progress messages usually mean the deployment is still starting. Tracebacks, memory errors, and invalid parameter messages usually need vLLM inference engine parameter changes. ## Startup Issues | Symptom | Likely cause | What to do | | --- | --- | --- | | Stuck in `preparing` | Runtime setup, quota, capacity, or GPU authorization issue | Check state details and available instance types | | Stuck in `deploying` with progress logs | Large model or slow model load | Wait, then check logs again | | `CUDA out of memory` | Model, context length, or KV-cache doesn't fit | Tune your vLLM engine parameters to fit the available CUDA memory, or use more GPUs or a GPU class with more VRAM | | Invalid vLLM parameter | Parameter isn't allowed for the selected engine version | Check parameter help and remove it | | Quota or capacity error | Not enough GPU quota or capacity in the zone | Check quota and GPU availability | | `error` without a clear log line | The failure happened before or outside vLLM startup | Collect state details and logs for support | If you lack a sufficient quota or the desired GPU type is not available in your preferred zone, open a support ticket so options for making that GPU available in that zone can be discussed and considered. Check available GPU types and authorization: ```bash exo dedicated-inference deployment instance-type -z "$ZONE" ``` Check allowed inference engine parameters for the default vLLM engine version: ```bash exo dedicated-inference deployment create --inference-engine-parameter-help ``` ## GPU Memory Errors A deployment can run out of memory even when the model weights fit. Long context windows also reserve memory for the KV-cache. That memory can be large. Try a lower context length: ```bash --inference-engine-params '--max-model-len=32768' ``` If supported in the context you're in, a lower-precision KV-cache can also reduce memory use: ```bash --inference-engine-params '--kv-cache-dtype=fp8' ``` If the model still doesn't fit, create a new deployment with more GPUs or a GPU type with more memory. `--gpu-count` and GPU type are fixed after creation. Scaling replicas helps concurrency, but it doesn't give one request more GPU memory. ## If the Model Is Not Created A deployment needs a model in the `created` state. Check models in the zone: ```bash exo dedicated-inference model list -z "$ZONE" ``` If the model is still `creating`, `downloading`, or `error`, fix the model import first. Common causes are a wrong Hugging Face model ID, missing access to a gated model, or unsupported model files. For gated or private models, see [Import Gated Models](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/deploy-gated-model/index.md). ## Request Errors If the deployment is `ready` but requests fail, check the client setup. | Error | Likely cause | What to do | | --- | --- | --- | | `401 Unauthorized` | Unknown or revoked [AI API Key](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) | Create a new key or update the client with its value | | `403 Forbidden` | The [AI API Key](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) does not cover this deployment | Update the key scope to include the deployment | | `404 Not Found` | Wrong endpoint URL or missing `/v1` path | Check the URL with `deployment show` | | `400 Bad Request` | Invalid request body | Check the JSON payload and model name | | `429 Too Many Requests` | More traffic than the deployment can handle | Reduce traffic or add replicas | | `500` | Runtime error during inference | Check state and logs | Create or update an [AI API Key](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) whose deployment scope includes this deployment. Use its value as a bearer token: ```bash Authorization: Bearer ``` Dedicated Inference endpoints are OpenAI-compatible. Use the deployment URL as the base URL and include the `/v1` path. ## Slow Responses Slow responses usually come from long prompts, high `max_tokens`, too much traffic, or limited GPU memory headroom. Add replicas when you need to handle more concurrent requests: ```bash exo dedicated-inference deployment scale "$DEPLOYMENT" 3 -z "$ZONE" ``` Replicas add more running copies of the same deployment. They help with concurrency. They don't make one heavy request faster. ## Collect Details for Support Before opening a ticket, collect state and logs: ```bash exo dedicated-inference deployment show "$DEPLOYMENT" -z "$ZONE" > deployment-info.txt exo dedicated-inference deployment logs "$DEPLOYMENT" -z "$ZONE" --tail 500 > deployment-logs.txt ``` Don't include AI API Keys, Hugging Face tokens, or other secrets. Open a support ticket through the [Exoscale Portal](https://portal.exoscale.com). ## Next Steps * [Use CLI Commands](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/cli-commands/index.md) * [Optimize Performance](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/optimize-performance/index.md) * [Optimize Deployment Costs](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/optimize-costs/index.md) * [AI API Keys](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) --- url: https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/optimize-costs/ --- # Optimize Deployment Costs Dedicated Inference cost comes mostly from running GPUs. The goal is simple: run the GPU capacity you need, only while you need it. This guide covers cost decisions. For command syntax, see [Use CLI Commands](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/cli-commands/index.md). ## What You Pay For There are two cost sources: * **GPU compute** for running replicas. This is the main driver. * **Object Storage** for model files kept after model creation. Smaller, but it adds up if you keep many large models. GPU usage for one deployment is: ```text Total GPUs = gpu-count × replicas ``` So `--gpu-count 2 --replicas 3` uses 6 GPUs. The same deployment at `--replicas 0` uses none. Object Storage cost stays even at zero replicas, because the model files remain stored. ## Right-Size the GPU Setup Start with the smallest GPU type and `--gpu-count` that runs your model with the context length and parameters your workload actually needs. This is where deployments get expensive too fast. Model size alone doesn't decide GPU memory. The context window, KV-cache, and precision all add to it. A model that looks small can still need a large GPU if it defaults to a very long context window. Before reaching for a bigger GPU, try trimming memory use. A lower `--max-model-len` shrinks the KV-cache, because it no longer reserves room for the full context window: ```bash --inference-engine-params '--max-model-len=32000' ``` `--gpu-count` is primarily for model fit. Use more than one GPU per replica only when the model plus your context settings don't fit on one. You can't change `--gpu-count` after creation, so pick it deliberately. Too low fails at startup or leaves no memory headroom. ```bash exo dedicated-inference deployment create "$DEPLOYMENT" \ --model-name "$MODEL" \ --gpu-type gpu3 \ --gpu-count 2 \ --replicas 1 \ --inference-engine-params '--max-model-len=32000' \ -z "$ZONE" ``` ## Scale Replicas for Traffic `--replicas` is the traffic lever. Each replica is another full copy of the GPU allocation, so a deployment created with `--gpu-count 2` uses 2 GPUs at 1 replica and 6 GPUs at 3 replicas. ```bash exo dedicated-inference deployment scale "$DEPLOYMENT" 3 -z "$ZONE" ``` ## Scale to Zero When Idle Scaling to zero stops GPU billing while keeping the deployment and its endpoint URL. [AI API Keys](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) are managed independently and can scope one or more deployments. ```bash exo dedicated-inference deployment scale "$DEPLOYMENT" 0 -z "$ZONE" ``` Bring it back when needed: ```bash exo dedicated-inference deployment scale "$DEPLOYMENT" 1 -z "$ZONE" ``` This fits development outside work hours, demos, paused projects, and scheduled idle periods. Coming back from zero takes a few minutes while the model reloads, so use it for production only when downtime is acceptable. ## Delete What You Won't Reuse Scale to zero is a pause. Delete is cleanup. Choose based on whether you still want that endpoint. Review the scopes of any [AI API Keys](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) that included a deleted deployment. Delete a deployment you no longer need: ```bash exo dedicated-inference deployment delete "$DEPLOYMENT" -z "$ZONE" ``` This stops GPU billing and removes the endpoint. The registered model stays stored, so you can deploy it again later. Model files keep costing storage on their own. List and delete models you no longer use: ```bash exo dedicated-inference model list -z "$ZONE" exo dedicated-inference model delete -z "$ZONE" ``` You can't delete a model while a deployment still uses it. Delete the deployment first. ## Reuse One Model Across Deployments Create a model once, then deploy it more than once, for example one deployment for testing and one for production: ```bash exo dedicated-inference model create "$MODEL" -z "$ZONE" ``` This avoids storing duplicate copies and keeps your workflow clean, since you can change or delete deployments without re-importing the model. ## Estimate Before You Commit Use the [Exoscale Advanced Calculator](https://www.exoscale.com/pricing/#calculator) before creating long-running deployments. ```text Monthly GPU hours = gpu-count × replicas × running hours ``` ## Monitor Costs Review billing and usage in the [Exoscale Portal](https://portal.exoscale.com): GPU compute, Object Storage for model files, running deployments, and unused models. Make it a habit to check development and test deployments weekly. Idle GPUs are easy to forget after experiments, demos, and benchmark runs. ## Cost Checklist * Use the smallest GPU setup that fits the model and context length and fits your performance needs. * Reduce context length when the full window isn't needed. * Use `gpu-count` for model fit, replicas for traffic. * Scale to zero when a deployment is idle. * Delete deployments and models you won't reuse. * Reuse one registered model across deployments. * Review costs in the Portal regularly. ## Next Steps * [Use CLI Commands](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/cli-commands/index.md) * [Optimize Performance](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/optimize-performance/index.md) * [Monitor and Troubleshoot](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/monitor-troubleshoot/index.md) --- url: https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/optimize-performance/ --- # Optimize Performance Use this guide when a deployment is too slow or can't handle enough concurrent requests. ## How the GPU spends its time Every inference request runs in two phases. They stress the hardware in opposite ways, and almost every tuning decision maps back to one of them. **Prefill** is when the model reads your prompt. All input tokens are processed in one large matrix multiplication, so the GPU's compute units are busy. This phase is FLOPS-bound. The time it takes is what users experience as Time to First Token (TTFT). **Decode** is when the model generates the reply, one token at a time. To produce each token, the GPU reads the full set of model weights from memory, does a small calculation, then waits for the next memory read. The math is fast. The memory transfer isn't. Decode is memory-bandwidth-bound, and its speed is what users experience as Inter-Token Latency (ITL). That split matters because the fixes are different: | Symptom | Bottleneck | Fix | | :--- | :--- | :--- | | First token arrives slowly | FLOPS | Reduce prompt length, use prefix caching, choose a higher-compute GPU class | | Tokens stream slowly | Memory bandwidth | Quantize weights and KV-cache, pool bandwidth across more GPUs | | Deployment crashes at startup | VRAM capacity | Reduce context length, quantize, or increase GPU count | ## Context length is a strong lever The KV-cache stores attention states for every token in the active context window. It lives in VRAM alongside the model weights, and it grows with every concurrent request. A model that looks small on a spec sheet can exhaust memory once you account for the cache. For a 70B model in FP16, each request at 4K context uses roughly 1.25 GB of KV-cache. At 128K context, that's 40 GB per request. The model weights don't change. If your workload doesn't need the model's full context window, cap it: ```bash exo dedicated-inference deployment create "$DEPLOYMENT" \ --model-name "$MODEL" \ --gpu-type gpu3 \ --gpu-count 2 \ --replicas 1 \ --inference-engine-params '--max-model-len=32768' \ -z "$ZONE" ``` ## Quantization Since decode is bottlenecked by memory reads, storing weights and KV-cache values in a smaller format directly translates to faster generation. To quantize the KV-cache at deployment time: ```bash --inference-engine-params '--kv-cache-dtype=fp8' ``` For weight quantization, if the model has already been published in AWQ or GPTQ format, you don't need to pass any flag because vLLM can detect it automatically. If you want to request quantization at runtime: ```bash --inference-engine-params '--quantization=awq' ``` ## Multi-GPU considerations Raising `--gpu-count` does two things: it splits model weights across cards so larger models fit, and it pools memory bandwidth across all cards. The default strategy is Tensor Parallelism. Each layer's weight matrices get sliced across all GPUs. Every card processes every token, but only its slice of each layer. After each layer, all cards synchronize to merge their partial results before moving to the next layer. That synchronization is a bottleneck to consider. On NVLink-connected GPUs, each synchronization runs at around 900 GB/s and costs single-digit milliseconds per pass. Over a PCIe connection, those same synchronizations can add more latency than the compute itself. Adding more GPUs can make things slower. A strategy worth knowing is for MoE models specifically is Expert Parallelism (`--enable-expert-parallel`). MoE models contain dozens to hundreds of expert networks but only a subset of them fire per token. Rather than replicating all experts on every GPU, EP assigns whole experts to specific GPUs. GPU 0 owns experts 0-15, GPU 1 owns experts 16-31, and so on. When a token is routed to expert 20, it gets dispatched to GPU 1 and the result is returned. Each card only stores and computes its own experts which cuts memory pressure and keeps matrix operations at a shape that hardware executes efficiently since it minimizes GPU-to-GPU communication. ## Replicas When the model fits and generates correctly but requests are queueing, add replicas: ```bash exo dedicated-inference deployment scale "$DEPLOYMENT" 3 -z "$ZONE" ``` Replicas are fully independent. No synchronization, no routing, no communication overhead between them. Just more copies of the same deployment handling separate requests in parallel. ```bash exo dedicated-inference deployment scale "$DEPLOYMENT" 3 -z "$ZONE" ``` ## Prefix caching If your application sends the same system prompt with every request, which most chatbots and RAG pipelines do, the model recomputes the KV-cache for that prompt from scratch every single time. Prefix caching fixes that. On the first request, vLLM computes the KV-cache for the shared prefix and stores it. Every subsequent request with the same prefix reuses those cached blocks, skipping prefill entirely for the shared portion. For lengthy prompts at scale, that's a significant reduction in both TTFT and GPU time per request. Enable it with: ```bash --inference-engine-params '--enable-prefix-caching' ``` ## Speculative decoding The GPU's compute units sit mostly idle during decode because the workload is memory-bound. Speculative decoding puts that spare compute to use. A small draft model generates several tokens ahead. The main model validates all of them in a single forward pass. Validation uses the same memory bandwidth as generating one token so accepted draft tokens are nearly free. ```bash --inference-engine-params '--speculative-config={"method":"mtp","num_speculative_tokens":3}' ``` The speedup depends on how often the draft guesses correctly. High-temperature sampling, diverse topics, and large batch sizes all reduce the acceptance rate and shrink the benefit. ## Measure before and after You can use a tool like [AIPerf](https://github.com/withcatai/aiperf) to benchmark your dedicated inference endpoints. It drives concurrent load against any OpenAI-compatible endpoint and reports TTFT, ITL, and throughput metrics that map directly to the bottlenecks covered in this guide. Use an [AI API Key](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) scoped to the deployment. ```bash aiperf profile \ --url "https://" \ --endpoint "/v1/chat/completions" \ --endpoint-type chat \ --model "" \ --header "Authorization:Bearer " \ --tokenizer "" \ --tokenizer-trust-remote-code \ --streaming \ --concurrency 10 \ --request-count 100 ``` Flags and default values change between AIPerf releases. Read the [AIPerf documentation](https://github.com/withcatai/aiperf) for the current recommended way to profile endpoints before running any benchmarks. Read the results against what you now know about the two phases: - **TTFT high and rising with concurrency** (prefill-bound). More concurrent requests means more prefill work competing for FLOPS. Trim prompts to reduce compute per request, enable prefix caching to skip recomputing shared prompt prefixes, or move to a higher-compute GPU class. - **ITL high regardless of concurrency** (memory-bandwidth-bound). Every decode step loads the full model weights from memory to generate a single token. Even one request hits this ceiling. Quantization shrinks weight size so each step reads less data. A GPU with higher memory bandwidth completes each read faster. - **Throughput lower than expected** (VRAM capacity-bound or GPU-to-GPU communication-bound). If the KV cache is full, vLLM can't grow the batch further. Quantize model weights to free VRAM for the cache, or quantize the KV cache itself so each entry takes less room. Reduce context length to shrink each request's cache footprint. If adding GPUs didn't help or made things worse, the bottleneck may be synchronization overhead rather than capacity: each extra card adds an all-reduce per forward pass, and over PCIe that cost compounds fast. On MoE models,consider Expert Parallelism to spreads expert weights across cards. ## Check logs after any change After updating parameters, check that the deployment came back up: ```bash exo dedicated-inference deployment show "$DEPLOYMENT" -z "$ZONE" ``` If it doesn't reach `ready`, read the logs: ```bash exo dedicated-inference deployment logs "$DEPLOYMENT" -z "$ZONE" --tail 100 ``` Look for invalid parameter errors, `CUDA out of memory`, model loading failures, and vLLM tracebacks. A deployment stuck in `deploying` might be loading a large model slowly, or it might have crashed during engine startup. The logs are the only way to tell the difference. For more on diagnosing failed deployments, see [Monitor and Troubleshoot](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/monitor-troubleshoot/index.md). ## Next steps - [Use CLI Commands](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/cli-commands/index.md) - [Monitor and Troubleshoot](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/monitor-troubleshoot/index.md) - [Optimize Deployment Costs](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/optimize-costs/index.md) --- url: https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/run-ocr/ --- # Run OCR This guide deploys [LightOnOCR-2-1B](https://huggingface.co/lightonai/LightOnOCR-2-1B) on Dedicated Inference and uses it to pull text out of images: invoices, receipts, forms, and scanned documents. LightOnOCR-2-1B is a small multilingual vision-language model built for OCR. It covers 11 languages and handles tables, multi-column layouts, and math notation. ## Prerequisites * Exoscale CLI (`exo`) [installed and configured](https://community.exoscale.com/reference/cli/exo/index.md) * an IAM API key with `compute` and `ai` access * enough GPU quota in your organization Set these variables for the examples: ```bash export ZONE="at-vie-2" export MODEL="lightonai/LightOnOCR-2-1B" ``` ## Step 1. Create the Model Register the model in your zone: ```bash exo dedicated-inference model create "$MODEL" -z "$ZONE" ``` Wait until it shows `created`: ```bash exo dedicated-inference model list -z "$ZONE" ``` ## Step 2. Deploy the Model LightOnOCR-2-1B is a 1B-parameter model, about 1.9 GiB, so one A5000 GPU is plenty. ```bash exo dedicated-inference deployment create ocr \ --model-name "$MODEL" \ --gpu-type gpua5000 \ --gpu-count 1 \ --replicas 1 \ --inference-engine-version 0.15.1 \ -z "$ZONE" ``` Wait until the deployment is `ready`, usually 2 to 3 minutes: ```bash exo dedicated-inference deployment show ocr -z "$ZONE" ``` ## Step 3. Get the Endpoint and Create an AI API Key ```bash exo dedicated-inference deployment show ocr -z "$ZONE" exo ai api-key create ocr-client --deployment -z "$ZONE" ``` The [AI API Key](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) value is returned once. Store it safely, then export both values for the next steps: ```bash export ENDPOINT_URL="https://.inference.at-vie-2.exoscale-cloud.com/v1" export API_KEY="" ``` ## Step 4. Run OCR on an Image The model uses the OpenAI-compatible chat completions API. Pass the image as a base64 data URI. You don't need a text prompt. The model extracts all text from the image on its own. Encode the image and build the request file: ```bash BASE64_IMG=$(base64 -i document.png) cat > request.json << EOF { "model": "lightonai/LightOnOCR-2-1B", "messages": [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "data:image/png;base64,${BASE64_IMG}" } } ] } ], "max_tokens": 4096, "temperature": 0.2, "top_p": 0.9 } EOF ``` Pass the body as a file with `-d @request.json`. Base64 image data inline tends to break shell escaping. ```bash curl -X POST "$ENDPOINT_URL/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $API_KEY" \ -d @request.json ``` For an invoice, the model returns the text with tables as HTML: ```json { "choices": [ { "message": { "content": "INVOICE #12345\nDate: 2026-03-26\n\n\n \n \n
ItemQtyPrice
Widget A219.99
\n\nTotal: 272.69" } } ] } ``` ## Get Better Results Image quality drives accuracy more than anything else: * Render PDFs at 200 DPI. * Aim for about 1540 pixels on the longest side. * Keep the original aspect ratio so text geometry stays intact. * PNG and JPEG both work. Use JPEG to keep large documents smaller. Supported languages: English, French, German, Spanish, Italian, Portuguese, Dutch, Russian, Chinese, Japanese, and Korean. ## Clean Up Scale to zero to stop GPU billing but keep the endpoint and key: ```bash exo dedicated-inference deployment scale ocr 0 -z "$ZONE" ``` Delete the deployment when you're done with the endpoint: ```bash exo dedicated-inference deployment delete ocr -z "$ZONE" ``` Then remove the model files from Object Storage if you no longer need them: ```bash exo dedicated-inference model delete -z "$ZONE" ``` ## Next Steps * [Use CLI Commands](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/cli-commands/index.md) * [Monitor and Troubleshoot](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/monitor-troubleshoot/index.md) * [Optimize Costs](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/optimize-costs/index.md) * [AI API Keys](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) --- url: https://community.exoscale.com/product/concrete-ai/dedicated-inference/service-boundaries/ --- # Service Boundaries - Dedicated Inference - [Limits and Quotas - Dedicated Inference](https://community.exoscale.com/product/concrete-ai/dedicated-inference/service-boundaries/limits-and-quotas/index.md) - Dedicated Inference Limitations and Quotas - [SLA - Dedicated Inference](https://community.exoscale.com/product/concrete-ai/dedicated-inference/service-boundaries/sla/index.md) - Dedicated Inference Service Level Agreement - [Pricing ↗︎](https://www.exoscale.com/pricing/#inference) --- url: https://community.exoscale.com/product/concrete-ai/dedicated-inference/service-boundaries/limits-and-quotas/ --- # Limits and Quotas - Dedicated Inference ## Limits Limits depend on the GPU type you want to use for your specific deployment. Some GPU categories require additional compliance requirements, including the [GPU End-User Certificate](https://portal.exoscale.com/organization/legal/terms). ## Quotas GPU quotas are per-organization limits on the number of GPU instances you can run concurrently. You can view your current quotas from the [Exoscale portal](https://portal.exoscale.com/organization/quotas). To request an increase, please contact support. ## Checking for GPU capacity and authorizations On top of having GPU quota: - You need to sign the GPU [End User Certificate](https://www.exoscale.com/end-user-certificate-gpu/) to be authorized to deploy models on RTX Pro 6000 GPUs - Dedicated Inference may be capacity-constrained on certain offerings You can use the `exo dedicated-inference deployment instance-type` command to check which GPUs you may deploy in any given zone: ``` ┼───────────────┼────────────┼──────────┼ │ FAMILY │ AUTHORIZED │ ZONE │ ┼───────────────┼────────────┼──────────┼ │ gpu3080ti │ true │ at-vie-2 │ │ gpua5000 │ true │ at-vie-2 │ │ gpurtx6000pro │ false │ ch-dk-2 │ │ gpu3 │ true │ de-fra-1 │ │ gpurtx6000pro │ true │ de-fra-1 │ │ gpurtx6000pro │ true │ hr-zag-1 │ ┼───────────────┼────────────┼──────────┼ ``` ## Additional Constraints Safetensors Model File Format : Model weights must be in the `safetensors` format. GGUF and other formats are not supported. Customer-managed sizing : Picking a GPU type and count is model-dependent and use-case dependent. As such, it is up to you to size your inference deployments. GPU Count Immutability : The `--gpu-count` parameter cannot be changed after deployment. To use a different GPU count, create a new deployment. Additional Runtime Dependencies : Models requiring additional Python packages, custom decoding logic, custom logits processors, or other runtime dependencies beyond the standard inference runtime are not currently supported. As a result, some models may remain unsupported even if their provider is approved for `trust-remote-code`. Check [Trusted Model Providers](https://community.exoscale.com/product/concrete-ai/dedicated-inference/how-to/trusted-models/index.md) for information about trusted model providers, remote code execution, and the provider review process. ## Availability GPU availability varies by zone and GPU type. See [GPU availability by zone](https://www.exoscale.com/gpu/#comparison-gpu) for the current GPU-by-zone matrix. | Zone | Country | City | Availability | | :--- | :--- | :--- | :---: | | __`at-vie-1`__ | Austria | Vienna | no | | __`at-vie-2`__ | Austria | Vienna | yes | | __`bg-sof-1`__ | Bulgaria | Sofia | no | | __`ch-dk-2`__ | Switzerland | Zurich | yes | | __`ch-gva-2`__ | Switzerland | Geneva | no | | __`de-fra-1`__ | Germany | Frankfurt | yes | | __`de-muc-1`__ | Germany | Munich | no | | __`hr-zag-1`__ | Croatia | Zagreb | yes | --- url: https://community.exoscale.com/product/concrete-ai/dedicated-inference/service-boundaries/sla/ --- # SLA - Dedicated Inference These product-specific SLAs define the Service Availability Target for Exoscale Dedicated Inference. It applies to each Client using the Service. Capitalized terms used herein but not defined herein shall have the meanings set forth in the Exoscale Terms and Conditions or EUSA, whichever applies. ## SLA Definitions | Metric | Target | Definition | | -------------------------------------------- | -------- | ------------ | | Dedicated Inference Endpoint Availability | >= 99.95% Monthly Uptime Percentage | The percentage of eligible requests to a Dedicated Inference endpoint that are served without an Exoscale-attributable server-side failure, calculated per Client and per Dedicated Inference endpoint during a calendar month. Availability is calculated as `(total eligible requests - failed eligible requests) / total eligible requests * 100`. Failed eligible requests are requests to the Client's Dedicated Inference endpoint that return an HTTP `5xx` response attributable to the Dedicated Inference service. | Only eligible requests are included in the Monthly Availability Percentage. The applicable exclusions are listed below. ## Product Specifications For detailed product features, technical specifications, and tutorials, please refer to the [Exoscale documentation](https://community.exoscale.com/product/concrete-ai/dedicated-inference/index.md). ## Client Responsibilities While Exoscale is responsible for operating and maintaining the Dedicated Inference endpoint, Clients remain responsible for the following: * Selection, deployment, and configuration of models and inference workloads. * Management of input and output data, including data classification, retention, and deletion. * Access control to inference endpoints, including [AI API Keys](https://community.exoscale.com/product/concrete-ai/quick-start/index.md), credentials, and network policies. * Validation of inference results and model behavior. * Backup and lifecycle management of models and related artifacts. * Compliance of datasets, prompts, outputs, and workloads with applicable laws and regulations. * Cost management, usage monitoring, and scaling configuration. The Service Level Agreement applies solely to Dedicated Inference Endpoint Availability and does not cover model performance, inference accuracy, or Client workload behavior. For a complete explanation of the shared responsibility framework between Exoscale and Clients, please see the [Exoscale Shared Responsibility Model documentation](https://community.exoscale.com/platform/compliance/shared-responsability-model/index.md). ## Exclusions No compensation shall be granted to the Client if the failure to comply with the Dedicated Inference Service Level Objective is due to any of the following reasons: * Client-side configurations, including model deployment parameters, scaling settings, network rules, access policies, authentication settings, authorization settings, or quota limitations. * Model behavior, inference quality, output correctness, inference latency, throughput, or data loss. * Invalid, malformed, unauthorized, forbidden, quota-related, or rate-limited requests. * Requests that are abusive, artificially generated to affect SLA measurement, part of denial-of-service activity, or above documented usage limits. For the complete list of SLA exclusions, please refer to the [General Exoscale Terms and Conditions](https://www.exoscale.com/terms/). ## Retribution Service credits apply only to Dedicated Inference Endpoint Availability as defined above and do not apply to model performance, inference latency, throughput, or Client workload behavior. The standard retribution and SLA credit schemes apply for the Dedicated Inference as described under [General Exoscale Terms and Conditions](https://www.exoscale.com/terms/) or [EUSA](https://www.exoscale.com/eusa/) whichever applies. The Service Credits shall be the sole remedy in case of non-meeting the SLA. In the event that the Monthly Uptime Percentage for Exoscale Compute Instances falls below the committed Service Level Objective, the Client will be eligible for service credits as follows: | Monthly Uptime Percentage | Service Credit Percentage of Monthly Service Fees for Affected Resources | | -------- | -------- | | from 99.95% to 98.3% | 50% | | below 98.3% | 100% | ### Claim Service Credit To request a service credit, refer to the Service Unavailability Credit section of the [Exoscale Terms and Conditions](https://www.exoscale.com/terms/) or the [EUSA](https://www.exoscale.com/eusa/), whichever applies. --- url: https://community.exoscale.com/product/concrete-ai/dedicated-inference/reference/ --- # Reference - Dedicated Inference - [API Dedicated Inference](https://community.exoscale.com/reference/api/ai/dedicated-inference/index.md) - All operations on Dedicated Inference. (external) - [CLI Dedicated Inference](https://community.exoscale.com/reference/cli/exo/dedicated-inference/index.md) - All commands on Dedicated Inference. (external) --- url: https://community.exoscale.com/product/concrete-ai/dedicated-inference/recipes/ --- # Recipes - Dedicated Inference Informational, concise guides showing how to integrate with external tools and solve specific scenarios. - [OpenCode Setup](https://community.exoscale.com/product/concrete-ai/dedicated-inference/recipes/opencode/index.md) - Configure OpenCode to use Dedicated Inference endpoints from Exoscale, then connect, select your provider, and pick your model. --- url: https://community.exoscale.com/product/concrete-ai/dedicated-inference/recipes/opencode/ --- # OpenCode Setup This guide is provided for informational purposes. Because it involves third party tooling we cannot guarantee things still work exactly as described here. Additional model providers can be added to OpenCode via configuration. Assuming you have deployed `MiniMaxAI/MiniMax-M2.5` in `hr-zag-1` Write the following json to the `~/.config/opencode/opencode.json` file: ```json { "$schema": "https://opencode.ai/config.json", "provider": { "exoscale": { "npm": "@ai-sdk/openai-compatible", "name": "Exoscale", "options": { "baseURL": "https://.inference.hr-zag-1.exoscale-cloud.com/v1" }, "models": { "minimax": { "name": "MiniMaxAI/MiniMax-M2.5" } } } } } ``` Adapt as necessary. Once configured: - run `opencode` - type `/connect` in the prompt - select "Exoscale" in the providers list and enter an [AI API Key](https://community.exoscale.com/product/concrete-ai/quick-start/index.md) scoped to your deployment. - type `/models` and pick your deployment from the list.