Environment Variables Configuration¶
This guide covers all environment variables available in inference-models for configuring model loading, caching, API access, and runtime behavior.
Quick Start¶
Set environment variables before importing inference-models:
# Set API key
export ROBOFLOW_API_KEY="your_api_key_here"
# Set model cache directory
export INFERENCE_HOME="/path/to/cache"
# Set device
export DEFAULT_DEVICE="cuda:0"
inference-models reads the process environment when it is first imported and
then loads a .env file from the current working directory without overriding
already-set process values. This matches the inference package, so package
import order cannot change the selected cache or offline mode.
Core Configuration¶
API Authentication¶
ROBOFLOW_API_KEY (or API_KEY)
Your Roboflow API key for accessing models.
Get your API key from: https://docs.roboflow.com/api-reference/authentication
ROBOFLOW_ENVIRONMENT
Environment to use: prod (default) or staging.
ROBOFLOW_API_HOST
Override API host URL (auto-set based on environment).
Model Cache¶
INFERENCE_HOME
Directory where downloaded models are cached. If it is unset,
MODEL_CACHE_DIR is used; if both are unset, the default is /tmp/cache.
OFFLINE_MODE
Startup-only switch for loading network-provider models exclusively from a
trusted, compatible local cache.
The first import of either inference or inference_models latches this value
for the process. Changing or removing the variable later does not change the
mode; restart the process instead. Child processes inherit the latch when they
inherit the parent environment with Inference's private marker intact. That
marker is trusted internal process state, not a security boundary against
arbitrary code already running in the process. A child launched with a
deliberately rewritten or sanitized environment is a new startup boundary, so
use operating-system or network-level isolation when a hard air gap is required.
The startup latch also enables the Hugging Face and Ultralytics dependency
offline controls before Inference imports those libraries, preventing their
built-in connectivity checks and online-gated behavior.
If HF_HOME is not explicitly configured, both packages set it before heavy
imports to $INFERENCE_HOME/hf_home, $MODEL_CACHE_DIR/hf_home, or
/tmp/cache/hf_home in that order. This keeps implicit Hugging Face backbone,
processor, and checkpoint downloads in the mounted cache across the
online-warm and fresh-offline phases.
Warm the cache online with the matching inference-models release and the same
model-loading constraints and runtime environment before enabling offline mode.
Legacy cache manifests do not contain the canonical owner, trust, dependency,
and compatibility metadata required by the offline loader and must be
re-warmed. A credential-free offline restart can use a cache warmed with a key
only when the current metadata proves one unambiguous canonical model identity.
A changed or rotated non-empty key requires an exact matching cache entry and
otherwise fails closed.
Device Selection¶
DEFAULT_DEVICE
Default device for model inference: cpu, cuda, cuda:0, etc.
API Configuration¶
Request Settings¶
API_CALLS_TIMEOUT
Timeout for API calls in seconds. Default: 5
API_CALLS_MAX_TRIES
Maximum retry attempts for API calls. Default: 3
IDEMPOTENT_API_REQUEST_CODES_TO_RETRY
HTTP status codes to retry (comma-separated). Default: 408,429,502,503,504
Backend Configuration¶
ONNX Runtime¶
ONNXRUNTIME_EXECUTION_PROVIDERS
Override ONNX execution providers, comma separated, no spaces.
Default: CUDAExecutionProvider,OpenVINOExecutionProvider,CoreMLExecutionProvider,CPUExecutionProvider
Prediction Parameter Defaults¶
These environment variables control the default values for prediction parameters across all models. Individual models may override these defaults. See Prediction Parameters for detailed information about each parameter.
General Detection Parameters¶
INFERENCE_MODELS_DEFAULT_CONFIDENCE
Default confidence threshold for filtering predictions. Default: 0.4
INFERENCE_MODELS_DEFAULT_IOU_THRESHOLD
Default IoU threshold for Non-Maximum Suppression. Default: 0.3
INFERENCE_MODELS_DEFAULT_MAX_DETECTIONS
Default maximum number of detections to return. Default: 300
INFERENCE_MODELS_INSTANCE_SEG_MASK_PROCESSING_CHUNK_SIZE
Number of instance-segmentation masks upscaled to original resolution per
slice when producing dense masks. Bounds postprocessing memory (the float32
working set is chunk x H x W instead of detections x H x W). Must be
>= 1. Default: 16
INFERENCE_MODELS_DEFAULT_CLASS_AGNOSTIC_NMS
Default for class-agnostic NMS. Default: false
General Vision-Language Model Parameters¶
INFERENCE_MODELS_DEFAULT_MAX_NEW_TOKENS
Default maximum number of tokens to generate. Default: 4096
INFERENCE_MODELS_DEFAULT_NUM_BEAMS
Default number of beams for beam search. Default: 3
INFERENCE_MODELS_DEFAULT_DO_SAMPLE
Default for sampling during generation. Default: false
INFERENCE_MODELS_DEFAULT_SKIP_SPECIAL_TOKENS
Default for skipping special tokens in output. Default: false
Model-Specific Overrides¶
Individual models can override the general defaults. Below are model-specific environment variables:
DeepLabV3+¶
INFERENCE_MODELS_DEEP_LAB_V3_PLUS_DEFAULT_CONFIDENCE
Default: 0.5
DINOv3¶
INFERENCE_MODELS_DINOV3_DEFAULT_CONFIDENCE
Default: 0.5
EasyOCR¶
INFERENCE_MODELS_EASYOCR_DEFAULT_CONFIDENCE
Default: 0.3
Florence-2¶
INFERENCE_MODELS_FLORENCE2_DEFAULT_MAX_NEW_TOKENS
Default: Inherits from INFERENCE_MODELS_DEFAULT_MAX_NEW_TOKENS
INFERENCE_MODELS_FLORENCE2_DEFAULT_NUM_BEAMS
Default: Inherits from INFERENCE_MODELS_DEFAULT_NUM_BEAMS
INFERENCE_MODELS_FLORENCE2_DEFAULT_DO_SAMPLE
Default: Inherits from INFERENCE_MODELS_DEFAULT_DO_SAMPLE
Grounding DINO¶
INFERENCE_MODELS_GROUNDING_DINO_DEFAULT_BOX_CONFIDENCE
Default: 0.5
INFERENCE_MODELS_GROUNDING_DINO_DEFAULT_MAX_DETECTIONS
Default: Inherits from INFERENCE_MODELS_DEFAULT_MAX_DETECTIONS
INFERENCE_MODELS_GROUNDING_DINO_DEFAULT_IOU_THRESHOLD
Default: 0.5
Moondream2¶
INFERENCE_MODELS_MOONDREAM2_DEFAULT_MAX_NEW_TOKENS
Default: 700
OWLv2¶
INFERENCE_MODELS_OWLV2_DEFAULT_CONFIDENCE
Default: 0.99
INFERENCE_MODELS_OWLV2_DEFAULT_IOU_THRESHOLD
Default: Inherits from INFERENCE_MODELS_DEFAULT_IOU_THRESHOLD
INFERENCE_MODELS_OWLV2_DEFAULT_MAX_DETECTIONS
Default: Inherits from INFERENCE_MODELS_DEFAULT_MAX_DETECTIONS
INFERENCE_MODELS_OWLV2_DEFAULT_CLASS_AGNOSTIC_NMS
Default: Inherits from INFERENCE_MODELS_DEFAULT_CLASS_AGNOSTIC_NMS
PaliGemma¶
INFERENCE_MODELS_PALIGEMMA_DEFAULT_MAX_NEW_TOKENS
Default: 400
INFERENCE_MODELS_PALIGEMMA_DEFAULT_DO_SAMPLE
Default: Inherits from INFERENCE_MODELS_DEFAULT_DO_SAMPLE
INFERENCE_MODELS_PALIGEMMA_DEFAULT_SKIP_SPECIAL_TOKENS
Default: true
Gemma 4¶
These variables set the defaults used by the Gemma 4 Hugging Face implementation (Gemma4HF; see Gemma 4) for prompt(), pre_process_generation(), and generate() (including sampling kwargs when do_sample is true).
INFERENCE_MODELS_GEMMA4_DEFAULT_MAX_NEW_TOKENS
Default: 512
INFERENCE_MODELS_GEMMA4_DEFAULT_DO_SAMPLE
Default: Inherits from INFERENCE_MODELS_DEFAULT_DO_SAMPLE
INFERENCE_MODELS_GEMMA4_DEFAULT_ENABLE_THINKING
Default: false (passed to the processor chat template as enable_thinking)
INFERENCE_MODELS_GEMMA4_DEFAULT_SKIP_SPECIAL_TOKENS
Default: true
INFERENCE_MODELS_GEMMA4_DEFAULT_TEMPERATURE
Default: 1.0 (used when do_sample is true)
INFERENCE_MODELS_GEMMA4_DEFAULT_TOP_P
Default: 0.95
INFERENCE_MODELS_GEMMA4_DEFAULT_TOP_K
Default: 64
Qwen2.5-VL¶
INFERENCE_MODELS_QWEN25_VL_DEFAULT_MAX_NEW_TOKENS
Default: 512
INFERENCE_MODELS_QWEN25_VL_DEFAULT_DO_SAMPLE
Default: Inherits from INFERENCE_MODELS_DEFAULT_DO_SAMPLE
INFERENCE_MODELS_QWEN25_VL_DEFAULT_SKIP_SPECIAL_TOKENS
Default: true
Qwen3-VL¶
INFERENCE_MODELS_QWEN3_VL_DEFAULT_MAX_NEW_TOKENS
Default: 512
INFERENCE_MODELS_QWEN3_VL_DEFAULT_DO_SAMPLE
Default: Inherits from INFERENCE_MODELS_DEFAULT_DO_SAMPLE
ResNet¶
INFERENCE_MODELS_RESNET_DEFAULT_CONFIDENCE
Default: Inherits from INFERENCE_MODELS_DEFAULT_CONFIDENCE
RF-DETR¶
INFERENCE_MODELS_RFDETR_DEFAULT_CONFIDENCE
Default: Inherits from INFERENCE_MODELS_DEFAULT_CONFIDENCE
INFERENCE_MODELS_RFDETR_DEFAULT_MAX_DETECTIONS
Default cap on RF-DETR instance-segmentation detections, applied by score
BEFORE masks are upscaled to original resolution (bounds mask memory).
Default: Inherits from INFERENCE_MODELS_DEFAULT_MAX_DETECTIONS
The following variables select RF-DETR TensorRT pipeline implementations when a client
cannot pass backend-specific from_pretrained arguments. Explicit arguments take
precedence over these environment variables.
See Inference-Path Optimization Architecture for the selection model and the complete RF-DETR execution flow.
INFERENCE_MODELS_RFDETR_PREPROCESSOR
Default: triton-universal-v1
Supported values: base, auto, threaded-exact-v1, triton-universal-v1.
INFERENCE_MODELS_RFDETR_PREPROCESSOR_MAX_WORKERS
Default: 4
Controls the bounded worker count used by threaded-exact-v1.
INFERENCE_MODELS_RFDETR_POSTPROCESSOR
Default: triton-fused-v1
Supported values: base, auto, triton-fused-v1.
Code that can pass backend-specific arguments may instead provide a composed, immutable execution plan:
from inference_models import AutoModel
from inference_models.models.rfdetr.optimization.execution_plan import (
RFDetrExecutionPlan,
)
plan = RFDetrExecutionPlan(
preprocessor_id="triton-universal-v1",
postprocessor_id="triton-fused-v1",
allow_compatibility_fallback=True,
)
model = AutoModel.from_pretrained(
"rfdetr-small",
backend="trt",
rfdetr_execution_plan=plan,
)
Public preprocessing synchronizes by default, so its result can be consumed by an
independent forward() call without relying on model-owned readiness state:
model = AutoModel.from_pretrained(
"rfdetr-small",
backend="trt",
rfdetr_execution_plan=plan,
)
preprocessed, metadata = model.pre_process(image)
raw_predictions = model.forward(preprocessed)
This invocation-boundary policy is intentionally separate from the execution plan.
Composed model(...) and infer() calls pass
independent_stage_execution=False to preprocessing internally, record a CUDA event,
and let forward() wait on that event without a host synchronization.
The plan also reserves independently selectable buffer-strategy, scheduler, and engine
plugin stages. Those stages currently accept only base. When supplied, an explicit
plan takes precedence and the implementation-selection environment variables are not
read.
When a selected optimized stage declares that it cannot preserve a model or request
contract, RF-DETR uses its declared base fallback and records the requested
implementation, effective implementation, and reason in logs and runtime metadata.
This policy applies consistently to preprocessing and postprocessing. Set
allow_compatibility_fallback=False in an explicit plan to require the selected
implementation or an error. Compilation, CUDA, allocation, and other execution
failures are never converted into fallbacks.
An all-False PreProcessingOverrides object is a no-op and remains compatible with
triton-universal-v1. Requests with any active preprocessing override use the declared
base fallback. A distinct request-level fallback reason is warned once per model
instance rather than once per inference.
When Triton is unavailable, uint8 universal preprocessing and fused postprocessing
declare a compatibility miss and use their base fallback. Floating-point tensor
preprocessing remains eligible for triton-universal-v1 because that input path uses
Torch operations and does not require Triton kernels.
Roboflow Instant¶
INFERENCE_MODELS_ROBOFLOW_INSTANT_DEFAULT_CONFIDENCE
Default: 0.99
INFERENCE_MODELS_ROBOFLOW_INSTANT_DEFAULT_IOU_THRESHOLD
Default: 0.3
INFERENCE_MODELS_ROBOFLOW_INSTANT_MAX_DETECTIONS
Default: Inherits from INFERENCE_MODELS_DEFAULT_MAX_DETECTIONS
SmolVLM¶
INFERENCE_MODELS_SMOL_VLM_DEFAULT_MAX_NEW_TOKENS
Default: 400
INFERENCE_MODELS_SMOL_VLM_DEFAULT_DO_SAMPLE
Default: Inherits from INFERENCE_MODELS_DEFAULT_DO_SAMPLE
INFERENCE_MODELS_SMOL_VLM_DEFAULT_SKIP_SPECIAL_TOKENS
Default: true
ViT¶
INFERENCE_MODELS_VIT_CLASSIFIER_DEFAULT_CONFIDENCE
Default: Inherits from INFERENCE_MODELS_DEFAULT_CONFIDENCE
YOLACT¶
INFERENCE_MODELS_YOLACT_DEFAULT_CONFIDENCE
Default: Inherits from INFERENCE_MODELS_DEFAULT_CONFIDENCE
INFERENCE_MODELS_YOLACT_DEFAULT_IOU_THRESHOLD
Default: Inherits from INFERENCE_MODELS_DEFAULT_IOU_THRESHOLD
INFERENCE_MODELS_YOLACT_DEFAULT_MAX_DETECTIONS
Default: Inherits from INFERENCE_MODELS_DEFAULT_MAX_DETECTIONS
INFERENCE_MODELS_YOLACT_DEFAULT_CLASS_AGNOSTIC_NMS
Default: Inherits from INFERENCE_MODELS_DEFAULT_CLASS_AGNOSTIC_NMS
YOLO-NAS¶
INFERENCE_MODELS_YOLONAS_DEFAULT_CONFIDENCE
Default: Inherits from INFERENCE_MODELS_DEFAULT_CONFIDENCE
INFERENCE_MODELS_YOLONAS_DEFAULT_IOU_THRESHOLD
Default: Inherits from INFERENCE_MODELS_DEFAULT_IOU_THRESHOLD
INFERENCE_MODELS_YOLONAS_DEFAULT_MAX_DETECTIONS
Default: Inherits from INFERENCE_MODELS_DEFAULT_MAX_DETECTIONS
INFERENCE_MODELS_YOLONAS_DEFAULT_CLASS_AGNOSTIC_NMS
Default: Inherits from INFERENCE_MODELS_DEFAULT_CLASS_AGNOSTIC_NMS
YOLOv5¶
INFERENCE_MODELS_YOLOV5_DEFAULT_CONFIDENCE
Default: Inherits from INFERENCE_MODELS_DEFAULT_CONFIDENCE
INFERENCE_MODELS_YOLOV5_DEFAULT_IOU_THRESHOLD
Default: Inherits from INFERENCE_MODELS_DEFAULT_IOU_THRESHOLD
INFERENCE_MODELS_YOLOV5_DEFAULT_MAX_DETECTIONS
Default: Inherits from INFERENCE_MODELS_DEFAULT_MAX_DETECTIONS
INFERENCE_MODELS_YOLOV5_DEFAULT_CLASS_AGNOSTIC_NMS
Default: Inherits from INFERENCE_MODELS_DEFAULT_CLASS_AGNOSTIC_NMS
YOLOv7¶
INFERENCE_MODELS_YOLOV7_DEFAULT_CONFIDENCE
Default: Inherits from INFERENCE_MODELS_DEFAULT_CONFIDENCE
INFERENCE_MODELS_YOLOV7_DEFAULT_IOU_THRESHOLD
Default: Inherits from INFERENCE_MODELS_DEFAULT_IOU_THRESHOLD
INFERENCE_MODELS_YOLOV7_DEFAULT_MAX_DETECTIONS
Default: Inherits from INFERENCE_MODELS_DEFAULT_MAX_DETECTIONS
INFERENCE_MODELS_YOLOV7_DEFAULT_CLASS_AGNOSTIC_NMS
Default: Inherits from INFERENCE_MODELS_DEFAULT_CLASS_AGNOSTIC_NMS
YOLOv8/v9/v11/v12 (Ultralytics)¶
INFERENCE_MODELS_YOLO_ULTRALYTICS_DEFAULT_CONFIDENCE
Default: Inherits from INFERENCE_MODELS_DEFAULT_CONFIDENCE
INFERENCE_MODELS_YOLO_ULTRALYTICS_DEFAULT_IOU_THRESHOLD
Default: Inherits from INFERENCE_MODELS_DEFAULT_IOU_THRESHOLD
INFERENCE_MODELS_YOLO_ULTRALYTICS_DEFAULT_MAX_DETECTIONS
Default: Inherits from INFERENCE_MODELS_DEFAULT_MAX_DETECTIONS
INFERENCE_MODELS_YOLO_ULTRALYTICS_DEFAULT_CLASS_AGNOSTIC_NMS
Default: Inherits from INFERENCE_MODELS_DEFAULT_CLASS_AGNOSTIC_NMS
INFERENCE_MODELS_YOLO_ULTRALYTICS_DEFAULT_KEY_POINTS_THRESHOLD
Default: 0.0
YOLOv10¶
INFERENCE_MODELS_YOLOV10_DEFAULT_CONFIDENCE
Default: Inherits from INFERENCE_MODELS_DEFAULT_CONFIDENCE
INFERENCE_MODELS_YOLOV10_DEFAULT_MAX_DETECTIONS
Default: Inherits from INFERENCE_MODELS_DEFAULT_MAX_DETECTIONS
Logging¶
LOG_LEVEL
Set the log level for the library. Default: WARNING
VERBOSE_LOG_LEVEL
Set the log level for verbose logging. Default: INFO
DISABLE_VERBOSE_LOGGER
Disable verbose logging. Default: false
DISABLE_INTERACTIVE_PROGRESS_BARS
Disable interactive progress bars. Default: false
Advanced Configuration¶
SAM3¶
INFERENCE_MODELS_SAM3_MASK_PROCESSING_CHUNK_SIZE
Number of SAM3 concept-segmentation masks upscaled to original resolution and
encoded per slice. Bounds postprocessing memory (working set is
chunk x H x W float32 instead of detections x H x W). Must be >= 1.
Default: 8
Input Validation¶
ALLOW_URL_INPUT
Allow URLs as image input. Used by models like OWL-V2, when access to larger
datasets provided as references is needed. Default: true
ALLOW_NON_HTTPS_URL_INPUT
Allow non-HTTPS URLs. Used by models like OWL-V2, when access to larger
datasets provided as references is needed. Default: false
ALLOW_URL_INPUT_WITHOUT_FQDN
Allow URLs without FQDN. Used by models like OWL-V2, when access to larger
datasets provided as references is needed. Default: false
WHITELISTED_DESTINATIONS_FOR_URL_INPUT
Comma-separated list of allowed destinations for URL input. Used by models like OWL-V2, when access to larger
datasets provided as references is needed. Default: None
BLACKLISTED_DESTINATIONS_FOR_URL_INPUT
Comma-separated list of allowed destinations for URL input. Used by models like OWL-V2, when access to larger
datasets provided as references is needed. Default: None
ALLOW_LOCAL_STORAGE_ACCESS_FOR_REFERENCE_DATA
Allow local storage access for reference data. Used by models like OWL-V2, when access to larger
datasets provided as references is needed. Default: true