Skip to content

💾 Cache Management

Understanding how inference-models caches data to improve performance and reduce redundant operations.

Overview

The library uses two types of caching:

  • Auto-Resolution Cache - Stores backend selection decisions to avoid repeated API calls and package negotiation
  • Model Package Cache - Stores downloaded model files (weights, configs, class names) to avoid re-downloading

Both caches are stored under $INFERENCE_HOME. When INFERENCE_HOME is unset, the library uses $MODEL_CACHE_DIR, then falls back to /tmp/cache/.

🔄 Auto-Resolution Cache

What Gets Cached

When you load a model with AutoModel.from_pretrained(), the library performs backend negotiation to select the optimal model package. This decision is cached to avoid repeating the process on subsequent loads.

All parameters that affect the negotiation are hashed together:

  • Weights provider (Roboflow, local, etc.)
  • Model ID
  • API key (hashed for security)
  • Requested backend preferences
  • Requested quantization
  • Requested batch size
  • Device configuration
  • ONNX execution providers
  • Other negotiation parameters

If any of these parameters change, the cache is bypassed and a fresh negotiation occurs.

Configuration

Default location: $INFERENCE_HOME/auto-resolution-cache/ (defaults to /tmp/cache/auto-resolution-cache/)

Cache expiration: 24 hours (1440 minutes) by default

Override cache location: 

export INFERENCE_HOME=/path/to/custom/cache

Change expiration time: 

export AUTO_LOADER_CACHE_EXPIRATION_MINUTES=60  # 1 hour

Disable caching: 

model = AutoModel.from_pretrained(
    "rfdetr-base",
    use_auto_resolution_cache=False
)

Purge cache from filesystem: 

rm -rf $INFERENCE_HOME/auto-resolution-cache/
# or if using default location:
rm -rf /tmp/cache/auto-resolution-cache/

📦 Model Package Cache

What Gets Cached

Downloaded model files (weights, configs, class names, etc.) are cached locally to avoid re-downloading on subsequent loads.

Cache Access and API Key Assumptions

Important: The default local access manager assumes that a canonically attributed model already stored on the local filesystem may be used by a credential-free process, even if the original download required authentication.

This means:

  • A credential-free offline load may reuse it only when current cache metadata resolves the requested alias to one unambiguous canonical model
  • A non-empty API key uses only its exact auto-resolution entry; changing the key does not use credential-independent fallback
  • The default library access manager does not add tenant authorization for otherwise eligible local files
  • In single-user environments, this is typically the desired behavior for convenience

Multi-tenant environments:

When running in multi-tenant or shared environments (e.g., on the Roboflow platform), an upstream guard layer should be implemented to ensure proper access control. The Roboflow platform ships with such guards that:

  • Verify user permissions before allowing cache access
  • Ensure client models remain isolated and secure
  • Prevent unauthorized access to cached models from other tenants

If you're deploying inference-models in a multi-tenant environment, you are responsible for implementing appropriate access control mechanisms at the application layer.

Directory Structure

Default location: $INFERENCE_HOME/models-cache/ (defaults to /tmp/cache/models-cache/)

Model IDs are slugified and hashed to create safe, unique, yet human-readable directory names. Package IDs (provided by the weights provider) are used as subdirectory names within each model directory.

Example structure: 

/tmp/cache/
├── models-cache/
│   ├── v2-yolov8n-640-0123456789abcdef0123456789abcdef/
│   │                                   # Slugified model ID + 128-bit hash
│   │   ├── onnxfp32/                   # Package ID from provider
│   │   │   ├── model.onnx -> ../../shared-blobs/e4f5a6b7...
│   │   │   └── class_names.txt
│   │   └── trtfp16/                    # Another package ID
│   │       └── model.engine -> ../../shared-blobs/c8d9e0f1...
│   └── v2-rfdetr-base-fedcba9876543210fedcba9876543210/
│       └── torchfp32/
│           └── model.pt -> ../../shared-blobs/a2b3c4d5...
└── shared-blobs/                       # Content-addressed blob storage
    ├── e4f5a6b7...                     # MD5 hash of file content
    ├── c8d9e0f1...
    └── a2b3c4d5...

🔗 Shared Blob Storage

When the weights provider supplies a content hash (MD5) for a file, the library stores the actual file in $INFERENCE_HOME/shared-blobs/ named after its hash, and creates symlinks from the model package directories.

Benefits:

  • Avoids duplicate downloads - If multiple models or packages share the same file (e.g., same weights with different configs), it's only downloaded once
  • Saves disk space - Shared files are stored once and linked multiple times
  • Helps in bandwidth-constrained environments - Particularly useful when working with multiple model variants or in offline/air-gapped deployments

Files without content hashes are stored directly in the model package directory.

Cache Expiration

Model package cache does not expire automatically - files remain until manually deleted.

Offline cache compatibility

Current package manifests record the exact cache owner, provider-resolved canonical model ID, source trust, dependency metadata, package-selection constraints, and a structured runtime compatibility fingerprint. OFFLINE_MODE=True only loads a package whose manifest matches the current request and runtime.

A request with a non-empty API key may use only the exact auto-resolution entry created for that key and set of loading parameters. A changed or rotated key does not use credential-independent fallback; reconnect and re-warm it. A credential-free offline restart may reuse compatible metadata only when every matching current entry is canonically attributed and all of them resolve to one canonical model identity. Ambiguous aliases fail closed.

New writes use a V2 model-cache path with a 128-bit identity digest. V1 paths with the older 32-bit digest are read-only and accepted only when a regular manifest proves the exact model owner. Ownerless or mismatched legacy entries are rejected. Before disconnecting a deployment, install the matching inference-models release and warm every required model again under the same backend, device, quantization, batch, ONNX-provider, and dependency settings that the offline process will use.

Purge model cache: 

rm -rf $INFERENCE_HOME/models-cache/
# or if using default location:
rm -rf /tmp/cache/models-cache/

Purge shared blobs: 

rm -rf $INFERENCE_HOME/shared-blobs/
# or if using default location:
rm -rf /tmp/cache/shared-blobs/

🚀 Next Steps