Serving Image Selection

Rule zero: HuggingFace images always win

1. Verified incompatibility — the model needs an architecture/modality/feature no available HuggingFace tag supports, confirmed against the catalog (not assumed).
2. No HuggingFace tag exists in the target region and mirroring is not an option.
3. The HuggingFace image is in "Known-broken images" below.

Where image URIs come from

Quick decision

Rerankers: TEI or vLLM?

• Encoder cross-encoders (BAAI/bge-reranker-*, mixedbread, most sentence-transformers rerankers) — BERT-family models with a classification head. config.json has architectures: [..ForSequenceClassification] on a TEI-supported encoder type. → TEI.
• Generative rerankers (Qwen/Qwen3-Reranker-*, and similar causal-LM judges) — decoder LLMs that score relevance via the logprob of a yes/no token. config.json has architectures: [..ForCausalLM]. → HuggingFace vLLM, deployed exactly like a text-generation LLM. TEI will load the architecture then reject the classifier model type (Qwen3 support in TEI is embeddings-only). Invocation pattern (raw completions API, max_tokens=1, logprobs scoring) is in hf-cloud-sagemaker-production-defaults.

curl -s https://huggingface.co/<model-id>/raw/main/config.json
# "architectures": ["Qwen3ForCausalLM"]              → vLLM
# "architectures": ["XLMRobertaForSequenceClassification"] → TEI

Workflow

1. Open https://aws.github.io/deep-learning-containers/reference/available_images/
2. Find the section for the right family (e.g. "HuggingFace vLLM Inference" for HuggingFace LLMs, "HuggingFace Text Embeddings Inference" for embeddings)
3. Pick the newest row marked SageMaker for the platform column — newest within that family. Do not switch to another family's section because it lists a higher engine version (see "Rule zero")
4. Substitute <region> with the user's region (from hf-cloud-aws-context-discovery)
5. For vLLM: also check the AMI requirement (see "vLLM AMI requirement" below)
6. Pass the URI to deploy.py --image-uri (real-time) or deploy_async.py --image-uri (async)

TEI: pick the right variant

• ml.g*, ml.p*, ml.inf* → GPU variant
• ml.c*, ml.m*, ml.t* → CPU variant

vLLM AMI requirement

Configuring the vLLM DLCs (HuggingFace vLLM and AWS vLLM)

Required for every HuggingFace LLM deployment

Tuning (optional)

Configuring TEI

CUDA / instance compatibility

VPC / NAT gateway problem

# macOS / Linux
PRIVATE_URI=$(python3 scripts/mirror_image.py \
    public.ecr.aws/deep-learning-containers/vllm:<tag> \
    vllm-mirror)

# Windows (PowerShell) — capture stdout into a variable
$PRIVATE_URI = python scripts\mirror_image.py `
    public.ecr.aws/deep-learning-containers/vllm:<tag> vllm-mirror

When the catalog page won't render, is stale, or is wrong

1. The catalog's source data on GitHub — the page is generated from one YAML file per version, listing exact tags, CUDA, and Python versions. List a family's files, then fetch the newest:
   bashCopy

   curl -s https://api.github.com/repos/aws/deep-learning-containers/contents/docs/src/data/huggingface-vllm
   curl -s https://raw.githubusercontent.com/aws/deep-learning-containers/main/docs/src/data/huggingface-vllm/0.21.0-gpu-sagemaker.yml

   Directory names match ECR repos (huggingface-vllm, huggingface-vllm-omni, huggingface-tei, vllm, djl-inference, ...).
2. Query ECR directly for current tags in the target region (works with credentials that can read the DLC registry; if it returns AccessDenied, use the YAML files):
   bashCopy

   aws ecr describe-images --registry-id 763104351884 --repository-name huggingface-vllm \
       --region <region> --query 'sort_by(imageDetails,&imagePushedAt)[-5:].imageTags' --output json

3. Release notes on the DLC GitHub repo.

Known-broken images (last checked July 2026)

Hub download time at first boot