Does Your Model Run on AWS Trainium? Look It Up

Introduction

If you work with Hugging Face models and are curious whether they can run on AWS Trainium, a new Space makes the answer trivial. It lists the most popular models, shows whether they run unmodified with the `torch‑neuronx` backend, and provides the exact script and environment needed to reproduce the results on your own Trainium instance.

How the Space works

• Model selection – The Space shows a curated "Selected" view of top‑downloaded and trending models. You can also switch to the Trending or Downloads tabs to search any model by name.
• Result columns – For each entry you’ll see:

- Pass/fail status - Number of Trainium cores used (sharding) - Wall‑time including compilation - Peak device memory

• Why a model is missing – If a model isn’t listed, the Space explains whether it was deliberately skipped (e.g., low priority) or truly incompatible with the current `torch‑neuronx` stack.
• Version control – You can pick the Neuron compiler (`neuronx‑cc`) and Transformers versions to see how results change across releases.

Running a model yourself

1. Pick a model – Click a row to expand the details. The Space displays the exact Neuron stack that was tested and a ready‑to‑run `run.py` script.
2. Set up the environment – On a Trainium2 (`trn2`) instance, use the AWS Neuron Deep Learning Container and install the listed packages.
3. Execute – Run the script with `python3 run.py`. The script handles distributed launch internally, so you don’t need `torchrun`.
4. Skip recompilation (optional) – The first run compiles the model graph. To avoid this step, sync the pre‑compiled NEFF cache from the public bucket:

``` hf buckets sync hf://buckets/aws-neuron/torch-neuronx-neff-cache/neuronxcc-<version> /tmp/neff_cache ``` The Space provides the exact command for the compiler version used.

What “runs” really means

• Sharding – Models are placed on the smallest number of Trainium cores that can hold their weights, typically using tensor‑parallelism. Mixture‑of‑experts models may use expert‑parallelism.
• Verification – After loading, the test generates 32 tokens. For single‑core runs, the output is compared to a CPU reference; for sharded runs, a lightweight judge model checks fluency.
• Scope – Currently only models that fit on up to four Trainium2 devices (16 cores) are evaluated. A PASS indicates successful loading and a short generation, not sustained throughput.

Accessing the raw dataset

All results originate from the public dataset `aws-neuron/torch-neuronx-model-support`. It stores a Parquet file for each combination of compiler and library versions. You can query it directly in Python: ```python import pandas as pd df = pd.readparquet("hf://datasets/aws-neuron/torch-neuronx-model-support/data") print(df[df.modelid == "Qwen/Qwen3-0.6B"][['neuronxcc','library','libraryversion','neuron_cores','result']]) ``` The dataset card explains each column and the meaning of the result codes.

Practical takeaways

• Quick lookup – Before spending time provisioning Trainium, check the Space to confirm support.
• Reproducible runs – Use the provided script and NEFF cache to replicate the benchmark on your own hardware.
• Future expansion – The current focus is on text‑generation and image‑text‑to‑text models, but support for diffusers, sentence‑transformers, and other tasks is planned.

--- Next time you wonder whether a Hugging Face model runs on AWS Trainium, simply search the Space, copy the script, and get running in minutes.