AWS brings SGLang to Trainium, with gpt-oss-120b as the demo

AWS brings SGLang to Trainium, with gpt-oss-120b as the demo

Amazon Web Services announced on November 12, on its official machine learning blog, that SGLang-Neuron is now generally available. Developers can use the same interface they already use to tune SGLang to deploy models on AWS's in-house Trainium chips and to generate rollout data for reinforcement learning. The software ships as an AWS Deep Learning Container, released jointly by RadixArk and Annapurna Labs, Amazon's chip team.

The release follows a beta from a month earlier. On October 22, the LMSYS blog published a beta of the SGLang Trainium plugin, which its authors called an “early milestone”, saying performance work was still under way and that an update would come at the SGLang Summit in November. With this release, AWS has put the plugin into an official container, moving it from an experimental repository to something developers can pull as an image.

How the plugin hooks into SGLang

SGLang-Neuron is an out-of-tree plugin, meaning its code does not live in the main SGLang repository. Once installed, it registers a platform called neuron through SGLang's plugin entry point, and setting the environment variable SGLANG_PLATFORM=neuron at launch selects it.

On the division of labor, SGLang core still handles the API server, the request lifecycle, RadixAttention and prefix-cache bookkeeping. The plugin supplies the Trainium-specific models, attention and MoE layers, and the graph executor. AWS says this lets the Trainium port and the SGLang mainline iterate independently, without waiting on each other's releases.

One layer further down is TorchNeuron, which registers Trainium as a native PyTorch device. SGLang-Neuron's model execution path uses standard PyTorch operators, swaps in kernels written with NKI (Neuron Kernel Interface) for performance-sensitive parts, and then hands everything to torch.compile to build an executable program.

The plugin also supports prefill-decode (PD) disaggregation: a prefill server computes the KV cache for the prompt and passes it over NIXL on the EFA network to a decode server, which goes on to generate the remaining tokens. Each server uses SGLang's own scheduler, while the plugin handles moving the cache between them.

What the official demo shows, and what it doesn't

The full example in the blog launches gpt-oss-120b on a single Trainium3 EC2 instance using the pytorch-inference-sglang-neuron container. The parameters include --tp-size 8, --ep-size 8, --kv-cache-dtype fp8_e4m3 and --context-length 10240, and it is called through the standard /v1/chat/completions endpoint.

That is as far as what can be confirmed goes. The announcement gives no throughput, latency or per-unit cost figures, and no comparison against SGLang running on GPUs. AWS describes Trainium as offering “strong economics for large-scale training and inference”, which is the vendor's own positioning; the material for this release contains no third-party tests to check it against. For which models and feature combinations have been validated, the blog points readers to the repository documentation, and the main text demonstrates only gpt-oss-120b.

What it means for teams in China

SGLang is widely used among teams in China that deploy open-weight models, and when models such as DeepSeek, Qwen and MiniMax launch, SGLang is often among the first inference frameworks to support them. For those teams, the direct benefit of SGLang-Neuron is low migration cost: the launch commands, the calling interface and the prefix-cache mechanism do not need to be relearned, and what changes is mainly the chip and the compilation stack underneath.

The limits are just as practical. The first is where the hardware is: the AWS blog does not say whether Trainium3 instances are offered in the China Regions (Beijing and Ningxia), so teams that want to try it will most likely need to open an account in an overseas region first. The second is model coverage: the example uses a single OpenAI open-weight model, and whether domestic open-weight models run out of the box, or need the plugin to add matching implementations, has to be checked against the documentation one model at a time. The third is performance: with no public benchmarks, teams hoping to cut costs by switching chips will have to run their own load tests and compare the results with the bills from their existing GPU clusters.

Taken together, SGLang-Neuron for now looks more like an extra option for teams that already run in AWS's overseas regions and want to bring inference costs down. Only when performance data appears at the SGLang Summit will it be possible to judge how far this option is from being able to replace part of a GPU fleet.

Sources: AWS Machine Learning Blog, LMSYS Blog, CocoLoop; container version, launch parameters and the example model follow the configuration demonstrated in the AWS blog.