Multi-Turn Reinforcement Learning on Amazon SageMaker AI: A hands-on guide

Introduction

Multi‑turn reinforcement learning (MTRL) lets you teach an AI agent to solve tasks that require a sequence of decisions, where only the final outcome provides a clear reward signal. Amazon SageMaker AI, together with Amazon Bedrock AgentCore, offers a serverless way to run MTRL at scale. This guide walks you through a hands‑on implementation using the popular Wordle game as a teaching example, illustrating the end‑to‑end pipeline: deploying the agent, preparing data, launching training, monitoring with MLflow, and evaluating the trained policy.

1. Architecture Overview

The system is split into two logical halves:

• SageMaker AI hosts the training job and the policy model (the model that decides the next action). It also provides an OpenAI‑compatible endpoint for inference.
• AgentCore runs the agent, the Wordle environment, and the reward function. A single Python file deployed to AgentCore contains the game logic, a tool (`guess_word`) that calls the policy model, and a scoring routine that returns a single reward at the end of each episode.

During each rollout, SageMaker AI sends a secret word to AgentCore. The agent repeatedly calls the policy model to generate guesses, receives per‑letter feedback, and continues until it either solves the puzzle or exhausts six attempts. After the episode finishes, AgentCore returns a scalar reward (positive for a fast solve, negative for a loss). The reward is then used by Group Relative Policy Optimization (GRPO) to update a LoRA adapter on the base model.

2. Preparing the Environment

Prerequisites

• Python 3.10+, `uv`, Node 20+, and the AgentCore CLI (`npm install -g @aws/agentcore`).
• `botocore` ≥ 1.43.37 on the machine that will launch the job.
• Two IAM roles: one for SageMaker AI and one for the AgentCore runtime (the latter trusts `bedrock-agentcore.amazonaws.com`).
• An S3 bucket in the same AWS Region where the training job will run.

Deploying the Agent

1. Write a single Python script that defines:

- The Wordle environment (secret word list, feedback logic). - The `guess_word` tool that forwards a prompt to the SageMaker endpoint. - The reward function (e.g., +1.2 for a solve in ≤3 guesses, –0.5 for a loss).

1. Use the AgentCore CLI to package and deploy the script:

```bash agentcore deploy --runtime-name wordle-mtrl --source ./agent.py ``` The CLI creates the runtime role automatically and returns the ARN needed for the training job.

3. Building the Dataset

MTRL requires only the secret word for each episode. Create a CSV or JSONL file with a single column, `secret_word`, containing a large list of five‑letter words (the open‑source SCOWL list works well). Upload the file to the S3 bucket you created earlier. No labels or action trajectories are needed because the reward is computed after the episode completes.

4. Launching the Training Job

Configure the job with the following key parameters:

• Base model – e.g., `gpt-oss-20b` (lowest token price and controllable reasoning effort).
• Region – US East (N. Virginia) or US West (Oregon) where both SageMaker and AgentCore are available.
• Token dimensions – SageMaker tracks pre‑fill, sample, and train tokens. These metrics appear in `DescribeJob` under `BillableTokenUsage`.
• GRPO settings – number of rollouts per secret word, learning rate, and LoRA rank.

Submit the job via the AWS SDK or CLI: ```bash aws sagemaker create-training-job \ --training-job-name wordle-mtrl-run \ --algorithm-specification TrainingImage=arn:aws:sagemaker:...:algorithm/mtrl,TrainingInputMode=File \ --role-arn arn:aws:iam::123456789012:role/SageMakerMTRLRole \ --input-data-config ChannelName=training,DataSource={S3DataSource={S3Uri=s3://my-bucket/wordlist.csv}} \ --hyper-parameters '{"basemodel":"gpt-oss-20b","agentarn":"arn:aws:bedrock:...:agentcore/wordle-mtrl"}' ``` The job will invoke AgentCore for each rollout, collect rewards, and update the LoRA adapter.

5. Monitoring and Debugging

• MLflow – Enable the MLflow tracking server in the job configuration to log metrics such as average reward, solve rate, and token usage per epoch.
• CloudWatch – AgentCore runtime emits CPU, memory, and duration metrics. Use these to estimate the additional $79 runtime cost observed in a 12‑hour run.
• Common pitfalls – Reward shaping that is too aggressive can lead to premature convergence; insufficient rollout diversity may cause over‑fitting to a narrow set of words.

6. Evaluating the Trained Policy

After training completes, deploy the LoRA‑adapted model to a SageMaker endpoint. Run a held‑out Wordle set and compute:

• Solve rate (percentage of words solved within six guesses).
• Average guesses per solved word.
• Cost per inference (prefill + sample tokens).

In the reference run, the solve rate improved from 61 % to 84 % at default reasoning effort, and from 15 % to 69 % when the model was constrained to low‑effort sampling, for a total token cost of roughly $339.

7. Cost Summary

| Dimension | Tokens | Price per 1M | Approx. Cost | |-----------|--------|--------------|--------------| | Prefill | 491 M | $0.12 | $59 | | Sample | 160 M | $0.30 | $48 | | Train | 643 M | $0.36 | $232 | | Total | 1.29 B | — | $339 |

Add the AgentCore runtime charge (~$79) for a total of about $418 for a 12‑hour training run.

Conclusion

Multi‑turn reinforcement learning on SageMaker AI provides a powerful, serverless pathway to teach custom agents for complex, sequential tasks. By isolating the environment and reward in AgentCore, you keep your data private while leveraging AWS’s managed training infrastructure. The Wordle example demonstrates the full workflow—from deployment and dataset preparation to training, monitoring, and cost analysis—so you can adapt the same pipeline to internal workflows such as expense reporting, ticket triage, or any domain where success is measured only at the episode’s end.