Orchestrating SageMaker AI endpoint auto scaling with AWS CDK

Orchestrating SageMaker Endpoint Auto‑Scaling with AWS CDK

Amazon SageMaker real‑time endpoints are a common way to serve AI models, but updating an endpoint’s configuration (instance type, count, or environment variables) can break the auto‑scaling resources managed by CloudFormation. The root cause is that SageMaker requires the scalable target to be deregistered before the endpoint can be modified, while CloudFormation expects the scaling resources to exist throughout the stack update. The result is a cascade of failures, manual workarounds, and sometimes a broken stack.

This article walks through a practical solution: a CDK construct that replaces the native `AWS::ApplicationAutoScaling::ScalableTarget` and `AWS::ApplicationAutoScaling::ScalingPolicy` resources with three custom resources. These resources orchestrate the deregistration, endpoint update, and re‑registration steps automatically, preserving the declarative nature of CloudFormation while eliminating the need for manual CLI commands.

---

Why the Native Approach Fails

When a SageMaker endpoint is created with auto‑scaling defined in CloudFormation, the stack contains two resources:

1. `AWS::ApplicationAutoScaling::ScalableTarget`
2. `AWS::ApplicationAutoScaling::ScalingPolicy`

If you later change the endpoint configuration (e.g., switch to a larger instance type) and redeploy, CloudFormation attempts to update the endpoint while the scalable target is still registered. SageMaker rejects the request with an error similar to:

``` The variant(s) "[AllTraffic]" must be deregistered as scalable targets … before they can be removed or have their instance type updated. ```

Manually deregistering the target via the CLI works for the endpoint update, but CloudFormation then loses track of the scaling resources. Subsequent deployments either skip re‑registration (leaving the endpoint without scaling) or fail with a resource not found error, often leaving the stack in an `UPDATEROLLBACKFAILED` state.

---

Solution Overview: Custom‑Resource Orchestration

The CDK construct introduces three custom resources that wrap the scaling lifecycle:

1. Deregister Scalable Target – Runs before the endpoint update. If no target exists (first deployment), the Lambda handler swallows the error.
2. Register Scalable Target – Executes after the endpoint reaches `InService`. It creates the scalable target.
3. Attach Scaling Policy – Fires once the target is queryable, applying the desired target‑tracking policy.

These resources are linked with explicit `node.addDependency` calls, guaranteeing the correct order. The asynchronous resources use the CDK `cr.Provider` pattern, which provides an `onevent` Lambda and an `iscomplete` waiter that polls every 30 seconds until the operation succeeds or a timeout (1 hour for registration, 15 minutes for policy attachment) is reached.

A single `ConfigHash` property is attached to each custom resource. Any change to the endpoint configuration or scaling parameters changes the hash, forcing CloudFormation to re‑invoke the Lambda functions and thus re‑run the whole orchestration automatically.

Resource Footprint

Although the construct defines only three custom resources, the underlying stack includes:

• 3 custom resources
• 2 Step Functions state machines (one per asynchronous operation)
• ~12 Lambda functions (handlers plus framework‑generated helpers)

You don’t need to maintain the framework‑generated functions; they are created automatically by the CDK provider.

---

Walkthrough: Deploying and Updating an Endpoint

Prerequisites

• AWS account with SageMaker, Lambda, IAM, Application Auto Scaling, and CloudWatch permissions
• AWS CDK v2 installed
• Python 3.12+ and `uv` for dependency management
• A SageMaker model artifact in S3 and a container image in ECR (or a built‑in algorithm)

Steps

1. Clone the sample repository

```bash git clone https://github.com/aws-samples/sample-sagemaker-endpoint-autoscaling.git cd sample-sagemaker-endpoint-autoscaling ```

1. Install dependencies

```bash uv sync ```

1. Configure the endpoint – Edit `config/config.yaml` with your model S3 path, ECR image, instance type, and scaling thresholds. The file groups endpoint and scaling settings together so a single `ConfigHash` drives the orchestration.
2. Deploy the stack

```bash cdk deploy ``` The deployment creates the SageMaker endpoint, the three custom resources, and the supporting Lambda/Step Functions infrastructure.

1. Verify – Check the SageMaker console; the endpoint should be `InService` and auto‑scaling metrics should appear in CloudWatch.
2. Test an update – Modify either the instance type or the scaling thresholds in `config.yaml` and run `cdk deploy` again. The custom resources automatically deregister the old scalable target, update the endpoint, then re‑register and attach the new scaling policy—all in one atomic stack update.

Observations

• No manual CLI steps are required after the initial deployment.
• The stack remains healthy; rollbacks succeed because the custom resources handle missing targets gracefully.
• All scaling changes are version‑controlled alongside the endpoint configuration, simplifying audit trails.

---

Practical Takeaways

• Encapsulate complex update sequences in reusable CDK constructs; this keeps your infrastructure code clean and declarative.
• Leverage `cr.Provider` for long‑running operations that need polling, such as waiting for a SageMaker endpoint to become `InService`.
• Use a hash‑based property to force re‑evaluation of custom resources whenever related configuration changes.
• Monitor the generated resources (Lambda functions, Step Functions) to understand cost and limits, especially in large environments.

By adopting this pattern, teams can confidently evolve SageMaker endpoint configurations and scaling policies without risking stack corruption or manual drift. The approach is fully automated, repeatable, and integrates seamlessly with existing CDK pipelines.