Production MLOps at GS Caltex with Amazon SageMaker Canvas, part 1

Production MLOps at GS Caltex with Amazon SageMaker Canvas – Part 1

GS Caltex, a South Korean energy and chemical firm, needed a way for its process engineers and ML engineers to collaborate without forcing engineers to write code. The solution combines Amazon SageMaker Canvas – a visual, no‑code environment for data preparation and model building – with Amazon SageMaker Pipelines for production‑grade orchestration. By automating the handoff between the two services, the company reduced model‑deployment lead time from roughly a week to a few hours and cut inference costs by moving from always‑on endpoints to 30‑minute batch jobs.

---

1. From Canvas to Production: Roles and Assets

| In SageMaker Canvas | In Production (SageMaker Pipelines) | |--------------------------|------------------------------------------| | Data preparation using Data Wrangler flows | Pre‑processing jobs executed via SageMaker Processing | | Trained model and evaluation metrics | Model version stored in SageMaker Model Registry | | Visual validation of predictions | Approved model deployed and scheduled for batch inference |

Process engineers focus on domain knowledge: they explore variables, clean data, and assess whether model outputs are useful. Their work lives in Canvas as Data Wrangler flows and a registered model.

ML engineers take those assets, configure the execution environment, and embed them in a repeatable pipeline that handles data ingestion, transformation, model inference, and result storage.

---

2. Architecture Overview

1. SageMaker Canvas – creates a Data Wrangler `.flow` file and registers the model in the Model Registry.
2. S3 – stores raw input data, the exported `.flow` file, and batch inference results.
3. SageMaker Processing – runs the exported flow in a container, reproducing the exact transformations validated by the process engineer.
4. SageMaker Pipelines – stitches together preprocessing, Batch Transform inference, and post‑processing steps.
5. Amazon EventBridge – triggers the pipeline every 30 minutes for scheduled batch inference.
6. CloudWatch – provides logs and step‑level status for monitoring and troubleshooting.

The pipeline reuses the same preprocessing logic and model artifact generated in Canvas, ensuring consistency between development and production.

---

3. Key Implementation Steps

a. Export and Reuse the Pre‑processing Flow

• In Canvas, export the Data Wrangler flow to SageMaker Processing (disable auto‑job configuration).
• Record the container image URI, the S3 URI of the `.flow` file, and the execution role.
• In the pipeline, add a ProcessingStep that points to the same container image and `.flow` file, preserving learned transformation parameters (e.g., scaling factors).

b. Model Approval and Registration

• After Canvas registers the model, ML engineers review the model artifact and approve it in the Model Registry.
• The approved version becomes the source for a Batch Transform step.

c. Build the Pipeline

```python from sagemaker.workflow.pipeline import Pipeline from sagemaker.workflow.steps import ProcessingStep, TransformStep

preprocess = ProcessingStep( name="Preprocess", processor=processor, # configured with the exported container image inputs=[ProcessingInput(source="s3://bucket/raw", destination="/opt/ml/processing/input")], outputs=[ProcessingOutput(source="/opt/ml/processing/output", destination="s3://bucket/processed")], job_arguments=["--flow", "s3://bucket/flow/myflow.flow"] )

transform = TransformStep( name="BatchInference", transformer=transformer, # points to the approved model version inputs=TransformInput(data="s3://bucket/processed", contenttype="text/csv"), outputpath="s3://bucket/inference" )

pipeline = Pipeline(name="GSCaltexMLOps", steps=[preprocess, transform]) ```

• The pipeline definition is version‑controlled and can be invoked programmatically.

d. Schedule with EventBridge

• Create an EventBridge rule with a *cron(0/30 ? )* expression.
• The rule’s target is the `StartPipelineExecution` API call, passing the pipeline name and any required parameters.

e. Monitoring and Failure Handling

• CloudWatch logs capture step‑level output; alarms can be set on error metrics.
• The pipeline’s step‑status is stored in the SageMaker Execution History, enabling quick rollback to a prior model version if needed.

---

4. Benefits Realized

| Metric | Before Automation | After Automation | |--------|-------------------|------------------| | Model‑deployment lead time | ~7 days (manual handoff) | < 4 hours (pipeline‑driven) | | Inference cost | Continuous endpoint usage | 30‑minute batch jobs, ~70 % cost reduction | | Engineer effort | Duplicate configuration work | Single source of truth (Canvas assets) | | Traceability | Ad‑hoc scripts | Full execution history in SageMaker Pipelines |

By reusing the exact preprocessing flow and model artifact, GS Caltex eliminated drift between development and production, while the scheduled batch approach matched the business’s need for periodic predictions without the overhead of always‑on endpoints.

---

5. Practical Takeaways

• Leverage Canvas for rapid prototyping – domain experts can iterate on data and models without writing code.
• Export flows as immutable artifacts – treat the `.flow` file and container image as versioned code.
• Use Model Registry as the single source of truth – approvals, versioning, and deployment all flow through it.
• Schedule pipelines with EventBridge – simple cron‑style triggers replace custom scheduler scripts.
• Monitor end‑to‑end – CloudWatch and SageMaker Execution History give visibility comparable to traditional CI/CD pipelines.

The next post will dive deeper into advanced topics such as dynamic input routing, multi‑region replication of the pipeline, and integrating custom post‑processing logic.

---

End of Part 1