Accelerating software delivery with agentic QA automation using Amazon Nova Act – Part 3

Accelerating Software Delivery with Agentic QA Automation – Part 3

In large‑scale development environments, the sheer volume of automated tests can become a management nightmare. After the initial rollout of a test‑automation platform, dozens of squads start contributing tests, and the organization quickly loses visibility into coverage, ownership, and health. This third installment of the series shows how Amazon Nova Act‑powered QA Studio tackles those challenges with two core capabilities: applications for organizing tests and reporting health metrics, and team‑based user management for fine‑grained access control.

---

1. Organizing Tests with Applications

What an Application Represents

An application in QA Studio is a logical container for the system under test—whether it’s a website, a web app, or a service front‑end. Each application groups together:

• Use cases (individual natural‑language tests introduced in Part 1)
• Test suites (regression collections from Part 2)
• Ownership information (the team responsible for the application)

Creating an application requires only a name, a base URL, and optionally a set of environments (dev, staging, prod). From the application’s detail page, teams can add new use cases, import existing ones, and assemble them into suites—all within the context of that specific system.

Health Metrics at a Glance

Every test execution is logged as a custom Amazon CloudWatch metric, dimensioned by application and environment. QA Studio aggregates these metrics into a concise health view:

• Pass rate for the selected time window
• Total executions and current failure count
• A health score (0‑100) that penalizes flaky tests, preventing a false sense of stability

The Metrics tab expands the view to trends, offering:

• 7‑day hourly or 30‑day daily charts of executions, successes, failures, and average duration
• Environment filters to compare staging vs. production behavior
• A flaky‑test ranking that surfaces intermittently failing use cases
• A recent‑failures list with direct links to Nova Act trajectory logs and session recordings for rapid triage

These dashboards let engineers scan an entire portfolio, identify stable applications, and prioritize those that need attention—without opening each test result individually.

---

2. Controlling Access with Teams and Roles

Team‑Centric Ownership

In QA Studio, an application belongs to a single team. Team members see only the applications they own, keeping the UI uncluttered and preventing accidental cross‑team modifications. This model scales because permissions are applied at the application level rather than on a per‑test basis.

Role Hierarchy

Each team member holds one of five cumulative roles: | Role | Permissions | |------|--------------| | Admin (studio‑wide) | Full access to all teams, applications, and users; can create teams and assign ownership | | Team Lead | Manages team members and creates applications for the team | | Editor | Creates/edits use cases and suites within the team’s applications | | Executor | Runs tests but cannot modify them | | Viewer | Read‑only access to results |

Because roles are cumulative, a Team Lead automatically inherits Editor, Executor, and Viewer capabilities. Access to a specific use case or suite follows the permissions of its parent application, simplifying governance.

Centralized User and Team Views

• Users page – Lists every account with its team and role, filterable by both. Administrators can quickly audit who holds Team‑Lead or Admin rights.
• Teams page – Shows each team’s member count and number of owned applications, giving a high‑level view of growth and workload distribution.
• Applications list – A studio‑wide catalog displaying base URLs, owning teams, test counts, and last execution status, useful for cross‑team visibility when needed.

These overview pages enable ops and leadership to monitor adoption, spot bottlenecks, and reassign ownership before coverage gaps become production issues.

---

3. Sharing Applications Across Teams

Real‑world development rarely fits into isolated silos. QA Studio supports shared ownership scenarios where a platform team maintains an application that multiple feature teams need to test against. Rather than transferring full edit rights, the platform team can grant specific roles (e.g., Executor) to other teams, allowing them to run tests while preserving control over the test definitions. This balances collaboration with accountability and keeps the source of truth centralized.

---

Practical Takeaways

• Structure tests around applications to provide clear ownership, contextual grouping, and health dashboards that scale with the number of tests.
• Leverage the role hierarchy to enforce least‑privilege access while still enabling cross‑team collaboration when necessary.
• Monitor health metrics (pass rate, flaky‑test ranking, recent failures) to maintain confidence in the CI/CD pipeline and to prioritize remediation effort.
• Use the centralized Users, Teams, and Applications pages for governance, audit, and capacity planning across the organization.

By adopting these patterns, engineering organizations can keep their QA automation ecosystem manageable, transparent, and aligned with business ownership—allowing the true benefits of agentic AI‑driven testing to shine through.