An Agent Can Pass Validation and Still Make an Unsafe Change

While designing AI-assisted operations for SWCraft, I ran into a boundary that MCP alone does not define.

SWCraft is an experimental project I am building: an AUTOSAR software architecture environment where the entire model lives in version-controlled YAML files, deterministic tooling validates the model and generates ARXML, diagrams and documentation from that single source of truth, and an LLM agent operates on the model through MCP tools. It is not a finished product, and this article is not its announcement. The more interesting subject is a problem that any AI-assisted engineering system has to face.

Suppose I ask an agent:

Add interface X to SWC Y.

The agent adds the interface to Y, then notices that SWC Z implements similar functionality for the rear axle. It concludes that the same interface should be added there as well.

From a conventional software engineering perspective, this may look helpful. The agent found an inconsistency and fixed it proactively.

In a safety-relevant architecture model, it is a harmful change.

The agent was authorized to modify Y. It was not authorized to modify Z.

This distinction is easy to lose when an agent has access to a broad set of MCP tools.

Capability Is Not Authority

MCP tells an agent which tools are available and how to call them. It does not, by itself, define which subset of those capabilities is authorized by the current user request.

If the server exposes tools such as:

add_interface
add_port
add_connector
add_runnable
delete_element

the agent can technically invoke all of them.

But the user request may authorize only one narrow operation against one particular component.

This creates an important distinction:

Available capability ≠ task authority

The presence of add_port means that the agent can add a port. It does not mean that the agent may add a port to any SWC it considers relevant.

This is not necessarily a flaw in MCP. Tool protocols operate at the execution interface. The semantic boundary of the user’s task belongs to the system orchestrating the agent.

That system must turn the user request into explicit and enforceable authority.

Why Validation Is Not Enough

A natural response is to validate the model after the agent finishes.

SWCraft already treats deterministic validation as a fundamental part of the architecture. But validation answers a different question:

Is the resulting model structurally and semantically consistent according to the implemented rules?

It does not answer:

Was the agent authorized to make every change contained in this model?

The agent may add a valid interface to Z. The model may still build successfully. All references may resolve. Port compatibility may remain correct.

The change can nevertheless invalidate assumptions outside those checks:

• communication topology may have changed;
• new data propagation paths may have appeared;
• timing or resource assumptions may no longer hold;
• component independence may have been affected;
• requirement traceability may now be incomplete;
• previous safety analysis may no longer describe the model.

A model can therefore be valid and still contain an unauthorized safety-relevant mutation.

Validation and authorization are separate controls. One cannot replace the other.

Defining Authority for a Task

The direction I am exploring for SWCraft is a task-scoped mutation boundary.

When the user submits a request, the system derives an immutable TaskScope before the agent begins executing mutation tools.

For example:

task:
  description: Add interface X to SWC Y

allowed_changes:
  - operation: create_interface
    interface: X

  - operation: add_port
    swc: Y
    interface: X

The agent may still read and analyse other parts of the architecture. It may inspect Z, compare similar components, calculate impact and explain inconsistencies.

But analysis does not grant mutation authority.

If the agent calls:

add_port(
    swc_name="Y",
    interface="X"
)

the operation is allowed.

If it calls:

add_port(
    swc_name="Z",
    interface="X"
)

the operation is rejected before execution.

The agent cannot modify its own TaskScope. Otherwise the same probabilistic component being constrained would also control the constraint.

Checking the Proposed Action Is Not Sufficient

Checking the tool name and its arguments catches the obvious case, but it does not provide a complete boundary.

A tool called for Y may have side effects elsewhere. It may update a connector, change a shared interface definition or modify another model element indirectly.

The system therefore needs to check both the proposed action and the actual result.

The mutation path should look approximately like this:

The important part is the transaction.

The change should not become part of the working model and then be checked afterwards. It should first be applied to an isolated model state. Only an accepted result should be committed.

Semantic Diff, Not File Diff

File boundaries are implementation details. Authorization boundaries belong to the architecture model.

One YAML file may contain several model elements. A legitimate operation may also update multiple files.

For that reason, checking that only an expected file changed is not enough. SWCraft needs to describe the actual model delta in domain terms:

created:
  interfaces:
    - X

modified:
  swcs:
    Y:
      ports_added:
        - interface: X

unchanged:
  swcs:
    - Z

The post-action check can then ask:

actual semantic delta ⊆ authorized task scope

The same check should also be performed against the cumulative delta of the whole task. Several individually allowed tool calls may produce an unauthorized result when combined.

What Happens When the Agent Discovers Another Necessary Change?

The agent should report it, not apply it.

For example:

Adding interface X to SWC Z may also be required for architectural consistency. Z is outside the authorized task scope, so no change was made. Scope expansion is required.

This preserves the useful part of agent intelligence. The agent can still discover relationships and propose improvements.

What it cannot do is convert its own reasoning into additional authority.

The user, an orchestration layer or an established engineering workflow must explicitly approve the expanded scope.

Starting Without Pretending to Have Solved Everything

At the current development stage, I do not need this mechanism to understand every possible natural-language request or prevent a measured percentage of failures.

The first useful version can be deliberately limited:

• recognise explicit SWC and interface names;
• cover the most common mutation tools;
• allow read-only exploration;
• deny mutations that do not match a known task scope;
• request clarification when the scope is ambiguous;
• record every authorization decision;
• verify the semantic delta after every mutation.

Its effectiveness can be measured later against real SWCraft tasks.

The architectural value appears earlier: every mutation now has to cross an explicit authorization boundary. Before that boundary exists, the agent’s authority is implicitly equal to its technical capabilities.

Even incomplete enforcement is useful during development because it gives the system a place where policies, domain rules and evidence can accumulate.

Deterministic Authority Around Probabilistic Reasoning

AI agents are useful precisely because they can reason beyond rigid predefined workflows. They can inspect context, identify relationships and propose actions that were not explicitly encoded in procedural logic.

But reasoning flexibility does not require execution flexibility.

In a critical engineering system, the agent should be free to think broadly while acting narrowly.

That leads to the principle I want SWCraft to enforce:

No mutation of the safety model becomes permanent until its actual semantic delta has been verified against the explicitly authorized boundaries of the current task.

The LLM may decide what to propose.

The surrounding deterministic system decides what is allowed to exist.

If you are building agent authorization boundaries around engineering models — in automotive or elsewhere — I would be interested in how you separate capability from authority. Connect with me on LinkedIn.