Everyone Is Building an AI AUTOSAR Toolchain. But How Should We Evaluate One?

Apparently, everyone is building an AUTOSAR toolchain now.

The recipe is becoming familiar. Load a set of requirements into an AI-assisted environment. Describe an architectural change in natural language. Let an agent produce a diagram, ARXML, source code, traceability links, and perhaps a few tests. End with a slide showing how much engineering time this could save.

To paraphrase Futurama:

I’m going to build my own AUTOSAR toolchain — with AI agents and generated artifacts.

Over the last few weeks, I have reviewed several proposals built around some version of this idea. Some of the demonstrations were genuinely impressive. The systems could navigate complex information, make coordinated changes across multiple artifacts, and expose engineering operations through a much more accessible interface.

But the same jump kept appearing in the discussion:

The system can generate the artifacts. Therefore, it can transform the development process.

That conclusion may eventually be correct. The demonstration does not establish it.

The missing part is not another agent, another prompt, or another generated artifact. It is a methodology for deciding what the demonstration actually proves.

That gap is why I started working on an experimental project called a3se, short for AI-enabled AUTOSAR Software Engineering Assessment. The name uses three A terms — AI, AUTOSAR, and assessment — followed by SE for software engineering. The project is not a finished product, and this article is not its announcement. The more important subject is the problem that made the project necessary:

How should we evaluate a system that claims to be an AI-enabled AUTOSAR toolchain?

Generating Files Is Not the Development Process

AUTOSAR development certainly produces files. But file generation is not the unit of engineering value.

The actual process is a chain of controlled transitions:

Every transition has conditions attached to it.

Which input is authoritative? Who owns the transformation? Which rules must hold before the result is accepted? How is identity preserved across artifacts? What happens when a reference cannot be resolved? How does a requirement change propagate into architecture, code, and tests? Which evidence allows somebody else to reproduce the decision later?

Introducing an LLM into the chain does not remove any of these questions. It adds new ones.

The result may now depend on the model and provider, the model version, system instructions, retrieved context, conversation history, enabled tools, permissions, repository state, and human approvals. If an agent is also allowed to create or modify generators, validators, build scripts, or pipelines, it is not merely using the engineering system. It is changing the system that must be evaluated.

A plausible ARXML file at the end tells us very little about that larger configuration.

First Define the Claim

The phrase “AI-enabled AUTOSAR toolchain” is too broad to assess.

One system may provide read-only navigation over an existing architecture model. Another may propose changes for human review. A third may directly modify authoritative architecture sources, regenerate ARXML and C code, update traceability, and prepare verification evidence.

These are not variations of the same capability. They carry different authority, failure modes, and assurance burdens.

Before evaluating a system, I would want its claim reduced to explicit engineering layers and operations:

• Which parts of the V-model are in scope?
• Can the system open, create, edit, delete, validate, generate, build, test, and export?
• Which operations are deterministic and which depend on a model response?
• Which artifacts can the AI read, propose changes to, or modify directly?
• Which use cases are production claims, and which are only experimental conveniences?

An architecture assistant should not fail because it does not generate production code it never claimed to generate. Conversely, a requirements-to-code system should not receive credit for the complete chain because one generated ARXML sample looked reasonable.

The assessment boundary must follow the claim.

The System Under Test Is Larger Than the Model

The system under test (SUT) is larger than the AI model. It is tempting to evaluate the model alone: give it prompts, inspect its answers, calculate a success rate.

That is useful for model evaluation, but it is not enough for engineering-system evaluation. The SUT and its operating context are closer to this:

The downstream engineering environment may sit outside the SUT boundary, but it remains part of the assessment context. Import and integration behavior can provide external acceptance evidence that the SUT cannot provide for itself.

A different prompt can change the result. So can a different tool definition, an unrecorded conversation, a stale local file, a permission change, or a script generated five minutes earlier by the same agent.

The assessment boundary must therefore include the complete operational configuration in which the AI acts. Otherwise, we test a visible demonstration while leaving its effective engineering state outside the boundary.

Six Questions a Demonstration Must Survive

I keep returning to six properties that separate an interesting generator from an engineering system.

1. Where does engineering truth live?

Requirements, architecture elements, interfaces, data types, traceability relationships, and configuration decisions need an explicit authoritative source.

Chat history cannot be that source. Neither can transient agent memory or a diagram that cannot be traced back to a versioned model.

Natural language may be a convenient way to propose a change. After acceptance, however, the resulting engineering facts must exist in inspectable, version-controlled artifacts with defined ownership.

2. Who decides that the result is correct?

An AI-generated artifact is a claim until it passes an independent acceptance mechanism.

For ARXML, well-formed XML and schema validation are necessary but insufficient. The result also has to satisfy AUTOSAR semantics, project-specific constraints, reference integrity, supported-scope rules, and the expectations of the real downstream toolchain.

The agent saying “validation passed” is not evidence. A validator generated on demand by the same agent and immediately trusted is not meaningfully independent either.

Acceptance needs predefined, versioned checks whose verdict does not depend on the model that produced the candidate.

3. Can the workflow be reproduced?

A production claim should survive a clean checkout.

An engineer who did not build the demonstration should be able to identify the authoritative sources, run the documented workflow, inspect the semantic changes, reproduce the accepted outputs, and diagnose a failure without reconstructing the process from somebody else’s chat history.

This does not require every generated file to be byte-identical. Some toolchains legitimately produce timestamps, reordered XML, or other irrelevant variation. It does require a predefined equivalence rule and a repeatable engineering result.

4. What happens when something goes wrong?

Positive demonstrations show the system receiving a well-formed request in a prepared environment.

Engineering assessments also need incompatible ports, missing references, unsupported constructs, stale traceability, unavailable validators, failed tool calls, exhausted model quotas, and network outages.

The desired outcome is not that the agent always finds a way forward. Sometimes the correct behavior is to stop.

A system must distinguish success from limitation. It must not silently repair an unsupported case, invent a missing reference, skip a failed validator, or convert unavailable evidence into a plausible answer.

5. Can the SUT generate and validate artifacts without the LLM?

For every claimed mandatory engineering operation, I would require a documented path that an operator can invoke with generative-model services and agent services unavailable. I refer to this as the no-LLM path.

That includes authoritative editing, validation, generation, build, test, and export wherever the SUT claims those operations as part of its scope. The path may use a structured editor, a versioned command-line tool, a typed API, a compiler, a rule engine, or another deterministic engineering tool. It must not require chat history, prompt wording, agent memory, or an on-demand model response.

This requirement does not make AI irrelevant. Chat may still be the faster and more accessible interface. The agent may interpret intent, propose a change, select tools, and explain the result. But an equivalent path to the same authoritative engineering state, governed by the same acceptance criteria, must remain available outside the conversation.

If the only path to ARXML generation is a request to an agent, or validation consists only of asking the agent whether the model is valid, the demonstration has not yet established an engineering toolchain. It has established an AI-agent harness around an engineering claim.

This is also a practical outage test. When the LLM service is unavailable, optional assistance may disappear. Authoritative engineering operations, access to accepted artifacts, and deterministic verdicts should not disappear with it.

6. Can the process outlive the demonstration?

Useful prototypes quickly become dependencies. Then ownership, maintenance, continuity, and recovery become part of correctness.

Who owns a generated parser or validator? Where is its specification? How is it regression-tested? What happens when the original author leaves, the model provider changes, or the proprietary AI service is unavailable?

If the workflow depends on undocumented local state or knowledge held only by the people who prepared the demonstration, the organization does not fully possess that workflow. It rents it from a particular runtime configuration and from the people who know how to reconstruct it.

A Possible Assessment Pattern

Once the claim, the SUT boundary, and the assessment context are explicit, the SUT can be treated as a black box and tested with inputs and oracles controlled by the assessor.

The basic pattern looks like this:

The reference should not be a bag of example files. It should describe typed engineering facts across the relevant layers: requirements, architecture, interfaces, behavior, code, tests, and traceability. For each assessed system, only the subset applicable to its declared scope would be selected.

The SUT developer should remain responsible for transferring those facts into any proprietary internal format and for documenting the operating procedure. That transfer is part of the claimed workflow, not something the assessor should silently solve with a custom adapter.

Before assessment execution begins, an input checkpoint must establish which reference facts actually reached the system. Otherwise, a missing output could be caused by a defective generator, a lossy import, or a fact that never entered the system at all. An assessment that cannot distinguish these causes is not very useful.

The exported outputs can then be normalized and compared semantically with the expected result. Names and serialization may differ. The engineering facts should not.

Test What the System Rejects

A valid reference project tests whether the system can reproduce an expected result. That is necessary, but it creates an easy path to a misleading success.

The assessment also needs deliberately difficult input classes.

Negative cases introduce invalid or incompatible engineering facts. The expected result is a controlled rejection or explicit limitation, not a plausible artifact that hides the problem.

Mutation cases seed a known defect into an otherwise valid project. They test whether the acceptance process detects the defect before the output is trusted.

Change-impact cases begin with a valid baseline, apply one defined change, and compare the expected and actual consequences across all claimed layers. Adding an interface should update the intended architecture and downstream artifacts without unrelated collateral changes. Removing a requirement should not leave stale traceability behind.

Continuity cases remove access to model services, chat history, or network connectivity, and repeat the workflow without the original operator. They test whether the underlying engineering process still exists outside the demonstration environment.

This is where the distinction becomes visible:

A generator is demonstrated by what it produces. An engineering system is characterized just as much by what it rejects, preserves, reproduces, and proves.

A Correct Artifact Is Still Not the Whole Result

Suppose the generated ARXML is semantically correct and imports into the downstream tool without errors.

That is strong evidence for the artifact. It is not automatically evidence for the complete process.

The artifact may have been produced from an unrecorded prompt. A human may have repaired it manually without that intervention being captured. A generated validator may contain the same mistaken assumption as the generator. The traceability report may refer to an earlier baseline. The workflow may be impossible to repeat without the person who prepared the demonstration.

For that reason, a single score would be misleading. The result should stay bounded by the assessed configuration, layers, operations, tool versions, evidence, and known limitations.

It should also separate different questions:

• Were the artifacts semantically acceptable?
• Was the process controlled and reproducible?
• Did the AI remain within its declared authority?
• Did failures stop safely?
• Can another engineer operate and maintain the system?

A system may perform well in one dimension and fail in another. Hiding that behind an overall percentage would lose the information an engineering decision actually needs.

What This Would Not Prove

Even a rigorous assessment of this kind would not automatically provide an Automotive SPICE assessment, functional-safety approval, cybersecurity approval, or tool qualification.

Those decisions have their own scope, authorities, and evidence requirements.

The narrower goal is still valuable: replace feature comparisons and curated demonstrations with a repeatable answer to a more disciplined question.

Under which declared boundaries and controls did this system produce acceptable results, and what evidence supports that conclusion?

That is a much less exciting sentence than “requirements to production code in ten minutes.”

It is also a sentence an engineering organization can act on.

Why I Started Building the Instrument

The current wave of AI-enabled AUTOSAR proposals is not a reason to reject the technology. It is a reason to improve how we evaluate it.

Natural-language interaction can make complex tools more accessible. Probabilistic reasoning can help engineers navigate large information spaces, assemble context, propose changes, and investigate inconsistencies. I use these capabilities myself, and I think they can remove substantial friction from engineering work.

But the easier artifact generation becomes, the more important independent acceptance becomes. When almost anyone can produce a plausible architecture diagram, ARXML fragment, generator, or test report, the differentiating capability is no longer generation alone. It is the engineering system around it.

That is the idea behind a3se. I have started with a trusted reference and deterministic validation primitives, but the complete assessment workflow still has to be built and tested against real, structurally different systems. The project may confirm the methodology, expose gaps in it, or force some of the proposed gates to change.

That uncertainty is part of the point. The criteria themselves should be tested, not merely declared.

Until then, I will remain interested in the AI AUTOSAR toolchains appearing in demonstrations. I just want a better way to tell which of them are engineering systems, which are useful assistants, and which are still theme park concepts.

If you are building or evaluating AI-enabled tooling for AUTOSAR development, I would be interested in comparing assessment criteria and failure cases. Connect with me on LinkedIn.