A Bug Report That Contains No Architecture

The Model I Cannot Have
The day I released SWCraft 0.19.0, every gate was green: tens of thousands of tests, on Linux and on Windows. That same day I ran the question forms the tool documents, plus the whole-layer reads an agent makes, on two real automotive models, and recorded four findings the gates had not caught.
Two of them are the reason for this article. One answer was sized by the model instead of by the question: asking what a change to the heaviest interface touches returned 60 KB, where a person wanted a paragraph. Another form grew with the long identifiers an import carries. Neither was exotic. Both were correct, by any test I had. The other two had nothing to do with shape: a recipe that was missing, and an import refused as designed.
Neither showed up in my test data, and the reason took me a while to accept. My fixtures are small and tidy, with short names. Even my large benchmark, with 150 components, 1,450 connectors and a 550-port hub taken from one real model’s proportions, stayed shallow, so the same summary came to 12 KB. Matching one large number was not enough. Real models differ in how deep their dependencies run, how long their identifiers are and where the weight sits: one had a component with 550 ports, the other an interface used by 597. Those are the shapes that break things.
So the defects that matter depend on the shape of a model, and the models with the shape belong to someone else. A real architecture cannot be attached to a bug report. It should not be. The file is confidential, the names are confidential, and often the fact that a given function exists is confidential too.
That leaves a tool vendor with a familiar sentence: it fails on my project, and I cannot show you my project.
What a Defect Actually Needs
Look at what those failures depended on. Not on the name of any component. Not on what any signal means. Not on any requirement’s text. They depended on structure, and much of structure is numbers: how many ports the largest component has, how heavily the worst interface is shared, how long the identifiers are, how many ports nothing connects to.
Numbers like these can leave the building, if they are the only thing that leaves.
That is the idea behind the model passport: one command, swcraft passport, reads a project and writes a small JSON file that states its shape. Counts, distributions and lengths. Here is part of one, from a synthetic model of mine:
"components": {
"by_declared_level": { "A": 22, "B": 23, "C": 9, "D": 12, "QM": 24 },
"hubs": [60, 40],
"ports": { "top": [60, 40, 4, 4, 4, 4, 4, 4], "total": 314 },
"runnables": 90,
"unconnected": { "max_on_one": 1, "total": 6 }
}
There are no element names, no values, no requirement text. You can read every number before you send it, because it prints to your terminal or to a file you choose. That matters more than any assurance of mine. The tool also knows, in memory, which component has the most ports, because a test needs that. That answer is never written out.
From Numbers Back to a Model
A passport on its own is only a description. To reproduce a defect I need a model I can run the tool on.
That is what Carchetype does. It is a separate, deterministic generator of synthetic automotive architectures. Give it a passport, and it writes a complete SWCraft project with the stated properties: a 550-port hub, a heavy interface, the same distribution of safety levels. It knows nothing about the source, because there is nothing about the source to know. The components have invented names, in invented domains.
The two tools are tied together only by the passport. SWCraft never imports Carchetype and does not call it by name; it invokes a configured generator command. After generating, SWCraft computes the passport of the result and compares it with the input leaf by leaf. If the numbers do not match, the model is not used.
The most important rule is the unglamorous one: a passport the generator cannot meet is refused, never approximated. An approximate model would be worse than none. It would pass or fail for reasons nobody could explain, and I would believe it.
Known Data, Checkable Answers
Generated data solves a problem I had not fully seen until this work, and it is bigger than the confidentiality one.
A test needs an expected answer. For a function that answers questions about a model, where does that answer come from? If it comes from the function under test, or from code that shares its logic, the test only proves the code agrees with itself.
A scenario in SWCraft’s functional suite names a specimen, the model generated from a committed passport. It drives one function through its real entry point, the real command or the real tool an AI agent calls. It compares the answer with what is computed from the passport and from the model’s files, never from the implementation. Each scenario is one JSON file stating one model and one question, which is the shape a bug report should have anyway.
This also gave me something I lacked: a catalogue with one row per product function, showing which functions no scenario of this kind reaches. Before, I could not have said that. I could only have counted tests.
The four-step path is simple:
- Model passport. The user writes it on their machine, reads it, and sends it.
- Carchetype specimen. A synthetic model with the stated shape.
- Reproducible scenario. The defect appears on a model anyone may see.
- Regression test. The specimen’s passport and the scenario are committed, and the defect stays fixed.
After step 1, nothing touches the original architecture, and nothing in step 1 leaves the owner’s machine without being read. The generated model is not even stored in the repository: the configuration that produces it is committed, and the model is regenerated on demand.
A Measurement That Means Something
There is a second benefit I did not plan for: a fixed specimen is a fixed ruler.
This week I fixed the way SWCraft loads a model, which was reading each file about twelve times per run. On a real workstation the check took long enough to notice. I measured it on the same large specimen, identified by its digest, on two very different machines. On a Windows runner, check-model went from 8.3 seconds to 2.3. On a Mac, from 2.3 seconds to 0.7. Because the data is identical by construction, those numbers compare like with like.
What This Does Not Do
A passport carries a shape, and only a shape. That limit is the point, and it is also a real limit.
It will not reproduce a defect that lives in the content: a broken definition in an imported file, a value out of range, two names that collide. It cannot help if the problem is in an export file rather than in the model, because the passport never sees the file. For a passport the generator cannot meet, I get a refusal and no specimen. A generated model is also not evidence about a real architecture’s quality; it tells you how the tool behaves on that shape, nothing more. Real models stay a calibration set for the chain, never an oracle.
A passport also records distributions, not the wiring. Two models with the same passport can be wired differently, and a defect of a wiring the numbers do not describe is not regenerated. The 60 KB answer is the warning here: it came from how deep the dependencies ran, and a passport records no depth.
A passport is also not free of information, and it cannot be. A file that said nothing about a model could not reproduce anything, so the shape has to stay. Everything else stays out: no names, no values, no text, no paths. What is left still shows how large a model is and how its safety levels are distributed, and for some projects that alone may be sensitive. Whether those numbers may leave is the owner’s decision, which is why you read the file before you send it.
A green functional scenario says the answer is right. It does not say an agent will ask the right question. That needs a different kind of test.
Why I Like This
Most ways of handling confidential data in a bug report ask someone to trust something: an anonymiser, a sanitising script, a vendor’s promise. Anonymised models keep more than people think, and you have to hope the tool found everything.
A passport turns that around. It does not remove the sensitive parts from your architecture. It never contained them, and the file is short enough to read in a minute.
If you work on tools for models you are not allowed to see, I would be curious how you solve this. And if your architecture breaks SWCraft, you now know what to send me: not your architecture.