Evaluate first, then ship

01

Overview: why evaluation comes before capability

This paper is Brello Research’s plan for AI evaluation before release: how we intend to test Brello Super Intelligence (Brello SI), which is in development, before each step up in capability.

We intend to test each new capability before anyone relies on it, and to hold a release until it meets criteria written down before testing began. The Brello Charter (version 1.0) makes this a commitment: new capability ships with its evaluations. This paper is the detailed plan behind the evaluation commitments summarised on our safety page.

The order matters because of what Brello SI is being designed to do: deep reasoning, long-term personal context and real work done on a person’s behalf. Each raises the cost of a mistake. An assistant that only answers can mislead. One that remembers can reveal something. One that acts can do something that can’t be undone. Testing after release measures harm that has already happened.

Privacy adds a second reason. Brello 1.0 has no server, so we never see its conversations, and the sealed compute we’re designing for Brello SI is intended to keep nothing and to allow no human access. That rules out the usual way of finding problems after launch, which is watching real use. More of the evidence has to be gathered before release.

We also want the bar fixed before results arrive, because criteria written after seeing the numbers tend to drift towards whatever the system already does. The structure borrows from the NIST AI Risk Management Framework, which organises the work into four functions: govern, map, measure and manage.1 We use it as a structure, not as a claim of conformance. Shevlane and colleagues argue that evaluations are critical for making responsible decisions about model training, deployment and security.2 Ours are intended to decide whether a release ships at all.

02

What Brello 1.0 already does

Brello 1.0, a prototype Android app that runs language models entirely on the phone, already runs eight safeguards. Six are checks written in code; two are instructions to the model, which it may not follow. None of them makes a small model correct. Each narrows one specific way a reply can go wrong.

The safeguards below are as shipped in version 1.0.0 (4 October 2026), and they are the eight listed on our safety page. They are not the gates in section 5, which are designed for Brello SI, but they shaped how we think about the larger system. Figure 1 places each one where it acts, from downloading a model to finishing a reply.

• Storage and memory checks. Before a download, Brello requires free space for the model, its speed cache and 300 MB of headroom, or shows “Not enough space” with exact numbers. On a phone with less memory than the model needs, it warns “It may run slowly or fail to start.” The person can still choose “Download anyway”, so the warning informs the choice rather than blocking it. How Brello matches a model to the phone’s memory is described in Fitting a model to the phone in your pocket.
• Crash-loop recovery. If a model crashes the app while loading, usually because the phone ran out of memory, Brello remembers. On the next launch it switches to another installed model and says why: “{Model} couldn’t start on this phone · Using {Other}”. This stops a crash loop. It doesn’t make the larger model fit.
• Asking before searching. Web search is off by default. When a question looks time-sensitive, with words such as “latest”, “today”, “price” or “score”, Brello pauses and asks: “Search the web for this?” The person chooses “Search the web” or “Answer offline”, and a question with a photo never triggers a search. If the person chooses to search, the search text goes directly from the phone to a search engine and up to four result pages are requested from their sites, which see a normal web request, including the phone’s IP address. The trigger is a list of words, so a question that needs fresh facts but uses none of them is answered offline. The design is described in Asking before going online.
• Uncertainty instruction. The system prompt is short on purpose, and it ends: “If you are not sure about something, say so instead of guessing.” This is an instruction, not a check, and it doesn’t guarantee that a small model will admit uncertainty. Small models copy the shape of their instructions explains why the prompt is short.
• Inline citations. When web results are present, the prompt adds: “Base your answer on them and cite the sources you rely on inline using their numbers, like [1] or [2]. If the results do not answer the question, say so briefly and answer from general knowledge.” Each citation in the answer becomes a tappable link to its source page. Asking for citations is an instruction; turning them into links is code. A citation shows which page the model says it used, not that the page supports the sentence. The retrieval pipeline behind these results is described in Answering from the open web, without a server.
• Repetition stopper. Every 48 characters, Brello checks whether the end of the reply is looping: a block repeated three or more times over at least 120 characters. If it is, Brello keeps the first copy and stops the model. The stopper catches repetition, not other kinds of wrong answer.
• Leaked-markup clean-up. Stray runtime tokens such as <think>, <|im_end|>, <end_of_turn>, <eos> and /no_think, and Gemma “channel” markers, are removed or routed to the “Thought process” panel.
• Empty replies. If the model reasoned but never answered while Think harder was off, the reasoning becomes the answer. A reply with no text shows “No response was generated. Try rephrasing.” instead of an empty space.

Several of these safeguards are covered by the app’s automated tests, alongside a live end-to-end test of web research. The lesson we carry forward is about where a safeguard lives. A check written in ordinary code does the same thing on every reply. An instruction in the prompt works only as well as the model follows it, which is why Figure 1 marks the two instructions differently from the six checks. Each finished reply also carries a meta line, such as “2.4s · On-device”, that records the time taken and whether the web was used, and the reply records which model wrote it. That line reports how an answer was made, not whether it is right, so we don’t count it as a safeguard. The Brello 1.0 system card gathers the prototype’s specifications in one place.

03

Six evaluation areas we intend to cover

We intend to evaluate Brello SI in six areas, each a plain question that a person using Brello would care about, and each re-tested with every release. For a general introduction to these methods, see AI safety evaluations, explained.

3.1 Capability on real tasks

This area asks whether Brello SI does the work people bring to it, at the quality it implies. Public benchmarks drift away from real use, and a published test set can end up in later training data, where models can memorise it.3 We intend to build task sets around the work Brello SI is being designed for, such as research with sources, writing and multi-step tasks. Where an answer can be checked automatically, it will be. Otherwise people will review it. Because what someone shares with Brello is used only to answer them, these task sets will be written for the purpose, never taken from conversations.

3.2 Honesty and calibration

This area asks whether Brello SI says it isn’t sure when it should, and only then. A calibrated system’s confidence matches its accuracy. Modern neural networks are often poorly calibrated,4 though large language models have been found to be well calibrated on some question formats, and can be trained to predict whether they know an answer.5 Whether that holds for Brello SI, in real use, has to be measured. We also intend to check that each citation supports the sentence it’s attached to. Our explainer Why AI makes things up covers the underlying problem.

3.3 Privacy leakage

This area asks whether the system reveals personal information it shouldn’t. Not training on conversations closes one route, since a model can’t memorise what it was never trained on. The routes that remain run through context: what Brello SI will carry into a task, and where it will send it. A search query could carry more of a question than it needs. A drafted message could include a detail from someone’s notes that the recipient shouldn’t see. The risk is measured, not hypothetical: in tests built on the theory of contextual integrity, Mireshghallah and colleagues found that two of the models they tested revealed private information in contexts where people would not, 39% and 57% of the time.6

We intend to test for leakage with privacy audits that borrow the canary method from memorisation research, which plants made-up secrets and then measures whether they come back out.7 Our audits will seed synthetic profiles with distinctive, made-up personal details, then search every output and every outbound request for them.

3.4 Misuse and harm

This area asks whether someone could use Brello SI to hurt other people, or themselves. The main tool here is red-teaming: deliberately trying to make a system behave badly. People can do it at scale. Ganguli and colleagues collected 38,961 red-team attacks on language models and released them for others to study.8 Models can do it too, with one language model generating test cases for another.9 Because we won’t see how Brello SI is used, misuse has to be looked for before release, not discovered after it.

3.5 Robustness to indirect prompt injection

This area asks whether Brello SI follows instructions planted in the text it reads. A web page can contain text aimed at the model rather than the reader. Section 4 covers this area in detail, because it shapes how we’re designing the whole retrieval path.

3.6 Safety of actions

This area asks whether every consequential action waits for the person. Anything that spends money, sends a message on someone’s behalf or deletes something will need their confirmation. We intend to test that this always holds: the confirmation appears every time, shows exactly what will happen (who receives what, how much is spent, what is deleted), and can’t be bypassed by any phrasing or by any text on a web page. We’re designing confirmation as a check outside the model, so a model that has been misled still can’t complete a consequential action on its own.

3.7 How evidence will be gathered

No single method covers all six areas, so each will draw on several. In Table 1, primary marks the evidence a release decision rests on, and supporting marks a second, independent check.

Staged rollout is never a primary source of evidence in this plan, and the reason is privacy. We won’t see conversations, so early access will tell us only what participants choose to report. Every primary method in Table 1 is designed to run before release, on material written for the purpose: task sets, synthetic profiles and attacks.

04

How Brello SI is being designed against indirect prompt injection

Brello SI is being designed to treat text from outside as data, never as instructions, and we intend to make that rule hold with four layers, because no single defence against indirect prompt injection is reliable alone. Prompt injection, explained introduces the attack in general terms.

In an indirect prompt injection, the attacker never talks to the assistant. They plant instructions where it will read them, in a web page or a document, sometimes hidden from people. Greshake and colleagues showed that this works against real applications built on language models, which, they argue, blur the line between data and instructions.10 OWASP lists prompt injection first among its 2025 risks for applications built on large language models (checked 5 October 2026).11

Brello 1.0 can only write an answer, so the worst a page can do is distort one, and its citations point back to the pages it used. Brello SI is being designed to hold personal context and to act, so the same page could try to make it reveal that context or take a step nobody asked for. Figure 2 follows one injected instruction through the four layers.

1. Label retrieved text as data. Retrieved passages will go into a marked, numbered block, kept apart from instructions. Brello 1.0 already does this: its web context is labelled “Web results (retrieved {date})”, and each passage carries its source number. Spotlighting goes further, transforming untrusted input so the model has a continuous signal of where it came from.12 Labels help a model tell sources apart. They don’t stop it from following what it reads.
2. Keep data out of the instruction channel. We intend instructions to come only from the person and from Brello, and retrieved text never to be placed where instructions go. Structured queries train a model to follow only its prompt channel and to ignore instructions in its data channel.13 CaMeL goes further, deriving the control flow from the trusted request alone, so retrieved data can’t change what the system does.14 We intend to test this layer with pages seeded with instructions, and to count any change in behaviour as a failure.
3. Let people check. Citations will link claims to their sources, so a distorted answer can be traced to the page that caused it. In Brello 1.0, the model is told to cite the sources it relies on, and each citation is a tappable link to the page. This protects only readers who follow the citations.
4. Confirm before acting. If an injection gets past every other layer, a consequential action will still stop for the person’s confirmation, shown in plain terms.

We intend to evaluate injection on the whole system, not the model alone, because many of the defences live in the code around it. AgentDojo, a public environment that tests tool-using agents against injections in untrusted data, shows what that end-to-end testing looks like.15 We expect to need our own suites as well, built around how Brello SI will read the web and what it will be able to do.

05

Release gates

Each Brello SI release, or new capability within one, will have to pass four gates in order: internal evaluations, adversarial red-teaming, invited early access and wider availability. Each gate will have pass criteria written before testing starts, and a failure at any gate will send the release back to be fixed and re-tested from the first gate.

The criteria come in a few types. A threshold is the minimum acceptable result in an area. No regressions means nothing gets worse than in the previous release without a stated reason. A hard stop is a failure that blocks a release however good everything else is. Completing a consequential action without confirmation is one. Figure 3 follows one release candidate through the gates, including a failure.

1. Internal evaluations. Automated suites and human review across all six areas, against every threshold, with no unexplained regression and no hard stop.
2. Adversarial red-teaming. People and automated methods will try deliberately to break the release. Every finding will have to be resolved: fixed, mitigated, or accepted as a known limit with a written reason.
3. Invited early access. A small group who know they’re using an early system, with a clear way to report problems. Serious reports will have to be resolved, and known limits written down, before the gate passes.
4. Wider availability. The release will open more widely with its evaluation summary published, and reports from people using Brello SI and from independent researchers will continue to be reviewed.

A failed gate will send the release back to the first gate, not to the one it failed, because a fix for one problem can cause another. After release, rollback will mean switching a capability off or returning to the previous version while the problem is fixed, so we’re designing each new capability to be switched off on its own. Releasing in stages has precedent: Solaiman and colleagues describe a staged release that left time between releases to analyse risks and benefits.16

06

What we intend to publish

Each Brello SI release will come with an evaluation summary that reports what was tested, how it was tested, what we found and what the release still gets wrong. The summaries are how we intend to meet the charter’s commitment to published evaluations (Brello Charter, version 1.0).

• What was tested. Which capabilities and versions, across which areas, and what was out of scope and why.
• How. The methods, how the test sets were built, and the pass criteria set before testing began.
• Results. Measured against those criteria, including the failures and what we did about them.
• Known limits. What the release still gets wrong, and what we couldn’t test.

Model cards established the practice of reporting a model’s evaluation results alongside its intended use and limits.17 Our summaries will apply the idea to the whole system, including the code around the model, where many of Brello’s safeguards live. Each summary will list every failed criterion and what we did about it.

Some material needs care. A detailed injection suite is also a set of instructions for attacking the system, and a published test set can stop measuring anything once it reaches training data. Where we hold something back for these reasons, the summary will say what was withheld and why. Anyone who finds a problem we missed can report it through our security and disclosure page.

07

Limitations of pre-release evaluation

This plan has not yet been run on Brello SI, and pre-release evaluation has limits that running it won’t remove.

• Criteria types, not values. We name the kinds of criteria each gate uses, not the thresholds. Those will be set for each capability before its testing starts, and reported in its evaluation summary.
• Passing shows only what was tested. An evaluation can show that a failure exists, never that none does. Red-teaming finds what its people and models think to try.
• We evaluate our own work. The people who build Brello will also design and run most of its evaluations. Publishing methods and inviting independent researchers reduce this conflict of interest but don’t remove it.
• Model graders have biases of their own. Where one model scores another’s output, it can favour the answer shown first, longer answers, or answers like its own.18 We intend to check model grading against human review before relying on it.
• Phones differ, and outputs vary. A device-first system runs on phones with different memory and chips, and Brello 1.0 already falls back from the GPU to the CPU when it has to. Its models also sample: Brello Pro and Brello Vision use temperature 1.0 and Brello Core uses temperature 0.7, so the same question can get different answers. Results will have to be reported as rates over repeated runs and classes of device, not as single outcomes.
• Confirmation depends on attention. A confirmation step protects people who read it. If it appears too often, people may approve by habit, so how often it appears is part of what we will have to evaluate.
• No defence against prompt injection is complete. The layers in section 4 are designed to reduce the risk. We don’t claim they remove it.

08

Open questions

Four questions about evaluating Brello SI remain open, and we expect the plan to change as we answer them.

8.1 Measuring what resists a number

Whether a long task was done well, whether “I’m not sure” came at the right moment, whether an answer was careful without being unhelpful: these depend on context and judgement. Automated metrics are proxies, and human review is slow and can be inconsistent. We’ll say which results rest on human judgement, and how that judgement was made.

8.2 Evaluating personal context without personal data

Long-term personal context is where Brello SI will differ most from Brello 1.0, and it’s the hardest part to test honestly. We can’t evaluate it on real people’s context, because we’ve committed to using that only to answer them. Synthetic profiles help, but they miss the mess of real lives. We don’t yet know whether some evaluations could run on a person’s own device, with their agreement, so that only a pass or a fail leaves it.

8.3 Setting thresholds before the evidence

Fixing criteria in advance protects against drift, but a threshold set too low lets problems through and one set too high blocks useful work. We don’t yet have a principled way to set thresholds for areas such as calibration or privacy leakage. When we set them, we intend to publish the reasoning alongside the numbers.

8.4 Independent replication

An evaluation we design, run and report ourselves is a claim, not a proof. Third-party auditing and red-teaming exercises are among the mechanisms researchers have proposed for making AI developers’ claims verifiable.19 We intend to publish methods in enough detail for others to reproduce them, and we’re designing Brello SI so that independent researchers can check what runs and where, as described in Private compute you can verify: the design space. Doing that without publishing a guide to attacking the system is something we’re still working out.

Until a threshold has a published rationale, we will report it as an open question rather than as a number.