For context, I play Silent for the shivs and Ironclad for the brute force. This project needed both decks, and I mostly brought the Ironclad one.

This post covers what got built, whose work it stands on, and what still doesn’t work.

Whose code this is

Most of the hard work isn’t mine. Not even Claude’s.

The Android launcher itself is iunius612’s code, unchanged. If it logs you into Steam, downloads the game, or loads your mods on Android, thank them. My fork adds two things on top, and “my” is doing a lot of work in that sentence.

Division of labour

For transparency:

A second Claude session drove adb on the tablet to run benchmarks. I relayed messages between the two like a very slow clipboard.

Android: my tablet was the problem

The Pad 5 has a 2560×1600 120 Hz screen and a Snapdragon 860. The game ran, but it felt choppy. The first guess was that the GPU couldn’t keep up, so the first fix was a “fast mode” that rendered at a lower resolution. That’s the Ironclad answer: hit the problem as hard as you can. It made the game look much worse and barely helped. The tablet-side measurements showed why: the fast mode kept the GPU at about 91% busy. Android’s own Game Mode downscaling was cheaper, at 74–80%.

The real culprit was boring: the game’s own 30 fps cap. On a 120 Hz panel its frames landed unevenly, and that reads as choppy. A 60 fps cap with vsync on, and the display at 60 Hz, was smooth with the GPU at mid clock.

The fix that worked was a Silent deck: no single big card, just a lot of small ones that add up. That became StS2 Launcher Perf, a build of upstream’s release APK with an add-on:

• A Perf settings panel: FPS cap, vsync, particle amount, render resolution, full-screen effects, and a frame-time log. The defaults change nothing.
• Settings live in a text file the app re-reads every 2 seconds, so the benchmarking session could switch settings over adb mid-fight.
• The launcher’s Korean-only dialogs translated to English, 243 text fragments in all.
• Its own package name, so it installs next to the original.

There’s no engine rebuild. A GitHub Actions workflow downloads upstream’s APK and checks its hash. It patches one DLL to load the add-on, edits the binary manifest, and re-signs the result. Four releases so far. The first one taught us that Assembly.Location is an empty string when the launcher loads a DLL from memory, so the add-on tried to load itself from /STS2Perf.dll. Apparently “that’s where I’d be if I existed” isn’t a valid path.

iOS: no JIT, no Mac

The Android launcher can’t just be recompiled for iOS. It runs the game’s code on a JIT and patches it while it runs, and iOS doesn’t allow JIT for sideloaded apps. So the iOS port turns everything inside out:

• The game’s sts2.dll is patched at build time with Mono.Cecil, with the same fixes the Android launcher applies at runtime: skip Steam’s own startup, skip crash reporting, and a few platform guards.
• All of it is compiled to native ARM64 with .NET NativeAOT, through Godot 4.5.1’s iOS export.
• A small native extension mounts SlayTheSpire2.pck from the app’s Documents folder at startup and checks it matches the build. You copy that file from your own Steam install. The repo has no game files, and the build is only uploaded inside an encrypted zip.

I don’t own a Mac, so all of this runs on GitHub’s macOS runners, which are free for public repos. Before touching the real game, the pipeline ran against a fake game: a tiny Godot project shaped like Slay the Spire 2, with the same class names, a fake Steam, and a fake pck. That caught most of the mistakes early. It didn’t catch all of them, which is what fake games are for.

Steam doesn’t trust London

To compile the game’s code, the build machine needs the game’s DLLs, so it logs into Steam as me. Steam sent the approval prompt to my phone. I approved it, and got this:

Fair. A server in London asking for my account while I’m on another continent does look like a malicious actor. The usual fix is to log in once on your own PC and give the build a saved token. That’s when I found out I no longer own a Windows PC.

What Claude built instead:

1. The build opens a GitHub issue that @mentions me and waits.
2. I reply with the 5-character code from the Steam app’s Steam Guard tab.
3. The build checks the issue every 2 seconds, which is as close as CI gets to a shiv build: small, fast, and a lot of them. It reads only my replies, deletes each one, signs in, and closes the issue.

My strategy for getting it working was pure Ironclad: press Run workflow, see what breaks, press it again. It took three runs:

• Run 8: the sign-in prompt popped up again, this time from San Francisco, and I tapped Approve after being told specifically not to. That turned out not to matter. What did matter: my code arrived after about 3 minutes, and Steam had already dropped the connection, because it closes connections that haven’t signed in within about a minute. Fix: wait for the code with no connection open, then connect and send it within seconds.
• Run 10: while a sign-in is pending, the Steam app shows its Approve/Deny screen on top of the code, so I couldn’t read it. I tapped Deny to get it out of the way, sent the code, and the sign-in worked. Then the patcher failed on the real sts2.dll: it couldn’t find GodotSharp.dll, which the game’s [Export(PropertyHint.Range, ...)] attributes refer to. The fake game had no such attribute. It has one now. Also fixed: the build now asks for the code before it starts any sign-in, so nothing covers it.
• Run 13: signed in, downloaded about 1.9 GB of depot files, patched, compiled, and produced a 52 MB StS2.ipa in about 14 minutes.

Installing it

The week I tried this, SideStore failed to sign in to my Apple ID with error -80045. According to its issue tracker, everyone was getting the same. Sideloadly on a PC with Apple’s iTunes and iCloud still worked. The app opened to this:

After a day of builds, a screen that correctly complains about a missing file is the best thing I’d seen. I copied the pck over, and the game started:

The version in the top-right corner is v0.107.1, the same as the build, so the app’s version check agreed with the pck.

And of course the first fight was Silent, with Cloak and Dagger in hand. Shivs on iOS, compiled ahead of time:

One more honest note. Before the real game, Claude built a tiny test app to check the toolchain on my phone. It closed right after the Godot splash. The real game, which is far bigger, did not. Nobody has gone back to find out why.

What doesn’t work yet

• Short freezes. Every so often the game stops for a moment and then carries on. As Ironclad, I barely notice. Halfway through playing nine shivs, I notice. The next build logs every frame over 100 ms with what happened during it: pipeline compiles, texture loads, or .NET garbage collection.
• No Steam. No cloud saves or achievements; saves get copied by hand between PC and phone.
• No mods. Mods are compiled code, and iOS won’t load new code.
• Every 7 days the free Apple ID signature expires and the app needs re-signing.
• Every game update needs a new build and a reinstall, for the same no-new-code reason.

Steam login with cloud saves inside the iOS app is possible. The Android launcher already does it with SteamKit2, a plain .NET library, so that’s next on the list after the freezes.

What I learned

Measure before fixing

The first Android fix made the game uglier without making it faster. A frame-time log and a GPU busy percentage would have saved a release.

Fake inputs find most bugs, real inputs find the rest

The fake game caught nearly everything before a single Steam login. The real game then found exactly one thing the fake game didn’t have.

Some bugs need Bash, most need shivs

Rerunning a build is the Ironclad approach, and it worked more often than it should have. The bugs that mattered, the Steam timeout and the missing GodotSharp, needed the Silent approach: a small, precise fix in exactly the right place.

Read the prompt before tapping Approve

Applies to Steam. Also to AI agents.

The code is at bhanuharya/sts-2-launcher-mobile. If you own the game and have a Windows PC or a Mac, the README walks through the iOS build. If you don’t own the game, none of this helps you, by design.

One last thing. Mega Crit has said an official mobile version of Slay the Spire 2 is coming “at some point”, just not during Early Access. Theirs will be better than mine: no GitHub issue full of Steam Guard codes, no re-signing every 7 days, and probably cloud saves. When it ships, I’ll happily buy it again. I just couldn’t wait. “At some point” is a lot longer than “one more run”, and I’ve been on one more run since June.

Now if you’ll excuse me, I have a Silent run to lose on a six-inch screen.

What Community Edition doesn’t cover

There are open-source engines for each of these: OpenGrep for code patterns, Gitleaks for secrets, Trivy for dependencies. Running them was quick. Getting their results into SonarQube in a form people would review took most of the time.

First try: external issues

Sonar can import other tools’ findings as “external issues”, so I started there. The first import: 77 findings exported, zero visible. I assumed the import had failed. It hadn’t. My rule names didn’t match Sonar’s external_<engine>:<rule> format, so the issues were there and my filter couldn’t see them.

With the names fixed, 249 of 255 findings showed up. The other six were in dotfiles Sonar doesn’t index. Then some quieter problems:

• narrow source scopes left out the dependency findings
• scanning the whole tree collided with sonar.tests
• everything landed under Issues, never under Security Hotspots

That last one decided it. External issues have no rule page and can’t go through hotspot review, and people scroll past them. So I wrote a plugin that registers the findings as native Sonar rules. The Dart post covers how that part works.

What I ended up with

SonarQube stays as it was. The missing engines run beside it, and their findings come in as native Sonar issues.

• sdt, a small Go CLI, runs the three engines and writes one findings.json. Every finding gets a fingerprint.
• A SonarQube plugin imports those findings as vulnerabilities and security hotspots, with descriptions and fix guidance, Dart included. To a developer they look like any other Sonar issue.
• A baseline built from the target branch’s fingerprints. Anything already there counts as existing, so only what you just added is new.
• Scanner errors fail the build. If an engine errors, the run exits with code 3. Every run writes a manifest with checksums.
• A Word report built from Sonar’s data, because that’s what gets attached to a release ticket.
• A Jenkins library, so one job can scan any repo, either a whole branch or one PR.

Sonar still runs its own analysis and quality gate on top.

Things that bit me

Jenkins env vars are case-insensitive. My job had a parameter called branch. The library exported BRANCH for the scan script, and the script kept failing with BRANCH: parameter null or not set. Jenkins merges environment names case-insensitively, so BRANCH got folded into the parameter’s lowercase key, and bash, which is case-sensitive, saw nothing. Same with pr_id and PR_ID. The fix: pass them as SDT_SCAN_* and rename them right before the script runs.

My first PR scan scanned the whole branch. I passed the base branch to the scanner and assumed it would only look at the diff. On a real PR it reported 117 findings, 115 of them dependency CVEs, from a PR that touched 15 files and no lockfiles. Only secret scanning was limited to the PR’s commits. The baseline fixed it. Scan the target branch first, then the PR:

[sdt] baseline: full scan of origin/main
baseline created: .secure-dev/baseline.json (119 entries)
[sdt] pull request adds 0 of 117 findings (the rest already exist on the target branch)

Sonar’s own PR analysis only looks at new code too, so the two now agree. A PR scan takes about twice as long because of the extra target scan.

The default quality gate fails every PR. Sonar’s default gate wants 80% coverage on new code. My scans don’t send a coverage report, so every PR failed on coverage, and people stop looking at builds that are always red. Next I’m setting up a security-only gate: no new vulnerabilities, and every new hotspot reviewed.

Merged PRs are awkward. Once a PR is merged, its branch is usually deleted and main already contains it, so a baseline from main finds nothing new. Checking a merged PR means comparing the two parents of the merge commit. I haven’t built that yet.

Where it is now

It runs on a local Jenkins and pulls repos from Bitbucket. One job has a dropdown for branch or pull-request. Findings go to SonarQube and each build keeps its reports. I still start PR scans by hand, because Bitbucket Cloud can’t reach a Jenkins on localhost.

It’s pre-release and the schema will change. So far it does what I wanted: developers stay in SonarQube, and a PR shows them only the security issues it added.

Code: github.com/bhanuharya/secure-development-tools

I wanted real security findings for Flutter apps on Community, showing up the same way Sonar’s own Java or Python security issues do. This post is how I got there with OpenGrep and a small plugin.

The pieces

OpenGrep runs in CI and writes JSON. The plugin doesn’t run anything itself. It reads that JSON during sonar-scanner.

Why I didn’t use SARIF

Sonar can import SARIF or its generic format as “external issues”. External issues show up, but they can’t be Security Hotspots, they don’t count in the CWE / OWASP reports, they have no rule page, and you can’t switch them on or off in a quality profile. In practice people scroll past them.

A plugin can register rules on the server like any language analyzer does. Then a finding is a normal Sonar issue: it has a rule description, a “How to fix it” tab, a CWE link, and hotspots go through the usual review.

A rule

Rules are plain OpenGrep YAML. The metadata block is what the plugin reads to build the Sonar rule:

- id: scp.dart.tls.disable-certificate-validation
  languages: [dart]
  severity: ERROR
  message: A badCertificateCallback that always returns true disables TLS certificate validation ...
  metadata:
    name: "TLS certificate validation should not be disabled"
    category: security
    cwe: ["CWE-295"]
    owasp: ["A02:2021 - Cryptographic Failures"]
    confidence: HIGH
    remediation: Remove the override, or validate the chain against a pinned certificate.
  pattern-either:
    - pattern: $CLIENT.badCertificateCallback = ($A, $B, $C) => true
    - pattern: $CLIENT.badCertificateCallback = ($A, $B, $C) { return true; }

The type comes from the metadata. confidence: HIGH or a taint rule (mode: taint) becomes a Vulnerability. Any other security rule becomes a Security Hotspot, since a person has to look at the context. owasp-mobile and masvs keys become tags, because Sonar has no mobile standard of its own.

The bundled pack has 18 rules so far. Each one has a vulnerable and a safe fixture for opengrep scan --test:

• TLS validation disabled, MD5/SHA-1, AES-ECB, zero keys, static IVs, insecure random
• SQL injection and untrusted WebView scripts (taint)
• http:// URLs, secrets in SharedPreferences or logs, hardcoded API keys
• WebView JavaScript channels, biometricOnly: false, deep-link handlers with no validation

You can point the server at your own rule directories too.

Things I didn’t expect

A plugin can’t add rules to another plugin’s built-in profile. The default Dart profile belongs to sonar-flutter. My rules can’t go into it, and a project uses one profile per language. So the setup includes a script that copies sonar-flutter’s profile, activates the OpenGrep rules in it, and makes it the default.

sonar-flutter runs flutter analyze itself by default. If flutter isn’t on the scanner’s PATH, the whole analysis fails. In CI I run dart analyze --format=machine first and point sonar-flutter at that file with sonar.dart.analyzer.mode=MANUAL and report.mode=MACHINE.

Zero-length diagnostics crash the import. Some dart analyze results come with a length of 0, file-name lints at line 1 for example. sonar-flutter throws on those and the whole scan fails. My scan script rewrites them to a one-character span before handing the file over. (“My” is doing some work in that sentence. Claude wrote it. I found the stack trace.)

Secrets in git history have no line to sit on. A key deleted two years ago is still in every clone, but the file isn’t in the checkout anymore. Those findings go on the project instead of a file, as their own rule (sdt:secret-in-history), so they don’t get dropped.

That last one comes from the wider setup: sdt runs Gitleaks and Trivy next to OpenGrep, and the plugin also imports secrets and vulnerable dependencies as native rules. A dependency nothing imports becomes a hotspot. One that’s imported becomes a vulnerability.

Limits

This is pattern matching, plus OpenGrep’s taint mode for a couple of rules. It is a lot shallower than a dedicated analyzer. If you have Developer Edition, use Sonar’s own Dart analysis, and treat these rules as an extra layer at most.

The plugin is tested in CI against SonarQube 9.9 LTA, 10.7 and the latest Community Build, with sonar-flutter 0.5.2. It’s early, and the rule pack is small. If you write Flutter on Community and have a rule idea, I’d like to hear it.

Three clocks

One grant starts three separate timers:

They stop separately:

• A grant can end while the credential still works.
• A credential can expire while the connection stays open.
• A database finishes a query it already accepted. The result can land after the revoke was reported done.

The runtime I have been building tracks this as five separate events:

One status word, four separate events. Late effects land in the last box.

I trust one sentence here: “admission closed and no sessions remain”. A status line that says revoked packs four open questions into one word.

Where each control stops

Secrets manager: custody

It decides who may read which secret. A Vault-shaped issuer can mint a short-lived database credential per task. That beats a long-lived password in an environment variable.

What it leaves open: the effect. A lease says when a credential dies. It says nothing about the query accepted a second earlier, or whether the task still gets its answer.

One more wrinkle in the same spot. Short-lived credentials for PostgreSQL ride on password authentication. If the fix is to move off password authentication, the clock I relied on may stop being the thing that enforces anything.

Gateway: admission

It sees a request, checks policy, forwards or refuses. The one I read checks HTTP method and path against a per-provider allowlist. A permitted host still needs a permitted operation.

The boundary: it decides at admission and the work lands later. Closing the door stops future admissions. Everything already inside keeps going.

Sandbox: reachability

The strongest setup I have read about: the agent container gets no network interface, drops every Linux capability, and mounts two Unix sockets for tool calls and model calls. Nothing else is reachable. The project aborts startup when the connectivity check fails, so it never slides quietly into a weaker mode.

The edge: the same screen that keeps the agent out of my network keeps it out of my test stack when the fixture resolves to loopback or a private range. The code says so directly. A fixture on loopback needs a specific option before it becomes reachable at all. So the adapter has to live somewhere the declared route permits, and that place can be awkward.

Two escape hatches

The same project documents one relaxation and ships another in code:

With the second one set, any claim that the agent cannot reach an undeclared route stops holding. The failure is quiet: the lab works better, the boundary disappears, results look the same.

Checking the citation

I treat “the code does this, at this file and line” as unverified until I fetch the source and grep the identifier. Here that turned up three kinds of rot:

1. Line moved. Stale, easy fix.
2. Identifier gone or renamed. The mechanism my plan relied on is absent from the version I would pin. Design problem.
3. Citation never existed. The expensive one, usually inside an otherwise careful document.

The habit that paid off: go to the enforcement site, the place that decides behaviour. A type existing proves nothing. In the gateway I read, the endpoint filter is declared in one file and called in the mitm proxy, on the request path, right before the credential swap. Stopping at the declaration gives a sentence that is true and misleading.

# five mentions in the tree, and one place that decides anything
$ grep -rn "isEndpointAllowed" src --include="*.ts" | wc -l
5
$ grep -rn "isEndpointAllowed(provider.config" src --include="*.ts"
src/docker/mitm-proxy.ts:1105:    if (!isEndpointAllowed(provider.config, method, path)) {

Five hits. Two are comments and one is an import. Exactly one runs on the request path. That is the one that decides whether the filter means anything.

What would settle it

Reading gives a hypothesis and seams worth testing. It cannot say whether a stopped grant still lands an effect. That is the question the post turns on.

The roles and the claim under test. A stop acts at the gateway, the effect lands at the service, the direct attempt from the sandbox is run as a case, and the observer is a separate process the task cannot see.

Small lab on purpose. One issuer, one service, one observer. Six rules for whether it can tell me anything:

• write down what each case should show before running it
• include a permitted operation that must succeed, so a later denial reads as policy
• keep the four lifecycle events separate
• observe from a process outside the one under test
• mark inconclusive cases as inconclusive, keep them out of the story
• run the direct attempt as a control, since topology is no substitute for watching it fail

The gateway earns its place once the small version produces something that needs it. Added first, it spends the budget on plumbing and reaches the interesting question late.

Where this stands

I have only read the docs and source at one revision. I have not pinned a release or run the lab, so I do not know whether an admitted operation can land after access is pulled. That is the next thing to test.

Jev doesn’t return text. You give it a state, a question and the answers you’ll accept. It returns the option it picked, a probability for every option, and a separate confidence score you’re meant to threshold.

I wanted to know if a local 8B could do the same job. Partly for privacy, since for email triage the input is the email. Partly because a local model has no per-call cost and doesn’t change under me. And partly because picking one of eight queues sounded like something an 8B should manage.

I tested it on synthetic email triage. The local model matched Jev on three of eight queues. The bigger problem was its confidence: it was just as sure of its wrong answers, so I couldn’t use it to decide what to route automatically.

The local setup

LFM2.5-8B-A1B, 8B parameters with about 1B active per token, quantised to Q4_K_M (4.9 GB). It runs under llama-server on the ThinkPad T14 that hosts my local lane, CPU only, started by hand when I need it:

./llama-server -m LFM2.5-8B-A1B-Q4_K_M.gguf \
  -c 65536 -t 8 -tb 8 -ngl 0 -fa on \
  --jinja --metrics -a lfm25-8b \
  -n 2048 --repeat-penalty 1.1 --repeat-last-n 256

Both sides get the same system prompt. A chat completion gives you none of the three things Jev returns, so most of the work was making the local model answer in the same shape:

{
  "model": "lfm25-8b",
  "messages": ["<system prompt>", "<the email>", {"role": "assistant", "content": "{\"label\": \""}],
  "temperature": 0,
  "max_tokens": 8,
  "grammar": "root ::= (\"billing\" | \"security\" | ... | \"__none__\") \"\\\"}\"",
  "continue_final_message": true,
  "add_generation_prompt": false,
  "logprobs": true,
  "top_logprobs": 20,
  "cache_prompt": true
}

• The pick. The assistant turn is pre-filled with {"label": ", and the grammar lists the allowed queues, so the model can’t answer anything else. With 8 tokens at temperature 0, a decision takes 4 output tokens. Without the grammar the same model used 454.
• The probabilities. llama-server has no per-option probability, so I rebuilt one from the top 20 token logprobs along the greedy path, then renormalised over the valid options. The renormalising is required: the server reports probabilities from before the grammar mask, so they don’t sum to one. It’s still an approximation. Exact scoring needs one forced call per option, and the completions endpoint returned zero prompt tokens for echo when I tried it.
• The confidence. There’s nothing separate to return, so it’s just the probability of the chosen option.

For comparison I also ran the model without the grammar or the prefill, with a 512-token cap, and pulled the label out of its answer.

Lab 1: 94 emails

My first conclusion was that the grammar added 11.7 points of accuracy. It didn’t. Nine of the no-grammar answers had no label in them at all, and those nine account for 9.6 of the 11.7 points. On the answers it did give, it was 57.6% against 63.8%, six emails out of 94. That’s noise.

The grammar bought parseable output and speed, 21.8 seconds per email down to 3.9. It didn’t buy accuracy.

The gate

To automate triage you pick a confidence threshold, route everything above it, and send the rest to a person. The number that matters is how many wrong answers get through.

At 0.8, both handle about four fifths of the mailbox. Jev lets 3 wrong answers through out of 94. The local model lets 20 through. Its average confidence on the answers it got right was actually higher than Jev’s, 0.964 against 0.941.

Wrong answers that clear the gate, lab 2. The local model covers the same share at every threshold because it has no answers below 90% confidence.

Lab 2: 205 emails

For lab 2 I changed three things, each aimed at a failure I’d seen in lab 1. Queue definitions were written as actions. The model picked from 25 plain descriptions, which code mapped onto the 8 queues. And the probabilities came from a full expansion over the token distribution, not just the greedy path. The set had 205 emails, 60 of them traps.

• Jev went from 84.4% to 85.9% with the descriptions. The local model got 65.4%, up from 63.8% on a harder set, so no real change.
• The local model got slower, 3.9 to 6.7 seconds per email, because the prompt grew.
• The descriptions made Jev worse on traps (43 of 60, down from 48) and raised its output tokens from 15,590 to 65,810.
• The local model’s ECE got worse, 0.339.

I tried fixing the calibration with temperature scaling on held-out data. ECE stayed between 0.32 and 0.36. Looking at coverage shows why: it’s 84.9% at 0.7, 0.8 and 0.9 alike. Every answer the local model gives a usable score for sits above 90% confidence, so there’s nothing in the middle to rescale. Another 31 of the 205 answers produced no usable distribution at all.

Confidence against accuracy, lab 2. The local model has no answers below 90% confidence.

I don’t think a prompt, a grammar or rescaling can fix a model that never reports doubt.

Per queue

The local model keeps up on newsletter, security and meeting. Complaints are the disaster: 24 of the 205 emails, and it got 1 of them. That one queue is most of the 19-point gap.

So a split might work: the local model takes its three good queues and everything else goes to Jev or a person.

Bugs in my own harness

I found and fixed ten bugs before or during the runs. Three of them would have given me confident, wrong conclusions:

• Three trap emails had their correct label pointing at the decoy queue. A phishing email was filed under billing in the answer key.
• The grammar’s closing brace comes in the same token as the end of the label, so a finished answer never matched an option name, and correct answers were scored as no answer.
• The coverage metric dropped rows with no distribution, which turned 84.9% coverage into 100% in one file while the report’s own table said 84.9%.

Four of the other seven:

• A typo in the grammar made every request return HTTP 400.
• A scorer read the token after the label instead of the label itself.
• A 64-token cap returned empty answers because the model was still thinking.
• The answer parser matched labels inside negations, so “the subject does not say newsletter” counted as a newsletter answer.

What this doesn’t show

• The emails are template-generated and 77 to 208 characters long. Real mail will score lower, so every accuracy here is a best case.
• It’s one model, one quantisation, one CPU and one email set. It says nothing about small models in general.
• jev-latest changes over time, so the Jev numbers can’t be reproduced against a fixed version.
• In lab 1 the local model beat Jev on traps, but there were only 14 traps. In lab 2, with 60, Jev won 48 to 31.
• The two demos at the top are self-reported, and neither reports accuracy.
• TypeSafe publishes no price for this, and their pricing page returns 404. The cost estimate below borrows the paper demo’s rate.

Cost isn’t the question

At the paper demo’s rate, Jev costs about eight cents per thousand emails. The local model is free per call but handled 909 emails an hour in lab 1 and 536 in lab 2, so 50,000 emails a month would take it 55 to 93 hours. At these volumes the money is small either way. What decides it is how many errors get through the gate.

Next

• Two or three examples in the prompt for the queues it confuses: internal against meeting, personal against no-fit.
• Run each email twice with the options in a different order, and gate on whether the two runs agree, since the model’s confidence is no use.
• A small fine-tuned classifier, which takes milliseconds per email on CPU and comes out of training calibrated.
• Pin one server slot so the shared system prompt is only processed once, and give the server all 12 threads. With four unpinned slots it’s probably reprocessed on every call, which would explain why lab 2 got slower.

For now I’d test the local model as a prefilter on its three good queues, with everything else going to Jev or a person. I wouldn’t let its confidence decide that split.

Then a 4B model came out as a GGUF, and I wondered if smaller might be better. I ran both on the same machine with the same prompts. The 4B read long prompts about seven times faster and used about half the VRAM. I kept the 23B anyway.

The box

• Intel i5-12400, 6 cores and 12 threads
• 32 GB of RAM
• RTX 3060 with 12 GB of VRAM, running the CUDA build of llama.cpp on Windows 11

The 23B takes 11,444 MiB of the card’s 12,288, so the two models can’t be loaded together. Every comparison meant stopping one and starting the other, about ninety seconds each time. Both ran on the same port so my other clients kept working.

The two models

GLM-4.7-Flash, the one I already run: 22.9B parameters, REAP-pruned, a community decensored build, quantised to IQ3_S mix at 3.66 bits per weight. 10.18 GB on disk, 131K context.

Spark-X2.5-4B from XHToken: 4.11B dense, 36 layers, one full attention layer for every three sliding-window ones. Q6_K, 3.37 GB. The most widely linked abliterated version only ships safetensors, so I ran a different abliteration of the same base that comes as a GGUF. With 131K context loaded it uses 6,425 MiB. The model card claims 1M context. I only tested recall up to 96K.

The tests

Temperature 0, same token caps for both:

• “reply with exactly: ok”
• two questions the 23B has gotten wrong before: does ice cream contain cream cheese, and is “icecream” a valid spelling
• a writing prompt of the kind that often gets refused, which is what I mostly use this model for
• speed on a short prompt and an 80K-token prompt, then the long prompt again to hit the cache
• a planted string to find at 41K and 96K tokens

Results

On short prompts both generated about 64 tok/s. Both found the planted string at both depths. Both replied with exactly “ok” and got 17 × 23 right.

The two facts went one each. The 4B got the ice cream question right. The 23B kept mixing up cream and cream cheese until it ran into the token cap. On the spelling question it was the other way round: the 23B got it right, and the 4B decided “icecream” was a valid compound word.

The 4B thinks before it answers

It’s a reasoning model. The answer comes back in one field and its reasoning in another. It spent 123 tokens of reasoning to say “ok”, and 1,282 on the spelling question.

The writing prompt came back empty at a 1,500-token cap. It was still empty at 3,000, after 12,249 characters of reasoning. At 6,000 it finished: 14,951 characters of reasoning, then a 5,736-character answer. It never refused. It just hadn’t finished thinking, so a client with a low token cap gets an empty response, not a shorter one.

At temperature 0 both runs were byte-identical, with the same token counts and the same reasoning lengths. Two runs is a small sample, but it means this setup can work as a regression check.

Why I kept the 23B

It answers directly without spending tokens on reasoning, it knows more, and I know how it behaves on the writing I use it for. The 4B’s one real advantage is reading long prompts fast. It’s on disk, and the swap scripts sit next to the launcher if I need to get through long documents quickly.

What actually cost me time

Not the benchmark. A model server started from an SSH session dies when that session closes. It passed a health check, then the port went dark, and I spent an hour blaming the model. The lane I actually use is started by a scheduled task, which is why it survives.

It’s also set to fail closed: if the lane is down, requests error out instead of quietly going somewhere else.

If the numbers hold on a third run, I’ll keep the 4B as a fast reader and stop trying to promote it :-)