Seven ways a security control can pass while doing nothing

We spent several days rebuilding 1lan, our reverse-tunnel service, so that each machine gets an allocated endpoint instead of one derived from its own IP address. The rebuild went fine. What was interesting was the bugs.

Seven of them, and they turned out to be the same bug wearing different clothes: a check that reports success while the thing it checks is absent. Not a control that fails loudly. A control that passes, logs nothing, and protects nothing.

They are worth writing down because most were invisible to their own test suites — the tests passed, the code was reviewed, and the property everyone believed in was simply not there.

1. The control that reports success and does nothing

We isolate tunnel tenants by pinning each machine’s SSH key to its own loopback address:

restrict,port-forwarding,permitlisten="127.0.2.8:*" ssh-ed25519 AAAA… nas

The intent is that this key may bind 127.0.2.8 and nothing else. Test it casually and it looks right: the tunnel comes up, the listener appears, no errors anywhere.

It was doing nothing. Under sshd’s default GatewayPorts no, a client-requested bind address is ignored — but permitlisten matches against what the client asked for, so the request is accepted, and then bound to 127.0.0.1 regardless. Every tenant pinned to a different address lands on the same one.

That is worse than a broken control. A broken control fails visibly and gets fixed. This one reports success, and you build on top of it. The fix is one line inside a Match block:

Match User tunnel
    GatewayPorts clientspecified

2. The commented-out default that satisfies your grep

Having found the setting, you go to apply it idempotently, and write the obvious thing:

grep -q GatewayPorts /etc/ssh/sshd_config || append_match_block   # WRONG

OpenBSD’s stock sshd_config ships a commented defaults section containing #GatewayPorts no at line 63 and #ClientAliveCountMax 3 at line 74. The grep matches the comment. On a completely unconfigured machine the check passes, the insert is skipped, and sshd keeps running with the default — while your deployment script reports success.

Anchor on something that can only be live config — a Match line, an indented setting — never on a keyword the defaults section also mentions.

3. “Configured” is not “configured correctly”

The next version guarded on whether the Match block existed at all:

grep -q "Match User tunnel" /etc/ssh/sshd_config || append_block   # ALSO WRONG

Every machine set up by an earlier version already had that block — carrying older settings and not the new one. So the machines that would have been declared “already configured” were precisely the ones that had been running longest, including the production relay. The newest machines would have been correct and the oldest silently wrong, which is the opposite of how anyone expects a fleet to fail.

Rewrite the block. Do not skip it.

4. Verifying the input instead of the outcome

Every version above shares a deeper mistake: they check the file that was written, not the behaviour that resulted. sshd will tell you its effective configuration for a given user:

sshd -T -C user=tunnel | grep gatewayports

Two traps here, both worth knowing. sshd -T lowercases keyword names, so grepping for GatewayPorts matches nothing and your verification reports failure forever against a setting that is correctly applied. And comparing sshd -T -C user=tunnel against a bare sshd -T proves something a single check cannot: that the setting is scoped to that account rather than enabled globally. A global GatewayPorts would work perfectly for the tunnel while quietly widening forwarding rights for every other account on the box.

5. Two self-consistent halves that disagree

Our allocator is written in Python; the renderer that consumes its output is POSIX shell. Both enforce the reserved-address range. They disagreed — Python excluded 127.0.0.0/16, the shell excluded only 127.0.0.* — which meant the allocator would hand out addresses the renderer refused, and every documented example contradicted both.

Both test suites passed the entire time. Each half was internally consistent, and nothing tested the contract between them. It surfaced only when a test asserted a specific documented address and got a refusal.

Any rule expressed in two languages needs a test that exercises the seam — run the real producer, feed the real consumer. And when you change such a rule, grep the whole tree for the old value including prose: stale comments outlived the code fix twice in our case, and a comment stating the old rule next to code implementing the new one is how the next person reintroduces it.

6. Hand-edited credentials fail in ways nothing reports

Three separate failures on one machine, all in authorized_keys, none of which produced any signal:

  • An unrestricted key sat in the tunnel account for five weeks. It had never authenticated once. Nobody had reason to look.
  • A key appended without a trailing newline welded two entries into one line. sshd parsed it as the first key plus a nonsense comment, so the second key never worked — and the person who added it had no reason to suspect anything.
  • The file was rewritten wholesale by a deployment script, meaning any hand-added entry would vanish at the next run, silently, whenever that happened to be.

The lesson is not “be careful with authorized_keys”. It is that a file edited by both humans and programs will eventually be wrong in a way neither notices. Generate it entirely, from a source of truth, or don’t generate it at all.

7. A guard that matches too much

Our tunnel client refuses to start if a tunnel is already running. The first implementation matched any established TCP connection to the relay — so an unrelated interactive SSH session satisfied it. A service restart then killed the real tunnel, matched the human’s session, concluded a tunnel was already up, and exited having left none. Every public service behind it returned 502.

The replacement matched the tunnel process specifically, which was correct until a migration required the old and new clients to run side by side — whereupon the old guard matched the new client’s tunnel and made it exit silently. The symptom would have been “the new client just doesn’t start.”

A guard is a claim about identity. Scope it to the thing it actually protects — in our case the machine’s own bind address, which nothing else can match.

What we changed about how we work

Write the test that makes the control refuse something. A control observed only succeeding has not been shown to work; it has been shown not to crash. Our test suites now assert the refusals — wrong key refused, wrong address refused, malformed registry rendering nothing at all — and those assertions found real bugs immediately.

Check the outcome, not the input. sshd -T over grepping a config file; the rendered artefact over the template; the effective permission over the line you wrote.

Assert the mechanism when the outcome can happen by accident. Twelve parallel token redemptions produced exactly one winner on our test database — which proved nothing, because SQLite’s global write lock serialises the operation for free. The property we actually depend on only exists under PostgreSQL. A green test can be green for the wrong reason.

Grep for the old value, including prose, whenever a rule changes.

None of this is exotic. All seven bugs were found by ordinary means — writing a test that asserts a refusal, comparing what two components believe, running the thing on real hardware and looking at what it did. The reason they lasted as long as they did is that each one looked like working software.

1lan is our self-hosted reverse-tunnel service; its design document, with the full evidence behind these findings, is published with the source. If your infrastructure has controls nobody has watched refuse anything, that is the kind of thing our security assessments are for.