A Zeratul Journey, Detour: NAND OOB, One Plaintext File, and a Router's Entire Credential Vault

· CaptainRon

A detour from A Zeratul Journey: Porting Thingino to the Jooan S7-U / A Zeratul Journey, Continued

Every LTE Thingino build runs into the same wall eventually: a direct connection, camera to viewer, no relay server in the middle. Getting there on this Zeratul-based 4G build meant sorting out what address family the modem was handing back.

The cellular side came up IPv6 only. My home network is IPv4 end to end, no IPv6 anywhere on the LAN, so a direct connection was dead before it started. The obvious next move was checking whether the ISP-supplied router could be talked into running real IPv6 on the LAN side too, which meant popping the case on the GPON ONT (the fiber terminal box) in the closet to see what was actually configurable in there.

That is how a look at IPv6 settings turned into six hours with a chip clip and a hardware programmer instead.

A NAND dump from a hardware programmer should just work. Point binwalk-ng at it, get a filesystem back. Instead I got a busybox binary that claimed to be 4 GB, a bin/ash symlink pointing at binary garbage, and directories that should have held a full root filesystem sitting empty. Multiple reads off the chip produced the exact same corruption, so this was not a flaky dump. Something in the extraction pipeline was wrong, and it took a detour through NAND page geometry, a Ghidra decompile of a vendor crypto library, and a router’s entire hardcoded credential store to find it.

A 4 GB busybox binary is not a real busybox binary

The target was a ZTE ZXHN F670L V9, a GPON ONT built around ZTE’s own ZX279128S SoC, an ARM Cortex-A9 part. The dump was 132 MiB, read raw off a single NAND package with a hardware programmer.

binwalk-ng found the JFFS2 filesystem signatures fine. Small partitions, under 200 nodes each, extracted clean. The two large rootfs partitions, 25 MB and 15,242 nodes each, did not:

$ stat bin/busybox
  size: 4294970154   Blocks: 656

4,294,970,154 bytes is suspiciously close to u32::MAX (4,294,967,295), off by a few thousand. du on the same file reported 328 KB of actual disk usage. The file was sparse: a huge logical size, almost no real data, and the data that was there sat in scattered two- and three-byte runs starting 12 KB into the file, no ELF header in sight.

The dump had spare area baked in

NAND flash reserves a small out-of-band (OOB) region per page for ECC and bad-block metadata, alongside the addressable main data. A raw programmer read that includes OOB produces a file shaped like [2048 bytes data][64 bytes OOB][2048 bytes data][64 bytes OOB]..., not a flat image.

The math confirmed it. 132 MiB total, divided by the chip’s 4 internal banks, is 33 MiB per bank. At a 2048+64 page geometry, 33 MiB of raw reads holds exactly 32 MiB of real data per bank, 128 MiB across all four, which is a normal NAND capacity. Sampling the presumed OOB position across the entire 132 MiB file showed uniform 0xFF at every occurrence, not the semi-random bytes real ECC data produces. The programmer’s raw dump mode was writing placeholder padding into the spare area, not skipping it.

binwalk-ng read the file as one continuous byte stream. JFFS2 node offset fields describe positions in the logical, OOB-stripped address space. Every 2048 bytes, the tool’s read position and the filesystem’s logical position drifted 64 bytes further apart. A handful of pages in, a small partition, the drift stayed small enough that the extractor still landed on real data. Thousands of pages in, a large partition, the drift compounded into exactly the kind of garbage the busybox binary showed: node headers parsed from the wrong bytes, sizes computed from noise, real content scattered wherever the misaligned offsets happened to point.

The fix was a single strip pass:

period = 2048 + 64
while chunk := f.read(period):
    out.write(chunk[:2048])

134,217,728 bytes out, exactly 128 MiB, exactly 4 times 32 MiB. Re-running binwalk-ng against the cleaned image gave page-aligned JFFS2 offsets and a clean extraction. busybox came back as a real 335,576-byte ARM ELF binary, and bin/ash pointed where it should.

This was a data-shape problem, not a binwalk-ng bug. But the tool has no way to know a dump carries OOB padding, and there is no way to tell it. An optional preprocessing flag, something like --nand-oob 2048:64, would turn this from a one-off script into a supported workflow. Anyone else pulling a raw NAND dump with a page-and-spare programmer is going to hit this exact corruption pattern.

One plaintext file was the root of trust for everything

With a clean rootfs in hand, several /etc config files stood out: db_default_cfg.xml and friends had entropy over 7.9 bits per byte and would not open as text. Encrypted, clearly, but by what key?

The trail led through libhardcode.so, an unstripped shared library whose name undersold how much it was hiding. A Ghidra decompile of its key derivation function, done in an afternoon instead of a slow hand disassembly of ARM Thumb-2, turned up:

key = SHA256(key_string)
iv  = SHA256(iv_string)[:16]
AES_cbc_encrypt(buf, out, len, key, iv, DECRYPT)

key_string and iv_string are not compiled in. They come from reading and lightly transforming the contents of a single file: /etc/hardcode. World-readable, -rw-r--r--, holding a 256-bit hex seed followed by a model string. That one file, present byte-for-byte identical in two firmware builds released over a year apart, is the entire root of trust for a directory called /etc/hardcodefile/, six encrypted blobs covering WiFi keys, TR-069 management credentials, GPON registration defaults, and SSH/FTP/web-admin passwords, plus, through a second key pulled from that same chain, 39 region-specific default-configuration databases, one per ISP that ships this router under its own branding.

Writing a Python decryptor against the recovered derivation opened every one of those files. All of it, on every device running this firmware, protected by one file anyone can read with cat.

The router’s own auth check does not run

Digging into the TR-069 (CWMP) remote-management path turned up a second, unrelated problem. The ConnectionRequest listener, the socket an ISP’s Auto Configuration Server uses to reach into a subscriber’s router, is guarded by an HTTP Digest check in httpd. Decompiling that check showed the real logic:

if (*(char *)(cfg_base + 0x1a0) == '\0' ||
    (strlen(username) == 0 && strlen(password) == 0)) {
    return 0;  // authentication succeeds, no digest check performed
}

A byte-by-byte reference scan across the entire binary found exactly one place that reads cfg_base + 0x1a0: the line above. Nothing writes to it, anywhere in the binary, and its compiled-in default is zero. There is no code path that ever flips this flag. The bypass branch runs on every boot, on every device, regardless of which of the 39 carrier profiles it shipped with.

The listener binds specifically to the WAN interface, confirmed by decompiling the socket setup, so this is not a LAN-only exposure. A firewall-rule audit across every iptables string in the management daemon showed Telnet, FTP, SSH, HTTP, and HTTPS explicitly blocked on both the LAN bridge and WiFi. The ConnectionRequest port was not in that list, on any interface.

None of this got IPv6 running on the LAN. That question is still open. But six hours with a chip clip turned into a NAND extraction bug worth fixing upstream, a credential store protected by nothing, and a remote-management bypass reachable from the WAN on every branded variant of this router, a good trade for an evening that started as a settings-page detour.

What you learned

  • A raw NAND dump from a page-and-spare programmer is not automatically a flat filesystem image. Check the total size against known page geometries (2048+64, 4096+128) before assuming an extractor bug.
  • Corruption that gets worse the deeper you scan, while staying clean in small regions, points at cumulative offset drift, not random damage.
  • A library named for what it does (libhardcode.so) is worth decompiling first.
  • When a “secret” file turns out to be world-readable and byte-identical across unrelated firmware releases, it is not a per-device secret. It is a constant, and everyone who has a copy of the firmware has it too.
  • A reference scan for who writes to a security-critical byte, not just who reads it, is the fastest way to tell a real check from dead code.