The Clock That Lost an Hour

· Paul Philippov

You flash Thingino onto a camera, open the live view, and there it is: the timestamp burned into the corner of the picture is wrong. Not scrambled, not missing — wrong by exactly one hour. The kitchen clock says 6:30; the camera says 5:30. The video is crisp, the stream is smooth, and only the little clock in the corner is telling a lie.

What follows is the story of how that one wrong hour got chased down, and why fixing it meant throwing out a pile of well-intentioned, separately-reasonable code and replacing it with a single small program that refuses to argue.

The camera that looked innocent

The first thing to check was the camera’s clock, and date reported the right time. The timezone is America/Toronto, resolved to the POSIX string EST5EDT,M3.2.0,M11.1.0. The burned-in timestamp and the one embedded in the stream both agree with date. On this camera, at this moment, everything is correct.

That is the first lesson of this kind of bug: a self-correcting system covers its tracks. The camera’s NTP daemon had already synced and stepped the clock back to reality. By the time you look, the crime scene is clean.

The clue was in the log

The break came from a second camera showing the same symptom, and two log lines:

Aug 14 06:33:22 onvif_simple_server: SetSystemDateAndTime: date -s "2026-08-14 05:33:24"
Aug 14 05:33:24 mini-snmpd: could not get ticks: time running backwards

At 06:33:22, something ran date -s "05:33:24" — setting the clock back a full hour. The next line is a system daemon noticing that time went backwards, which no daemon enjoys.

That “something” turned out to be an ONVIF client. ONVIF is the management protocol a network video recorder or home-automation box uses to boss cameras around, and one of those clients had decided to set this camera’s clock. It sent a time one hour behind, and the camera applied it without a second thought.

Why one hour? The client’s daylight-saving setting was off. It sent the time in standard time (UTC-5) when the correct local time was daylight time (UTC-4). The camera trusted the client. And the wrong value stuck, because this camera’s NTP daemon was sitting at stratum 16 — unsynchronized — so nothing ever pulled the clock back.

The moment the problem changed shape

At this point the bug looks like a bad NVR. A client with daylight saving turned off reached over the network and moved the clock. The obvious fix is to flip that NVR’s setting, close the ticket, move on.

That answer did not sit right. The NVR was wrong, sure — yet the camera had let itself be moved. Any device on the network that speaks ONVIF could reach in and set the clock, and the firmware would nod and apply it. That is not a bad NVR. That is a camera with no opinion about its own time.

Which raises the real question: who, on this camera, owns the clock?

Everyone, it turned out — which is the same as no one.

Five voices, no conductor

Five independent subsystems were writing the clock and the timezone, each with its own idea of what was right and its own private code for doing it. ntpd held the UTC clock, or was supposed to. DHCP pushed a timezone and an NTP server with every lease. ONVIF pushed the clock whenever a client asked. The WebUI could write all three — the timezone, the NTP server, and the clock — from its own settings pages. And the Home Assistant integration had a command pipe sitting there, ready to do the same the moment anyone wired it up.

Each one was reasonable in isolation, and together they could stomp on each other. The ONVIF handler applied whatever the client sent, with no sanity check and no reconcile with NTP. The WebUI’s manual time set was a bare date -s, not even the hardware-clock sync the ONVIF path had. That made two ways to set the clock with two different behaviors, and neither asked NTP for a second opinion.

The timezone resolver — the bit that maps a friendly name like America/Toronto onto the POSIX string the C library understands — was copy-pasted into two scripts, so the copies could drift apart. Precedence, the question of who wins when DHCP and the user disagree, was enforced by two unrelated mechanisms: a boolean flag in thingino.json, and the file permission 444 used as a secret handshake meaning “the user pinned this”.

And a timezone change triggered a full reboot -f. The camera restarted itself to apply something the C library already knew how to re-read.

Why a gate, not another bandaid

The first instinct, once you see the ONVIF handler apply a wrong time and never check with NTP, is to add a check right there — one line that says “after a manual set, re-sync NTP.” A five-minute fix that would have felt good and fixed nothing.

Because the ONVIF handler was not the problem — it was one of five copies of the problem. Patch it, and the WebUI still sets the clock with a bare date -s. Patch that, and DHCP still carries its own timezone logic with its own idea of precedence. You would be tightening five screws that keep loosening, forever.

The disease was not “this one writer is sloppy.” It was “there is no single owner, so every writer reimplements the rules, and the rules drift.” The cure has to be structural: make every writer pass through one gate, and put all the rules behind that gate.

That gate was a small program named timectl.

The uClibc-ng catch

One more discovery shaped the gate before it was built. Time handling is shaped by the C library, and Thingino runs on uClibc-ng, which handles timezones differently from glibc.

glibc reads the IANA timezone database under /usr/share/zoneinfo, compiled files that encode every political timezone change in history. uClibc-ng does none of that. It has two sources of truth: the TZ environment variable, or a single /etc/TZ file holding one POSIX TZ string.

A POSIX TZ string like EST5EDT,M3.2.0,M11.1.0 describes a fixed offset and one simple daylight-saving rule. It cannot express a city whose rules changed a dozen times over a century. For a camera, that is a fine trade — but the whole design hangs on that one file being right.

And there is a trap hiding in it. When TZ is set in a process’s environment, uClibc-ng uses it and caches it. When it is unset, uClibc-ng re-reads /etc/TZ on every localtime() call — which is why a timezone change can take effect with no reboot, provided no daemon ever carries TZ in its environment.

That “provided” was the landmine. Daemons started at boot were clean. A daemon restarted from an SSH session or a WebUI action inherited an exported TZ from the shell profile and cached that value forever. The same timezone change that applied live on one camera would silently never apply on another, depending on how the daemon had been restarted. The reboot hammer existed because nobody trusted the live reload, and nobody trusted the live reload because the TZ leak made it unreliable.

One gate: timectl

So the gate had to do three things, and each one comes straight from the chase.

One authority per state. ntpd owns the clock. A manual time set is a hint: apply it, then immediately reconcile with NTP, so a client that sends garbage gets corrected instead of persisting. This is the bandaid from before, promoted to a rule — because now there is one place where “set the clock” can happen, and the reconcile lives there instead of being a step each writer had to remember.

A precedence ladder. Every write carries a source: default, dhcp, or user. A lower source cannot override a higher one. DHCP can fill in a timezone, but it cannot overwrite one you chose in the WebUI. The old “ignore DHCP timezone” checkbox became pin-timezone, which marks the current timezone as yours without changing it. One rule replaces the boolean flag and the 444 handshake.

No reboot, ever. With TZ banished from daemon environments, every daemon re-reads /etc/TZ live, so a timezone change applies the moment it is written. The reboot hammer becomes unnecessary the moment the leak that justified it is closed.

The whole gate fits in a screenful:

timectl status                      # key=value dump for anything that reads time
timectl refresh                     # boot: resolve /etc/timezone -> /etc/TZ
timectl set-timezone <name> [--source user|dhcp]
timectl set-ntp <srv>...   [--source user|dhcp]
timectl set-time <date>             # hint + NTP reconcile
timectl pin-timezone                # keep current timezone, ignore DHCP
timectl unpin-timezone              # allow DHCP to change it again
timectl sync                        # one-shot NTP sync

The WebUI, the DHCP hooks, the tzselect picker, the wifi portal, and ONVIF all became one-line clients of it. The duplicated resolver, the 444 permission hack, the reboot hammer, and the stale flag are gone. The full design lives in docs/timezone.md.

What you learned

  • When a device’s clock is wrong, read the log for who last wrote it before touching the timezone. A date -s line next to “time running backwards” is the whole case file.
  • A self-healing system covers its tracks. The camera that looked correct had already been corrected by NTP; the one still wrong was the one whose NTP was unsynchronized.
  • Give each piece of shared state one owner. Five writers with good intentions are five ways to stomp on each other.
  • Precedence belongs in one place. A boolean here and a file permission there become a policy nobody can reason about.
  • Know your C library’s timezone story. uClibc-ng reads one /etc/TZ file, not a zoneinfo database, and the TZ environment variable always wins — keep it out of daemon environments.