> Is a long-period sawtooth what I should expect to see?
It should be noise rather than a sawtooth, but temperature changes are easily
visible so there may be patterns in the noise.
Note that temperature comes from the environment outside the box such as sun
through a window or the cycling of heating/air-conditioning as well as
internal heating due to the workload.
----------
Long story:
The crystals used in consumer electronics have two properties we are
interested in: accuracy and stability. Accuracy is how well the measured
frequency matches the number printed on the can. Stability is how well the
frequency stays the same even if it isn't accurate. (Long term aging is a
3rd property that might be interesting, but probably not for this discussion.)
Typical specs for oscillator packages are 20, 50, 100 ppm. That includes
everything; initial
accuracy, temperature, supply voltage, aging...
With a bit of software, you can correct for the in-accuracy. ntpd calls it
drift. It just takes some extra low order bits on the arithmetic doing the
time calculations. In the simple minded case, if you thought you had a 100
MHz crystal, you need to change that to something like 100.000324.
The major source of in-stability is temperature. Ballpark is the drift
changes by 1 PPM per degree C. (HP used to make a thermometer that used a
crystal as the sensor. You can measure frequency very accurately.)
So how do we calculate the drift? The general idea is simple. Measure the
time offset every N seconds, plot the graph, and fit a straight line. The
slope of that line is the drift. The units cancel out. Parts-per-million is
a handy scale.
How do you turn that handwaving description into code? One easy way is to
use 2 points and pick
the right N, then run the answer through a low pass
filter. In that context, there are two competing sources of error. If N is
too small, the errors on each individual measurement of the offset time will
dominate. If N is too big, the actual drift will change while you are
measuring it. In the middle is a sweet spot.
http://www.leapsecond.com/pages/adev-avg/
The ballpark is somewhere between 10 and 10,000 seconds.
ntpd adjusts the value of the polling interval to get the best results. That
turns into the N above.
If you turn on logging for loopstats, that will record both offset and drift.
(and the polling interval) It's easy to feed to gnuplot.
----------
There are various worms in this can.
While ntpd is adjusting the polling interval, it is assuming that the drift
is not changing. It gets confused if your drift
changes abruptly, say
because you started some big chunk of work on a machine that's usually idle
and that raises the temperature.
ntpd writes out the drift every hour or so. (Less often if it hasn't changed
much to reduce the workload on flash file systems.) On startup, ntpd reloads
the old value.
If you look at the boot time startup messages, you will find things like:
klogd: Detected 2792.915 MHz processor.
klogd: Detected 2793.311 MHz processor.
Those differ by 140 PPM. On this scale, that's huge. The problem is that
the Linux code that measures the speed of the CPU clock isn't repeatable.
(It may be better to delete the drift file before starting ntpd. I haven't
done much testing. It depends on your OS. YMMV...)
If you restart ntpd, it should start with a close old drift value and quickly
converge to the newer slightly different value. If you reboot, expect it to
converge to a new/different drift value and that may take a while depending
on how different the basic calibration factors are.