Re: [gpsd-dev] U-Blox M8, PPS and NTPsec - GPSd logic enhancement

[Martin Boissonneault]

Good Sunday Gary!

  Before I continue, I'll get a bit in the context of this Pi, why GPSd and what I was trying to do since I got it in January 2019:

  This Raspberry Pi 3B+ serves as an ADSB feeder to a few networks. It needed accurate time to help correlate MLAT messages, so I installed NTPd (Classic). I then read about NTPsec and switched. I had to compile from source tarball because I could not find the package in the Raspian repos.

  I then decided to add a GNSSr to improve precision, this one:
Uputronics - uBLOX MAX-M8Q Breakout for Active Antennas

  I interfaced it (over serial to /dev/ttyAMA0, PPS to gpio 18) to the Pi with a prototyping hat on which I installed a 3.3v regulator+capacitors to power the u-Blox. Of course, some configuration was required on the Pi for this to work.

  I have kernel PPS working and NTPsec can use it, but I wanted to add time information using GPSd for times when Internet is inaccessible. The GPSd package on the repos is 3.16.4, so I installed from the 3.18.1 tarball. Using the files in the systemd folder of the tarball, I set up automatic start of GPSd.

  Until you are satisfied with the GPSd configuration and operation, NTPsec configuration changes will wait.

My main references to get NTPsec/GPSd working where GPSD Time Service HOWTO, Quick start NTP on the Raspberry Pi and The Raspberry Pi as a Stratum-1 NTP Server.

On 2019-04-06 18:48, Gary E. Miller wrote:

Yo Martin!

On Sat, 6 Apr 2019 02:02:50 -0400
Martin Boissonneault <address@hidden> wrote:

Once again, you have still not said where SHM(1) and SHM(2) come
from.  

Well, the only answer I have is that it's from GPSd.

OK, so what are you telling gpsd to do?  Here is how to
find the running gpsd command line, as root:

# pstree -paul | fgrep gpsd

The result shows  -n /dev/gps0 /dev/pps0

I have a single GNSS attached to the Pi.

USB, serial, i2c?

Serial over /dev/ttyAMA0 through a link named gps0, but I also made a link named gpsd0. The /dev/pps0 also has a link named gpspps0.

Why SHM(1)? GPSd was started from the
command line, not by systemctl.

Mixing systemd and command lines ways to start gpsd is very hard.
The "pstree" above should help disentangle what is really running.

At some point I needed to troubleshoot so I did run GPSd from command-line, but only for the troubleshooting session. The normal method is with systemd, and I use systemctl to stop/start GPSd. As I write, systemd/systemctl is the ONLY method I use.

That includes the script I use to copy modified NTPsec and GPSd configuration files to the proper folders.

If memory is right, it showed as
SHM(1) instead of SHM(2). The fix was too easy to care as to why. I
just added the SHM(1) for those times it does. It's disabled now.

Except if you are randomly getting SHM(1) or SHM(2) then things are
not going well.

It's not random. Since I have exclusively used systemctl to start/stop GPSd, it has always been SHM 0 and 2.

A common mistake to try to fix things in ntpd, but is it very 
important that gpsd is working perfectly before starting ntpd.

Always debug the first failure.

Agreed. Nothing worse than changing multiple things at a time. The circle of confusion grows. Unfortunately, I got the GNSSr after I began using NTPd. And the combo has been giving me nice accuracy for the last few months.

Modified:
#refclock shm unit 1 refid gPPS prefer

Right now, GPSd gives me SHM(0) and SHM(2) as visible from ntpshmmon.
As it should.

No, SHM(2) says something bad is happening.

# GPS Serial data reference (NTP0)
#refclock shm unit 0 refid GNSS flag1 0 time1 0.0 minpoll 0
maxpoll 8 stratum 10 noselect
#refclock shm unit 0 refid GNSS flag1 0 time1 0.025 minpoll 0
maxpoll 8 stratum 10 noselect
# stratum 10  

Why is this commented out?  You need SHM(0) to get the right second
for PPS.  

I fail to understand how I need SHM(0) if I'm to make it /noselect/
in ntp.conf?

You need it to know the GPS is working.

Eventually you will remove the noselect and add a large offset so
it never gets selected.  But let's not get ahead of you first failure.

It's the serial timestamp, not the PPS from GPSd?

Correct.  The PPS does not tell you the date, just the top of the
seccond.

I think I don't understand SHM(0) and SHM(2) properly then. I thought that the SHMs carried accurate time as determined by GPSd from the GNSSr's PPS and serial messages. So, SHM(2) carried the accurate time at the PPS  edge as determined from the GNSS time messages? In a way, SHM(2) would be time calculated from serial messages but corrected to the edge of PPS by GPSd? Hard to put in words. English is not my first language...

Or maybe the SHM timestamps are relative to the local timer, and NTPd has to correlate all that into true time. I'll go read more about NTP SHM. References welcome!

The time pulse on GPIO 18 does not carry any time, only Top Of Second, that I know.

"noselect: Marks the server as unused, except for display purposes.
The server is discarded by the selection algorithm." 
From:https://docs.ntpsec.org/latest/ntp_conf.html

Really confusing.

And display purposes will tell us more about your problem.  Later
we'll worry about selection.

what is conencted to SHM(2) and SHM(2)?  

Both SHM(1) and SHM(2) are from GPSd.  

Obviously, but what is gpsd configured to send as SHM(1) and SHM(2)?
And what happend to SHM(0)?  Very strange configuration.  

Depending on how I start GPSd, I get SHM(0) and (SHM(1) /or /SHM(2),
not both). If systemctl starts GPSd, it's gonna be SHM(2). Anyway, I 
commented out the SHM(1) refclock as it's not normally there.

You can't debug a random system.  Pick one, get it to work.

The right ones on my system are SHM(0) and SHM(2). If I get anything else, something has happened and it's broken.

Don't think about SHM(1), I haven't seen it for many months. Not since I only use systemd/systemctl to start/stop NTPd and GPSd.

They are the SHM related to the
GNSS PPS from the offsets I see. Both are the same PPS from the
GNSS attached.  

You have two PPS sources?  What does that look like on your gpsd
command line?  

Only one PPS source (GPIO pin), plus the serial port (no PPS on it).

But your PPS flips between SHM(1) and SHM(2).  Figure out why.

That has been fixed for many months now. I only see SHM(0) and (2). If you really want me to, I can try to get GPSd to give me SHM(0) and (1), but this is not normally happening.

--/etc/default/gpsd:
# Default settings for the gpsd init script and the hotplug wrapper.

That file is NOT part of gpsd.  It is not supported here.  And it you
are starting manually, this file is not used at all.

So, pick one: use systemd, or use some other command line method.

I have been using systemd/systemctl for the last few months exclusively. The systemd files are in the systemd folder of the tarball. I might have adjusted them to fit my system.

Raspbian has both SysV and systemd... I use systemd whenever I am given the choice, but if SysV is selected by the package, that will be used.

# Devices gpsd should collect to at boot time.
# They need to be read/writeable, either by user gpsd or the group
dialout. #DEVICES="/dev/ttyAMA0"
DEVICES="/dev/gps0 /dev/pps0"

I assume you set /dev/gps0 as a link to /dev/ttyAMA0?

Exactly. I also have /dev/gpsd0 as a link to /dev/ttyAMA0, and /dev/gpspps0 linked to /dev/pps0.

# Other options you want to pass to gpsd
GPSD_OPTIONS="-n"

GPSD_SOCKET="/var/run/gpsd.sock"

Since you are hard coding the GPS, and you are not doing hotplug, then
you do not want the soacket.

I'd be glad to disable that, but I haven't given time to learn how to do it. I had much to learn to get to this point, Linux is fairly new to me. I've done what I could in three months of learning. Of course, not full-time, it's all done health permitting.

--/usr/lib/systemd/system/gpsd.service:

Also not part of gpsd, and unsupported here.

Taken from the systemd in the tarball. Manually installed.

[Unit]
Description=GPS (Global Positioning System) Daemon
Requires=gpsd.socket
# Needed with chrony SOCK refclock
After=chronyd.service
Before=ntpd.service

Running chronyd and ntpd?  How do you manage that?

Absolutely not!! NTPsec only!!! The file came with Chronyd references.

# Give a few seconds for init, that way NTPd will have valid data on
start ExecStart='/bin/sleep 10'

That is a joke...

More my attempt to have GPSd ready and serving before starting NTPd. Suggestions welcome!

Only one shows up, usually SHM(2).  

"usually" is not a good thing.  That shows a major problem.  Fix
that before continuing.  

It shows only when I start GPSd from the command-line. Haven't seen
it for many months because I am using systemctl to stop/start GPSD,
and it is SHM(2) now. SHM(1) is a ghost from testing.

Make up your mind: systemd or command line.

systemd. All NTPd/GPSd start/stop is handled by systemd. I start/stop with systemctl when needed. NO other method is used to run them. The only exception is command-line for debugging, and only for the troubleshooting session.

I also have SHM(0) from GPSd, and that one jumps ~100ms every 45
minutes. I assume SHM(0) is from the serial messages. It's
disabled.  

Don't disable that.  You need that to provide the proper second for
the PPS.  Just "noselect" it.  

If I do so, how can NTPsec use it?

Have to confirm it is good, and how far off.  Then add a fudge so it
never gets selected, unless nothing else is available.  The remove the
unselect, and that helps steer the PPS to the right second.

But, one step at a time.

Let's get GPSd good before. We'll come back to NTPd

"noselect: Marks the server as unused, except for display purposes.
The server is discarded by the selection algorithm." 
From:https://docs.ntpsec.org/latest/ntp_conf.html

Makes me really confused to "need" it, but having it "for display 
purposes".

Don't try to over think it, small steps to where you need to go.

All right, u-Blox has been reset to defaults with CFG-CFG, Revert to 
default. I have everything backed up.

sudo systemctl disable gpsd - to prevent interference

sudo ubxtool -e BINARY -f /dev/gps0
sudo ubxtool -d NMEA -f /dev/gps0

Prolly best to swap those.  For a while you have both BINARY and NMEA,
at 9600, and that may very well cause buffers to overflow.

sudo ubxtool -s 9600 -S 115200 -f /dev/gps0
sudo ubxtool -s 115200 -f /dev/gps0

Get the speed up, first.  To avoid buffer drop.

Done. I now set speed first and then reconfigure NMEA and UBX messages.

Sort of.  Stop doing "sudo".  Just become root.

I find it more convenient for scripts, and that was from one. Sure, I could sudo <script>, but it had some issues when I tried with some of my past scripts. I don't remember the specifics, but this way of doing fixed my issues.

As for becoming root, that changes my current folder. My scripts are under a subfolder of my home folder, and I'd have to cd back to that folder again and again. Sure, putting all those scripts in the root home folder would fix that, but then I would not be able to modify them from WinSCP because of permissions. I could fix the permissions, but what is the point in the end? I'd have stuff spread all over the place, and right now, it's all tidy under my ~/projects subfolders and working fine.

Of course, for one-shot commands, becoming root is the way... Don't worry, I'll get better with time...

sudo systemctl enable gpsd

Which just undid all that work. 

Either use systemd, or don't.  You can't
mix them.

man systemctl says it's from systemd?!?
systemctl - Control the systemd system and service manager
DESCRIPTION
       systemctl may be used to introspect and control the state of the
       "systemd" system and service manager. Please refer to systemd(1) for an
       introduction into the basic concepts and functionality this tool
       manages.

What am I misunderstanding?

Now, about /sudo ubxtool -d SBAS -f /dev/gps0/: it does NOT disable
SBAS.

Well, duh.  You just started systemctl, so /dev/gps0 is no longer
available.

No, I mean running ubxtool -d SBAS -f /dev/gps0 BEFORE starting GPSd does not change the status of SBAS in CFG-GNSS. It is not sending. Confirmed from serial sniffing. I have no idea why, something to look into later. Bug?

ubxtool: disable SBAS
sent:
UBX-CFG-GNSS:
 Ver: 0 ChHw; 0 ChUse: ff, Blocks: 1
 gnssId: SBAS TrkCh: 1 maxTrCh: 3, Flags: 0x1 01 00 00
  L1C/A

I get $GPTXT,01,01,01,More than 100 frame errors, UART RX was disabled*6E in the u-Center console sometimes.

Once GPSd runs, ubxtool -d SBAS 127.0.0.1:2947:/dev/gps0 DOES work. I get ACK-ACK for 0x06-0x3E.

I'm trying to use known tools, but I still have to use u-Center to 
disable SBAS and set Timing-friendly settings.

No, you do not need u-center.  But you gotta learn how to use
ubxtool correctly.  Let gpsd start first.  Let it autobaud, configure,
etc.  Then use ubxtool, without the -f, to configure the GPS.  You have
to order all wrong.

GPSd will set it's own configuration on start, including enabling SBAS incorrectly (described below). Not good for timing, but them I'm not supposed to interfere. I have to run ubxtool -d SBAS 127.0.0.1:2947:/dev/gps0 at every start of GPSd now. The jumper prevents GPSd to do enable SBAS. I'm fixing GPSd with hardware. GPSd resets the GNSSr to saved config on stop. So, I configure to my liking and save. It would be so nice if the required message list was easy to find or configurable.

I know GPSd is meant to be transparent and return the configuration to what it was before it started, and I normally would appreciate it. Unfortunately, the default is not what I want outside of GPSd use. I must configure it and save. u-Center makes it 100x easier than ubxtool.

Keep in mind that I want it up fast so NTPd has time available on start. Autobauding and reconfiguration is an annoyance in that context, but only in that context.

It would be perfect if I could get the GNSS configured as GPSd expects it with speed preset BEFORE GPSd starts. No wasted time on autobauding and reconfiguring. Of course, GPSd should not reset it while exiting. That would be with -b?

Now, I set:
CFG-ANT: Set according the the hardware.
CFG-GNSS: Disable SBAS, Enable Galileo. Sets CFG-SBAS to Disabled.
(Read to the end, GPS/QZSS are the only enabled.)
CFG-ITFM: Set antenna to "active", Enabled.
CFG-NAV5: Dynamic: Stationary, Mode: 2D, set HAE to that of my
location. CFG-NMEA: NMEA version 4.1, Numbering 1-Extended. (Doesn't
matter, the messages have been turned off.)
CFG-PMS: Full Power. Sets CFG-RXM Power to 0-Continous.
CFG-PRT: For 1-UART1: Protocol out 0+1 UBX+NMEA.NMEA messages are off 
anyway.
CFG-TP5: For 0-TIMEPULSE: Set Cable Delay to 15ns, that's for my
antenna. CFG-CFG: Save current configuration.

Why set NMEA version, then turn off NMEA?  Stop doing pointless things...

Because I might enable some NMEA messages for use by other programs I have on Windows. There is no harm in having the base configuration set and turning the messages off?  I don't see it as pointless, I get the GNSSr ready for future NMEA use. It is unused, but not pointless to me.

None of that will have a noticeable change in your ntp stability.

Stop over complicating...

Which part? Why? I'm listening. 

Once again, gpsd did not do that, but you really do not want to use
Galileo and Glonass when doing timing.  So says the offical u-blox
doc.  

Ghost in the machine:
When the jumper is connected, SBAS gets reactivated.

And you wonder why I keep saying don't do that?

I don't want SBAS for timing. When the jumper is on, it turns on. No jumper, no SBAS. And ubxtool -d SBAS -f /dev/gps0 does not turn it off.

GPSd does turn it on on start. I logged the GPSd start and stop commands sent.

(BUG) - And incorrectly:

22:47:35  [19:07:41.54]  0000  B5 62 06 16 08 00 03 07 03 00 00 00 00 00 31 E5  µb............1å.
UBX-CFG-SBAS
mode        0x03    SBAS Mode
    1   enabled     SBAS Enabled (1) / Disabled (0) - This field is deprecated; use UBX-CFG-GNSS to enable/disable SBAS operation
    1   test        SBAS Testbed: Use data anyhow (1) / Ignore data when in Test Mode (SBAS Msg 0)
           
usage       0x07    SBAS Usage
    1   range       Use SBAS GEOs as a ranging source (for navigation)
    1   diffCorr    Use SBAS Differential Corrections
    1   integrity   Use SBAS Integrity Information
   
maxSBAS     0x03    Maximum Number of SBAS prioritized tracking channels (valid range: 0 - 3) to use (obsolete and superseded by UBX-CFG-GNSS in protocol versions 14+).

scanmode2   0x00
scanmode1   0x00000000

In CFG-SBAS, the test bit must be OFF. Otherwise, information from SBAS sats in testing will be used. Here, it means that SBAS not for production use will be used! Not good.

Fixed string (caveat below), in 0x00 format:
B5 62 06 16 08 00 01 07 03 00 00 00 00 00 2F D5

Take note that this byte string is not the best for everyone, like me. It sets SBAS in Autoscan mode. I can see one SBAS sat from Europe, but it is inappropriate for selection as it would not carry information for my geographic area. The recommended setting would include only the current WAAS svns, sv131, 135, 138. This setting should be configurable by the user.

I had disabled the ability for GPSd to send to my GNSSr and it fixed
the problem, but I now see GPSd is not at fault. Constellations get 
disabled, although indirectly. It's very odd that only SBAS is
affected. could be a u-Blox bug. Dang. I now need to find out why...
Possibly some ground loop or RF, my Arduino a few feet away sometimes
spontaneously reboots, and it's connected to the same computer.

I think you are over thinking it.  More likely a glitch when you move the
jumper causes a reset.

GPSd does turn on SBAS on start. I captured the commands sent on GPSd's start today. So, my jumper fixes the issue.

Yesterday, my capture attempt was looking for a delayed command, but it is sent right at the start. Acquiring SBAS takes some time, that is why it did not catch it.

For now, I'll leave the jumper OFF, as it's already configured. After 
that ghost, I disabled GPS/QZSS so they are the only ones enabled.
SBAS disabled, config checked and saved.

Note that u-blox doc says: don't do that!  GPS and QZSS should both be
on or both off.

Oops, I meant everything BUT GPS/QZSS. I wrote that part too late at night and couldn't re-read it properly ;-)

OK, then I must ask what does GPSd expect?  

I guess you keep missing the part where I say to set your GPS with
gpsctl or ubxtool.  

Not every setting I need to set is easily accessible from those
tools.

Think about that.  Maybe it is not easy because it is the wrong thing
to do?

Well, I disagree with that. By default, the GPSd+u-Blox M8 combination is not optimal for timing purposes. What option do you offer to set antenna cable length delay.?

Again, keep in mind that ubxtool -d SBAS -f /dev/gps0 is not disabling SBAS in my setup even if it says it does. I tried it with GPSd stopped, of course.

Likely a bug, but I have to work around it.

Besides, the tool does not matter, as long as the
configuration is good. That is why I described my changes above..

Which do not look good...

I ran "ubxtool -e BINARY -f /dev/gps0" above. It enabled NAV-POSECEF, 
NAV-DOP, NAV-VELECEF, NAV-SBAS, plus those I already had, NAV-SOL, 
NAV-TIMEGPS. It prefers NAV-SVINFO to NAV-SAT, I have no problem with
that.

Before or after running gpsd?  

Before.

If before you run gpsd, then the settings
are lost.  

Well, not quite, now that it is been set and saved. The configuration does not change because I ran it, let it change it and injected my adjustments with a save command. GPSd does send a reset command, but it only brings back the same configuration that I did save. It resets to same.

We can't really support configurations that are not well tested
and part of our regressions.  

I don't mind giving full GNSS control back to GPSd, but then, why
was it disabling Galileo and Glonass?  

No, it was not.  And anyway that is the right thing to do.  

No, GPSd does not send the commands. I even recorded the stream to 
check. It somehow gets "disabled", but not by GPSd. That is something 
somewhat solved. :-/

If gpsd not sending commands, then something is badly broken.

GPSd does send commands to the u-Blox, and they are ACK-ACK.

I'm only speaking about SBAS getting re-enabled by GPSd and Galileo/Glonass getting ignored as a result. I was saying I could not find a command sent by GPSd to cause THAT issue. I've found that command today. I was searching for something different yesterday. I changed my snooping method and the commands have been logged.

GPSd does send UBX-CFG-SBAS, SBAS gets incorrectly enabled (test is enabled) and as a result of the integrity information carried by the SBAS sats visible to me (sv138), anything other than GPS is deemed unhealthy. Thus, Galileo and Glonass are ignored.

Too early to look at ntpmon, your pool servers not running yet.  

Well, they where up, and after a while, they where released. NTPd
picks the best and drops the unneeded extras?  

No.  

When NTPd starts, it does use a associate to a bunch of pool servers. 
They are later pruned by NTPsec as needed. I just use the pool 
statement. I don't mess with anything else. If they get pruned, I
have no hand in it. That fits what I read there: 
https://docs.ntpsec.org/latest/discover.html

I think you misunderstand what it is saying, but low priority.

SHM(1)          .gPPS.           0 l  64    0 0ns      0ns
954ns  

Why is SHM(1) broken?  

Already answered: It's there only for when GPSd is started from the 
command-line while debugging. Wierd, but it works, and then it goes
back to SHM(2) when systemctl starts it. Disabled now anyway.

Which tells me that when you run those two ways you get very different
results.  That flip-flopping is confusing you.

Yes, starting GPSd differently does change what SHM(0)+(SHM(1) or SHM(2) I get. No idea why. I can send you a backup image if it can help figure it out.

For me, the SHM(1) issue is closed. GPSd works, it is stable, no surprise. I am going with what GPSd figures it should be giving me. As long as it is consistent and usable for the purpose, I'm ok with it. For the last few months, after my troubleshooting sessions, I ALWAYS get SHM(0) and SHM(2) with systemd.

I'd care if it "flipped" from one run to the other. It doesn't.

After another day of working on this, time to rest.

Have a nice day!
Martin