Re: [gpsd-dev] new tcp tests fail on NetBSD

[Eric S. Raymond]

Greg Troxel <address@hidden>:
> 
> At release-3.9-645-g810b437, on NetBSD 6, i386, all regression test pass
> except for 2. (I have not tried anything live.)  I can't look into these
> until at least tonight.

Well, it's good news that the other failures are fixed.

> Has anyone seen these tests pass on other than Linux?

I don't have anything else to test on, sorry.

> The interesting part of the log, minus the ^M characters that gnus
> objects to:

So, it looks like trailing data is not passing to the message parser.  That's
not good; it may indicate that BSD-hosted GPSDs won't handle TCP data sources
correctly.

I'll explain in detail what these are testing, and give you some 
diagnostics to run.

We had a tracker bug report #36409: "GPSD fails to start get GPS data from
tcp://location:port".  The original bug report is at

https://savannah.nongnu.org/bugs/?36409

tcp-torture.log reproduced the bug.  Which wasn't trivial - it
actually took me two days of work to up-gun the test framework enough
to do it!  tcp-test.log is the same test with some internal write
delays removed.  I doubt the delays are relevant here since both tests
seem to fail in the same way.

Here is the plain TCP test:

# Name: Generic NMEA
# Submitter: Eric S. Raymond <address@hidden>
# Date: 2013-11-03
# Transport: TCP
# Notes: The most significant thing about this test load is not the content,
#        but the magic cookie that tells it to feed the daemon over TCP
#        rather than a pseudo-tty. We include spme leading garbage as a
#        robustness check for the packetizer.  There is a UDP test load 
#        that feeds the same content.
,1.7,-30.40,M,-13.9,M,,*7D
$GPGGA,193221.00,2037.72792,N,08704.08478,W,1,04,1.7,-30.40,M,-13.9,M,,*7D
$GPGSA,A,3,10,28,09,13,,,,,,,,,03.4,01.7,03.0*00
$GPGSV,3,1,12,28,14,150,41,09,15,254,41,10,43,192,47,13,06,081,36*7A
$GPGSV,3,2,12,02,56,323,,04,41,024,,12,31,317,,17,31,085,*72
$GPGSV,3,3,12,05,15,318,,24,02,246,,33,08,096,,35,45,118,*7D
$GPRMC,193221.00,A,2037.7279,N,08704.0848,W,00.1,201.8,231207,01,W,A*2D
$GPZDA,193223.00,23,12,2007,00,00*69

Both, uniquely in the test suite, exercise the TCP source feature.  

Ordinarily, the test framework feeds each log to a daemon instance
through the master end of a pty after telling the daemon to open the
slave end using the normal mechanism of a message through gpsd's
control socket. The daemon doesn't know it's seeing anything but a
normal serial device being hotplugged.  (Alas, though, TIOCMIWAIT on a
pty hangs, so I can't do synthetic PPS events.)

"Transport: TCP" field tells the test framework not to do what it
ordinarily does. Instead, it declares a TCP source to the daemon,
passing it a synthetic URL with a private-use IP address.  When the
daemon does a TCP socket open to this address, it's actually talking
to a serverlet running in a thread in the test framework. which feeds
it the test log in the same way it would over a pty.

The most significant clue here may be that the very similar UDP test
in udp-test.log is *not* failing.

# Name: Generic NMEA
# Submitter: Eric S. Raymond <address@hidden>
# Date: 2013-11-03
# Transport: UDP
# Notes: The most significant thing about this test load is not the content,
#        but the magic cookie that tells it to feed the daemon over UDP
#        rather than a pseudo-tty. We include spme leading garbage as a
#        robustness check for the packetizer.  There is a TCP test load 
#        that feeds the same content.
,1.7,-30.40,M,-13.9,M,,*7D
$GPGGA,193221.00,2037.72792,N,08704.08478,W,1,04,1.7,-30.40,M,-13.9,M,,*7D
$GPGSA,A,3,10,28,09,13,,,,,,,,,03.4,01.7,03.0*00
$GPGSV,3,1,12,28,14,150,41,09,15,254,41,10,43,192,47,13,06,081,36*7A
$GPGSV,3,2,12,02,56,323,,04,41,024,,12,31,317,,17,31,085,*72
$GPGSV,3,3,12,05,15,318,,24,02,246,,33,08,096,,35,45,118,*7D
$GPRMC,193221.00,A,2037.7279,N,08704.0848,W,00.1,201.8,231207,01,W,A*2D
$GPZDA,193223.00,23,12,2007,00,00*69

Notice that this test is identical except for "Transport: UDP".  Works
much like the TCP test, only the serverlet-like thingy is actually simpler
because it uses UDP and doesn't have to do any connection-establishment
rituals.

I think the first thing to do is establish whether we have a failure in
gpsd or the test framework.  Fortunately, that won't be hard.

I've attached Jiri Pinkava's script, tcp-test.py, from the original 
bug report.  Try this:

1. Run python tcp_test.py in one window.  This will run a little TCP
   server reasdy to ship a few NMEA sentences.

2. Run gpsd -N -D4 tcp://127.0.0.1:50100 in another window.  This gpsd
   will stash the server address, ready to get data from it when a client
   does  a watch.

3. In a third window, run gpspipe -rw.  This will ship a watch request
   to the gpsd instance, causing it to open a connection to the stashed
   server address and read the data it is ready to present.

4. Look at the JSON output. Disregard the VERSION, DEVICES, WATCH, and
   DEVICE lines at the beginning.  What you want to do is compare the
   remainder to the contents of test/daemon/tcp-torture.py.  Which I'll
   reproduce here:

$GPGGA,193221.00,2037.72792,N,08704.08478,W,1,04,1.7,-30.40,M,-13.9,M,,*7D
{"class":"TPV","tag":"GGA","mode":3,"lat":20.628798667,"lon":-87.068079667,"alt":-30.400}
$GPGSA,A,3,10,28,09,13,,,,,,,,,03.4,01.7,03.0*00
{"class":"TPV","tag":"GSA","mode":3,"lat":20.628798667,"lon":-87.068079667,"alt":-30.400,"epv":69.000}
$GPGSV,3,1,12,28,14,150,41,09,15,254,41,10,43,192,47,13,06,081,36*7A
$GPGSV,3,2,12,02,56,323,,04,41,024,,12,31,317,,17,31,085,*72
$GPGSV,3,3,12,05,15,318,,24,02,246,,33,08,096,,35,45,118,*7D
{"class":"SKY","tag":"GSV","xdop":0.76,"ydop":1.60,"vdop":3.00,"tdop":0.99,"hdop":1.70,"gdop":3.70,"pdop":3.40,"satellites":[{"PRN":28,"el":14,"az":150,"ss":41,"used":true},{"PRN":9,"el":15,"az":254,"ss":41,"used":true},{"PRN":10,"el":43,"az":192,"ss":47,"used":true},{"PRN":13,"el":6,"az":81,"ss":36,"used":true},{"PRN":2,"el":56,"az":323,"ss":0,"used":false},{"PRN":4,"el":41,"az":24,"ss":0,"used":false},{"PRN":12,"el":31,"az":317,"ss":0,"used":false},{"PRN":17,"el":31,"az":85,"ss":0,"used":false},{"PRN":5,"el":15,"az":318,"ss":0,"used":false},{"PRN":24,"el":2,"az":246,"ss":0,"used":false},{"PRN":120,"el":8,"az":96,"ss":0,"used":false},{"PRN":122,"el":45,"az":118,"ss":0,"used":false}]}
$GPRMC,193221.00,A,2037.7279,N,08704.0848,W,00.1,201.8,231207,01,W,A*2D
{"class":"TPV","tag":"RMC","mode":3,"time":"2007-12-23T19:32:21.000Z","ept":0.005,"lat":20.628798333,"lon":-87.068080000,"alt":-30.400,"epx":11.444,"epy":24.060,"epv":69.000,"track":201.8000,"speed":0.051}
$GPZDA,193223.00,23,12,2007,00,00*69

If what you see is the same as this, then gpsd is actually getting all the
data from Jiri's fake TCP source, and there's a port bug somewhwere in
my test framework.

If what you see is different, there's a port bug in gpsd itself 
somewhere.  The good news is the amount of code that differs from
the (working) UDP case is very small.
-- 
                <a href="http://www.catb.org/~esr/";>Eric S. Raymond</a>

tcp_test.py
Description: Text Data

signature.asc
Description: Digital signature