[gpsd-dev] Feature request (very long, sorry about that)

[Roy Barkas]

Feature request:

There are at least two companies who currently provide tcp/ip feeds of satellite captured AIS data.

One of the companies (probably both of them) prepend some information onto the front of the !AIVDM message.

The prepended information includes a timestamp that is set to the datetime that the message was received at the satellite.
Depending on the state of the vendors system and network the datetime stamp may vary quite a bit from the datetime that the
message is received by a subscriber.  If a subscriber assumes that the current date/time when the message is received 
at their system is "good enough" they'll soon discover that they are building potentially significant errors into their
data set.  As you're aware many AIS messages contain only the second of the current minute, not the minute, hour or date.

There is also information identifying which satellite of the vendors constellation received the data, a message sequence 
number and some other bits of data.

Currently, I receive the message from the satellite vendor, strip out the !AIVDM sentence and send only that part to gpsd via a
named pipe.  A separate program connects to gpsd on port 2947 and accepts json output which it then further processes.
To further complicate things I'll soon be adding a dynamically sized FIFO in the middle between the two programs.  So it doesn't 
seem practical for me to extract the metadata and by some black magic re-unite it with the decoded AIVDM data downstream.
My architecture exists as it is because of the incredible volume of data that I receive from the satellite vendor.

Obviously when the satellite data is passed through gpsd for decoding, the 'metadata' referred to above is lost.  The output contains the data that 
was in the !AIVDM message only (which is correct for the current design).

My proposal is to add a feature to gpsd that will allow metadata to be transparently passed through gpsd so that it can appear on the translated output 
along with the AIVDM data.

One way to do this would be to allow the gpsd input to take an alternative form in addition to the current text string format.
For instance, gpsd could interpret an AIVDM string in the way it currently does, or alternatively it could accept a json string
that contained (for example)

{
    "payload":"the AIVDM string",
    "datetime":"2013-01-23 23:00:04"
    "satellite":"24B"
}        

The idea being that a gpsd user would unpack the message as received from the vendor and reformat it in a particular way
that gpsd could tolerate.  gpsd would translate the "payload" no differently than it does now.  After gpsd has translated
the payload it would simply append the other objects found within it's input stream to it's own output.  No interpretation of the
'metadata' by gpsd would be required, it would simply add the additional items to it's output in the same form that they were provided.

So the output might look like:  (standard gpsd ais json output with two items added at end)

{"class":"AIS","device":"/var/www/tracker/pipes/ais_Q_server_pipe","type":3,"repeat":0,"mmsi":351201000,
 "scaled":false,"status":5,"turn":0,"speed":0,"accuracy":true,"lon":91062450,"lat":-19742820,"course":3340,
 "heading":174,"second":39,"maneuver":0,"raim":false,"radio":0,"datetime":"2013-01-23 23:00:04","satellite":"24B"}

 It might be wise to enforce some naming convention on the added data elements to avoid collisions with native 
 gpsd names - perhaps the names could require 2 starting underscores - i.e. "__datetime" or some such mechanism.
 
 In my opinion it would be better to require the user to format the json message according to some ruleset like 
 the one I've proposed rather than to try to change gpsd so that it can natively recognise all of the potential formats that
 various vendors might come up with.
 
 I suspect that the use case I present isn't the only or the last such suggestion / request that you'll see, 
 particularly for AIS data.
 
 I'd post some samples of the data as received except that it's covered by a non-disclosure.
 I'd work on adding the feature myself if I had any C language experience but alas...
 
 I'll be interested to see what the architect / developers think of my proposal.  Apologies for the length but I thought it
 better to be complete than abreviated.
 
 Cheers;
 
 Roy Barkas
 Australia.