[gpsd-dev] Virtual sensors and gpsd architectures

[Olivier Cornu]

Playing around gpsd's architecture and the design challenge of
elegantly introducing virtual sensors somewhere in the loop between
real sensors and final clients, i came up with a few graphs (hopefully
worth N words).
So i engaged in summarizing the different alternatives, using these
graphs as illustration.

In the following, I take for granted that the proper abstraction to
solve this whole problem is the _virtual_sensor_ (VS): it's a simple
"mechanical" concept, thus with a clean interface, neutral and
abstract enough to allow for pretty much anything.
Most of what comes would still apply to a different concept though, in
case you can't stand the VS paradigm.


VIRTUAL SENSORS

A virtual sensor can do any kind of computation based on other sensors
data and provide the result in a sensor format.
For example, it may import data from a web service such as
http://marinetraffic.com/ and publish a corresponding stream of gpsd
AIS messages. It appears to clients as another JSON DEVICE, that can
be activated, POLLed, WATCHed, etc.
Most often, it works with local sensor(s) data to provide an altered
(or compound) stream, enforcing one particular policy: stricter fix
requirements, best-effort location, sensor divergence alarm… It
subscribes to other DEVICE streams like any gpsd client.

In a typical setup, several virtual sensors are available: client apps
get to choose which sensor (virtual or not) they listen to, because
different clients may have different needs.
They simply connect to gpsd, query the available DEVICE list and pick
their favorite(s) sensor(s).
When a VS is WATCHed, it gets activated (unless it already was). Upon
activation, it activates and WATCHes local "slave" sensors, if any; It
releases them when it gets inactivated.


GPSD ARCHITECTURE

The previous threads have made it clear gpsd is a «mechanical
multiplexer»: it gathers data from sensors and broadcast them to
clients.
In a classic multi-producer/multi-consumer scenario, its sensor data
flow basically looks like this:

        S1  S2  S3        Sensors
          ↘ ↓ ↙
           gpsd           Multiplexer
          ↙ ↓ ↘
        C1  C2  C3        Clients

Technically speaking though, gpsd is both a multiplexer (gathering
input signal) and a demultiplexer (broadcasting output signal):

        S1  S2  S3        Sensors
          ↘ ↓ ↙
     ╭-    mux            Multiplexer
gpsd |      ⇓
     ╰-   demux           Demultiplexer
          ↙ ↓ ↘
        C1  C2  C3        Clients

The distinction is important to make when talking about virtual
sensors, because they both:
 - receive sensors data
 - provide data to clients
Thus, computationally speaking, they optimally come after mux and before demux:

       S1  S2  S3        Sensors
         ↘ ↓ ↙
          mux            Multiplexer
           ⇓
           VS            Virtual Sensor
           ⇓
         demux           Demultiplexer
         ↙ ↓ ↘
       C1  C2  C3        Clients


INTERNAL VIRTUAL SENSORS

In such a scenario, virtual sensors follow the optimal computational
logic and insert themselves into gpsd, between the mux and demux
layers.
A VS manager may handle multiple VS and run then on demand. The sensor
data flow is the following:

          mux            Multiplexer
           ⇓
   VS1 ⇆  vsm  ⇄ VS2    Virtual Sensors, VS Mixer
           ⇓
         demux           Demultiplexer

Note that, in such a design, physical sensor data and related VS data
can be broadcast at the same time.
Virtual sensors can even be chained (use data from another VS) without
introducing noticeable lag either.


EXTERNAL VIRTUAL SENSORS

In this scenario, virtual sensors are pushed back into another
process, plugging itself twice to gpsd: as a sensor and as a client.

        S1  S2  VS_out    Sensors
          ↘ ↓ ↙   ⇑
           gpsd    ║      Mux/Demux
          ↙ ↓ ↘   ║
        C1  C2  VS_in     Clients

Note that, in such a design, virtual sensors lag compared to their
source sensors: transport costs aside, data must got through an
additional gpsd mux-demux cycle.
Consecutively, the lag gets worse if VS are chained: each new VS layer
adding its own cycle, unless there is a single VS manager process
hosting all chained VS.


VIRTUAL SENSOR PROXY

An alternative external design has been proposed by esr: the VS runs
on top of gpsd, as « a proxy daemon that launches GPSD reporting to a
private port, reads the JSON from that, does whatever (...) you want,
and ships results to port 2947 as if it were GPSD ».

        S1  S2  S3        Sensors
          ↘ ↓ ↙
           gpsd           Mux/Demux
            ↓
           vsd            VS Mixer and (Mux/)Demux
          ↙ ↓ ↘
        C1  C2  C3        Clients

Note that, in such a design, there is no lag between physical sensor
data and VS data: they both suffer the same additional mux-demux cycle
lag introduced by vsd.
The vsd daemon indeed needs to duplicate gpsd features, such as a
demux to serve clients on demand -- and a mux if data were to come
from several gpsd instances. In fact, such daemon would basically be
an internal VS/gpsd design, stripped of its physical sensor logic.


COMPARATIVE MERITS

Internal VS run fastest, consume less resources and introduce no lag.
Some VS drivers can be shipped with gpsd: they are important enough
and require the same overall quality other gpsd code requires.
But an internal VS manager also touches gpsd core code, so it's also
the most pervasive alternative. If sensor and client interfaces are
well defined, it should be a rather light touch though (in fact, VS
implementation provides a good check they are).

On the other hand, external VS are the least intrusive: they can live
without requiring any change in core code, at least ideally. They
still require gpsd quality though and should rather remain in the
project's control (at least the VS manager bit).
External VS run slower, consume more resources and introduce lag. But
they can provide a remote mechanism for client apps to publish the
additional/custom VS they'd need.

The proxy design is not well suited to the virtual sensor paradigm: it
is doomed to duplicate lots of gpsd code and impersonate the port 2947
service, which is dangerous. With its slave gpsd instance, it would be
better suited to a gspd controller paradigm.


CONCLUSION

Eventually, it would be best to offer both options as they address
different scenarios.
Yet, in the end, my guess is that:
 - most users would rather see such code shipped with a gpsd seal, and
run optimally
 - most client apps would rather focus on their part and leave that
code to gpsd as well
 - consecutively, internal use would far exceed external use

Unsurprisingly, project maintainers would rather start external to
test the logic, and maybe integrate it later when it's mature enough;
An individual contributor like myself would rather start internal to
test the logic rapidly, and maybe export it to a remote process when
it's mature enough.
A git branch would allow us to take the fast path and multiply tests
without prematurely impacting everyone. Added benefit: it would be
guaranteed to get merged without unnecessary delays, because esr hates
to see code rotting there… ;-)


--
Olivier Cornu