On 29/09/20 12:34, Marc-André Lureau wrote:
> That would not be backwards compatible as you would have to set all
> optional fields. Every time the command grows a new optional argument,
> all clients would have to specify it; if a command becomes optional,
> you'd have to add Some() around it. So I would say that...
>
>
> Not necessarily, with ..default::Default()
That's true, I always forget about .. (though you'd still have to add
Some() for now-optional fields).
> > Less idiomatic, but it also works around the optional arguments and
> > ordering issue.
>
> ... the builder pattern is not a workaround: it's the best and most
> common Rust idiom to achieve what QAPI expresses as optional fields.
> Likewise for keyword-only arguments in Python.
>
> Except QAPI makes all fields potentially optional (and unordered),
> that's not idiomatic.
Yes, for some APIs you can always add hand-written, more idiomatic
versions. Or you could mark them as fixed-arguments in the schema and
let the code generator do that (but then you need to add a compatibility
check). But that would be an explicit choice, not something required by
the transport.
That's my point, if we agree on keeping arguments & members non-optional and ordered, we are doing 50% of the job for nicer automated bindings.
Second half would probably be involving some version information in the schema.
> D-Bus is machine-level oriented, it's easy to bind to various languages,
> it can be pretty efficient too. It's not designed to be a good network
> RPC. QMP tries to be a bit of both, but is perhaps not good enough in
> either.
No, only tries to be a good network RPC; not a particularly efficient
one, but future-proof. And it mostly succeeds at that---with one
notable exception: JSON parsers that mess up with numbers bigger than 2^53.
> If you want to "reinvent" QMP, instead of focusing on D-Bus you should
> take a look at alternative IDLs and protocols (D-Bus is one but there's
> also Protobuf and Flexbuffers), see how QAPI declarations would map to
> those protocols, see how you would deal with extensibility, and rank
> them according to various criteria. For example:
>
> * JSON "just works" but needs a custom code generator and imposes some
> extra complexity on the clients for the simplest commands
>
> * D-Bus has a good ecosystem and would keep simple commands simpler but
> has issues with upgrade paths and is uglier for complex commands
>
> * Protobufs probably would also just work and would have better code
> generators, but would require some kind of lint to ensure
> backwards-compatibility
>
> Again, the issues we are discussing are not specific to binding QMP over
> D-Bus. Binding QMP to various languages has similar problems.
Marc-André, we are totally in agreement about that! The problem is that
you have already decided what the solution looks like, and that's what
I'm not sure about because your solution also implies completely
revisiting the schema.
Did I? Which schema change are you (or I) implying? Versioning the interface? It's necessary at the client level, unless everything is dynamic, after introspection, which makes automated static bindings impractical.
I say there are many candidates (the ones I know are Protobuf and
Flexbuffers) for serialization and many candidates for transport (REST
and gRPC to begin with) in addition to the two {QMP,JSON} and
{DBus,DBus} tuples. We should at least look at how they do code
generation before deciding that JSON is bad and DBus is good.
Contrary to what you believe I am not focusing so much on DBus here :) It took about 200 loc to bind it, effortlessly (compared to sys<->rs conversion). All it does is to expose the same API we have in the generated C somehow (similar static types & functions - not all as a{sv} opaque dictionaries).
It's easy for QEMU to generate a good static binding for C, because the version always matches. For a client, you wouldn't be able to write a similar idiomatic API in C, Rust, Python or Go, unfortunately.
Iow, I am not trying to sell DBus, I would like to make it easier to bind QMP in general. (although I do believe that DBus is a better protocol than QMP for local IPC, yes. And gRPC is probably better for remoting)
> I would rather make those problems solved at the server level, that
> doesn't require any change to QMP today, just a more careful
> consideration when making changes (and probably some tools to help
> enforce some stability).
Problem is, "more careful consideration when making changes" is not a
small thing. And other RPCs have evolved in a completely different
space (REST APIs for web services) that have chosen the same tradeoffs
as QMP, so why should we not learn from them?
I don't buy that generalization. A very recent protocol in this space, that aims to be a good low-level RPC on Linux (for containers, cloud etc) is varlink. (In many ways, we could compare it to QMP, but it lacks some important features, like events)
varlink does non-optional members and versioning the same way I propose here, for what I could tell. Although they use JSON, and have similar transport "benefits", this basic rule allow them to have very decent automated binding in various languages, without resorting to unorthodox solutions, ex:
https://github.com/varlink/rust/blob/master/examples/example/src/main.rs