Re: [RFC 0/4] Support interactions between TCG plugins

[Andrew S. Fasano]

From: Alex Bennée <alex.bennee@linaro.org>
> Andrew Fasano <fasano@mit.edu> writes:
> 
> > Hello,
> >
> > I'm requesting comments on the following series of patches expanding the
> > TCG plugin system to add the "QEMU Plugin-to-Plugin (QPP)" interface
> > that allows for interactions between TCG plugins. The goal of this
> > interface is to enable plugins to expand on other plugins and reduce
> > code duplication. This patch series includes documentation and
> > significant comments, but a high-level summary is below along with a
> > discussion of the current implementation as well as the benefits and
> > drawbacks of these changes.
> 
> Thanks for a very detailed cover letter. My initial thoughts are if we
> are trying to reduce code duplication what about simply using a library
> and linking it to the final plugin. I guess it depends on how
> computational effort has been spent in calculating a particular piece of
> state and if that is avoided by having this IPC mechanism instead of
> just repeating the calculation.

Thanks for the detailed feedback Alex, I think I understand and agree with the
changes you've suggested. I'll work on those over the next couple weeks and
send along patches (or another RFC, let me know what you prefer) when they're
ready. And perhaps some follow up questions that come up along the way.

> 
> > **Summary**
> >
> > The QPP interface allows two types of interactions between plugins:
> >
> > 1) Exported functions: A plugin may wish to allow other plugins to call
> > one of the functions it has defined. To do this, the plugin must mark
> > the function definition as publicly visible with the QEMU_PLUGIN_EXPORT
> > macro and place a definition in an included header file using the
> > QPP_FUN_PROTOTYPE macro. Other plugins can then include this header and
> > call the exported function by combining the name of the target plugin
> > with the name of the exported function.
> >
> > For example, consider a hypothetical plugin that collects a list of
> > cache misses. This plugin could export two functions using the QPP
> > interface: one to allow another plugin to query this list and another
> > to empty the list. This would enable the development of another plugin
> > that examines guest CPU state to identify process changes and reports
> > the cache misses per process. With the QPP interface, this second plugin
> > would not need to duplicate any logic from the first.
> 
> Thinking of this concrete example I guess the process change detection
> is a fairly expensive operation that might be tuned to a particular
> architecture? Is this something Panda currently derives from plugins
> instead of the core QEMU code?

Process change detection can be somewhat complex. PANDA has a callback in
the core emulation code named on_asid_changed that fires whenever the address
space layout identifier (ASID) is changed in the CPUState. This fires on most
process changes for some architectures (x86), but this sort of event is too
low-level to always be correct and varies by architecture.

To provide accurate detection of when processes change, we have a PANDA plugin
for operating system introspection which, when given details of the guest OS,
can provide a callback on the process change as well as some APIs for querying
details about running processes in the guest. This callback and these APIs
are exposed to other plugins through our inter-plugin interface.

> 
> > 2) Callbacks: Multiple plugins may wish to take some action when some
> > event of interest occurs inside a running guest. To support modularity
> > and reduce code duplication, the QPP callback system allows this logic
> > to be contained in single plugin that detects whenever a given event
> > occurs and exposes a callback with a given name. Another plugin can then
> > request to have one of its own functions run whenever this event occurs.
> > Additional plugins could also use this same callback to run additional
> > logic whenever this event occurs.
> >
> > For example, consider (again) a hypothetical plugin that detects when
> > the current guest process changes by analyzing the guest CPU state. This
> > plugin could define a callback named "on_process_change" and trigger
> > this callback event whenever it detects a process change. Other plugins
> > could then be developed that take various actions on process changes by
> > registering internal functions to run on this event.
> >
> > These patches and examples are inspired by the PANDA project
> > (https://panda.re and https://github.com/panda-re/panda), a fork of QEMU
> > modified to support dynamic program analysis and reverse engineering.
> > PANDA also includes a large plugin system with a similar interface for
> > interactions between plugins. I'm one of the maintainers of PANDA
> > and have seen how the ability for plugins to interact with
> > other plugins reduces code duplication and enables the creation of many
> > useful plugins.
> 
> Would another use-case be to export the PANDA APIs so you could use the
> existing plugins on an upstream QEMU?
> 

PANDA was designed and built by a bunch of researchers who wanted to quickly
get their research done (myself included). I think the design of its APIs
reflect this and aren't up to par with the code quality expected in QEMU. In
particular, the PANDA APIs directly expose the CPUState to plugins
and allow them to read and write any fields in it. This means PANDA plugins
are only compatible with the version of the emulator they are designed and
built against.

Although the QEMU plugin interface is smaller than the one provided by PANDA,
I think the QEMU design is much better. I'd be happy to help expand
the callbacks and APIs available to QEMU plugins to get to a point
where we can port some PANDA plugins to be in-tree QEMU plugins. I suspect
only a small subset (maybe about 15) of the ~45 PANDA plugins we have would be
worth merging, but it would depend on the types of plugins you'd want in-tree,
if they needed to work for all QEMU-supported architectures, and if plugins
would be allowed to modify guest state.

If anyone is interested, the full list of PANDA plugins and their source code
can be found in our repository at
https://github.com/panda-re/panda/tree/dev/panda/plugins/.
If any of these seem particularly interesting, I'd be happy to figure out
what (if any) new APIs they'd require, help build those APIs in QEMU, and then
change the plugin and try to get it merged into QEMU. I will point out
that some of them are built atop non-trivial emulation changes that would
likely not be wanted here (such as using LLVM IR instead of TCG to support
dynamic taint analysis).

> > **Implementation Overview**
[snip]
> While having an example plugin for debugging is useful I think having a
> more useful in-tree use-case is going to be important to justify the
> merging of a quite large API into the code base.

This sounds good to me, but I worry that QEMU plugins aren't (yet) able to
access enough information to build analyses complex enough to really
demonstrate the value of this type of interface. If plugins were able
to read guest registers, I could easily throw together a more powerful example
set of plugins such as the syscalls identification and logging plugins
described in my original message below.

Perhaps this is an argument for holding off on finalizing the details of
inter-plugin interactions until the core QEMU plugin interface is a bit
larger? I'd be okay with that as your response here shows that you're open to
the idea of allowing plugins to interact with other plugins which I see
as a key feature for building complex analyses in this interface. Knowing
that this is coming in the future (i.e., that the analyses I'm interested in
doing will eventually be possible with QEMU plugins) means I'd be willing to
contribute to other parts of the plugin interface before then.

> > **Request for Comments**
[snip]

> I'm certainly keen to get more plugins merged into the tree. I'm aware
> there are some gaps in the plugin API at the moment. The most pressing
> one is getting access to register values which requires some
> re-factoring of the core code. I have very incomplete WIP branch while I
> consider the API (not wired to plugins yet):
> 
>   https://gitlab.com/stsquad/qemu/-/tree/introspection/registers
> 

I'm really glad to see you're trying to expose that information to plugins,
it's definitely a blocker for many analyses! I had tried out a different
implementation of this using the gdb_read_register functions. I don't
particularly like the code I wrote and it was painful to use, but it
was quite short and didn't require changes outside of the plugin code.
On the off chance that there's something useful in it, you can see the code at
    
https://github.com/AndrewFasano/futurepanda/commit/1c46cc89bf3e825f559a70ccb84b4f034cd2490b



> > I see a handful of potential issues that I'll highlight below:
> > 1) Unstable APIs: these inter-plugin interactions do not specify API
> >    versions. If the behavior of a callback or exported function
> >    changes without the type signature changing, it may lead to subtle
> >    errors in other (unchanged) plugins that were using these functions.
> >
> >    If the signature of a plugin's callback or exported function,
> >    changes, build time errors would be raised and necessary
> >    changes could be made to in-tree plugins. However, out-of-tree
> >    plugins would break.
> 
> I'm not overly concerned about out-of-tree plugins for the time being as
> long as we keep the in-tree examples tested and working. That said we do
> implement an API versioning scheme for the core plugin API. Maybe
> something similar can be done for plugin APIs?

Glad to hear that. I'll consider this as I think about the changes you've
suggested in the code.

[snip]
> > 3) Decentralized interactions. This approach allows plugins to directly
> >    interact with other plugins. An alternative design could register
> >    exported functions and callbacks with the core plugin logic in
> >    the main QEMU object and have all the interactions go through there.
> >    Would a centralized design where plugins send requests through
> >    the core plugin logic in the QEMU binary be better that allowing
> >    direct interactions between the shared objects built for the
> >    plugins?
> 
> I shall see when I read the code but it does make me wander if we are
> just implementing a dynamic linker by another name. Maybe something like
> the callback/event system would be better marshalled by QEMU itself? I
> wonder if the dlload mechanism could just automatically make all
> non-static plugin functions exportable and then just complain if we get
> a clash or missing symbol when it is loaded?
> 

I think I'm coming around to the idea that this is a better approach. I'll
go in more details in response to your feedback on the code.


> > Does it seem like an interface like this would be worth merging? If so,
> > I'd welcome any and all suggestions on how to improve these changes.
> 
> I'm certainly open to it but I do think we will need some more concrete
> examples using such an API before we could consider merging it. I don't
> want to merge something that would only be used by out-of-tree plugins
> because it would impose a maintenance burden for no project gain.

I'm more than happy to build some plugins that use these APIs. My goal here
is to move as much code as would be welcome into QEMU. Just wanted to figure
out these APIs before writing plugins that use them. In the future when
we've figured out more details of this API, I'll be sure to include some
additional plugins with the patch.

If there's any documentation or prior discussion about what types of plugins
are wanted in QEMU, I'd appreciate it if someone could share that. I assume
you don't want a full platform for dynamic program analysis to live in tree
and require maintenance for obscure features that are rarely used. But if
you're interested in things that might be useful for a reasonable fraction of
the users who build QEMU with plugins enabled, I think there's a lot of
valuable capabilities that could be ported over from PANDA.

Thanks so much for the reply and for your review of the code,
Andrew

> 
> >
> > Thank you,
> > Andrew Fasano
> >
> > Andrew Fasano (4):
> >   docs/tcg-plugins: describe QPP API
> >   tcg/plugins: Automatically define CURRENT_PLUGIN
> >   tcg/plugins: Support for inter-plugin interactions
> >   tcg/plugins: Add example pair of QPP plugins
> >
> >  contrib/plugins/Makefile     |   4 +-
> >  contrib/plugins/qpp_client.c |  42 ++++++
> >  contrib/plugins/qpp_client.h |   1 +
> >  contrib/plugins/qpp_srv.c    |  33 +++++
> >  contrib/plugins/qpp_srv.h    |  17 +++
> >  docs/devel/tcg-plugins.rst   |  76 +++++++++++
> >  include/qemu/plugin-qpp.h    | 252 +++++++++++++++++++++++++++++++++++
> >  plugins/core.c               |  11 ++
> >  plugins/loader.c             |  24 ++++
> >  plugins/plugin.h             |   5 +
> >  plugins/qemu-plugins.symbols |   1 +
> >  11 files changed, 465 insertions(+), 1 deletion(-)
> >  create mode 100644 contrib/plugins/qpp_client.c
> >  create mode 100644 contrib/plugins/qpp_client.h
> >  create mode 100644 contrib/plugins/qpp_srv.c
> >  create mode 100644 contrib/plugins/qpp_srv.h
> >  create mode 100644 include/qemu/plugin-qpp.h
> 
> 
> --
> Alex Bennée
>