Re: [PATCH v2 3/9] qemu-thread: Avoid futex abstraction for non-Linux

[Akihiko Odaki]

On 2025/05/11 0:02, Paolo Bonzini wrote:
> On 5/10/25 10:51, Akihiko Odaki wrote:
> > Add special implementations of qemu_event_set() and qemu_event_wait()
> > using pthread primitives. qemu_event_wait() will ensure qemu_event_set()
> > finishes, and these functions will avoid complex barrier and atomic
> > operations.
>
> Unfortunately not...
>
> > Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
> > Tested-by: Phil Dennis-Jordan <phil@philjordan.eu>
> > Reviewed-by: Phil Dennis-Jordan <phil@philjordan.eu>
> > ---
> >   util/qemu-thread-posix.c | 45 +++++++++++++++++ 
> > +---------------------------
> >   1 file changed, 18 insertions(+), 27 deletions(-)
> >
> > diff --git a/util/qemu-thread-posix.c b/util/qemu-thread-posix.c
> > index 13459e44c768..805cac444f15 100644
> > --- a/util/qemu-thread-posix.c
> > +++ b/util/qemu-thread-posix.c
> > @@ -319,28 +319,6 @@ void qemu_sem_wait(QemuSemaphore *sem)
> >   #ifdef __linux__
> >   #include "qemu/futex.h"
> > -#else
> > -static inline void qemu_futex_wake(QemuEvent *ev, int n)
> > -{
> > -    assert(ev->initialized);
> > -    pthread_mutex_lock(&ev->lock);
> > -    if (n == 1) {
> > -        pthread_cond_signal(&ev->cond);
> > -    } else {
> > -        pthread_cond_broadcast(&ev->cond);
> > -    }
> > -    pthread_mutex_unlock(&ev->lock);
> > -}
> > -
> > -static inline void qemu_futex_wait(QemuEvent *ev, unsigned val)
> > -{
> > -    assert(ev->initialized);
> > -    pthread_mutex_lock(&ev->lock);
> > -    if (ev->value == val) {
> > -        pthread_cond_wait(&ev->cond, &ev->lock);
> > -    }
> > -    pthread_mutex_unlock(&ev->lock);
> > -}
> >   #endif
> >   /* Valid transitions:
> > @@ -363,7 +341,7 @@ static inline void qemu_futex_wait(QemuEvent *ev, 
> > unsigned val)
> >   void qemu_event_init(QemuEvent *ev, bool init)
> >   {
> > -#ifndef __linux__
> > +#ifndef CONFIG_LINUX
> >       pthread_mutex_init(&ev->lock, NULL);
> >       pthread_cond_init(&ev->cond, NULL);
> >   #endif
> > @@ -376,7 +354,7 @@ void qemu_event_destroy(QemuEvent *ev)
> >   {
> >       assert(ev->initialized);
> >       ev->initialized = false;
> > -#ifndef __linux__
> > +#ifndef CONFIG_LINUX
> >       pthread_mutex_destroy(&ev->lock);
> >       pthread_cond_destroy(&ev->cond);
> >   #endif
> > @@ -386,6 +364,7 @@ void qemu_event_set(QemuEvent *ev)
> >   {
> >       assert(ev->initialized);
> > +#ifdef CONFIG_LINUX
> >       /*
> >        * Pairs with both qemu_event_reset() and qemu_event_wait().
> >        *
>
> The user of this code will not have mutexes so some care is needed.  You 
> have:
>
>      if (!qatomic_load_acquire(&done)) {
>          qemu_event_reset(&ev);
>          if (!qatomic_load_acquire(&done))
>              qemu_event_wait(&ev);
>      }
>
> and on the other side
>
>      qatomic_store_release(&done, 1);
>      qemu_event_set(&ev);
>      --> qatomic_set(&ev.value, EV_SET);
>
> I don't think this is correct without the memory barrier in 
> qemu_event_set(), though I am not actually sure how you'd
> add it.
>
> The problem is, I don't see anything that prevents this:
>
>                  set done
>                  qemu_event_set()
>                    pthread_mutex_lock()
>                    ev.value = SET
>      qemu_event_reset()
>        ev.value |= FREE
>        smp_mb__after_rmw()
>                    // store buffer not flushed yet,
>                    // so other thread doesn't see "done"
>      read done
>      pthread_mutex_lock()
>                    pthread_cond_broadcast()
>                    pthread_mutex_unlock()
>      while ev.value != SET
>          // hang
>          pthread_cond_wait()
>
> The barrier in qemu_event_reset() is not enough, you need one on each side.

This is insightful; I will restore smp_mb() in qemu_event_set().

> > +#else
> > +    pthread_mutex_lock(&ev->lock);
> > +    if (qatomic_read(&ev->value) != EV_SET) {
>
> Apart from the above this needs to be a while(), because condition 
> variables also have spurious wakeups.

I will fix it with the next version too.

Regards,
Akihiko Odaki

> Paolo
>
> > +        pthread_cond_wait(&ev->cond, &ev->lock);
> > +    }
> > +    pthread_mutex_unlock(&ev->lock);
> > +#endif
> >   }
> >   static __thread NotifierList thread_exit;