Thanks! Yes I'm getting great results with RPi Zero, using an I2S DAC. I will look into FreeRTOS, ChibiOS, eCos.
I'm running FluidSynth on similar but slightly more powerful hardware. The plain linux boot time (via SD-card) from power on to start of my custom kernel drivers, fluidsynth and my main program takes about 4-5 seconds. And I haven't taken some of the more invasive steps to reduce the Linux boot time yet. And I'm not really sure it's even relevant... as most of the startup time of my embedded system comes from loading the Soundfonts from disk and fluidsynth parsing them into memory. But the dynamic-sample-loading option has helped in reduced that load time considerably.
I'm not using a ready-made ARM distribution, but build everything from scratch using buildroot. It's a fantastic tool that lets you build extremely streamlined Linux systems that also boot very quickly. I think the key is: remove everything you don't need. Both in the system and in the kernel. The guys from bootlin (formerly Free Electrons) who developed buildroot also have good information about boot time reduction:
https://bootlin.com/doc/training/boot-time/
And maybe as an additional note: on my embedded platform, I'm using the preempt-rt realtime patches for the Linux kernel. They give you a real-time capable Linux system with bounded latencies, configurable IRQ priorities etc via the standard Linux API. No need for a separate RTOS, no need for new or distinct APIs. I haven't tried other RTOS solutions though, so can't really compare the different approaches.
Cheers,
Marcus