Re: [Paparazzi-devel] Libopencm3 - STM32-F1 change interrupt priority

[Gautier Hattenberger]

Hello,

We are currently working at ENAC on a solution based on ChibiOS (scheduling) and libopencm3 (peripherals). This is not a so easy solution (we are having some headaches right now...) but the main idea was to avoid adding a new architecture (ChibiOS HAL) and reuse working code as much as possible.
Current code is here: https://github.com/enacuavlab/paparazzi/tree/chibios

We have heard on this list that people are also trying a FreeRTOS and a full ChibiOS implementation.

Extra question for Piotr: we are also running into priority issues right now. Default priorities are higher than ChibiOS scheduler, so we will make them configurable in order to lower them. How was chosen the current hard-coded values ? After some clever thinking or randomly ?

Gautier

Le 05/09/2013 00:39, Michal Podhradsky a écrit :

Hi all,

after carefully examining the code and thinking about possible solutions, we think that, based on our experience, the best and most reliable way how to fix the timing issue would be to port Paparazzi to a RealTime OS. And we are willing to do that.

I noticed that there was a discussion a while ago about it:
http://lists.gnu.org/archive/html/paparazzi-devel/2013-01/msg00158.html
and
http://lists.gnu.org/archive/html/paparazzi-devel/2012-01/msg00185.html
and
http://paparazzi.517.n7.nabble.com/New-Cortex-M4-hardware-platform-td534.html

Is anybody already working on porting Paparazzi onto RTOS? If so, how does it look? If not, do you have any recommendations/preferences which RTOS would be best for ARM Cortex-M3/M4 based autopilots? Lisa M and its derivatives... We want to be able to run the AP at up to 1kHz, so STM32F4 would be most suitable for this.

Looks like those support STM32F4 chips:

FreeRTOS (doesnt use GCC)

eCos RTOS

ChibyOS/RT

RTOS from P4 autopilot
...

Any other ideas/preferences/comments?

Regards
Michal

On Tue, Aug 27, 2013 at 2:55 PM, Michal Podhradsky <address@hidden> wrote:

Hi Felix,

just a little update.

I tried that trick with increased SYS_TIME_FREQUENCY, but haven't noticed any difference.

I had a suspicion that this jitter was caused mainly by interrupts, so I disabled all uart interrupts (commented out https://github.com/paparazzi/paparazzi/blob/master/sw/airborne/arch/stm32/mcu_periph/uart_arch.c#L182)

Surprisingly there was no noticeable difference in the jitter, so it won't be that easy.

As far as scheduling goes, the right way imho would be to first look at the data flow diagram and make sure there are only the necessary parts. Second, look at the tasks/events and take care of the possibly problematic parts of the code. Interrupts should have defined priorities (not all at 0). Events should only set flags for periodic tasks, if possible avoid data processing (tasks do that).

Periodic tasks rely on RTC timers, possibly use more timers for finer control and also timeouts (if the task takes too long time).

It won't be much fun, but I ll look into it in the following days.

I don't have much experience with RT systems, so suggestions are gladly welcome.

Since Paparazzi is for flying things, it should be designed with hard RT constraints in mind...

Regards
M

On Fri, Aug 23, 2013 at 12:20 PM, <address@hidden> wrote:

Hi Felix,

thanks for the explanation about the sys_mon, now it makes more sense:)

I ran a couple of quick tests (just looking at the telemetry data and
sys_mon module, no data logging):
A) rotorcraft running @512Hz + GX3 as ahrs @500Hz + int_euler for
stabilization -> runs just fine (master 5.0)
B) rotorcraft running @512Hz + GX3 as ahrs @500Hz + float_euler for
stabilization -> VERY BAD timing (probably because of the floating point
calculations at high rate?) (master 4.9)
C) rotorcraft running @512Hz + GX3 as ahrs @125Hz + float_euler for
stabilization (master 4.9) -> OK(!), note that floating point calculations
runs at the same rate as in B), there is just 4x more UART traffic...

Maybe rotorcraft code is written in a way it can handle the tasks and
timing better?

M

As for yesterday tests w. fixedwing, here is the configuration for aspirin:
  <firmware name="fixedwing">
    <target name="ap"             board="lisa_m_2.0"/>
    <configure name="FLASH_MODE" value="JTAG"/>

    <define name="AHRS_USE_GPS_HEADING"/>
    <configure name="PERIODIC_FREQUENCY" value="500"/>
    <define name="MODULES_FREQUENCY" value="500"/>

    <subsystem name="control"/>

<!-- Communication -->
    <subsystem name="telemetry"     type="transparent"/>

    <subsystem name="radio_control" type="ppm">
       <configure name="RADIO_CONTROL_PPM_PIN" value="UART1_RX"/>
    </subsystem>

<!-- Sensors -->
    <subsystem name="imu"      type="aspirin_v2.2"/>
    <subsystem name="ahrs"      type="int_cmpl_quat"/>
    <subsystem name="gps"       type="ublox"/>
    <subsystem name="navigation" type="extra"/>
    <subsystem name="ins" type="alt_float" />
  </firmware>

<!-- Modules -->
  <!--modules main_freq="500"-->
  <modules>
    <!-- Airspeed Sensor -->
    <load name="airspeed_adc_adv.xml">
    </load>

    <!-- System Monitor -->
    <load name="sys_mon.xml"/>
    <load name="battery_monitor.xml"/>

    <!-- UGEAR BLACKBOX -->
    <load name="ugear_blackbox_uart.xml"/>
  </modules>


On Fri, Aug 23, 2013 at 8:15 AM, Felix Ruess <address@hidden> wrote:

> Hi Michal,
>
> I added some more information to
> http://paparazzi.enac.fr/wiki/Module/System_monitor. I hope that answers
> your question about the periodic_cycle...
>
> Nice job on the testing/plotting so far ;-) We clearly need to have a
> closer look at this...
> I didn't observe any variations like that in my setups so far (will check
> again tonight when I have access to the hardware again).
> There are quite a number of reasons why this could happen...
> E.g. if have an algorithm that takes a lot of time, it could delay the
> next periodic function. As we don't have any preemption (yet) this is some
> sort of cooperative scheduling.
> It could be triggered by an event like new gyro data and hence run in the
> event loop, or be one of the autopilot periodic functions like running
> guidance (e.g. if you try to run that in float without a floating point
> unit it will probably take too long)
>
> What configuration (stabilization, ahrs, ins, etc..) were you using for
> the aspirin tests?
>
> Cheers, Felix
>
>
> On Fri, Aug 23, 2013 at 2:50 AM, Michal Podhradsky <
> address@hidden> wrote:
>
>> Hi Felix,
>>
>> so I looked into it a bit more. First test with system monitor, using
>> fixedwing with GX3 at 500Hz, and PERIODIC_FREQUENCY=500Hz.
>> The periodic time is an average, so it is exactly at 2ms. The max
>> periodic cycle is however over the periodic time -> something is going on
>> there.
>>
>> Although I am not sure how to interpret the periodic_cycle. The average
>> is 1.25ms, which represents 800Hz, but I am not aware of any loop going
>> that fast.
>>
>> I made a simple logger module to monitor system time on the autopilot. It
>> runs at the same frequency as the periodic frequency, and sends through
>> UART current system time  (get_sys_time_usec() from
>> arch/stm32/mcu_periph/sys_time_arch.h). I ran couple of tests with
>> different configurations.
>>
>> Test 1: -> data1.png
>> Aspirin IMU, 60Hz periodic frequency. Just to get the baseline.
>> After first 80sec I plugged in GX3 configured to just output data @500Hz
>> (loads of data on UART). The data from GX3 weren't processed any further
>> though.
>>
>> I calculated difference between the received timestamps and converted
>> that to frequency. Data1 shows the result. You can see that most of the
>> data come with frequency between 58 and 62Hz, which is +-3% around the
>> defined periodic frequency. It is not great, but sufficient for now. Then
>> every half second you get a timestamp at 56.6/64Hz (that is +-6% which is
>> not acceptable precision).
>>
>> At T=~80sec I plugged in GX3 and you can see that the frequency of
>> timestamps starts to oscillate. Not much, but it definitely observable.
>>
>>
>> Test 2: -> data2.png
>>  GX3, 60Hz periodic frequency. GX3 configured to output attitude data
>> @125Hz which is slightly higher, but its only effect is a higher load on
>> CPU. See data2
>> Most of the time the measured frequency is right 60Hz, but every 250ms
>> the loop gets delayed (you can see the frequency drops to 58Hz and then
>> jumps to 62Hz (probably the next loop is somehow faster).
>>
>> So at lower frequencies everything works just fine.
>>
>>
>> Test3 -> data3a.png and data3b.png
>> GX3, 500Hz periodic frequency, GX3 @500Hz.
>>
>> Now the things starts to be messy. You can see that the system frequency
>> is all over the place, and there is some periodicity in the behaviour. The
>> mean is still 500Hz, but the actual frequency is between 400Hz and 600Hz.
>>
>> Test4 -> data4.png
>> Aspirin @500Hz, basically the same behaviour as with GX3.
>>
>> Since the problem with timing is the same with Aspirin as with GX3, it
>> probably wont be UART/SPI drivers/isr routines.
>>
>> Any idea which other part of code could delay the loop like that?
>>
>> Cheers
>> M
>>
>>
>> On Wed, Aug 21, 2013 at 2:34 AM, Felix Ruess <address@hidden>wrote:
>>
>>> Hi Michal,
>>>
>>> That's rather bad... I don't have any timing problems here, but then I'm
>>> not blowing heaps of data through UART...
>>> You can also use the sys_mon module [1] to check if the timing of your
>>> main periodic is ok.
>>> The priorities should probably be configurable.... can you make some
>>> tests if this helps or if it is something else?
>>>
>>> [1] http://paparazzi.enac.fr/wiki/Module/System_monitor
>>>
>>> Cheers, Felix
>>>
>>>
>>> On Wed, Aug 21, 2013 at 1:02 AM, Michal Podhradsky <
>>> address@hidden> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I have a question about timing on Lia 1.1 (stm32f1). I recently noticed
>>>> that the timing on the MCU is not solid.
>>>> I made a simple module with a periodic function which toggles an LED,
>>>> ran it at PERIODIC_FREQUENCY and then plugged in a scope to see if the LED
>>>> switching has a constant frequency.
>>>>
>>>> I am using PERIODIC_FREQUENCY=512Hz, and the frequency moves around
>>>> 30%, and occasionally bounces down to ~300Hz. If necessary, I can post
>>>> video from the scope if needed.
>>>>
>>>> Seems to be caused by interrupt priorities (e.g. excessive traffic on
>>>> one of the UARTS can delay the main loop etc). The function
>>>> nvic_set_priority() which mentioned Esden is used (master) only on a few
>>>> peripherals (i2c, spi, adc, ppm), most often setting the priority to 0.
>>>>

>>>> According to Cortex M3 programming manual, "*If software does not

>>>> configure any priorities, then all exceptions with a configurable priority

>>>> have a priority of 0.*"

>>>> if all the configurable interrupts have the same (highest) priority,
>>>> then there is no way to move the more important interrupts (e.g. UART with
>>>> lots of data coming through) any higher.
>>>>
>>>> The possible solution seems to be to set explicitly isr priority on
>>>> each peripheral (maybe an issue for github?).
>>>>
>>>> What do you think of that? Did you have similar issues with the timing?
>>>>
>>>> Regards
>>>> M
>>>>
>>>>
>>>>
>>>> On Mon, Apr 22, 2013 at 10:14 AM, Michal Podhradsky <
>>>> address@hidden> wrote:
>>>>
>>>>> Hi Esden,
>>>>>
>>>>> thanks for clarification!
>>>>>
>>>>> M
>>>>>
>>>>>
>>>>> On Fri, Apr 19, 2013 at 4:01 PM, Piotr Esden-Tempski <
>>>>> address@hidden> wrote:
>>>>>
>>>>>> Hi Michal,
>>>>>>
>>>>>> I think you are mistaking the vector table slot numbers there for
>>>>>> priorities. These are the positions of the function pointers inside the
>>>>>> vector table. If you change those numbers then things will obviously
>>>>>> explode.
>>>>>>
>>>>>> (just for reference, here the defines get their weak functions
>>>>>> assigned: sw/ext/libopencm3/lib/stm32/f1/vector_nvic.c, and here the vector
>>>>>> is being put together: sw/ext/libopencm3/lib/cm3/vector.c)
>>>>>>
>>>>>> IRQ priorities are set using the NVIC_IPR register, or using
>>>>>> nvic_set_priority function.
>>>>>>
>>>>>> I hope this helps.
>>>>>>
>>>>>> Cheers Esden
>>>>>>
>>>>>> P.S. both files nvic.h and vector_nvic.c are generated using the
>>>>>> sw/ext/libopencm3/scripts/irq2nvic_h from
>>>>>> sw/ext/libopencm3/include/libopencm3/stm32/f1/irq.yaml so you should not
>>>>>> edit those files by hand.
>>>>>>
>>>>>> On Apr 19, 2013, at 2:12 PM, Michal Podhradsky <
>>>>>> address@hidden> wrote:
>>>>>>
>>>>>> > Hi folks,
>>>>>> >
>>>>>> > I have a question about interrupt priorities for STM32F1 chip (Lia
>>>>>> 1.1/Lisa_M 2.0).
>>>>>> >
>>>>>> > In sw/ext/libopencm3/include/libopencm3/stm32/f1/nvic.h are defined
>>>>>> priorities for user interrupts. However, if I try to change the priority
>>>>>> for example for NVIC_USART2_IRQ (let's say make it higher priority than
>>>>>> NVIC_USART1_IRQ), the  code compiles, but then the program hangs up
>>>>>> instantly in usart_isr interrupt routine (debugged with JTAG).
>>>>>> >
>>>>>> > Can the priorities be set somewhere else or is it a feature to have
>>>>>> "hardcoded" priorities?
>>>>>> >
>>>>>> > Thanks
>>>>>> > Michal
>>>>>> > _______________________________________________
>>>>>> > Paparazzi-devel mailing list
>>>>>> > address@hidden
>>>>>> > https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Paparazzi-devel mailing list
>>>>>> address@hidden
>>>>>> https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
>>>>>>
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Paparazzi-devel mailing list
>>>> address@hidden
>>>> https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Paparazzi-devel mailing list
>>> address@hidden
>>> https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
>>>
>>>
>>
>> _______________________________________________
>> Paparazzi-devel mailing list
>> address@hidden
>> https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
>>
>>
>
> _______________________________________________
> Paparazzi-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
>
>

--- Select 'Retrieve' in options menu to retrieve whole e-mail ---

_______________________________________________
Paparazzi-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/paparazzi-devel

_______________________________________________
Paparazzi-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/paparazzi-devel