[avr-libc-dev] [bug #51255] Calculations in util/setbaud.h are incorrect

[Ian]

URL:
  <http://savannah.nongnu.org/bugs/?51255>

                 Summary: Calculations in util/setbaud.h are incorrect
                 Project: AVR C Runtime Library
            Submitted by: dense505
            Submitted on: Sat 17 Jun 2017 05:56:37 AM UTC
                Category: Header
                Severity: 3 - Normal
                Priority: 5 - Normal
              Item Group: Header files
                  Status: None
        Percent Complete: 0%
             Assigned to: None
        Originator Email: 
             Open/Closed: Open
         Discussion Lock: Any
                 Release: 2.0.0
           Fixed Release: None

    _______________________________________________________

Details:

Calculations are wrong, the error becomes severe at high baud rates.  Correct
formula in attached file includes correct calculation and finesse of (ratio *
100 + 50)/100 to achieve correct rounding to nearest integer value, not just
truncation.

This should also address bug #37758 as the issue is more general than just a
problem for Xmega devices.  I personally find the updated formula version
easier to follow than the suggestion in bug #37758.

table of results from attached spreadsheet, looks weird with the limited width
to display here, hope it comes out at your end.  The corrected formula agrees
exactly with the corresponding table in an Atmel datasheet.

Osc     16000000                                        error % after rounding  
Baud rate       UBRR    bad UBRR        error % Rounded UBRR    Rounded bad 
UBRR        Correct UBRR    bad
UBRR
1200    832.3   833.9   0.19    832     834     0.04    -0.20
2400    415.7   417.2   0.36    416     417     -0.08   -0.32
4800    207.3   208.8   0.72    207     209     0.16    -0.80
9600    103.2   104.7   1.45    103     105     0.16    -1.78
14400   68.4    69.9    2.19    68      70      0.65    -2.27
19200   51.1    52.6    2.94    51      53      0.16    -3.75
28800   33.7    35.2    4.45    34      35      -0.82   -3.79
38400   25.0    26.5    5.99    25      27      0.17    -7.82
57600   16.4    17.9    9.17    16      18      2.21    -10.02
76800   12.0    13.5    12.48   12      14      0.17    -16.46
115200  7.7     9.2     19.53   8       9       -4.16   -17.18
230400  3.3     4.8     44.91   3       5       10.19   -49.69
250000  3.0     4.5     50.00   3       5       0.00    -66.67
500000  1.0     2.5     150.00  1       3       0.00    -200.00




    _______________________________________________________

File Attachments:


-------------------------------------------------------
Date: Sat 17 Jun 2017 05:56:37 AM UTC  Name: setbaud corrected.h  Size: 7kB  
By: dense505

<http://savannah.nongnu.org/bugs/download.php?file_id=40929>
-------------------------------------------------------
Date: Sat 17 Jun 2017 05:56:37 AM UTC  Name: setbuad errors.xlsx  Size: 14kB  
By: dense505

<http://savannah.nongnu.org/bugs/download.php?file_id=40930>

    _______________________________________________________

Reply to this item at:

  <http://savannah.nongnu.org/bugs/?51255>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.nongnu.org/