Re: [lwip-devel] [RFC PATCH 0/5] LWIP stack integration

[Simon Glass]

Hi,

On Wed, 7 Jun 2023 at 10:47, Ilias Apalodimas
<ilias.apalodimas@linaro.org> wrote:
>
> Hi Maxim,
>
> On Tue, 6 Jun 2023 at 17:33, Maxim Uvarov <maxim.uvarov@linaro.org> wrote:
> >
> > Greetings,
> >
> > I implemented the tftp client (that was quick due to lwip has example app 
> > for tftp), and did some more measurements.
> > I uploaded patches here if somebody want to do his own measurements:
> > https://github.com/muvarov/uboot-lwip
> >
> > measure 1:
> > 976K - total (total means lwip with all 3 commands ping, tftp, wget)
> > 971K - total - tftp  (total, but disable/minus tftp)
> > 965K - total - tftp - wget (disable tftp and wget)
> > 963K - total - tftp - wget - ping (disable tftp, wget, ping)
> > 931K - no lwip
> >
> > result 1: lwip tftp (+ udp protocol) protocol 976-971k = 5kb
> > result 2: lwip ping command 965- 963 = 2kb
> > result 3: lwip wget command 971- 965 = 6kb
> > result 4: lwip core stack with apps 976 - 931 = 45kb
>
> So tftp = 5kb, wget = 6kb ping =2kb and lwip = 32kb
>
> >
> > measure 2:
> > 890K - no CONFIG_NET_CMD
> > 930K - + lwip tftp only
> > 937K - + full lwip (ping wget tftp)
> >
> > result 1: 937-890=47kb ( lwip + all 3 commands)
> > result 2: 937-930=7kb  (ping and lwip command)
>
> I am not sure I understand this measurement. How is this different
> from measurement 1 where the entire binary was 976K?
>
> >
> > measure 3:
> > 904K - no lwip, CMD_NET_TFTP=y
> > 900K - no lwip, CMD_NET_TFTP=n
> > result 1: original u-boot tftp command 904-900=4kb
> > 890K - no lwip, CMD_NET=n
> > result 2: 900-890=10k original u-boot net/IP stack.
> >
> > My findings for all that measurements and lwip configuration:
> > 1. The original u-boot net stack (packet process and up layers) is 10k vs 
> > lwip 40k (the very minimal settings were 30k).
> > 2. Network applications size is about the same 4kb for tftp original 
> > command 5kb for lwip.
> > 3. It's quite easy to reuse LWIP examples to implement the same 
> > functionality for the U-boot.
> > 4. I still think that there are other criterias which might have more 
> > priority than size (bug free code, code reuse, development speed,  
> > compatible API to posix and etc).
>
> Yes, there are other criteria and certainly having a complete network
> stack might be worth it in many cases, but we need to keep in mind
> 30kb might be a lot for some systems.
>
> I personally think this is decent and we can optimize lwip more in the
> future.  Tom, Simon, how about adding lwip as 'experimental' and
> making it depend on !CMD_NET or something similar?

That seems OK to me, but we don't really want two network stacks, so
we'd need to set an expectation that we would move to lwip.

I wonder why it is so large? I saw mention of it supporting multiple
buffers and perhaps having a fuller implementation of the protocols.
But it makes U-Boot's stack seem super-slim in comparison. I wonder if
lwip could support just a single buffer and reduced functionality in
other areas?

Regards,
Simon



>
> Thanks
> /Ilias
> >
> > BR,
> > Maxim.
> >
> > On Thu, 25 May 2023 at 02:18, Simon Goldschmidt <goldsimon@gmx.de> wrote:
> >>
> >> Hi Maxim, Tom,
> >>
> >> On 24.05.2023 16:05, Maxim Uvarov wrote:
> >> > On Tue, 23 May 2023 at 03:23, Tom Rini <trini@konsulko.com> wrote:
> >> >
> >> >> On Mon, May 22, 2023 at 12:40:49PM -0400, Maxim Uvarov wrote:
> >> >>> On Mon, 22 May 2023 at 10:20, Tom Rini <trini@konsulko.com> wrote:
> >> >>>
> >> >>>> On Mon, May 22, 2023 at 04:33:57PM +0300, Ilias Apalodimas wrote:
> >> >>>>> Hi Maxim
> >> >>>>>
> >> >>>>> On Mon, 22 May 2023 at 12:01, Maxim Uvarov <maxim.uvarov@linaro.org>
> >> >>>> wrote:
> >> >>>>>>
> >> >>>>>> My measurements for binary after LTO looks like:
> >> >>>>>>
> >> >>>>>> U-boot WGET |  LWIP WGET + ping |  LWIP WGET| diff bytes| diff %
> >> >>>>>> 870728            |  915000                    | 912560          |
> >> >>>> 41832    | 4.8
> >> >>>>>
> >> >>>>>
> >> >>>>> I think you'll need to analyze that a bit more.  First of all I don't
> >> >>>>> think the '+ping' tab is useful.  What is is trying to achieve?
> >> >>>>
> >> >>>
> >> >>> To show the  difference of extra bytes if we add a ping app on top.
> >> >>>
> >> >>>
> >> >>>>> - How was LWIP compiled?
> >> >>>>
> >> >>>
> >> >>> It has a really huge configuration. I tried to turn off everything off
> >> >>> everything what can impact on size but still make http app work:
> >> >>> #define LWIP_HAVE_LOOPIF                0
> >> >>> #define LWIP_NETCONN                    0
> >> >>> #define LWIP_SOCKET                     0
> >> >>> #define SO_REUSE                        0
> >> >>> #define LWIP_STATS                      0
> >> >>> #define PPP_SUPPORT                     0
> >> >>>
> >> >>> Disabling loopback:
> >> >>> #define LWIP_NETIF_LOOPBACK 0
> >> >>> can lower to 912288 bytes.
> >> >>>
> >> >>> And it's the same compilation option (optimization for size) as the 
> >> >>> main
> >> >>> u-boot. I will do more experiments, but I think the goal is not to turn
> >> >> off
> >> >>> everything.
> >> >>>
> >> >>>
> >> >>>>> - Was ipv6 supported?
> >> >>>>
> >> >>>
> >> >>> No.  I.e. when I sent results it was enabled on the compilation stage 
> >> >>> but
> >> >>> not used. I just checked that size remains the same if IPv6 is not even
> >> >>> compiled.
> >> >>>
> >> >>>
> >> >>>>> - Can we strip it down even further?
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>> There is always room for optimization. I think I tried to turn off
> >> >>> everything that is configurable with defines. I can play with disable 
> >> >>> IP
> >> >>> reassembly and things like that or figure out which functions have more
> >> >>> size and if it's possible to exclude them.
> >> >>>
> >> >>>
> >> >>>>>   In general please give as much information as you can with what we
> >> >>>>> gain in functionality from LWIP with those extra bytes of code.
> >> >>>>
> >> >>>>
> >> >>> The main idea is to reuse a maintainable IP stack outside of U-boot.
> >> >> LWIP
> >> >>> can give a nice separation between IP stack code and network 
> >> >>> application
> >> >>> code.  I.e. application should not take care about any TCP details  
> >> >>> (SYN,
> >> >>> ACK, retransmission, reassembly etc) and should open connection and use
> >> >>> functions similar to recv() and send() to transfer data. Data means
> >> >>> application data, no network packets. And LWIP allows
> >> >>> us to do that.
> >> >>> Because LWIP has an API similar to sockets, it has to be very easy to
> >> >> port
> >> >>> a linux application to LWIP. Then you can test it with a tap device. 
> >> >>> Then
> >> >>> copy sources to U-boot, add a small integration layer (cmd command to
> >> >>> call), compile and use.
> >> >>>
> >> >>> So my suggestion was:
> >> >>> -  do not maintain new network stack code in the current U-boot. Use 
> >> >>> lwip
> >> >>> sources as an external project.  All bugs related to network stack go 
> >> >>> to
> >> >>> lwip project first, then sync with U-boot.
> >> >>> - maintain network apps code* or
> >> >>>    -- inside U-boot. Write our own code for application and maintain it
> >> >>> inside U-boot.
> >> >>>    -- inside LWIP. Add examples to LWIP which are suitable for both
> >> >> U-boot
> >> >>> and LWIP.
> >> >>>
> >> >>> * Let's define a U-boot network application as a cmd command. It might 
> >> >>> be
> >> >>> ping, wget (http or https download), telnet, arp dns etc..
> >> >>>
> >> >>> Let's consider the real use case, like HTTPS download client. We need 
> >> >>> to
> >> >>> enable TLS connection, validate certificates, then do http download.
> >> >>> Looking at the current code of wget command it's quite difficult to
> >> >>> implement this due to the application having some protol level things. 
> >> >>> On
> >> >>> the other side we can find embedTLS examples to do https download on
> >> >>> sockets. If LWIP socket API is ported then the only thing you need to 
> >> >>> do
> >> >> is
> >> >>> change socket() -> lwip_socket(), recv()->lwip_recv(),
> >> >> send()->lwip_send()
> >> >>> and etc, even function names are similar. If LWIP socket API is not
> >> >>> supported, then use callback API for recv() and send(), which are also
> >> >>> easy.
> >> >>>
> >> >>> So yes we add extra bytes, but that will allow us to write more complex
> >> >>> apps, use standard debug tools, use applications with very minimal
> >> >>> integration changes, use help from the LWIP community to fix protocol
> >> >> bugs,
> >> >>> etc..
> >> >>> Bunch of things already implemented there:
> >> >>> - ipv6
> >> >>> - dhcp
> >> >>> - snmp
> >> >>> - igmp
> >> >>> - dns
> >> >>> - tcp and udp and raw.
> >> >>> - loopback
> >> >>> - netconn
> >> >>> - socket
> >> >>> - stats
> >> >>> - ppp
> >> >>> (I just followed configurable defines).
> >> >>>
> >> >>>
> >> >>> And please make sure to disable the previous support, my guess fro that
> >> >>>> much growth is that you didn't.
> >> >>>>
> >> >>>
> >> >>> # CONFIG_PROT_TCP is not set
> >> >>> # CONFIG_PROT_UDP is not set
> >> >>> # CONFIG_UDP_CHECKSUM is not set
> >> >>> # CONFIG_UDP_FUNCTION_FASTBOOT is not set
> >> >>> # CONFIG_CMD_WGET is not set
> >> >>
> >> >> I think you need to step back and figure out a better way to measure the
> >> >> size change and growth.
> >> >>
> >> >> I am not interested in a path that long term means two networking stacks
> >> >> in U-Boot.
> >> >>
> >> >> I am not interested in massively growing the overall binary size for
> >> >> every platform.  Given how much larger just TCP support is, that's
> >> >> strongly implying a huge growth for the older use cases too.
> >> >>
> >> >> But I also suspect given the overall reputation that LWIP enjoys,
> >> >> there's something amiss here.
> >> >>
> >> >> --
> >> >> Tom
> >> >>
> >> >
> >> > +cc lwip-devel@ mailing list, maybe they have something to add.
> >>
> >> I do think using lwIP instead of "inventing yet another IP stack" is a
> >> good idea! However, in terms of code size, lwIP will lose against what's
> >> in U-Boot at present. And this is only natural, as lwIP is a "full-size"
> >> stack supporting multiple concurrently running applications while the
> >> current IP stack in U-Boot is rather "crippled" down to just what the
> >> implementor needed at the time of writing.
> >>
> >> One example of this is that (if I remember correctly), U-Boot only has
> >> one single network packet buffer, while lwIP has support for multiple
> >> buffers. When speaking of TCP (forgive me if I'm wrong, I've lost track
> >> of that development in U-Boot about 3 years ago), we're comparing "we
> >> have implemented everything we need so that it just kind of works" to
> >> "we can easily add a HTTPS client to download something over the
> >> internet just by enabling more compile options".
> >>
> >> Also, when comparing lwIP to U-Boot TCP code size, keep in mind that
> >> U-Boot TCP (at least that of some years ago) is far from complete when
> >> compared to lwIP!
> >>
> >> lwIP is meant to be highly configurable and we're always open to add yet
> >> more options to leave out more code when it's not needed. However, I
> >> think there are some design decisions that will make lwIP larger than
> >> the current IP stack in U-Boot. To me, that's a natural result of having
> >> a "generic code" approach vs "developed to our needs". However, while
> >> DHCP + BOOTP and even a simple network console was rather easy to
> >> implement, I would not recommend implementing your own HTTPS download
> >> but rather using the existing lwIP + apps for that.
> >>
> >> In the end, I cannot take the decision from you. In my opinion, lwIP
> >> would be the better decision in terms of future work load and
> >> compatibility, but in the short run, it *will* lead to bigger binaries
> >> at least in some setups. And I do know from my past that it sometimes
> >> has been a pain to try and stuff a new U-Boot release into the existing
> >> space of flash or RAM, so that's not an easy decision.
> >>
> >> If you do take the lwIP approach however, let us know if we can help!
> >>
> >> Regards,
> >> Simon
> >>
> >> >
> >> > My measurements say that the current U-boot IP stack + wget command adds 
> >> > an
> >> > additional 9 Kbytes.
> >> > The  minimal configuration of LWIP with wget command is 30 Kbytes.
> >> > (compiled out all asserts, debugs, not used protocols etc.).
> >> >
> >> > And the most bigger functions are tcp in/out itself:
> >> >   * These functions are generally called in the order (ip_input() ->)
> >> >   * tcp_input() -> * tcp_process() -> tcp_receive() (-> application).
> >> >
> >> > +tcp_input                                      -    4364   +4364
> >> > https://git.savannah.nongnu.org/cgit/lwip.git/tree/src/core/tcp_in.c#n118
> >> > +tcp_receive                                    -    3444   +3444
> >> > https://git.savannah.nongnu.org/cgit/lwip.git/tree/src/core/tcp_in.c#n1154
> >> > +tcp_write                                      -    2192   +2192
> >> > https://git.savannah.nongnu.org/cgit/lwip.git/tree/src/core/tcp_out.c#n393
> >> > +ip4_reass                                      -    2096   +2096
> >> > https://git.savannah.nongnu.org/cgit/lwip.git/tree/src/core/ipv4/ip4_frag.c#n503
> >> > +tcp_output                                     -    1616   +1616
> >> > https://git.savannah.nongnu.org/cgit/lwip.git/tree/src/core/tcp_out.c#n1241
> >> >
> >> > If we transfer current net commands to lwip then we can decrease the 
> >> > size,
> >> > because of functions reuse.
> >> > And if we turn on all features in lwip it will be about 50 Kbytes.
> >> >
> >> > BR,
> >> > Maxim.
> >> >
> >> >
> >> > _______________________________________________
> >> > lwip-devel mailing list
> >> > lwip-devel@nongnu.org
> >> > https://lists.nongnu.org/mailman/listinfo/lwip-devel