Re: [lwip-devel] [RFC PATCH 0/5] LWIP stack integration

[Ilias Apalodimas]

Hi Maxim,

On Tue, 6 Jun 2023 at 17:33, Maxim Uvarov <maxim.uvarov@linaro.org> wrote:
>
> Greetings,
>
> I implemented the tftp client (that was quick due to lwip has example app for 
> tftp), and did some more measurements.
> I uploaded patches here if somebody want to do his own measurements:
> https://github.com/muvarov/uboot-lwip
>
> measure 1:
> 976K - total (total means lwip with all 3 commands ping, tftp, wget)
> 971K - total - tftp  (total, but disable/minus tftp)
> 965K - total - tftp - wget (disable tftp and wget)
> 963K - total - tftp - wget - ping (disable tftp, wget, ping)
> 931K - no lwip
>
> result 1: lwip tftp (+ udp protocol) protocol 976-971k = 5kb
> result 2: lwip ping command 965- 963 = 2kb
> result 3: lwip wget command 971- 965 = 6kb
> result 4: lwip core stack with apps 976 - 931 = 45kb

So tftp = 5kb, wget = 6kb ping =2kb and lwip = 32kb

>
> measure 2:
> 890K - no CONFIG_NET_CMD
> 930K - + lwip tftp only
> 937K - + full lwip (ping wget tftp)
>
> result 1: 937-890=47kb ( lwip + all 3 commands)
> result 2: 937-930=7kb  (ping and lwip command)

I am not sure I understand this measurement. How is this different
from measurement 1 where the entire binary was 976K?

>
> measure 3:
> 904K - no lwip, CMD_NET_TFTP=y
> 900K - no lwip, CMD_NET_TFTP=n
> result 1: original u-boot tftp command 904-900=4kb
> 890K - no lwip, CMD_NET=n
> result 2: 900-890=10k original u-boot net/IP stack.
>
> My findings for all that measurements and lwip configuration:
> 1. The original u-boot net stack (packet process and up layers) is 10k vs 
> lwip 40k (the very minimal settings were 30k).
> 2. Network applications size is about the same 4kb for tftp original command 
> 5kb for lwip.
> 3. It's quite easy to reuse LWIP examples to implement the same functionality 
> for the U-boot.
> 4. I still think that there are other criterias which might have more 
> priority than size (bug free code, code reuse, development speed,  compatible 
> API to posix and etc).

Yes, there are other criteria and certainly having a complete network
stack might be worth it in many cases, but we need to keep in mind
30kb might be a lot for some systems.

I personally think this is decent and we can optimize lwip more in the
future.  Tom, Simon, how about adding lwip as 'experimental' and
making it depend on !CMD_NET or something similar?

Thanks
/Ilias
>
> BR,
> Maxim.
>
> On Thu, 25 May 2023 at 02:18, Simon Goldschmidt <goldsimon@gmx.de> wrote:
>>
>> Hi Maxim, Tom,
>>
>> On 24.05.2023 16:05, Maxim Uvarov wrote:
>> > On Tue, 23 May 2023 at 03:23, Tom Rini <trini@konsulko.com> wrote:
>> >
>> >> On Mon, May 22, 2023 at 12:40:49PM -0400, Maxim Uvarov wrote:
>> >>> On Mon, 22 May 2023 at 10:20, Tom Rini <trini@konsulko.com> wrote:
>> >>>
>> >>>> On Mon, May 22, 2023 at 04:33:57PM +0300, Ilias Apalodimas wrote:
>> >>>>> Hi Maxim
>> >>>>>
>> >>>>> On Mon, 22 May 2023 at 12:01, Maxim Uvarov <maxim.uvarov@linaro.org>
>> >>>> wrote:
>> >>>>>>
>> >>>>>> My measurements for binary after LTO looks like:
>> >>>>>>
>> >>>>>> U-boot WGET |  LWIP WGET + ping |  LWIP WGET| diff bytes| diff %
>> >>>>>> 870728            |  915000                    | 912560          |
>> >>>> 41832    | 4.8
>> >>>>>
>> >>>>>
>> >>>>> I think you'll need to analyze that a bit more.  First of all I don't
>> >>>>> think the '+ping' tab is useful.  What is is trying to achieve?
>> >>>>
>> >>>
>> >>> To show the  difference of extra bytes if we add a ping app on top.
>> >>>
>> >>>
>> >>>>> - How was LWIP compiled?
>> >>>>
>> >>>
>> >>> It has a really huge configuration. I tried to turn off everything off
>> >>> everything what can impact on size but still make http app work:
>> >>> #define LWIP_HAVE_LOOPIF                0
>> >>> #define LWIP_NETCONN                    0
>> >>> #define LWIP_SOCKET                     0
>> >>> #define SO_REUSE                        0
>> >>> #define LWIP_STATS                      0
>> >>> #define PPP_SUPPORT                     0
>> >>>
>> >>> Disabling loopback:
>> >>> #define LWIP_NETIF_LOOPBACK 0
>> >>> can lower to 912288 bytes.
>> >>>
>> >>> And it's the same compilation option (optimization for size) as the main
>> >>> u-boot. I will do more experiments, but I think the goal is not to turn
>> >> off
>> >>> everything.
>> >>>
>> >>>
>> >>>>> - Was ipv6 supported?
>> >>>>
>> >>>
>> >>> No.  I.e. when I sent results it was enabled on the compilation stage but
>> >>> not used. I just checked that size remains the same if IPv6 is not even
>> >>> compiled.
>> >>>
>> >>>
>> >>>>> - Can we strip it down even further?
>> >>>>>
>> >>>>
>> >>>
>> >>> There is always room for optimization. I think I tried to turn off
>> >>> everything that is configurable with defines. I can play with disable IP
>> >>> reassembly and things like that or figure out which functions have more
>> >>> size and if it's possible to exclude them.
>> >>>
>> >>>
>> >>>>>   In general please give as much information as you can with what we
>> >>>>> gain in functionality from LWIP with those extra bytes of code.
>> >>>>
>> >>>>
>> >>> The main idea is to reuse a maintainable IP stack outside of U-boot.
>> >> LWIP
>> >>> can give a nice separation between IP stack code and network application
>> >>> code.  I.e. application should not take care about any TCP details  (SYN,
>> >>> ACK, retransmission, reassembly etc) and should open connection and use
>> >>> functions similar to recv() and send() to transfer data. Data means
>> >>> application data, no network packets. And LWIP allows
>> >>> us to do that.
>> >>> Because LWIP has an API similar to sockets, it has to be very easy to
>> >> port
>> >>> a linux application to LWIP. Then you can test it with a tap device. Then
>> >>> copy sources to U-boot, add a small integration layer (cmd command to
>> >>> call), compile and use.
>> >>>
>> >>> So my suggestion was:
>> >>> -  do not maintain new network stack code in the current U-boot. Use lwip
>> >>> sources as an external project.  All bugs related to network stack go to
>> >>> lwip project first, then sync with U-boot.
>> >>> - maintain network apps code* or
>> >>>    -- inside U-boot. Write our own code for application and maintain it
>> >>> inside U-boot.
>> >>>    -- inside LWIP. Add examples to LWIP which are suitable for both
>> >> U-boot
>> >>> and LWIP.
>> >>>
>> >>> * Let's define a U-boot network application as a cmd command. It might be
>> >>> ping, wget (http or https download), telnet, arp dns etc..
>> >>>
>> >>> Let's consider the real use case, like HTTPS download client. We need to
>> >>> enable TLS connection, validate certificates, then do http download.
>> >>> Looking at the current code of wget command it's quite difficult to
>> >>> implement this due to the application having some protol level things. On
>> >>> the other side we can find embedTLS examples to do https download on
>> >>> sockets. If LWIP socket API is ported then the only thing you need to do
>> >> is
>> >>> change socket() -> lwip_socket(), recv()->lwip_recv(),
>> >> send()->lwip_send()
>> >>> and etc, even function names are similar. If LWIP socket API is not
>> >>> supported, then use callback API for recv() and send(), which are also
>> >>> easy.
>> >>>
>> >>> So yes we add extra bytes, but that will allow us to write more complex
>> >>> apps, use standard debug tools, use applications with very minimal
>> >>> integration changes, use help from the LWIP community to fix protocol
>> >> bugs,
>> >>> etc..
>> >>> Bunch of things already implemented there:
>> >>> - ipv6
>> >>> - dhcp
>> >>> - snmp
>> >>> - igmp
>> >>> - dns
>> >>> - tcp and udp and raw.
>> >>> - loopback
>> >>> - netconn
>> >>> - socket
>> >>> - stats
>> >>> - ppp
>> >>> (I just followed configurable defines).
>> >>>
>> >>>
>> >>> And please make sure to disable the previous support, my guess fro that
>> >>>> much growth is that you didn't.
>> >>>>
>> >>>
>> >>> # CONFIG_PROT_TCP is not set
>> >>> # CONFIG_PROT_UDP is not set
>> >>> # CONFIG_UDP_CHECKSUM is not set
>> >>> # CONFIG_UDP_FUNCTION_FASTBOOT is not set
>> >>> # CONFIG_CMD_WGET is not set
>> >>
>> >> I think you need to step back and figure out a better way to measure the
>> >> size change and growth.
>> >>
>> >> I am not interested in a path that long term means two networking stacks
>> >> in U-Boot.
>> >>
>> >> I am not interested in massively growing the overall binary size for
>> >> every platform.  Given how much larger just TCP support is, that's
>> >> strongly implying a huge growth for the older use cases too.
>> >>
>> >> But I also suspect given the overall reputation that LWIP enjoys,
>> >> there's something amiss here.
>> >>
>> >> --
>> >> Tom
>> >>
>> >
>> > +cc lwip-devel@ mailing list, maybe they have something to add.
>>
>> I do think using lwIP instead of "inventing yet another IP stack" is a
>> good idea! However, in terms of code size, lwIP will lose against what's
>> in U-Boot at present. And this is only natural, as lwIP is a "full-size"
>> stack supporting multiple concurrently running applications while the
>> current IP stack in U-Boot is rather "crippled" down to just what the
>> implementor needed at the time of writing.
>>
>> One example of this is that (if I remember correctly), U-Boot only has
>> one single network packet buffer, while lwIP has support for multiple
>> buffers. When speaking of TCP (forgive me if I'm wrong, I've lost track
>> of that development in U-Boot about 3 years ago), we're comparing "we
>> have implemented everything we need so that it just kind of works" to
>> "we can easily add a HTTPS client to download something over the
>> internet just by enabling more compile options".
>>
>> Also, when comparing lwIP to U-Boot TCP code size, keep in mind that
>> U-Boot TCP (at least that of some years ago) is far from complete when
>> compared to lwIP!
>>
>> lwIP is meant to be highly configurable and we're always open to add yet
>> more options to leave out more code when it's not needed. However, I
>> think there are some design decisions that will make lwIP larger than
>> the current IP stack in U-Boot. To me, that's a natural result of having
>> a "generic code" approach vs "developed to our needs". However, while
>> DHCP + BOOTP and even a simple network console was rather easy to
>> implement, I would not recommend implementing your own HTTPS download
>> but rather using the existing lwIP + apps for that.
>>
>> In the end, I cannot take the decision from you. In my opinion, lwIP
>> would be the better decision in terms of future work load and
>> compatibility, but in the short run, it *will* lead to bigger binaries
>> at least in some setups. And I do know from my past that it sometimes
>> has been a pain to try and stuff a new U-Boot release into the existing
>> space of flash or RAM, so that's not an easy decision.
>>
>> If you do take the lwIP approach however, let us know if we can help!
>>
>> Regards,
>> Simon
>>
>> >
>> > My measurements say that the current U-boot IP stack + wget command adds an
>> > additional 9 Kbytes.
>> > The  minimal configuration of LWIP with wget command is 30 Kbytes.
>> > (compiled out all asserts, debugs, not used protocols etc.).
>> >
>> > And the most bigger functions are tcp in/out itself:
>> >   * These functions are generally called in the order (ip_input() ->)
>> >   * tcp_input() -> * tcp_process() -> tcp_receive() (-> application).
>> >
>> > +tcp_input                                      -    4364   +4364
>> > https://git.savannah.nongnu.org/cgit/lwip.git/tree/src/core/tcp_in.c#n118
>> > +tcp_receive                                    -    3444   +3444
>> > https://git.savannah.nongnu.org/cgit/lwip.git/tree/src/core/tcp_in.c#n1154
>> > +tcp_write                                      -    2192   +2192
>> > https://git.savannah.nongnu.org/cgit/lwip.git/tree/src/core/tcp_out.c#n393
>> > +ip4_reass                                      -    2096   +2096
>> > https://git.savannah.nongnu.org/cgit/lwip.git/tree/src/core/ipv4/ip4_frag.c#n503
>> > +tcp_output                                     -    1616   +1616
>> > https://git.savannah.nongnu.org/cgit/lwip.git/tree/src/core/tcp_out.c#n1241
>> >
>> > If we transfer current net commands to lwip then we can decrease the size,
>> > because of functions reuse.
>> > And if we turn on all features in lwip it will be about 50 Kbytes.
>> >
>> > BR,
>> > Maxim.
>> >
>> >
>> > _______________________________________________
>> > lwip-devel mailing list
>> > lwip-devel@nongnu.org
>> > https://lists.nongnu.org/mailman/listinfo/lwip-devel