[Bug-wget] [GSoC Update] Week 11

[Didik Setiawan]

================================= SUMMARY ======================================
My public fork of Wget2 project is available here [0]. I will continuously keep
pushing my work so anyone interested can track me there. Feel free to
participate in the discussions going on merge request with my mentors [1]. Your
feedback are highly appreciated.


=============================== INTRODUCTION ===================================
The purpose of this project is to use Libmicrohttpd as test suite for Wget2.  I
plan to do this by do some changes on function wget_test_start_server() also
wget_test_stop_server() on src/libtest.c of Wget2. With this approach, I don't
need to change existing test suite which call the internal server code through
functions mentioned above.
I've count there are 33 test file which use wget_test_start_server(). I must
ensure all the test passed when I've done the transition.
For installation prerequisite, I make that Libmicrohttpd support will needed
when we need to run `make check` or run test suite. Then I need to modify
configure.ac. I will give proper warning about this requirement. There is a
section in README.md where I must explain to user to provide Libmicrohttpd to
make all test running correctly.
With Libmicrohttpd I can add new test using feature that not yet implemented in
old server code, but ready on Libmicrohttpd, such as HTTP digest 
authentication. 

Mentors:
Darshit Shah <address@hidden>
Ander Juaristi <address@hidden>


================================== UPDATES =====================================
Things which were done until this week:

 * Started working on wget_test_start_server() to use Libmicrohttpd as HTTP
   server. Workflow to resolve this:
   - Remove initial process for HTTP server socket
   - Create _http_server_start() function, wrapper for Libmicrohttpd. There is
     also function _answer_to_connection() which use to create proper HTTP
     response
   - Use select method (MHD_USE_SELECT_INTERNALLY) for threading model in
     Libmicrohttpd to get better compatibility
   - http_server_port seized automatically using Libmicrohttpd function by
     passing MHD_DAEMON_INFO_BIND_PORT or MHD_DAEMON_INFO_LISTEN_FD parameter to
     MHD_get_daemon_info() 
   - Using iteration to parse urls data in _answer_to_connection(). This
     guarantee we can pass any variadic data to Libmicrohttpd and prevent
     segmentation fault 
   - Fix _answer_to_connection() function to create proper HTTP response:
     - Handle arguments (query string) on URL
     - Handle redirection
     - Handle directory creation
     - Handle URL with IRI object
     - Handle URL with IDN hostname
     - Handle URL with query string which contains space
     - Handle If-Modified-Since header
     - Handle Byte Range request
     - Handle HTTP basic authentication 
     - Handle HTTPS
     - Handle chunked transfer encoding Notes: I use
       MHD_create_response_from_callback() from Libmicrohttpd to generate
       chunked transfer method. For simulation with bad chunked data, I use
       MHD_create_response_from_buffer() with proper header and use connection
       close. I think the result of this method is most closely approached with
       the result from old HTTP server
 * Fix compiler warning -Wunused-param
 * Use persistent connection (Connection: keep-alive) instead multiple
   connection (Connection: close). This prevent hiding several problems of the
   connection between HTTP server and client
 * Fix CI testing on Travis OSX: must install Libmicrohttpd from source to get
   support for HTTPS. Maybe the Libmicrohttpd package provided by brew not
   compiled with HTTPS support
 * Fix CI testing on Gitlab MinGW64. That problem caused by variable clash
   between Wget2 and MinGW64 header have been reported and fixed
 * Add CI testing to simulate Wget2 build without Libmicrohttpd installed. This
   is just a copy of main build process but without 'make check' part, so the
   Wget2 binary can be built with skipped testing [2][3]


================================= NEXT STEPS ===================================
Things which would be done in the coming week:

 * Add HTTP digest authentication test
 * Add other tests which seems to possible to add in this time frame
   (compression test, IPv6 test, etc). I need more suggestion about this


[0]: https://gitlab.com/gnuwget/wget2/tree/tmp-use-mhd
[1]: https://gitlab.com/gnuwget/wget2/merge_requests/265 
[2]: https://gitlab.com/dstw/wget2/tree/mhd-not-install
[3]: https://github.com/dstw/wget2/tree/mhd-not-install

Regards,
Didik Setiawan