[Bug-wget] [GSoC Update] Week 10

[Didik Setiawan]

================================= SUMMARY ======================================
My public fork of Wget2 project is available here [0]. I will continuously keep
pushing my work so anyone interested can track me there. Feel free to
participate in the discussions going on merge request with my mentors. Your
feedback are highly appreciated.


=============================== INTRODUCTION ===================================
The purpose of this project is to use Libmicrohttpd as test suite for Wget2.  I
plan to do this by do some changes on function wget_test_start_server() also
wget_test_stop_server() on src/libtest.c of Wget2. With this approach, I don't
need to change existing test suite which call the internal server code through
functions mentioned above.
I've count there are 38 test file which use wget_test_start_server(). I must
ensure all the test passed when I've done the transition.
For installation prerequisite, I make that Libmicrohttpd support will needed
when we need to run `make check` or run test suite. Then I need to modify
configure.ac. I will give proper warning about this requirement. There is a
section in README.md where I must explain to user to provide Libmicrohttpd to
make all test running correctly.
With Libmicrohttpd I can add new test using feature that not yet implemented in
old server code, but ready on Libmicrohttpd, such as HTTP digest authentication
and concurrent request checking. 

Mentors:
Darshit Shah <address@hidden>
Ander Juaristi <address@hidden>


================================== UPDATES =====================================
Things which were done until this week:

 * Started working on wget_test_start_server() to use Libmicrohttpd as HTTP
   server. Workflow to resolve this:
   - Remove initial process for HTTP server socket
   - Create _http_server_start() function, wrapper for Libmicrohttpd. There is
     also function _answer_to_connection() which use to create proper HTTP
     response
   - Use select method (MHD_USE_SELECT_INTERNALLY) for threading model in
     Libmicrohttpd to get better compatibility
   - http_server_port seized automatically using Libmicrohttpd function by
     passing MHD_DAEMON_INFO_BIND_PORT or MHD_DAEMON_INFO_LISTEN_FD parameter to
     MHD_get_daemon_info() 
   - Using iteration to parse urls data in _answer_to_connection(). This
     guarantee we can pass any variadic data to Libmicrohttpd and prevent
     segmentation fault 
   - Fix _answer_to_connection() function to create proper HTTP response:
     - Handle arguments (query string) on URL
     - Handle redirection
     - Handle directory creation
     - Handle URL with IRI object
     - Handle URL with IDN hostname
     - Handle URL with query string which contains space
     - Handle If-Modified-Since header
     - Handle Byte Range request
     - Handle HTTP basic authentication 
     - Handle HTTPS
     - Handle chunked transfer encoding Notes: I use
       MHD_create_response_from_callback() from Libmicrohttpd to generate
       chunked transfer method. For simulation with bad chunked data, I use
       MHD_create_response_from_buffer() with proper header and use connection
       close. I think the result of this method is most closely approached with
       the result from old HTTP server.
 * Fix compiler warning -Wunused-param
 * Use persistent connection (Connection: keep-alive) instead multiple
   connection (Connection: close). This prevent hiding several problems of the
   connection between HTTP server and client
 * Fix CI testing on Travis OSX: must install Libmicrohttpd from source to get
   support for HTTPS. Maybe the Libmicrohttpd package provided by brew not
   compiled with HTTPS support.


================================= NEXT STEPS ===================================
Things which would be done in the coming week:

 * Fix CI testing on Gitlab MingW64
   - MingW64: Need a script for complete setup Libmicrohttpd
 * Add HTTP digest authentication test


[0]: https://gitlab.com/gnuwget/wget2/tree/tmp-use-mhd

Regards,
Didik Setiawan