Re: [Bug-wget] [GSoC Proposal] Python3-based FTP Server for Test Suit and Test Suit Refactoring

[Zihang Chen]

Hi.

I just uploaded my proposal for GSoC 2014 on melange. Looking forward to
your feedback :)


2014-03-07 22:43 GMT+08:00 陈子杭 (Zihang Chen) <address@hidden>:

> Sure.
>
>
> 2014-03-07 22:42 GMT+08:00 Darshit Shah <address@hidden>:
>
>
>>
>>
>> On Fri, Mar 7, 2014 at 3:40 PM, 陈子杭 (Zihang Chen) <address@hidden>wrote:
>>
>>> Oh, right. Didn't come up with the second way. I'll take that into
>>> consideration. Thanks.
>>> BTW, how should the final proposal look like? Should I make a pdf file
>>> out of latex or just plain text? I'm a first-time GSoCer.
>>>
>>> You'd have to use HTML. Since the proposals need to be sent through the
>> melange website which has a WYSWYG editor I guess
>> There's time for that though. Proposals open on 10th March. Check out the
>> format then and send in one.
>>
>>>
>>>
>>> 2014-03-07 22:35 GMT+08:00 Darshit Shah <address@hidden>:
>>>
>>>
>>>>
>>>>
>>>> On Fri, Mar 7, 2014 at 3:32 PM, 陈子杭 (Zihang Chen) <address@hidden>wrote:
>>>>
>>>>> Thanks for your comments! :D
>>>>>
>>>>>
>>>>> 2014-03-07 22:17 GMT+08:00 Darshit Shah <address@hidden>:
>>>>>
>>>>> Hi Zihang,
>>>>>>
>>>>>> The proposal seems good.
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 7, 2014 at 11:39 AM, 陈子杭 (Zihang Chen) <
>>>>>> address@hidden> wrote:
>>>>>>
>>>>>>> * this is only a draft, so it's not written according to the GNU GSoC
>>>>>>> template. When discussion is done, I'll get a formal one done (with
>>>>>>> schedule etc.) *
>>>>>>>
>>>>>>> Briefing
>>>>>>> ====
>>>>>>>
>>>>>>> As is known, a new Python3-based test suite was implemented to
>>>>>>> replace the
>>>>>>> old one. And a configurable HTTP server utilizing `http.server` and
>>>>>>> `http.BaseHTTPRequestHandler` was also implemented. Thus the HTTP
>>>>>>> server
>>>>>>> can be configured to respond certain headers according to the
>>>>>>> configuration
>>>>>>> in separate test cases.
>>>>>>>
>>>>>>> However, a configurable FTP server was not required to be
>>>>>>> implemented,
>>>>>>> which breaks the consistency of the test suite of wget. So it's
>>>>>>> necessary
>>>>>>> that a configurable FTP server be built.
>>>>>>>
>>>>>>> Initial Design
>>>>>>> ====
>>>>>>>
>>>>>>> The test suite consists of two parts: `WgetTest.py` (module that
>>>>>>> actually
>>>>>>> runs the tests), servers (configurable HTTP and FTP server). When
>>>>>>> running
>>>>>>> test cases, `HTTPTest` instances collect configurations from test
>>>>>>> cases and
>>>>>>> start the servers with them so that the HTTP handler will handle the
>>>>>>> incoming requests according to the configurations (also known as
>>>>>>> rules).
>>>>>>>
>>>>>>> About the FTP server, I think the handler architecture can be
>>>>>>> applied here:
>>>>>>> implement multiple handlers corresponding to the FTP commands and
>>>>>>> register
>>>>>>> them in server instances automatically. When registering handlers,
>>>>>>> the
>>>>>>> server should also inform the handlers about the configurations.
>>>>>>> When new
>>>>>>> FTP command arrives, the server invoke the corresponding handler,
>>>>>>> and then
>>>>>>> the handler look up the rules itself and do the expected actions.
>>>>>>> When
>>>>>>> doing real FTP actions, I intend to use third-party FTP library like
>>>>>>> the
>>>>>>> current FTP server (I fear sockets so best not directly touch them).
>>>>>>>
>>>>>>> There's two things that are unclear:
>>>>>>> * the necessity of implementing all the FTP command. I presume that
>>>>>>> wget
>>>>>>> wouldn't utilize all of them.
>>>>>>>
>>>>>> If you're using a library, e.g. pyftpdlib, I don't see why you'd have
>>>>>> to implement
>>>>>> any commands on your own.
>>>>>>
>>>>> If the server is configurable, I think I'll have to intercept the
>>>>> commands so I can apply rules on them. When I say implement FTP command, I
>>>>> mean creating handle for the command, and invoke library in the handle. I
>>>>> think that's what mislead you probably.
>>>>>
>>>> Oh. Okay then. In that scenario, you could probably add the command
>>>> handlers
>>>> iteratively as and when they are required by a test case.
>>>>
>>>> Another way would be to look at Wget's FTP client implementation and
>>>> see what is the
>>>> superset of commands that it may send. If you're not comfortable with
>>>> C, I would help
>>>> you out with that.
>>>>
>>>>>    Simply use the library for it.
>>>>>>
>>>>>>> * what will the configurations be, do I need to design the tests
>>>>>>> myself? Or
>>>>>>> should I also port the Perl scripts to Python3?
>>>>>>>
>>>>>> Porting the existing Perl tests should give you a good idea of the
>>>>>> various
>>>>>> configurations that will be needed. Looking up the Wget source is
>>>>>> also often
>>>>>> a good idea. Either ways, some of us can help out with identifying
>>>>>> and writing
>>>>>> test cases when the time comes.
>>>>>>
>>>>>>
>>>>>>> Test Suite Refactoring
>>>>>>> ====
>>>>>>>
>>>>>>> While reading source, I don't find the test suite pythonic. Multiple
>>>>>>> parts
>>>>>>> of it seems unmaintainable. If the test suite can be refactored into
>>>>>>> a
>>>>>>> better coded one,  maintainability can further be achieved.
>>>>>>>
>>>>>>> Yes. I agree the current python based test suite might advantage
>>>>>> from a
>>>>>> few refactorings. There are a few places where the test suite is
>>>>>> quite un-pythonic
>>>>>> and fixing those would be a nice idea.
>>>>>>
>>>>>> Examples of unpythonicness:
>>>>>>> * docstrings out of classes or methods
>>>>>>> * mutable objects as default argument values
>>>>>>> * multiple use of operator is (controversial)
>>>>>>> * writing `_replace_substring (WgetTest.py:119)` instead of
>>>>>>> utilizing the
>>>>>>> format specification mini-language
>>>>>>>
>>>>>>> Examples of unmaintainability:
>>>>>>> * referencing attributes which are defined in subclasses
>>>>>>> (`WgetTest.py:61`
>>>>>>> references `self.servers` which is defined in its subclass
>>>>>>> `WgetTest.py:254`)
>>>>>>> * inappropriate variable names (in `WgetTest.py:254` `self.servers`
>>>>>>> actually becomes an integer while it regarded normally as a list of
>>>>>>> servers)
>>>>>>>
>>>>>> Yes, the variable names are horrible aren't they? That should be
>>>>>> refactored. (Shouldn't
>>>>>> take much time ).
>>>>>>
>>>>>>> * no intuitive, self-declarative module hierarchy
>>>>>>>
>>>>>> The module hierarchy is slightly complicated. Making it more
>>>>>> intuitive would help.
>>>>>>
>>>>>>>
>>>>>>> About me
>>>>>>> ====
>>>>>>> My name's Zihang Chen. I'm a 3rd undergraduate student from China
>>>>>>> (timezone: UTC+8). Please forgive me for any mistaken uses of the
>>>>>>> English
>>>>>>> language. I have been using Python for 3 years. I mostly code for
>>>>>>> myself
>>>>>>> (assignment, interest etc.) Never worked for any open source
>>>>>>> organizations
>>>>>>> before, so I'm thrilled to apply for GSoC. You can visit my GitHub
>>>>>>> page here<https://github.com/mad4alcohol>
>>>>>>>
>>>>>>> .
>>>>>>>
>>>>>>> Looking forward to your comments. Thanks very much.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanking You,
>>>>>> Darshit Shah
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Chen Zihang,
>>>>> Computer School of Wuhan University
>>>>> ---
>>>>> 此致
>>>>> 陈子杭
>>>>> 武汉大学计算机学院
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanking You,
>>>> Darshit Shah
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Chen Zihang,
>>> Computer School of Wuhan University
>>> ---
>>> 此致
>>> 陈子杭
>>> 武汉大学计算机学院
>>>
>>>
>>
>>
>> --
>> Thanking You,
>> Darshit Shah
>>
>>
>
>
> --
> Regards,
> Chen Zihang,
> Computer School of Wuhan University
> ---
> 此致
> 陈子杭
> 武汉大学计算机学院
>
>


-- 
Regards,
Chen Zihang,
Computer School of Wuhan University
---
此致
陈子杭
武汉大学计算机学院