Re: [Fastcgipp-users] UTF-8 POST value

[Axel von Bertoldi]

Did some testing and found the following:

When using narrow characters (i.e. <char>) I can replicate Alexey's problem regardless of the request method (GET or POST), the content type (application/x-www-form-urlencoded or multipart/form-data), or the method of retrieving the variable (requestVarGet or directly accessing the contents of Environment::Posts). In all these cases the correct string is returned, but it's with is 2 instead of 1 as expected.

Not sure where exactly the problem is here, but I guess it's because Ñ can't be represented using one narrow character. Eddie, is this correct?

When using wide characters (<wchar_t>), it's a different story: In all but one of the above described combinations, the correct value and length are returned (on my computer at least). The exception is when the request method is POST and the content type is application/x-www-form-urlencoded, in this case garbage is returned (when retrieving the data in either way).

The problem in this case is occurring in fillPostsUrlEncoded, but may point to somewhere else. fillPostsUrlEncoded is short and basically copies the post data into a string as follows:

> std::basic_string<charT> queryString;
> boost::scoped_array<char> buffer(new char[size]);
>
> memcpy (buffer.get(), data, size);
> charToString (buffer.get(), size, queryString);
>
> doFillPostsUrlEncoded(queryString);

I think the problem might be in charToString (or my use of it) as that's where the data is corrupted. Eddie, any thoughts WRT this? Will do further testing.

Alexey, I'm not sure what to suggest other than to make sure when you define your Request class, to make sure the template parameter is a wchar_t. Like so

> class MyRequest: public Fastcgipp::Request<wchar_t> {
> ...
> }

On Fri, Mar 12, 2010 at 1:23 PM, Alexey Savelyev <address@hidden> wrote:

Hi Axel,

Yes.

HTML

<form action="" method="post">
    <input value="" name="q" type="input">
    <input type="submit">
</form>

HTTP packet dump is

POST /fastcgi?t=12345678901234567890123456789012 HTTP/1.1
Host: 172.16.206.130:999
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: http://172.16.206.130:999/fix.php
Content-Type: application/x-www-form-urlencoded
Content-Length: 8

q=%D1%8F

FastCGI packet

SCRIPT_FILENAME/home/www/frontendx/fix.php..HTTP_X_TOKENtesttoken..SCRIPT_FILENAME/home/www/frontendx/fix."QUERY_STRINGt=12345678901234567890123456789012..REQUEST_METHODPOST.!CONTENT_TYPEapplication/x-www-form-urlencoded..CONTENT_LENGTH8..SCRIPT_NAME/fix..REQUEST_URI/search.php?t=12345678901234567890123456789012..DOCUMENT_URI/fix
.DOCUMENT_ROOT/home/www/frontendx..SERVER_PROTOCOLHTTP/1.1..GATEWAY_INTERFACECGI/1.1..SERVER_SOFTWAREnginx/0.7.65..REMOTE_ADDR172.16.206.1..REMOTE_PORT52781..SERVER_ADDR172.16.206.130..SERVER_PORT999..SERVER_NAMEaccounting.evoplus.com..REDIRECT_STATUS200..HTTP_HOST172.16.206.130:999.[HTTP_USER_AGENTMozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6.?HTTP_ACCEPTtext/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8..HTTP_ACCEPT_LANGUAGEen-us,en;q=0.5..HTTP_ACCEPT_ENCODINGgzip,deflate..HTTP_ACCEPT_CHARSETISO-8859-1,utf-8;q=0.7,*;q=0.7..HTTP_KEEP_ALIVE115.
HTTP_CONNECTIONkeep-alive.!HTTP_REFERERhttp://172.16.206.130:999/fix.php..!HTTP_CONTENT_TYPEapplication/x-www-form-urlencoded..HTTP_CONTENT_LENGTH8.................q=%D1%8F................

---
Alexey

On 10-03-12 1:37 PM, "Axel von Bertoldi" <address@hidden> wrote:

Hi Alexey,

In your first example, how are you submitting the data from the browser?

Eddie, I'll look at this over the next couple of days.

a.

On Fri, Mar 12, 2010 at 11:26 AM, Alexey Savelyev <address@hidden> wrote:

Hi all,

I’m evaluating fastcgi++ library for use in our next project. The library itself looks
solid but I’ve faced strange behavior/bug with UTF-8 characters.

environment.requestVarGet(KEY_KW,val_kw);

wstring val_kw;
wstring val_kw;

kw_encoded=Url::encode(val_kw);
out << "POST [" << val_kw << "]<br>";
out << "LEN  [" << val_kw.length() << "]<br>";
out << "PENC [" << kw_encoded << "]<br>";

val_kw=L"я";
kw_encoded=Url::encode(val_kw);
out << "TEXT [" << val_kw << "]<br>";
out << "LEN  [" << val_kw.length() << "]<br>";
out << "ENC  [" << kw_encoded << "]<br>";

output:
POST [Ñ ]
LEN  [2]
PENC [%c3%91%c2%8f]

TEXT [я]
LEN  [1]
ENC [%d1%8f]

My concern is length of POST variable. I post single character and expect it to be a single w_char character.
And clues?

library version is fastcgi++-2.0beta-d0c390fa.


---
Alexey