Re: How is http request via /dev/tcp different from that of curl?

[Peng Yu]

http://www.dest-unreach.org/socat/doc/socat.html#EXAMPLES


Let’s take socat as an example. I see none of the examples shown above
allows you make a TCP request, get the response, process it, then send
another request. The process part must allow arbitrary operations. It has
fixed ways of processing the responses but not arbitrary ones.

Please clarify which example it allows you process it arbitrarily.

On Sun, May 19, 2024 at 7:35 PM alex xmb sw ratchev <fxmbsw7@gmail.com>
wrote:

> if u gimme an official example
> say , some simple http2 keep alive server url , u name it ..
> i can show u
>
> On Mon, May 20, 2024, 02:33 alex xmb sw ratchev <fxmbsw7@gmail.com> wrote:
>
>> two comments about ur first post
>> first , the last -H with newline and json data looks suspicious , .. that
>> it may not be the last sent ( in the list ) , whenever it makes a difference
>>
>> and second , ur bash doesnt contain that json
>>
>> On Mon, May 20, 2024, 02:28 alex xmb sw ratchev <fxmbsw7@gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Sun, May 19, 2024, 22:23 Peng Yu <pengyu.ut@gmail.com> wrote:
>>>
>>>> That is a simple example in my original post. For example, in a real
>>>> application, I may get some info from the server and then send a new
>>>> request based on the info I received. This will require full fledged
>>>> programming ability.
>>>>
>>>> The tools that you don’t seem to have such functionality. Nor it is
>>>> easy to access external programs in those tools like bash.
>>>>
>>>
>>> no ...
>>> its not as u describe it
>>>
>>> u use http2 keep alive
>>> u , with every coding language and tool , have to send tx1 then read rx1
>>> then again for tx2 rx2
>>> if http'ed correctly , u can do all these
>>> u dont , it seems to me , cause u got some points complety wrong
>>>
>>> the tools , i said here , as also bash , do make tcp or udp or whatever
>>> conns
>>> example : socat or tcpclient or bash opens tcp conn to httpd
>>> this , only this , the tools do
>>> they dont parse the conns data much
>>> this is UR JOB , not conn making apps
>>> its : if u code right it works
>>> again : the tools make tcp conns
>>> if u dont exit the conn , or the httpd , it may still be connected
>>> if u get the sent headers and reads right , it all works
>>>
>>> On Sun, May 19, 2024 at 11:14 AM alex xmb sw ratchev <fxmbsw7@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Sun, May 19, 2024, 13:46 Peng Yu <pengyu.ut@gmail.com> wrote:
>>>>>
>>>>>> I said I need to process the responses after the sentence you quoted.
>>>>>> This means that I can do whatever operations with those responses. Can
>>>>>> those tools have the ability allowing me to do so?
>>>>>>
>>>>>
>>>>> they make tcp conn , what u printf in is ur thing
>>>>> and when to read or printf more first
>>>>>
>>>>> i dont get ur http example much
>>>>>
>>>>> On Sat, May 18, 2024 at 3:29 AM alex xmb sw ratchev <fxmbsw7@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> both tools i mentioned can put multiple user http requests ,
>>>>>>> depending on httpd 2's
>>>>>>>
>>>>>>> they connect , and send user data , and recieve
>>>>>>> say u can send 3 requests , then read ..
>>>>>>>
>>>>>>> ita not ' those tools dont support multi request ' - thats wrong
>>>>>>> knowledge
>>>>>>>
>>>>>>> On Fri, May 17, 2024, 20:03 Peng Yu <pengyu.ut@gmail.com> wrote:
>>>>>>>
>>>>>>>> Please use proper grammar. I am not following you.
>>>>>>>>
>>>>>>>> On Fri, May 17, 2024 at 10:06 AM alex xmb sw ratchev <
>>>>>>>> fxmbsw7@gmail.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, May 17, 2024, 13:44 Peng Yu <pengyu.ut@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I know. But I need to establish one tcp connection and submit
>>>>>>>>>> multiple requests using the same connection. I don’t want the 
>>>>>>>>>> connection be
>>>>>>>>>> closed while I process the response. None external tools can achieve 
>>>>>>>>>> this.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> what , ur dreaming ..
>>>>>>>>> if u http request , and the httpd is valid , u can send all kinds
>>>>>>>>> of http supported multi-things*
>>>>>>>>> no matter what tool all work
>>>>>>>>> its up to u having to make the http request n headers
>>>>>>>>> if u do well , and the httpd is not too junk , it ll work , no
>>>>>>>>> matter which tool
>>>>>>>>>
>>>>>>>>> On Fri, May 17, 2024 at 5:40 AM alex xmb sw ratchev <
>>>>>>>>>> fxmbsw7@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> other tools to make tcp scripts are socat
>>>>>>>>>>> http://www.dest-unreach.org/socat/ and tcpclient and tcpserver
>>>>>>>>>>> from ucspi-tcp https://cr.yp.to/ucspi-tcp.html
>>>>>>>>>>>
>>>>>>>>>>> On Fri, May 17, 2024, 01:25 Peng Yu <pengyu.ut@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> $ curl 'http://httpbin.org/get' -H 'User-Agent: Mozilla/5.0
>>>>>>>>>>>> (Macintosh; Intel Mac OS X 10.15; rv:125.0) Gecko/20100101
>>>>>>>>>>>> Firefox/125.0' -H 'Accept: application/json' -H
>>>>>>>>>>>> 'Accept-Language:
>>>>>>>>>>>> en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate' -H 'Referer:
>>>>>>>>>>>> http://httpbin.org/' -H 'DNT: 1' -H 'Sec-GPC: 1' -H
>>>>>>>>>>>> 'Connection:
>>>>>>>>>>>> keep-alive'
>>>>>>>>>>>> {
>>>>>>>>>>>>   "args": {},
>>>>>>>>>>>>   "headers": {
>>>>>>>>>>>>     "Accept": "application/json",
>>>>>>>>>>>>     "Accept-Encoding": "gzip, deflate",
>>>>>>>>>>>>     "Accept-Language": "en-US,en;q=0.5",
>>>>>>>>>>>>     "Dnt": "1",
>>>>>>>>>>>>     "Host": "httpbin.org",
>>>>>>>>>>>>     "Referer": "http://httpbin.org/";,
>>>>>>>>>>>>     "Sec-Gpc": "1",
>>>>>>>>>>>>     "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15;
>>>>>>>>>>>> rv:125.0) Gecko/20100101 Firefox/125.0",
>>>>>>>>>>>>     "X-Amzn-Trace-Id":
>>>>>>>>>>>> "Root=1-66469337-0aab56e96f4fee271f6a09ce"
>>>>>>>>>>>>   },
>>>>>>>>>>>>   "origin": "xxx.xxx.xxx.xxx",
>>>>>>>>>>>>   "url": "http://httpbin.org/get";
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> I want to replicate the same thing in bash below. But the
>>>>>>>>>>>> result is
>>>>>>>>>>>> not the same. How is the bash http request via /dev/tcp
>>>>>>>>>>>> different from
>>>>>>>>>>>> that of curl?
>>>>>>>>>>>>
>>>>>>>>>>>> exec {fd}<>/dev/tcp/httbin.org/80
>>>>>>>>>>>>
>>>>>>>>>>>> echo -e "GET /get HTTP/1.1\r
>>>>>>>>>>>> Host: httpbin.org\r
>>>>>>>>>>>> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15;
>>>>>>>>>>>> rv:125.0)
>>>>>>>>>>>> Gecko/20100101 Firefox/125.0\r
>>>>>>>>>>>> Accept: application/json\r
>>>>>>>>>>>> Accept-Language: en-US,en;q=0.5\r
>>>>>>>>>>>> Accept-Encoding: gzip, deflate\r
>>>>>>>>>>>> Referer: http://httpbin.org/\r
>>>>>>>>>>>> DNT: 1\r
>>>>>>>>>>>> Sec-GPC: 1\r
>>>>>>>>>>>> Connection: keep-alive\r
>>>>>>>>>>>> \r" >&"$fd"
>>>>>>>>>>>>
>>>>>>>>>>>> function httpprotocol/recv {
>>>>>>>>>>>>   while IFS= read -u "$fd" -r; do
>>>>>>>>>>>>     if [[ $REPLY =~ ^([^:]+):\ *(.*)$'\r'$ ]]; then
>>>>>>>>>>>>       headers[${BASH_REMATCH[1],,}]=${BASH_REMATCH[2]}
>>>>>>>>>>>>     elif [[ $REPLY = $'\r' ]]; then
>>>>>>>>>>>>       LC_ALL=C read -u "$fd" -r -d '' -N
>>>>>>>>>>>> "${headers[content-length]}" body
>>>>>>>>>>>>       break
>>>>>>>>>>>>     fi
>>>>>>>>>>>>  printf -v REPLY '%q' "$REPLY"
>>>>>>>>>>>>  echo "$REPLY"
>>>>>>>>>>>>   done
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> declare -A headers
>>>>>>>>>>>> $ httpprotocol/recv
>>>>>>>>>>>> $'HTTP/1.1 200 OK\r'
>>>>>>>>>>>> $'accept-ch: Sec-CH-UA, Sec-CH-UA-Platform,
>>>>>>>>>>>> Sec-CH-UA-Platform-Version, Sec-CH-UA-Mobile\r'
>>>>>>>>>>>> $'cache-control: max-age=0, private, must-revalidate\r'
>>>>>>>>>>>> $'connection: close\r'
>>>>>>>>>>>> $'content-length: 475\r'
>>>>>>>>>>>> $'content-type: text/html; charset=utf-8\r'
>>>>>>>>>>>> $'date: Thu, 16 May 2024 23:18:56 GMT\r'
>>>>>>>>>>>> $'server: nginx\r'
>>>>>>>>>>>> $'set-cookie: sid=ab0e69e8-13da-11ef-911e-ea384f7d4599; path=/;
>>>>>>>>>>>> domain=.httpbin.org; expires=Wed, 04 Jun 2092 02:33:03 GMT;
>>>>>>>>>>>> max-age=2147483647; HttpOnly\r'
>>>>>>>>>>>>
>>>>>>>>>>>> $ declare -p headers
>>>>>>>>>>>> declare -A headers=([server]="nginx" [connection]="close"
>>>>>>>>>>>> [content-type]="text/html; charset=utf-8"
>>>>>>>>>>>> [cache-control]="max-age=0,
>>>>>>>>>>>> private, must-revalidate"
>>>>>>>>>>>> [set-cookie]="sid=ab0e69e8-13da-11ef-911e-ea384f7d4599; path=/;
>>>>>>>>>>>> domain=.httpbin.org; expires=Wed, 04 Jun 2092 02:33:03 GMT;
>>>>>>>>>>>> max-age=2147483647; HttpOnly" [content-length]="475"
>>>>>>>>>>>> [accept-ch]="Sec-CH-UA, Sec-CH-UA-Platform,
>>>>>>>>>>>> Sec-CH-UA-Platform-Version, Sec-CH-UA-Mobile" [date]="Thu, 16
>>>>>>>>>>>> May 2024
>>>>>>>>>>>> 23:18:56 GMT" )
>>>>>>>>>>>> $ echo "$body"
>>>>>>>>>>>> <html><head><title>Loading...</title></head><body><script
>>>>>>>>>>>> type='text/javascript'>window.location.replace('
>>>>>>>>>>>> http://httpbin.org/get?ch=1&js=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJKb2tlbiIsImV4cCI6MTcxNTkwODczNiwiaWF0IjoxNzE1OTAxNTM2LCJpc3MiOiJKb2tlbiIsImpzIjoxLCJqdGkiOiIydjgxbW1wajdvc2w5b2M4M2swMHBwNmMiLCJuYmYiOjE3MTU5MDE1MzYsInRzIjoxNzE1OTAxNTM2NTQzNzg3fQ.Bc07CLhZ_HLw5QwfQoWotZPfoegQMRYc84bD-Wqg96k&sid=ab0e69e8-13da-11ef-911e-ea384f7d4599')
>>>>>>>>>>>> ;</script></body></html>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Peng
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Peng
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Peng
>>>>>>>>
>>>>>>> --
>>>>>> Regards,
>>>>>> Peng
>>>>>>
>>>>> --
>>>> Regards,
>>>> Peng
>>>>
>>> --
Regards,
Peng