[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Bug-wget] Fwd: Regarding wget to download webpage
|
From: |
Bhargavi N |
|
Subject: |
Re: [Bug-wget] Fwd: Regarding wget to download webpage |
|
Date: |
Tue, 31 Jan 2012 10:06:33 +0530 |
Hello Henrik,
Thanks Thanks Thanks a LOT!!
Thanks Entire wget team for helping me so generously. Even in your busy
schedules.
Issue is now resolved.
I used Live HTTP headers add on as suggested by you and it really helped me.
Initially i passed all the headers as displayed in the capture window,
with "--header" option of wget.
Later i understood that session details have to be captured.
I used "--keep-session-cookies" option of wget, since sessions details have
to be saved.
I saved the session details in cookies.txt, which i used later to load
ADHOC query page and run the query.
Below are the commands i used to get the query results.
wget -O logon.html --keep-session-cookies --save-cookies cookies.txt
--post-data 'loginID=mylogon&password=mypasswd'
https://www.collabnet.timeinc.net
wget -O query_res.html --load-cookies cookies.txt "
https://www.collabnet.timeinc.net/servlets/AdHocQuery?query=select+count%28*%29+from+HELM_USER&Button=Run+query
"
Special thanks to you and whole team for sparing your time to help me out
with the issue.
Please let me know if i can be part of your team or i can be of any help to
you.
HAVE NICE DAY !!
With wishes,
Bhargavi
On Tue, Jan 24, 2012 at 3:20 PM, Henrik Holst
<address@hidden>wrote:
> I looked at your web site and it does not perform standard http
> authentication so --username and --password cannot be used to logon to
> that page.
>
> You have to supply the username and password using --post-data there
> aswell. If you had followed my advice to use the Live Headers extension
> with Firefox you would have seen exactly what to do, please use that
> tool aswell as learn some http basics and you will soon learn how to
> perform what you want. Because since we do not have access to that site
> of yours (no username or password) we as a community will have quite
> some hard time to tell you exactly how to proceed since we cannot test
> things at our end.
>
> Anyways, as I wrote I tested to perform a logon attempt with the site
> and with Live HTTP Headers extension active I could see that the
> authentication should be performed like this:
>
> wget --post-data "detour=https%3A%2F%2Fwww.collabnet.timeinc.net%
> 2F&loginID=username&password=password&Login=Login" --save-cookies
> cookies.txt "https://www.collabnet.timeinc.net/servlets/TLogin";
>
> Replace the "username" and "password" in the post data with your account
> details.
>
> However since I have no account on that site of yours, I do not know if
> this really works and whether the detour=xxx thing is really needed and
> whether you have to also add a Referer: header or not to the request.
>
> So if I where you:
> 1. Install Live HTTPS Headers extension in Firefox
> 2. Open it's capture window
> 3. Perform a logon and a sql query
> 4. Study the results in the capture window and try to mimic what you see
> with wget.
>
> /HH
>
>
> tis 2012-01-24 klockan 10:11 +0530 skrev Bhargavi N:
> > Hello Henrik,
> >
> >
> > I am saving the log on details in the cookies same way as you have
> > mentioned.
> >
> >
> > But cookie file looks empty..
> >
> >
> > Next i loaded the cookie.txt with the
> > URL: https://www.collabnet.timeinc.net/servlets/AdHocQuery
> >
> >
> > I saved the results page and then opened in firefox, i get the same
> > logon page to ollabnet instead of results page.
> >
> >
> > Steps to run the query:
> >
> >
> > 1) I need to logon to collabnet website:
> > https://www.collabnet.timeinc.net
> >
> >
> > 2) In the collabnet website, we have an option to run the query:
> >
> >
> > https://www.collabnet.timeinc.net/servlets/AdHocQuery
> >
> >
> > I need to go to this page and then need to provide SQL query in the
> > query text area.
> >
> >
> > Then need to submit the button "Run Query", which will submit the
> > query.
> >
> >
> > 3) Next need to download the results page to my local directoy on
> > LINUX box.
> >
> >
> > Please help me regarding this.
> >
> >
> > Thanks !
> >
> >
> > Regards,
> > Bhargavi
> >
> >
> > On Mon, Jan 23, 2012 at 12:43 PM, Henrik Holst
> > <address@hidden> wrote:
> > Ok,
> >
> > most probably the first site (where you logon) returns a
> > cookie which you must present to the other site (where you
> > perform the sql-query). So
> >
> > wget --username xx --password yy "https://collabnet website"
> > -O /dev/null --save-cookies cookies.txt
> >
> > That will logon and save the resulting cookie in the
> > cookies.txt file. Next you have to send this cookie and your
> > query to the other site:
> >
> > whet --load-cookies cookies.txt "https://the query site"
> > --post-data "the query" -O result
> >
> > And the result should be in the result file.
> >
> > Now as Angel Gonzales wrote you probably have to send the
> > query using the form like --post-data "query=Select%20*%20From
> > %20table" where the sql query in question has to be
> > url-encoded (something that wget cannot do for you) which is
> > mostly replave all spaces with %20 and also all occurences of
> > +, &, " and ' with their respective %hex code.
> >
> > A easy way to see exactly which formfield to use in the
> > post-data is to install the Live Headers extension to Firefox
> > and open it's capture window and peform the query on the site
> > using Firefox, you'll then get to see the exact query, what
> > the GET string is, what the post-data is etc. It's a very good
> > start to go from.
> >
> >
> > /HH
> >
> > 2012/1/20 Bhargavi N <address@hidden>
> > I meant i am unable to logon with to the website.
> >
> >
> > Sorry, to confuse you with lot of questions.
> >
> >
> > I have to logon to collabnet website first, providing
> > username and passwd.
> >
> >
> > Next i want to go to servlet page where i can provide
> > SQL query in the textarea of the form.
> >
> >
> > Then i need to submit the form and download the
> > results page to my local directory in linux box.
> >
> >
> > I will be running wget command on linux box.
> >
> >
> > Please help me regarding this.
> >
> >
> > Thanks !
> >
> >
> > Regards,
> > Bhargavi
> >
> >
> >
> > On Fri, Jan 20, 2012 at 7:20 PM, Bhargavi N
> > <address@hidden> wrote:
> > Hi Henrik,
> >
> >
> > Thanks for the reply.
> >
> >
> > But when i run as you have suggested me, then
> > i get the login page of the collabnet
> > website..
> >
> >
> > instead of the servlet page where i can enter
> > sql.
> >
> >
> > Please help me !
> >
> >
> > I am unable to logon to that page 1st.
> >
> >
> > Thanks!
> >
> >
> > Regards,
> > Bhargavi
> >
> >
> > On Fri, Jan 20, 2012 at 4:44 PM, Henrik Holst
> > <address@hidden> wrote:
> > Well, it normally should work with
> > this:
> > wget
> > "
> https://collabnet.net/servlets/adhocquery"; --user="username"
> --password="password" --post-data "select * from emp" -O output_filename
> >
> > But then according to the man page,
> > wget only supports post-data this is
> > in urlencoded format like
> > "key=value&key2=value2" so wget might
> > not support sql queries at all.
> > Perhaps cURL works better for your use
> > case.
> >
> > /HH
> >
> >
> > 2012/1/20 Bhargavi N
> > <address@hidden>
> > Including the team to this
> > mail chain..
> >
> > ---------- Forwarded message
> > ----------
> > From: Bhargavi N
> > <address@hidden>
> > Date: Fri, Jan 20, 2012 at
> > 4:30 PM
> > Subject: Re: [Bug-wget]
> > Regarding wget to download
> > webpage
> > To: Dayanand Kumbhar
> > <address@hidden>,
> > address@hidden,
> > address@hidden
> >
> >
> >
> > Hello Everyone,
> >
> > I am extremely thankful to
> > all of you for the help
> > regarding wget. But
> > still i am unable to get the
> > right page downloaded.
> >
> > Requirement:
> >
> > I need to run adhoc sql query
> > on remote web page. ie;
> > collabnet. The
> > webpage has text area where i
> > can enter sql query and then
> > click submit.
> >
> > Once i submit sql runs and
> > results are displayed in the
> > page.
> >
> > This REMOTE page is call to
> > servlet ie;
> >
> https://collabnet.net/servlets/adhocquery for example.
> >
> > I want to pass SQL query as
> > form data through POST/GET
> > method and get the
> > sql results page downloaded to
> > my local directory on LINUX
> > box. I am
> > running wget on LINUX host on
> > commandline / shell script.
> >
> > I will invoke wget with the
> > URL
> >
> https://collabnet.net/servlets/adhocquery and
> > formdata ie; SQL as
> > --post-data "select * from
> > emp". Finally i want to
> > download the sql query results
> > page to my local directory.
> >
> > I tried all the options that
> > you suggested me, but still
> > nable to get the
> > right page. Every time it is
> > displaying me the index page
> > for collabnet,
> > which i do not need. I need to
> > run the servelet page with the
> > formdata
> > passed to it.
> >
> > Then i want to get the results
> > page downloaded to my local
> > directory on my
> > linux host.
> >
> > Please help me regarding this.
> >
> > Again thanking all of you for
> > your reply and helping me out.
> >
> > Thanks !
> >
> > Regards,
> > Bhargavi
> >
> > On Wed, Jan 18, 2012 at 12:40
> > PM, Dayanand Kumbhar
> > <address@hidden>wrote:
> >
> > > Hi
> > >
> > > i am assuming you have to
> > post the query string
> > > You can use this
> > >
> > > wget -qO-
> > http://example.com/page.php
> > --post-data "SQL QUERY" | cat
> > >
> > >
> > > On Wed, Jan 18, 2012 at
> > 12:24 PM, Bhargavi N
> > <address@hidden> wrote:
> > >
> > >> Hello Team,
> > >>
> > >> My name is Bhargavi and i
> > am working on UNIX for over
> > 7years.
> > >>
> > >> I work on project, that
> > needs to download results from
> > remote web page.
> > >> This web page has adhoc sql
> > querying option. I need to run
> > this sql query
> > >> on webpage and get the
> > results downloaded using wget.
> > >>
> > >> How can i do that ?
> > >>
> > >> Please suggest me way to
> > download the results page
> > using wget.
> > >>
> > >> Thanks !
> > >>
> > >> Warm Regards,
> > >> Bhargavi
> > >>
> > >
> > >
> > >
> > > --
> > > Dayanand Kumbhar
> > >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
>
Message not available