lynx-dev
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Lynx-dev] Viewing Dave's Trailer page with Lynx 2.8.7dev.5: error


From: Patrick Vervoorn
Subject: Re: [Lynx-dev] Viewing Dave's Trailer page with Lynx 2.8.7dev.5: error
Date: Thu, 16 Apr 2009 15:02:18 +0200 (CEST)
User-agent: Alpine 2.00 (DEB 1167 2008-08-23)

On Thu, 16 Apr 2009, Chuck Houpt wrote:

Hi Patrick,

It looks like the website is browser sniffing, and rejecting anything with 'libwww' or 'libcurl' in the User-Agent field. Here are some tests with curl that show the blocking of libwww/libcurl, but not 'foobar':

$ curl -A 'libwww' -I http://www.davestrailerpage.co.uk/
HTTP/1.1 406 Not Acceptable

$ curl -A 'libcurl' -I http://www.davestrailerpage.co.uk/
HTTP/1.1 406 Not Acceptable

$ curl -A 'foobar' -I http://www.davestrailerpage.co.uk/
HTTP/1.1 200 OK

Possibly the site is trying to deter rogue robots and scrapers, but filtering based on the User-Agent field won't be very effective, since it is easily changed (for example from the Lynx options page).

Aha, I did not know that. I changed the User-Agent field in my lynx.cfg to just plain 'Lynx' and the site is displaying again.

I also forwarded your comments to Dave himself, perhaps he can do something about it. This limitation was introduced when he recently moved the site to a new host, so perhaps it's something which was introduced by his new hoster.

Thanks for the explanation in any case. As to the other response: my apologies for not searching through the archives before posting (emailing) my question; too late for that now though...

Cheers - Chuck
http://chuck.habilis.net

Regards,

Patrick.

On Apr 16, 2009, at 4:06 AM, Patrick Vervoorn wrote:

Hello,

My apologies for sending this 'out of the blue', since I did not subscribe to the lynx-dev list.

However, I'm seeing a problem which is rather Lynx-specific, since viewing the same URL, on the same machine, but using Lynx, works rather ok.

The full story is in the email below which I sent to Dave, but since w3m works fine, I'm not sure it's a problem with the site...

Regards,

Patrick.

---------- Forwarded message ----------
Date: Sun, 12 Apr 2009 17:01:51 +0200 (CEST)
From: Patrick Vervoorn <address@hidden>
To: address@hidden
Subject: Viewing Trailer page with Lynx: error

Hi there,

When I try to view your site after the latest move with Lynx, I invariably get the following error-message:

(This is when viewing both the main site at: http://www.davestrailerpage.co.uk/ but also happens when I try to view one of the sub-pages directly, like: http://www.davestrailerpage.co.uk/about.html)


                               Not Acceptable

 An appropriate representation of the requested resource / could not be
 found on this server.

 Additionally, a 404 Not Found error was encountered while trying to use
 an ErrorDocument to handle the request.



When using something like w3m, it is displayed. Using a graphical browser like Firefox also works fine.

Is there anything you can do from your side to be able to view your Trailer page using Lynx?

Best regards,

Patrick.


_______________________________________________
Lynx-dev mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/lynx-dev





reply via email to

[Prev in Thread] Current Thread [Next in Thread]