http_client.c hardly speaks HTTP/1.0

[killerbees19]

I have the following problem with BitlBee: Sometimes my DNS returns wrong results. I don't know why, but that's another story and not the primary problem.

Ok, as example: The DNS returns my public server IP and/or localhost as result. BitlBee uses this IP for the Twitter HTTP-Requests. Now BitlBee gets a 301 status (Redirect) from my server, but it can't or won't understand it. Now BitlBee sends more than 50 (!) HTTP-Requests/second to my webserver. Perfect, a mini DoS. There is no end, my webserver logfile was bigger than 4 GB. I had to kill all BitlBee processes, to fix it :-(

Here is a short snippet from the logfile:

188.40.248.*** - - [29/Jun/2010:14:32:20 +0200] "GET /rss.php?mode[blog]=topics&mode[news]=topics&limit=10 HTTP/1.0" 301 320 "-" "-"
188.40.248.*** - - [29/Jun/2010:14:32:22 +0200] "GET /statuses/home_timeline.xml?cursor=-1 HTTP/1.0" 301 267 "-" "BitlBee 1.2.7 Linux/x86_64"
188.40.248.*** - - [29/Jun/2010:14:32:22 +0200] "GET /statuses/home_timeline.xml?cursor=-1 HTTP/1.0" 301 267 "-" "BitlBee 1.2.7 Linux/x86_64"
188.40.248.*** - - [29/Jun/2010:14:32:22 +0200] "GET /statuses/home_timeline.xml?cursor=-1 HTTP/1.0" 301 267 "-" "BitlBee 1.2.7 Linux/x86_64"

[...]

You can see, it was no problem for the IRC Bot (1st line; RSS-Feed requested), it got the wrong DNS result too, sent a request to localhost, got a wrong result and ignored it.

But BitlBee tries it again, again and again, ...
Any ideas, patches, ...? thx in advance :-)


Regards, Christian

PS: Sorry for my bad english ;-)
PPS: Yes, this is the official reply to the Twitter messages here: ​http://twitter.com/c_schroetter/status/17509357145 | ​http://twitter.com/BitlBee/status/17519286997 | ​http://twitter.com/c_schroetter/status/17602742760 | ​http://twitter.com/c_schroetter/status/17603087510 | ​http://twitter.com/c_schroetter/status/17604756683

[wilmer]

Hey Chris, thanks a lot for the report!

Not sure why the retries are happening since IIRC I put a limit on the number of redirects the HTTP lib will follow. I'll investigate.

[wilmer]

Alright, I finally took a look at this. Not sure what's going on, I created a 301 page somewhere myself but it won't follow the redirect even *once*.

After some more poking, I noticed that I only implemented handing of absolute redirects. Now I'm also stuck in an infinite loop. Turns out that I never wrote code to avoid infinite redirects. *ouch* :-(

[wilmer]

changeset:devel,610 adds a redirection TTL.

[killerbees19 <christian@…>]

Replying to wilmer:

changeset:devel,610 adds a redirection TTL.

The patch looks good, nice work! thx :-)

[wilmer]

Thanks for the report, also. :-) It's pretty bad that this code didn't have such a simple but important thing for so long. There are some other flaws that need fixing (like parsing the Content-Length: header immediatley), I'll try to work on that too.

That's less important though.

[wilmer]

changeset:devel,991 cleans this up a little bit. Basic HTTP/1.1 support, enough to not get confused anymore when webservers want to do keepalives or chunked encoding.