Modify

#838 closed defect (fixed)

Twitter stops fetching tweets after 5am CET

Reported by: sjefen6@… Owned by:
Priority: normal Milestone:
Component: Twitter Version: devel
Keywords: Cc:
IRC client+version: Client-independent Operating System: Linux
OS version/distro: Ubuntu-server 11.04

Description

I have noticed that twitter stops getting tweets arround 5am CET on Mondays. I have seen this behaviur on:

  • Sunday September 11 [05:45:36] (my first occurrence)
  • Monday September 19 [05:02:01]
  • Monday September 26 [05:09:59]
  • Monday October 3 [05:18:25]
  • Friday October 7 [05:19:01]
  • Monday October 10 [04:54:14]

Time is of the last fetched tweet.

Some times did I see a tweet between 5am and last tweet. Example Monday October 3:

[04:57:28] <extralife> Hehe. Nard.  http://t.co/f5KpcFfV <yfrog.com/nw6n9hyj>
[05:10:25] <kevinrose> LAS -> SFO next stop home! cc: @summertomato @toasterpup  http://t.co/npd1Ub7e <instagr.am/p/O87-M/>
[05:18:25] <extralife> RT @rwwoodward: @extralife @nicolespag @missiondeep Final Score episode 69 show notes now available...  http://t.co/sg3tjJVB <finalscoreshow.com>

Doing:

account twitter off
account twitter on

seems to get it going again.

I am able to send tweets but I do not get the "<root> You: $tweet" message.

Wilmer mentioned in IRC that disabling fetch_mentions might be a workaround. I have not tested it as I did not see the pattern until now.

Attachments (2)

bitlbee-twitter-stall.patch (2.7 KB) - added by Artem Savkov <artem.savkov@…> at 2011-11-19T09:18:15Z.
bitlbee-twitter-stall-wo-slow.patch (2.4 KB) - added by Artem Savkov <artem.savkov@…> at 2011-11-19T09:18:29Z.

Download all attachments as: .zip

Change History (15)

comment:1 Changed at 2011-10-11T07:37:24Z by lkraav <leho@…>

yep same here, cc-ing

comment:2 Changed at 2011-10-18T04:00:44Z by wilmer

I wrote a work-around for this in changeset:devel,814.

I need more info before I can fix this for real. My current guess is something like a HTTP timeout. Anyone who sees this happen a lot, please get me some traffic dumps..

comment:3 Changed at 2011-10-27T23:08:43Z by jjudin+bitlbee@…

I also noticed this behavior with Twitter by using multiple accounts (with development branch changeset number 811) and one of them would stop giving me new tweets after a while when other one would work fine. And it didn't seem to be related to the time of the day. Now that I disabled fetch_mentions from all of them, all my Twitter accounts have been working OK for over a week.

comment:4 Changed at 2011-10-27T23:20:55Z by wilmer

Actually I've seen it happen even with fetch_mentions on by the way.

The most recent version in bzr has a workaround BTW (one that doesn't require disabling mentions). Still hoping to really fix the problem though, of course..

comment:5 Changed at 2011-11-07T16:18:27Z by ilf

I hit it again, running 821. In my experience, this is not really connected to a certain time of the day.

comment:6 Changed at 2011-11-07T16:26:07Z by sjefen6

happens to me everytime arround 5am on Mondays (CET) even last week when Norway had changed to wintertime and US still was on summertime.

comment:7 Changed at 2011-11-07T16:46:14Z by jjudin+bitlbee@…

It might have something to do with connections not timeouting when they should. I noticed that my Twitter accounts were not updating after my Bitlbee machine's ISP had some connection troubles for an hour or two. This could be easily tested by just disconnecting from the network for a while and then checking if Twitter is updating at all after it.

comment:8 Changed at 2011-11-07T17:24:36Z by Wilmer van der Gaast <wilmer@…>

Yes, pretty sure it's related to fetch failures, just not sure which kind and no time to test so far. BTW at least the workaround should be kicking in now?

comment:9 Changed at 2011-11-19T09:17:53Z by Artem Savkov <artem.savkov@…>

Just to sum up what have been discussed on irc for history: The bug is caused by network problems, http_dorequest not always calling callback functions and current flag system. Current fix is not perfect, because twitter still stalls on partial success (i.e. one of the requests succeeds before next attempt and the second one fails).

I'm attaching two experimental patches.

  • bitlbee-twitter-stall.patch - the one fixing both the current fix and the issue when callback functions were never called on failed connection. I've tested this one for some time and it works for me.
  • bitlbee-twitter-stall-wo-slow.patch - the one fixing the issue when callback functions were never called on failed connection and removing current workaround completely. I tested it somehow and it seems to work, but I've just came up with the idea that this might be actually enough to fix the whole issue. So I'll run it for a day or two(that should be enough since my isp is pretty nasty :) and report back with the results.

Changed at 2011-11-19T09:18:15Z by Artem Savkov <artem.savkov@…>

Attachment: bitlbee-twitter-stall.patch added

Changed at 2011-11-19T09:18:29Z by Artem Savkov <artem.savkov@…>

comment:10 Changed at 2011-11-20T08:58:48Z by Artem Savkov <artem.savkov@…>

I've ran the second patch overnight. It hit both cases of the 'connection failed' (i.e. with both requests failing to connect and one of them). Seems to hold up nicely.

comment:11 Changed at 2011-11-20T15:28:16Z by Wilmer van der Gaast <wilmer@…>

Awesome. Thanks for your work so far! I've also now realised you and trac3r on #bitlbee are one and the same person which is useful to know! (-:

Will be back in Europe in a few days..

comment:12 Changed at 2011-11-24T08:19:16Z by wilmer

changeset:devel,829. Thanks Artem!

comment:13 Changed at 2011-11-25T07:40:55Z by wilmer

Resolution: fixed
Status: newclosed

Modify Ticket

Action
as closed The ticket will remain with no owner.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.