Twitter stops fetching tweets after 5am CET

[sjefen6@…]

I have noticed that twitter stops getting tweets arround 5am CET on Mondays. I have seen this behaviur on:

• Sunday September 11 [05:45:36] (my first occurrence)
• Monday September 19 [05:02:01]
• Monday September 26 [05:09:59]
• Monday October 3 [05:18:25]
• Friday October 7 [05:19:01]
• Monday October 10 [04:54:14]

Time is of the last fetched tweet.

Some times did I see a tweet between 5am and last tweet. Example Monday October 3:

[04:57:28] <extralife> Hehe. Nard.  http://t.co/f5KpcFfV <yfrog.com/nw6n9hyj>
[05:10:25] <kevinrose> LAS -> SFO next stop home! cc: @summertomato @toasterpup  http://t.co/npd1Ub7e <instagr.am/p/O87-M/>
[05:18:25] <extralife> RT @rwwoodward: @extralife @nicolespag @missiondeep Final Score episode 69 show notes now available...  http://t.co/sg3tjJVB <finalscoreshow.com>

Doing:

account twitter off
account twitter on

seems to get it going again.

I am able to send tweets but I do not get the "<root> You: $tweet" message.

Wilmer mentioned in IRC that disabling fetch_mentions might be a workaround. I have not tested it as I did not see the pattern until now.

[lkraav <leho@…>]

yep same here, cc-ing

[wilmer]

I wrote a work-around for this in changeset:devel,814.

I need more info before I can fix this for real. My current guess is something like a HTTP timeout. Anyone who sees this happen a lot, please get me some traffic dumps..

[jjudin+bitlbee@…]

I also noticed this behavior with Twitter by using multiple accounts (with development branch changeset number 811) and one of them would stop giving me new tweets after a while when other one would work fine. And it didn't seem to be related to the time of the day. Now that I disabled fetch_mentions from all of them, all my Twitter accounts have been working OK for over a week.

[wilmer]

Actually I've seen it happen even with fetch_mentions on by the way.

The most recent version in bzr has a workaround BTW (one that doesn't require disabling mentions). Still hoping to really fix the problem though, of course..

[ilf]

I hit it again, running 821. In my experience, this is not really connected to a certain time of the day.

[sjefen6]

happens to me everytime arround 5am on Mondays (CET) even last week when Norway had changed to wintertime and US still was on summertime.

[jjudin+bitlbee@…]

It might have something to do with connections not timeouting when they should. I noticed that my Twitter accounts were not updating after my Bitlbee machine's ISP had some connection troubles for an hour or two. This could be easily tested by just disconnecting from the network for a while and then checking if Twitter is updating at all after it.

[Wilmer van der Gaast <wilmer@…>]

Yes, pretty sure it's related to fetch failures, just not sure which kind and no time to test so far. BTW at least the workaround should be kicking in now?

[Artem Savkov <artem.savkov@…>]

Just to sum up what have been discussed on irc for history: The bug is caused by network problems, http_dorequest not always calling callback functions and current flag system. Current fix is not perfect, because twitter still stalls on partial success (i.e. one of the requests succeeds before next attempt and the second one fails).

I'm attaching two experimental patches.

• bitlbee-twitter-stall.patch - the one fixing both the current fix and the issue when callback functions were never called on failed connection. I've tested this one for some time and it works for me.
• bitlbee-twitter-stall-wo-slow.patch - the one fixing the issue when callback functions were never called on failed connection and removing current workaround completely. I tested it somehow and it seems to work, but I've just came up with the idea that this might be actually enough to fix the whole issue. So I'll run it for a day or two(that should be enough since my isp is pretty nasty :) and report back with the results.

[Artem Savkov <artem.savkov@…>]

I've ran the second patch overnight. It hit both cases of the 'connection failed' (i.e. with both requests failing to connect and one of them). Seems to hold up nicely.

[Wilmer van der Gaast <wilmer@…>]

Awesome. Thanks for your work so far! I've also now realised you and trac3r on #bitlbee are one and the same person which is useful to know! (-:

Will be back in Europe in a few days..

[wilmer]

changeset:devel,829. Thanks Artem!