Random disconnect on KVM VPS

[André Friedrich <info@…>]

Hi,

on my virtual KVM-VPS only MSN disconnects every 5-10 minutes with "error while reading from server". I've recompiled it several times, no improvement. After installing bitlbee on a OpenVZ Container, all seems to work well. What's the matter here? You need special logs?

[Wilmer van der Gaast <wilmer@…>]

Are those OpenVZ and KVM instances running on the same machine and network? This is, AFAIK, not a KVM-specific issue. I myself have it on my machine since about a week ago, and so do others. Yet many others have no troubles like this at all.

I'm not sure what's going on here, doing more frequent pings doesn't help at least. :-/

[André Friedrich <info@…>]

No, the KVM-Machine is hosted at hetzner.de - the other one at accelerated.de. I'm wondering why only MSN is doing that ... Twitter, ICQ and AIM are ok[[BR]]

Things i've already done:

• Reinstall VPS
  
• New Kernel
  
• Recompile Bitlbee
  
• Other Login-Server
  
• Reboot few times VPS
  

Is there maybe a debug-Mode why the connection aborts?

[wilmer]

Ah, hetzner, that's where I have my BitlBee running and I'm also seeing these disconnects. Many others aren't, like you on the other box..

I've run a tcpdump to see what's going on:

00:49:46.049795 IP 178.63.244.1.55258 > 65.54.49.140.1863: Flags [P.], seq 35:40, ack 57, win 561, options [nop,nop,TS val 893568548 ecr 24026640], length 5
        0x0000:  4500 0039 d36d 4000 4006 4e4e b23f f401  E..9.m@.@.NN.?..
        0x0010:  4136 318c d7da 0747 4701 efcb c403 7571  A61....GG.....uq
        0x0020:  8018 0231 192f 0000 0101 080a 3542 c624  ...1./......5B.$
        0x0030:  016e 9e10 504e 470d 0a                   .n..PNG..
00:49:46.173795 IP 65.54.49.140.1863 > 178.63.244.1.55258: Flags [P.], seq 57:65, ack 40, win 65330, options [nop,nop,TS val 24027112 ecr 893568548], length 8
        0x0000:  4500 003c 504a 4000 7206 9f6e 4136 318c  E..<PJ@.r..nA61.
        0x0010:  b23f f401 0747 d7da c403 7571 4701 efd0  .?...G....uqG...
        0x0020:  8018 ff32 98a4 0000 0101 080a 016e 9fe8  ...2.........n..
        0x0030:  3542 c624 514e 4720 3433 0d0a            5B.$QNG.43..
00:49:46.173795 IP 178.63.244.1.55258 > 65.54.49.140.1863: Flags [.], ack 65, win 561, options [nop,nop,TS val 893568579 ecr 24027112], length 0
        0x0000:  4500 0034 d36e 4000 4006 4e52 b23f f401  E..4.n@.@.NR.?..
        0x0010:  4136 318c d7da 0747 4701 efd0 c403 7579  A61....GG.....uy
        0x0020:  8010 0231 6f3b 0000 0101 080a 3542 c643  ...1o;......5B.C
        0x0030:  016e 9fe8                                .n..
00:50:29.173795 IP 178.63.244.1.55258 > 65.54.49.140.1863: Flags [P.], seq 40:45, ack 65, win 561, options [nop,nop,TS val 893579329 ecr 24027112], length 5
        0x0000:  4500 0039 d36f 4000 4006 4e4c b23f f401  E..9.o@.@.NL.?..
        0x0010:  4136 318c d7da 0747 4701 efd0 c403 7579  A61....GG.....uy
        0x0020:  8018 0231 192f 0000 0101 080a 3542 f041  ...1./......5B.A
        0x0030:  016e 9fe8 504e 470d 0a                   .n..PNG..
00:50:29.513795 IP 178.63.244.1.55258 > 65.54.49.140.1863: Flags [P.], seq 40:45, ack 65, win 561, options [nop,nop,TS val 893579414 ecr 24027112], length 5
        0x0000:  4500 0039 d370 4000 4006 4e4b b23f f401  E..9.p@.@.NK.?..
        0x0010:  4136 318c d7da 0747 4701 efd0 c403 7579  A61....GG.....uy
        0x0020:  8018 0231 192f 0000 0101 080a 3542 f096  ...1./......5B..
        0x0030:  016e 9fe8 504e 470d 0a                   .n..PNG..
00:50:30.193795 IP 178.63.244.1.55258 > 65.54.49.140.1863: Flags [P.], seq 40:45, ack 65, win 561, options [nop,nop,TS val 893579584 ecr 24027112], length 5
        0x0000:  4500 0039 d371 4000 4006 4e4a b23f f401  E..9.q@.@.NJ.?..
        0x0010:  4136 318c d7da 0747 4701 efd0 c403 7579  A61....GG.....uy
        0x0020:  8018 0231 192f 0000 0101 080a 3542 f140  ...1./......5B.@
        0x0030:  016e 9fe8 504e 470d 0a                   .n..PNG..

...

00:53:22.913795 IP 178.63.244.1.55258 > 65.54.49.140.1863: Flags [P.], seq 40:45, ack 65, win 561, options [nop,nop,TS val 893622764 ecr 24027112], length 5
        0x0000:  4500 0039 d378 4000 4006 4e43 b23f f401  E..9.x@.@.NC.?..
        0x0010:  4136 318c d7da 0747 4701 efd0 c403 7579  A61....GG.....uy
        0x0020:  8018 0231 192f 0000 0101 080a 3543 99ec  ...1./......5C..
        0x0030:  016e 9fe8 504e 470d 0a                   .n..PNG..
00:53:23.037795 IP 65.54.49.140.1863 > 178.63.244.1.55258: Flags [R], seq 3288561017, win 0, length 0

First a successful ping-pong. Then, 43s later the next one, which does not get acked (on the TCP level even), and 3m later a TCP reset. Keepalives more than once a minute really should be sufficient, so I don't know what's going on. Either MS is having serious capacity issues or this is just classical sabotage. Have you tried using a different client from the Hetzner box yet?

[André Friedrich <info@…>]

No, i haven't test it with another client yet ... I think i don't know any other ones :p
But some MTR told me, that the tracetroute is a completely other. Maybe this is the fault?
So please tell me a client to check it :)

[Wilmer van der Gaast <wilmer@…>]

You can try bitlbee-libpurple instead.

So you mean that the route to the MSN servers changes entirely?? Yikes!

[wilmer]

... to be more precise, you see a route change around the time the server stops sending TCP ACKs? Wow.

[André Friedrich <info@…>]

No, not the complete route changes - the route was completely different with accelerated.de.

I've done some MTR ... Hetzner has a lot of Paketloss on the way to MSN :-) I think I'll have to write to the support, haven't I? Maybe the firewalls filter those TCP ACKs?

[André Friedrich <info@…>]

Seems to be, that the Routing is broken. Talking again to Hetzner Support.

[André Friedrich <info@…>]

The Routing of the Init7-Peering was broken - Hetzner chose another one, now all is perfect. Thank you, Wilmer!

[Wilmer van der Gaast <wilmer@…>]

Oh, wow, that was the issue?

More people were reporting problems like this, I wonder if they were all on Hetzner boxes. Thanks a lot for chasing this down!

[ander.punnar@…]

Same problems here. Random disconnects and other not so annoying problems. VPS, Parallels Virtuozzo, in Estonian main hub.

[André Friedrich <info@…>]

Please, post the output of an mtr with at least 1000 passes on messenger.hotmail.com. Maybe it's the same problem here?

[wilmer]

Is it just me or did the problem return for you too?

[André Friedrich <info@…>]

Sorry for late response ... for me, it seems all ok. You already made an mtr? Maybe there's again packetloss at init7?

[André Friedrich <info@…>]

Yeah - there's the same problem again. For more information check: ​http://www.init7.com/status/ ... afaik you'll be routed through de-cix.