BitlBee can't establish SSL connections with NSS, works when starting via "bitlbee -D -n"

[dskulinska@…]

I am running a local bitlbee server version 3.0.1 on a GNU/Linux system (Fedora 14).

Three accounts have been set up for a user (ICQ, Jabber, Google Talk). When BitlBee is run in daemon mode as local user "bitlbee", the ICQ login succeeds, but both XMPP logins (Jabber, GoogleTalk) fail:

@root | jabber(XXX@XXX) - Logging in: Connected to server, logging in

@root | jabber(XXX@XXX) - Logging in: Converting stream to TLS

@root | jabber(XXX@XXX) - Login error: Could not connect to server

@root | jabber(XXX@XXX) - Logging in: Signing off..

However, when BitlBee is run via "bitlbee -n -v -D" (for debugging), all accounts are successfully connected, using the same configuration for the user (explicitly dictated via "/etc/bitlbee/bitlbee.conf").

I find it odd that starting the BitlBee server with an option ("-n") that should not affect the configuration leads to a different result. If this problem is due to my own ignorance, I would welcome any hints and pointers, otherwise I would be happy to help to resolve this issue on my own. I did find it difficult, however, to track down the problem due to the lack of debugging output. Am I missing something?

Regards

Dasia

[wilmer]

XMPP failing and ICQ working may be because ICQ doesn't need SSL anywhere while your Jabber logins seem to go wrong at the STARTTLS stage.

I do indeed also find this very odd. And without more debugging info from the SSL layer I don't know what to do.. So -n is the only flag you changed? Which SSL lib is this BTW?

[dskulinska@…]

deferred call to ssl_init() and otr_init() until after fork()

[dskulinska@…]

Thank you for the quick reply and pointing towards the SSL library, which turned out to be a fruitful hint. The "-n" flag was indeed the only flag changed (I used to add "-v", but it did not produce any more log output).

I am using bitlbee 3.0.1 as packaged for Fedora 14 (x86_64 architecture), which links against libnss3.so from NSS version 3.12.9.

I have checked out bitlbee branch "-r 3.0.1" from the repository and built two versions,

• bitlbee-3.0.1-nss, configured with "--ssl=nss", linking against libssl3.so from NSS 3.12.9, and
• bitlbee-3.0.1-gnutls, configured with "--ssl=gnutls", linking against libgnutls.so from GnuTLS 2.8.6.

The reported problem can be reproduced with bitlbee-3.0.1-nss, i.e. Jabber logins fail when forking into background (run with "-D"), but work in "non-fork mode" (giving "-n -D" on the command line). Interestingly, bitlbee-3.0.1-gnutls works in both cases.

When running bitlbee-3.0.1-nss in daemon mode, the call to SSL_ForceHandshake() in ssl_connected() (lib/ssl_nss.c) fails. Further investigation revealed that there might be a problem regarding NSS and the use of fork():

• ​http://osdir.com/ml/mozilla.crypto/2002-08/msg00016.html (might be out-dated, but addresses problems using NSS and forking)
• ​http://groups.google.com/group/mozilla.dev.tech.crypto/browse_thread/thread/9143e90465750611/ebc0c287a4cafab3
• ​http://www.mail-archive.com/dev-tech-crypto@lists.mozilla.org/msg08189.html (mentions a change in behavior for NSS for at least NSS 3.12.9 and above, which will not warn about a detected fork())

I am not familiar at all with the various SSL libraries or their initialization process, so I am not sure if these are at all related. A quick patch (see attached file) that moves the call to ssl_init() further down until after fork()ing into background seems to solve at least part of the problem. Now the Google Talk login succeeds, but the login to a second account (a private server running ejabberd) fails with a timeout (after requesting the buddy list -- the initial SSL handshake succeeds, however).

Regards

[wilmer]

Ah, thanks for tracking that down!

I wonder if this fixes the problem for ForkDaemon mode? I suppose not..

[castaway@…]

I'm having a similar issue with 3.0.5 right now, this is using gnutls (not nss), built on gentoo with: USE="gnutls jabber msn oscar plugins ssl yahoo -debug -ipv6 -libevent -nss -otr -purple (-skype) -test -twitter -xinetd"

I previously had 3.0.3 installed which did the same thing (same options) .. it seems to eventually, after about 10 rounds of connecting, disconnecting, figure it out and stay connected:

17:43 <@root> fb - Reconnecting in 5 seconds..
17:43 <@root> fb - Logging in: Connecting
17:43 <@root> fb - Logging in: Connected to server, logging in
17:43 <@root> fb - Logging in: Converting stream to TLS
17:43 <@root> fb - Logging in: Connected to server, logging in
17:43 <@root> fb - Logging in: Authentication finished
17:43 <@root> fb - Logging in: Server changed session resource string to 
              `BitlBee_f0e1effa_4BF60DA422B82'
17:43 <@root> fb - Logging in: Authenticated, requesting buddy list
17:43 <@root> fb - Logging in: Logged in
17:43 -!- Netsplit over, joins: <names>

17:43 <@root> fb - Error: Error while reading from server
17:43 <@root> fb - Signing off..
17:43 -!- Netsplit localhost.localdomain <-> chat.facebook.com quits: 
          <names>
17:43 <@root> fb - Reconnecting in 5 seconds..

etc, around and around

[anonymous]

I can confirm that the issue exists in 3.0.5 (3.0.5 from EPEL), and that running with the -n -D switch fixes the issue.

Using Daemon or ForkDaemon does not seem to make a difference.

NSS is the default SSL library on Fedora and RHEL based distributions, so there are likely to be a number of affected users.

[Matěj Cepl <mcepl@…>]

Please distinguish between using Fedora patches and NSS (if you don't have Fedora packaged bitlbee, you most likely don't have NSS), patches in #714 have not been merged yet and they need first review from RH security people (and apparently rewrite).

Also, if you have Fedora package, the only supported mode of running bitlbee is with the init scripts provided with the package. Doing anything else will most likely hit SELinux and others. If you use bitlbee in other mode on Fedora, then please don't file bug reports to bugzilla.redhat.com.

[anonymous]

I have the same issue with 3.2. Bitlbee is compiled with --ssl=gnutls.

[dx]

So it turns out, in the last week two people using CentOS 6.5 and RHEL 6.5 joined #bitlbee about this exact issue, although with different error messages (Unusuable response (sic) with facebook OAuth2, and Error during Passport authentication: Error while writing HTTP request with MSN, both of them HTTPS)

And we discovered the -n workaround when trying to debug it, too.

And then we noticed it was a NSS issue.

And then I pretty much replicated the research that the OP did in comment:2, and did a bunch of pointless debugging (Fun fact: debug builds of NSS do a CHECK_FORK() assert that does abort() when they detect a fork, instead of failing silently).

And then I realized that this wasn't a NSS bug and we were really doing it wrong.

And then I came up with a patch that looks very similar to the one posted here.

And then I noticed that we have this goddamn bug, with major priority, that I also tagged as "patch" seven days ago but somehow managed to avoid reading.

SIGH.

Sorry for the massive delay, redhat users!

Anyway, the patch posted here does solve it for daemon mode, but will fail for forkdaemon. Libpurple had a similar issue with forks, so it does its initialization in irc.c. I think it's an appropriate place to put both otr and ssl init, but will need to test some more.

Also, comment:4 and comment:7 use gnutls, those are unrelated issues.

[dx]

Changed subject to make it clearer that this is *the* nss bug

[mcepl@…]

I am here.

[mcepl@…]

Replying to mcepl@…:

I am here.

Hmm, I am not, how to get myself to CC?

[Matěj Cepl <mcepl@…>]

Couple of questions:

• if there are hords of CentOS/Fedora users with broken bitlbee, why I don't have anything in ​bugzilla.redhat.com?

• we have in EPEL-6 ​bitlbee-3.2.1-3 so I would be more interested in crashes reproduced with that. Could somebody who can reproduce this test with the updated EPEL package could try also this ​experimental build (in next 14 days or so, then it gets garbage collected), please?

[dx]

Replying to Matěj Cepl <mcepl@…>:

Couple of questions:

• if there are hords of CentOS/Fedora users with broken bitlbee, why I don't have anything in ​bugzilla.redhat.com?

I already replied on irc so copypasting here for completeness:

BitlBee+nss only breaks when using forkdaemon or daemon without the -n parameter - centos only ships xinetd units, while fedora's systemd service unit starts it with -D -n (the exact workaround, but might be a coincidence) and the systemd socket unit is like xinetd, I think.

So in practice this issue only happens when someone decides to ignore what your packages include, and start it with "bitlbee -F" or "bitlbee -D"

/end copypasta

Also, just checked the source of bitlbee-3.2.1-3.el5 - the patch named bitlbee-systemd.patch changes the systemd service unit to use -D -n instead of -F -n - but it might be just so the unit can be turned into a Type=simple one, which is why i call it 'coincidence'. Fun stuff.

Replying to Matěj Cepl <mcepl@…>:

• we have in EPEL-6 ​bitlbee-3.2.1-3 so I would be more interested in crashes reproduced with that.

I don't redhat but i've already reproduced this in my arch linux with 3.2.1 / current bzr, just by enabling nss. Not a crash, btw. Only ssl connections dying silently and returning weird-ass error messages.

Replying to Matěj Cepl <mcepl@…>:

Could somebody who can reproduce this test with the updated EPEL package could try also this ​experimental build (in next 14 days or so, then it gets garbage collected), please?

Took me a while to find out how to get a source package out of that link (or any package for that matter. had to pick one of the architectures in descendants, and there are package links in the output section).

That seems to be the exact patch posted in this ticket, which, as i said in comment:9, fixes daemon and breaks forkdaemon.

Will attach fixed patch soon™

[Matěj Cepl <mcepl@…>]

Replying to dx:

So in practice this issue only happens when someone decides to ignore what your packages include, and start it with "bitlbee -F" or "bitlbee -D"

The main reason why we prefer -D on Fedora (and consequently RHEL/EPEL) is that it works way better with SELinux (bitlbee is tightly confined on Fedora and it is much more simple to write a policy for one process than for randomly popping up ones). We will add README.Fedora explaining this to the Fedora/EPEL package.

​https://bugzilla.redhat.com/show_bug.cgi?id=1098801

Also, just checked the source of bitlbee-3.2.1-3.el5 - the patch named bitlbee-systemd.patch changes the systemd service unit to use -D -n instead of -F -n - but it might be just so the unit can be turned into a Type=simple one, which is why i call it 'coincidence'. Fun stuff.

Yes, this issue happens only when somebody changes Fedora default configuration (which works perfectly as far as I know) without understanding consequences (SELinux issues and this bug are only I can recall at the moment, but I cannot guarantee there are no others).

I don't redhat but i've already reproduced this in my arch linux with 3.2.1 / current bzr, just by enabling nss. Not a crash, btw. Only ssl connections dying silently and returning weird-ass error messages.

Cool, thank you.

That seems to be the exact patch posted in this ticket, which, as i said in comment:9, fixes daemon and breaks forkdaemon.

Will attach fixed patch soon™

Thank you.

[dx]

A hopefully working and secure version of the patch!

[dx]

Okay that "soon™" took a bit too long.

Copy-pasting from the commit message (included in the patch):

This might look like a simple diff, but those 'lies' made this not very straightforward.

The NSS bug itself is simple: NSS detects a fork happened after the initialization, and refuses to work because shared CSPRNG state is bad. The bug has been around for long time. I've been aware of it for 5 months, which says something about this mess. Trac link: #785 [this ticket]

This wasn't a big deal because the main users of NSS (redhat) already applied a different patch in their packages that workarounded the issue somewhat accidentally. And this is the ticket for the 'lies' in unix.c: #1159

Basically a conflict with libotr that doesn't happen anymore. Read that ticket for details on why ignoring those comments is acceptable.

Anyway: yay!

[dx]

Patch submitted, setting as 'accepted'.

[robert@…]

FYI: bitlbee-3.2.2-2.fc22 is shipping this patch and switches from daemon to forkdaemon mode; older Fedora releases (Fedora < 22) are still at daemon mode for the moment.

[robert@…]

As of today, bitlbee-3.2.2-3.fc19, bitlbee-3.2.2-3.fc20, bitlbee-3.2.2-3.fc21 and bitlbee-3.2.2-3.fc22 are shipping this patch to unbreak NSS. However Fedora < 22 are still at daemon mode (and without libpurple), only Fedora >= 22 is forkdaemon (and with additional libpurple).

[dx]

Applied in bzr rev 1038