Non-local account/settings storage

[Jelmer Vernooij]

It would be nice to be able to store user data (buddy name -> nickname mappings, account data) on an LDAP database. This would particularly be useful for sharing data between various public servers.

[Jelmer Vernooij]

I'd like to work on LDAP support, so assigning to me.

[wilmer]

1.1 should probably have some nice configuration storage backend, so we can implement this and #23 at once in a nice and clean way.

[Jelmer Vernooij]

I've put a branch up at ​http://jelmer.vernstok.nl/oss/bitlbee/bzr/storage/ that adds an abstraction layer for the storage support (and contains just one backend at the moment, which reads and writes the old BitlBee data files). Once this is merged into mainline, I'll have a look at LDAP support.

[wilmer]

Just read the diffs, looks good!

BTW, maybe it could be nice to allow multiple storages to be used for at least loading settings, this could be convenient for conversion from the old format to the (planned) new file format. And maybe also for migration from the usual files to LDAP.

For saving, one storage backend should be enough.

[Jelmer Vernooij]

right, that makes sense. This is quite easy to implement, but I'm a bit doubtful as to what the easiest way to configure such a thing would be. Adding a 'storage_migrate' option that specifies a storage backend to migrate from would make most sense to me. One of the things to keep in mind is that a smooth upgrade (without any required interference from the user) would be nice to have and should be possible. What do you think?

[Jelmer Vernooij]

Wilmer just merged some changes that add a 'account storage' and a 'account storage migrate' option. Left now is the actual LDAP backend, preferably in storage_ldap.c

[Jelmer Vernooij]

I'm currently stuck waiting for an OID assignment from IANA so I can fill those into the schema.

[Jelmer Vernooij]

Ok, we've been allocated an OID by IANA: 25873

[wilmer]

​http://code.bitlbee.org/contrib/arnau-storage/

Thanks arnau!

[wilmer]

I'm trying out a feature freeze, I only want to fix bugs for the 1.2 release and postpone actual features until a later release.

[wilmer]

For the record, this is probably going to be merged in some near future. Some changes have to be made though, most of all we need a more atomic way of saving settings. This is a little bit complicated in LDAP because it'd be nice to avoid having to recreate the whole tree for a user every time we're saving settings.

[mail@…]

any news?

[wilmer]

Whenever I think about this lately, I think LDAP is not the right solution.

Although I like how LDAP would allow us to have the server not share any informaiton unless the client provides the right password, I'm less interested in having to maintain two storage backends, and whenever I add a new kind of info (settings, but also for example channels and per-channel settings as introduced by 3.0), have to update both backends.

Time has moved on in the meantime and there are many distributed storage systems available now. We could just use something like Ceph to store userdata, and for security we'd A) lock down the servers, obviously and B) encrypt the *full* user files instead of just the IM passwords. And the backend can just use the same XML generated and parsed by storage_xml so once written, it'll support whatever the XML backend supports.

By adding a layer in between the BitlBee client and the database, we could actually still get the same level of security like LDAP (i.e. a little daemon running on the same machines hosting the database that will give the userfile in exchange for the right password).

[wilmer]

Trying out a discussion about this design on ​https://plus.google.com/u/1/104643827569815543173/posts/Eeuob4vv6kw

[dx]

TICKET EUTHANASIA TIME!

[wilmer]

Nope! :-)

But let's change the title to not mention LDAP since that part is definitely a NOPE.

I was thinking of this one again. Maintaining public servers is a massive pain and I want to make it easier to run them in a post-2000 way. Throw them into a stateless container, have them read user settings from an external source.

Maybe use etcd or so as a backend? A friend and I worked on an API for this a while ago, and there's the discussion above.

[dx]

• Summary changed to Quantum nonlocality

Someone wrote a mysql storage engine in #65, could probably adopt that module, fix it up a bit, and make it one of the options.

etcd looks cool, and I hear people use that with docker often, which is also cool.

Or maybe it's just a fad, just like... LDAP and XML :D (Oh god what)

There's also redis, which i'm not sure if it's the best option for "configuration" storage (whatever that means), but it's so lightweight for servers and the protocol is so trivial to implement that, why not?

[Dennis Kaarsemaker <dennis@…>]

Redis isn't really suitable for persistent storage. I'm currently working on a hybrid mysql/ldap storage backend (ldap for authentication, mysql for account/settings storage). The thing that's really "getting in the way" there is the tight coupling between authentication and data storage. I might submit a PR soon to loosen that coupling a bit if you think that is a good idea.