Twitter DM channels don't show messages sent in other clients

[count-bitlbee.org@…]

I sometimes use a phone or webclient to view the full timeline including pictures (can't see them in weechat-curses).

I also respond to DMs in these clients, and the messages sent by me don't become visible via bitlbee later :(

[wilmer]

This may not be that simple to do since I *think* your own messages are not repeated in the stream.

[anonymous]

Hmm.

​https://dev.twitter.com/streaming/overview/messages-types#Direct_Messages is a bit unclear in that regard.

In doubt checking ​https://dev.twitter.com/rest/reference/get/direct_messages/sent since the last displayed message before displaying the next might help, which could also be polled ever minute or so additionally ...?

[dx]

I looked into this as part of #1156.

It's complicated. Your messages are repeated in the stream, but that means *all* your messages are repeated in the stream, and the messages we sent just have a target, a body, and no IDs. When our own messages come back over the stream we have no way to identify them, since twitter often changes their contents too (for example, turning urls into t.co, normalizing unicode, stripping whitespace)

In other words, I see no decent way to do this without echoing absolutely everything, much like how the #twitter channel echoes a "You: ..." when you tweet. That's awful enough for the twitter channel, but worse for DMs.

[wilmer]

Don't we get the Tweet ID back on successful posts which could maybe be used for deduping? Or do DMs not get IDs assigned to them?

[anonymous]

what about deduping by timestamp? i.e. expect a message sent in the same second as one sent by us a dupe of ours?

maybe making it configurable to see the dupes (i.e. get a view of what was actually posted after something was sent by us) would also be desirable.

[dx]

Don't we get the Tweet ID back on successful posts which could maybe be used for deduping? Or do DMs not get IDs assigned to them?

Ah yeah I knew I was forgetting a part. So yes, we get an ID back on success. The real problem was that the tweet may (and often does) arrive over the stream *before* we get the reply of the http request with the ID. So my idea of a fix was to use some sort of 'lock', and put incoming DMs from your own username on hold until all outgoing DM creation http requests are answered. That's where the "It's complicated" comes from. It's a hassle.

[count-bitlbee.org@…]

Okay, that seems like a (sound) architecture decision on Twitter's side - Publish a message to all streams before the HTTP call for its Push returns.

How about (optionally) dropping messages from our own account IF we have a Push going on, and showing all others?

While sucessfully de-duping, this would let us only miss messages from foreign clients if they're sent at exactly the same time as we're sending one - right now, we're missing all of them. Considering the speed of the Twitter API, this should be an acceptable tradeoff.

I'd consider this a vast improvement, at very moderate cost. Not sure whether it would require architectural changes.

Let me know if I can help in any way.

[wilmer]

Yeah, I think it's reasonable to just suppress own-messages if the user has posted something over the last ... 10s? The streaming API is fast.

Yes you may lose things that way if you're posting things on two clients at the same time but people who do bizarre things like that shouldn't complain when that has odd side-effects.

The real problem was that the tweet may (and often does) arrive over the stream *before* we get the reply of the http request with the ID.

Darn, good point, streaming API is even faster than I thought then. Yeah we could do the locking/buffering but I'd rather just keep it simple with the idea above?

[count-bitlbee.org@…]

I'm still convinced just suppressing (own) messages while we have a message outgoing would be enough, no need for further locking, delays, checking previous posts or other complexity.

[wilmer]

Suppressing them for, say, 10s, comes down to the same thing I think, since "while we have a message outgoing" is somewhat vague - when exactly are we done, and what guarantees do we have on when the message is echoed back over the stream? It may happen 1s after the POST is finished.

So just a 10s or so blackout will do the job best I think.

[count-bitlbee.org@…]

Hmm. "While we have a message outgoing" would be "we have a POST request active", and suppressing message originating from our account while that's the case should be possible. After the POST is done we know the message ID, and could de-dup anything new by that ...

Sorry if I'm annoying about this, but I kinda consider time-based decisions somewhat arbitrary.

[wilmer]

Heh ok, there's that of course, the message ID.

That's two mechanisms though so more complexity. I don't care terribly much but I do like to keep things simple. Theoretically your proposal is still time-based and the only truly clean solution would be to enqueue incoming messages so ID-based deduping can be done when all in-flight POSTs are out.

But that's even more complex so ew! Keep in mind that the only messages we'd drop during the 10s blackout would be your own, so unless you have a habit of simultaneously sending DMs from different devices (including BitlBee), which sounds like a rather eccentric hobby to me, it's not an issue.

[count-bitlbee.org@…]

Mine is more order and data based, but yeah ;)

Starting to see my own messages from another client in-channel would already be a big plus no matter how it's achieved.