Twitter: tweets with embedded tweets sometimes(?) not shown

[levu@…]

Sometimes tweets with embedded tweets (= link to a tweet inside) don't get shown. I experienced this now a hand full of times over the last days, starting when twitter announced their "retweet with comment" feature, which essentially is just tweeting with a link to some other tweet inside.

Some other tweets with links to a tweet inside were displayed indeed.

I looked into the twitter API, a tweet object looks the same it looked some weeks ago, so I guess there were no changes.

Since there were no missing bitlbee-tweet-IDs (those 2digit-hex-numbers) in the time range the tweet was posted, this is not some client filtering or so.

Version is BitlBee 3.4-0-g61d21e5-release-3.4-git API version 030400

[dx]

It could be some client filtering. Maybe it's considering them as retweets (with no extra comment) and deduplicating them as such, because they were already retweeted / quoted before.

[wilmer]

Yes, would be interesting to see whether that's explained by dx' arguments. Though the format appears to be different. An example I've just grabbed from my stream:

​http://p.gaa.st/view/51702364

Nested message is a "quoted_status", not a "retweeted_status". BitlBee doesn't handle that at all. (Also, note the lack of useful results for ​https://www.google.co.uk/search?q=twitter+api+"quoted_status" .. nice documentation guys! :-/)

I wonder whether these can be nested, I'll assume not. But could it be native-retweeted? It *looks* like it can't be: I've just clicked on a message like that, and it shows me the dialog where I can add a comment, but the quoted Tweet is shown as just the added text and a twitter.com/... URL at the end.

[dx]

I really have no idea. Given that pastebin, my theory is wrong, and everything should work.

The developer forums don't mention this at all, the REST api doesn't return quoted_status for the same tweet wilmer linked.

Maybe a server side issue while they roll out this feature?

I'd appreciate any reports that this is still happening, and "me too" comments in general, if anyone else is having this issue.

[dx]

I poked two random twitter users who appeared in my tracking stream complaining about twitter issues, and i thought they referred to this one, but nope (both were about old bugs which are fixed already)

Sooo as far as I can see, no one else reported this bug (To be fair, it's hard to notice it)

But yeah, I'm giving this a few weeks, if it doesn't happen anymore, this could be closed blaming the server.

[wilmer]

Either that or repurpose this bug for decoding these Tweets properly. I'm already tired of clicking on the twitter.com links because seeing the added comment only is super-unhelpful. :-(

[wilmer]

Parsing this now, this is going to be the format:

00:19:03 < BitlBeeTest2> [13] ok nog eentje <https://t.co/eQaBK01Xvk> [@Wilmer: After some hours on 
                         the UK roads,I don't see why people fear self-driving cars. Can't be hard 
                         to have computers drive at least that "well".]

Funny discovery: It appears that not all feeds include a "quoted_status" property yet. I'm getting it from the streaming API only for now. It's not documented so who knows what they'll do with it/for how long this is going to work..

[dx]

Sorry, ticket takeover cancelled!

While reviewing wilmer's new stuff I noticed that I was also missing *some* quoted tweets, the original issue here. I posted this thread in the developer forums:

​https://twittercommunity.com/t/some-retweets-with-comments-are-missing-filtered-out-from-user-streams/36826

I'm still mostly blaming the servers, but who knows. Hopefully some twitter employee will give it a look.

[wilmer]

Sorry, ticket takeover cancelled!

*shakes fist*

It indeed seems most likely that they're not appearing in the streaming API sometimes. As I said above there's some oddity with them already anyway. Lame. :-(

I can restore my hack to dump every message on the stream to confirm this if necessary. Or just temporarily disable stream deduping.

[wilmer]

So Twicca beat me to it, they did a release yesterday with quoted_status support. And not just in the streaming API...

No source code available to easily see what's their trick.

[dx]

No source code available to easily see what's their trick

I had a look. It's not quoted_status, it's just a REST api request to /1.1/statuses/show/<id>.json right after the request to home_timeline. Meh.

[dx]

whoops, sorry for that

[dx]

More relevant to this ticket: added a post to ​that thread, in which I believe I've found what causes this. It applies only to quoted statuses that start with a @mention, and as far as I can see it's definitely twitter's fault

@OP: can you confirm that the tweets that disappeared were quoting other tweets that started with @?

Because apparently all the examples I have of missing tweets are my own, no one else reported it. But it's reproducible, so it's gotta be this.

[levu@…]

I'll have a look into it, couldn't reproduce it on the first try, but I'll keep you informed.

[dx]

Quoting myself from the twitter thread, 15 days ago:

Did some more tests, now with just curl + signatures for the userstream generated with OAuth Tool.

And, uh, either it's fixed, or my theory that it's related to the fact that they started with mentions is wrong. Now I kinda regret not testing more than two times. I repeated it 12 times, all of them showed up, both in the curl stream and my client

I think I'm just going to keep looking for more of those like i did before (comparing stream results with twitter web contents) for a few more days until I get bored of wasting time with this.

And I haven't seen this happen again.

Closing this as "fixed" because wilmer did something, and I like to close stuff as "fixed" when something was done.