Changes in / [92d3044:cc6fdf8]

Files:

• debian/copyright (modified) (2 diffs)
• doc/user-guide/commands.xml (modified) (4 diffs)
• lib/Makefile (modified) (1 diff)
• lib/http_client.c (modified) (16 diffs)
• lib/http_client.h (modified) (7 diffs)
• lib/json.c (added)
• lib/json.h (added)
• lib/json_util.c (added)
• lib/json_util.h (added)
• lib/oauth2.c (modified) (5 diffs)
• lib/ssl_gnutls.c (modified) (8 diffs)
• nick.c (modified) (2 diffs)
• protocols/msn/msn.c (modified) (1 diff)
• protocols/msn/ns.c (modified) (1 diff)
• protocols/nogaim.c (modified) (2 diffs)
• protocols/nogaim.h (modified) (1 diff)
• protocols/twitter/twitter.c (modified) (16 diffs)
• protocols/twitter/twitter.h (modified) (5 diffs)
• protocols/twitter/twitter_http.c (modified) (6 diffs)
• protocols/twitter/twitter_http.h (modified) (1 diff)
• protocols/twitter/twitter_lib.c (modified) (40 diffs)
• protocols/twitter/twitter_lib.h (modified) (3 diffs)

debian/copyright

@@ -8 +8 @@
 
 Mainly Copyright 2002-2012 Wilmer van der Gaast.
-Some parts are borrowed from Gaim (version 0.58) <http://gaim.sf.net/>.
-For the copyrights on those parts, please read the Gaim source code.
+
+
+Bits of third party code, also GPLed:
+* Some parts (mostly protocols/oscar/) are borrowed from Gaim (version
+  0.58), now known as Pidgin <http://www.pidgin.im/>.
+* protocols/yahoo/ is libyahoo2 <http://libyahoo2.sf.net/>.
+
+Other license (but GPL-compatible):
+* lib/json.[ch] is from <https://github.com/udp/json-parser> and licensed
+  under the modified BSD license.
+
 
 BitlBee License:
@@ -33 +42 @@
 
 The SGML-formatted documentation is written by Jelmer Vernooij
-<jelmer@nl.linux.org> under the GNU Free Documentation License:
+<jelmer@samba.org> under the GNU Free Documentation License:
 
 ============================================================================
-                GNU Free Documentation License
-                   Version 1.1, March 2000
+Copyright (c)  2002-2012 Jelmer Vernooij, Wilmer van der Gaast.
 
- Copyright (C) 2000  Free Software Foundation, Inc.
-        51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA.
- Everyone is permitted to copy and distribute verbatim copies
- of this license document, but changing it is not allowed.
-
-
-0. PREAMBLE
-
-The purpose of this License is to make a manual, textbook, or other
-written document "free" in the sense of freedom: to assure everyone
-the effective freedom to copy and redistribute it, with or without
-modifying it, either commercially or noncommercially.  Secondarily,
-this License preserves for the author and publisher a way to get
-credit for their work, while not being considered responsible for
-modifications made by others.
-
-This License is a kind of "copyleft", which means that derivative
-works of the document must themselves be free in the same sense.  It
-complements the GNU General Public License, which is a copyleft
-license designed for free software.
-
-We have designed this License in order to use it for manuals for free
-software, because free software needs free documentation: a free
-program should come with manuals providing the same freedoms that the
-software does.  But this License is not limited to software manuals;
-it can be used for any textual work, regardless of subject matter or
-whether it is published as a printed book.  We recommend this License
-principally for works whose purpose is instruction or reference.
-
-
-1. APPLICABILITY AND DEFINITIONS
-
-This License applies to any manual or other work that contains a
-notice placed by the copyright holder saying it can be distributed
-under the terms of this License.  The "Document", below, refers to any
-such manual or work.  Any member of the public is a licensee, and is
-addressed as "you".
-
-A "Modified Version" of the Document means any work containing the
-Document or a portion of it, either copied verbatim, or with
-modifications and/or translated into another language.
-
-A "Secondary Section" is a named appendix or a front-matter section of
-the Document that deals exclusively with the relationship of the
-publishers or authors of the Document to the Document's overall subject
-(or to related matters) and contains nothing that could fall directly
-within that overall subject.  (For example, if the Document is in part a
-textbook of mathematics, a Secondary Section may not explain any
-mathematics.)  The relationship could be a matter of historical
-connection with the subject or with related matters, or of legal,
-commercial, philosophical, ethical or political position regarding
-them.
-
-The "Invariant Sections" are certain Secondary Sections whose titles
-are designated, as being those of Invariant Sections, in the notice
-that says that the Document is released under this License.
-
-The "Cover Texts" are certain short passages of text that are listed,
-as Front-Cover Texts or Back-Cover Texts, in the notice that says that
-the Document is released under this License.
-
-A "Transparent" copy of the Document means a machine-readable copy,
-represented in a format whose specification is available to the
-general public, whose contents can be viewed and edited directly and
-straightforwardly with generic text editors or (for images composed of
-pixels) generic paint programs or (for drawings) some widely available
-drawing editor, and that is suitable for input to text formatters or
-for automatic translation to a variety of formats suitable for input
-to text formatters.  A copy made in an otherwise Transparent file
-format whose markup has been designed to thwart or discourage
-subsequent modification by readers is not Transparent.  A copy that is
-not "Transparent" is called "Opaque".
-
-Examples of suitable formats for Transparent copies include plain
-ASCII without markup, Texinfo input format, LaTeX input format, SGML
-or XML using a publicly available DTD, and standard-conforming simple
-HTML designed for human modification.  Opaque formats include
-PostScript, PDF, proprietary formats that can be read and edited only
-by proprietary word processors, SGML or XML for which the DTD and/or
-processing tools are not generally available, and the
-machine-generated HTML produced by some word processors for output
-purposes only.
-
-The "Title Page" means, for a printed book, the title page itself,
-plus such following pages as are needed to hold, legibly, the material
-this License requires to appear in the title page.  For works in
-formats which do not have any title page as such, "Title Page" means
-the text near the most prominent appearance of the work's title,
-preceding the beginning of the body of the text.
-
-
-2. VERBATIM COPYING
-
-You may copy and distribute the Document in any medium, either
-commercially or noncommercially, provided that this License, the
-copyright notices, and the license notice saying this License applies
-to the Document are reproduced in all copies, and that you add no other
-conditions whatsoever to those of this License.  You may not use
-technical measures to obstruct or control the reading or further
-copying of the copies you make or distribute.  However, you may accept
-compensation in exchange for copies.  If you distribute a large enough
-number of copies you must also follow the conditions in section 3.
-
-You may also lend copies, under the same conditions stated above, and
-you may publicly display copies.
-
-
-3. COPYING IN QUANTITY
-
-If you publish printed copies of the Document numbering more than 100,
-and the Document's license notice requires Cover Texts, you must enclose
-the copies in covers that carry, clearly and legibly, all these Cover
-Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on
-the back cover.  Both covers must also clearly and legibly identify
-you as the publisher of these copies.  The front cover must present
-the full title with all words of the title equally prominent and
-visible.  You may add other material on the covers in addition.
-Copying with changes limited to the covers, as long as they preserve
-the title of the Document and satisfy these conditions, can be treated
-as verbatim copying in other respects.
-
-If the required texts for either cover are too voluminous to fit
-legibly, you should put the first ones listed (as many as fit
-reasonably) on the actual cover, and continue the rest onto adjacent
-pages.
-
-If you publish or distribute Opaque copies of the Document numbering
-more than 100, you must either include a machine-readable Transparent
-copy along with each Opaque copy, or state in or with each Opaque copy
-a publicly-accessible computer-network location containing a complete
-Transparent copy of the Document, free of added material, which the
-general network-using public has access to download anonymously at no
-charge using public-standard network protocols.  If you use the latter
-option, you must take reasonably prudent steps, when you begin
-distribution of Opaque copies in quantity, to ensure that this
-Transparent copy will remain thus accessible at the stated location
-until at least one year after the last time you distribute an Opaque
-copy (directly or through your agents or retailers) of that edition to
-the public.
-
-It is requested, but not required, that you contact the authors of the
-Document well before redistributing any large number of copies, to give
-them a chance to provide you with an updated version of the Document.
-
-
-4. MODIFICATIONS
-
-You may copy and distribute a Modified Version of the Document under
-the conditions of sections 2 and 3 above, provided that you release
-the Modified Version under precisely this License, with the Modified
-Version filling the role of the Document, thus licensing distribution
-and modification of the Modified Version to whoever possesses a copy
-of it.  In addition, you must do these things in the Modified Version:
-
-A. Use in the Title Page (and on the covers, if any) a title distinct
-   from that of the Document, and from those of previous versions
-   (which should, if there were any, be listed in the History section
-   of the Document).  You may use the same title as a previous version
-   if the original publisher of that version gives permission.
-B. List on the Title Page, as authors, one or more persons or entities
-   responsible for authorship of the modifications in the Modified
-   Version, together with at least five of the principal authors of the
-   Document (all of its principal authors, if it has less than five).
-C. State on the Title page the name of the publisher of the
-   Modified Version, as the publisher.
-D. Preserve all the copyright notices of the Document.
-E. Add an appropriate copyright notice for your modifications
-   adjacent to the other copyright notices.
-F. Include, immediately after the copyright notices, a license notice
-   giving the public permission to use the Modified Version under the
-   terms of this License, in the form shown in the Addendum below.
-G. Preserve in that license notice the full lists of Invariant Sections
-   and required Cover Texts given in the Document's license notice.
-H. Include an unaltered copy of this License.
-I. Preserve the section entitled "History", and its title, and add to
-   it an item stating at least the title, year, new authors, and
-   publisher of the Modified Version as given on the Title Page.  If
-   there is no section entitled "History" in the Document, create one
-   stating the title, year, authors, and publisher of the Document as
-   given on its Title Page, then add an item describing the Modified
-   Version as stated in the previous sentence.
-J. Preserve the network location, if any, given in the Document for
-   public access to a Transparent copy of the Document, and likewise
-   the network locations given in the Document for previous versions
-   it was based on.  These may be placed in the "History" section.
-   You may omit a network location for a work that was published at
-   least four years before the Document itself, or if the original
-   publisher of the version it refers to gives permission.
-K. In any section entitled "Acknowledgements" or "Dedications",
-   preserve the section's title, and preserve in the section all the
-   substance and tone of each of the contributor acknowledgements
-   and/or dedications given therein.
-L. Preserve all the Invariant Sections of the Document,
-   unaltered in their text and in their titles.  Section numbers
-   or the equivalent are not considered part of the section titles.
-M. Delete any section entitled "Endorsements".  Such a section
-   may not be included in the Modified Version.
-N. Do not retitle any existing section as "Endorsements"
-   or to conflict in title with any Invariant Section.
-
-If the Modified Version includes new front-matter sections or
-appendices that qualify as Secondary Sections and contain no material
-copied from the Document, you may at your option designate some or all
-of these sections as invariant.  To do this, add their titles to the
-list of Invariant Sections in the Modified Version's license notice.
-These titles must be distinct from any other section titles.
-
-You may add a section entitled "Endorsements", provided it contains
-nothing but endorsements of your Modified Version by various
-parties--for example, statements of peer review or that the text has
-been approved by an organization as the authoritative definition of a
-standard.
-
-You may add a passage of up to five words as a Front-Cover Text, and a
-passage of up to 25 words as a Back-Cover Text, to the end of the list
-of Cover Texts in the Modified Version.  Only one passage of
-Front-Cover Text and one of Back-Cover Text may be added by (or
-through arrangements made by) any one entity.  If the Document already
-includes a cover text for the same cover, previously added by you or
-by arrangement made by the same entity you are acting on behalf of,
-you may not add another; but you may replace the old one, on explicit
-permission from the previous publisher that added the old one.
-
-The author(s) and publisher(s) of the Document do not by this License
-give permission to use their names for publicity for or to assert or
-imply endorsement of any Modified Version.
-
-
-5. COMBINING DOCUMENTS
-
-You may combine the Document with other documents released under this
-License, under the terms defined in section 4 above for modified
-versions, provided that you include in the combination all of the
-Invariant Sections of all of the original documents, unmodified, and
-list them all as Invariant Sections of your combined work in its
-license notice.
-
-The combined work need only contain one copy of this License, and
-multiple identical Invariant Sections may be replaced with a single
-copy.  If there are multiple Invariant Sections with the same name but
-different contents, make the title of each such section unique by
-adding at the end of it, in parentheses, the name of the original
-author or publisher of that section if known, or else a unique number.
-Make the same adjustment to the section titles in the list of
-Invariant Sections in the license notice of the combined work.
-
-In the combination, you must combine any sections entitled "History"
-in the various original documents, forming one section entitled
-"History"; likewise combine any sections entitled "Acknowledgements",
-and any sections entitled "Dedications".  You must delete all sections
-entitled "Endorsements."
-
-
-6. COLLECTIONS OF DOCUMENTS
-
-You may make a collection consisting of the Document and other documents
-released under this License, and replace the individual copies of this
-License in the various documents with a single copy that is included in
-the collection, provided that you follow the rules of this License for
-verbatim copying of each of the documents in all other respects.
-
-You may extract a single document from such a collection, and distribute
-it individually under this License, provided you insert a copy of this
-License into the extracted document, and follow this License in all
-other respects regarding verbatim copying of that document.
-
-
-7. AGGREGATION WITH INDEPENDENT WORKS
-
-A compilation of the Document or its derivatives with other separate
-and independent documents or works, in or on a volume of a storage or
-distribution medium, does not as a whole count as a Modified Version
-of the Document, provided no compilation copyright is claimed for the
-compilation.  Such a compilation is called an "aggregate", and this
-License does not apply to the other self-contained works thus compiled
-with the Document, on account of their being thus compiled, if they
-are not themselves derivative works of the Document.
-
-If the Cover Text requirement of section 3 is applicable to these
-copies of the Document, then if the Document is less than one quarter
-of the entire aggregate, the Document's Cover Texts may be placed on
-covers that surround only the Document within the aggregate.
-Otherwise they must appear on covers around the whole aggregate.
-
-
-8. TRANSLATION
-
-Translation is considered a kind of modification, so you may
-distribute translations of the Document under the terms of section 4.
-Replacing Invariant Sections with translations requires special
-permission from their copyright holders, but you may include
-translations of some or all Invariant Sections in addition to the
-original versions of these Invariant Sections.  You may include a
-translation of this License provided that you also include the
-original English version of this License.  In case of a disagreement
-between the translation and the original English version of this
-License, the original English version will prevail.
-
-
-9. TERMINATION
-
-You may not copy, modify, sublicense, or distribute the Document except
-as expressly provided for under this License.  Any other attempt to
-copy, modify, sublicense or distribute the Document is void, and will
-automatically terminate your rights under this License.  However,
-parties who have received copies, or rights, from you under this
-License will not have their licenses terminated so long as such
-parties remain in full compliance.
-
-
-10. FUTURE REVISIONS OF THIS LICENSE
-
-The Free Software Foundation may publish new, revised versions
-of the GNU Free Documentation License from time to time.  Such new
-versions will be similar in spirit to the present version, but may
-differ in detail to address new problems or concerns.  See
-http://www.gnu.org/copyleft/.
-
-Each version of the License is given a distinguishing version number.
-If the Document specifies that a particular numbered version of this
-License "or any later version" applies to it, you have the option of
-following the terms and conditions either of that specified version or
-of any later version that has been published (not as a draft) by the
-Free Software Foundation.  If the Document does not specify a version
-number of this License, you may choose any version ever published (not
-as a draft) by the Free Software Foundation.
-
-
-ADDENDUM: How to use this License for your documents
-
-To use this License in a document you have written, include a copy of
-the License in the document and put the following copyright and
-license notices just after the title page:
-
-      Copyright (c)  YEAR  YOUR NAME.
-      Permission is granted to copy, distribute and/or modify this document
-      under the terms of the GNU Free Documentation License, Version 1.1
-      or any later version published by the Free Software Foundation;
-      with the Invariant Sections being LIST THEIR TITLES, with the
-      Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.
-      A copy of the license is included in the section entitled "GNU
-      Free Documentation License".
-
-If you have no Invariant Sections, write "with no Invariant Sections"
-instead of saying which ones are invariant.  If you have no
-Front-Cover Texts, write "no Front-Cover Texts" instead of
-"Front-Cover Texts being LIST"; likewise for Back-Cover Texts.
-
-If your document contains nontrivial examples of program code, we
-recommend releasing these examples in parallel under your choice of
-free software license, such as the GNU General Public License,
-to permit their use in free software.
+Permission is granted to copy, distribute and/or modify this document
+under the terms of the GNU Free Documentation License, Version 1.1 or
+any later version published by the Free Software Foundation; with no
+Invariant Sections, with no Front-Cover Texts, and with no Back-Cover
+Texts.  A copy of the license is included in the section entitled `GNU
+Free Documentation License''.
 ============================================================================

doc/user-guide/commands.xml

@@ -835 +835 @@
         <bitlbee-setting name="commands" type="boolean" scope="account">
                 <default>true</default>
+                <possible-values>true, false, strict</possible-values>
 
                 <description>
@@ -853 +854 @@
 
                         <para>
-                                Anything that doesn't look like a command will be treated as a tweet. Watch out for typos! :-)
+                                Anything that doesn't look like a command will be treated as a tweet. Watch out for typos, or to avoid this behaviour, you can set this setting to <emphasis>strict</emphasis>, which causes the <emphasis>post</emphasis> command to become mandatory for posting a tweet.
                         </para>
                 </description>
@@ -1023 +1024 @@
 
         </bitlbee-setting>
+
+        <bitlbee-setting name="stream" type="boolean" scope="account">
+                <default>true</default>
+
+                <description>
+                        <para>
+                                For Twitter accounts, this setting enables use of the Streaming API. This automatically gives you incoming DMs as well.
+                        </para>
+                        
+                        <para>
+                                For other Twitter-like services, this setting is not supported.
+                        </para>
+                </description>
+
+        </bitlbee-setting>
         
         <bitlbee-setting name="target_url_length" type="integer" scope="account">
@@ -1338 +1354 @@
 
         <bitlbee-setting name="show_ids" type="boolean" scope="account">
-                <default>false</default>
+                <default>true</default>
 
                 <description>

lib/Makefile

@@ -13 +13 @@
 
 # [SH] Program variables
-objects = arc.o base64.o $(EVENT_HANDLER) ftutil.o http_client.o ini.o md5.o misc.o oauth.o oauth2.o proxy.o sha1.o $(SSL_CLIENT) url.o xmltree.o
+objects = arc.o base64.o $(EVENT_HANDLER) ftutil.o http_client.o ini.o json.o json_util.o md5.o misc.o oauth.o oauth2.o proxy.o sha1.o $(SSL_CLIENT) url.o xmltree.o
 
 LFLAGS += -r

lib/http_client.c

@@ -193 +193 @@
 }
 
+static gboolean http_handle_headers( struct http_request *req );
+
 static gboolean http_incoming_data( gpointer data, int source, b_input_condition cond )
 {
         struct http_request *req = data;
-        int evil_server = 0;
-        char buffer[2048];
-        char *end1, *end2, *s;
+        char buffer[4096];
+        char *s;
         size_t content_length;
         int st;
@@ -218 +219 @@
                                    packets that abort connections! \o/ */
                                 
-                                goto got_reply;
+                                goto eof;
                         }
                 }
                 else if( st == 0 )
                 {
-                        goto got_reply;
+                        goto eof;
                 }
         }
@@ -239 +240 @@
                 else if( st == 0 )
                 {
-                        goto got_reply;
-                }
-        }
-        
-        if( st > 0 )
+                        goto eof;
+                }
+        }
+        
+        if( st > 0 && !req->sbuf )
         {
                 req->reply_headers = g_realloc( req->reply_headers, req->bytes_read + st + 1 );
                 memcpy( req->reply_headers + req->bytes_read, buffer, st );
                 req->bytes_read += st;
-        }
+                
+                st = 0;
+        }
+        
+        if( st >= 0 && ( req->flags & HTTPC_STREAMING ) )
+        {
+                if( !req->reply_body &&
+                    ( strstr( req->reply_headers, "\r\n\r\n" ) ||
+                      strstr( req->reply_headers, "\n\n" ) ) )
+                {
+                        size_t hlen;
+                        
+                        /* We've now received all headers, so process them once
+                           before we start feeding back data. */
+                        if( !http_handle_headers( req ) )
+                                return FALSE;
+                        
+                        hlen = req->reply_body - req->reply_headers;
+                        
+                        req->sblen = req->bytes_read - hlen;
+                        req->sbuf = g_memdup( req->reply_body, req->sblen + 1 );
+                        req->reply_headers = g_realloc( req->reply_headers, hlen + 1 );
+                        
+                        req->reply_body = req->sbuf;
+                }
+                
+                if( st > 0 )
+                {
+                        int pos = req->reply_body - req->sbuf;
+                        req->sbuf = g_realloc( req->sbuf, req->sblen + st + 1 );
+                        memcpy( req->sbuf + req->sblen, buffer, st );
+                        req->bytes_read += st;
+                        req->sblen += st;
+                        req->sbuf[req->sblen] = '\0';
+                        req->reply_body = req->sbuf + pos;
+                        req->body_size = req->sblen - pos;
+                }
+                
+                if( req->reply_body )
+                        req->func( req );
+        }
+        
+        if( ssl_pending( req->ssl ) )
+                return http_incoming_data( data, source, cond );
         
         /* There will be more! */
@@ -255 +299 @@
                                  http_incoming_data, req );
         
-        if( ssl_pending( req->ssl ) )
-                return http_incoming_data( data, source, cond );
-        else
-                return FALSE;
-
-got_reply:
+        return FALSE;
+
+eof:
+        req->flags |= HTTPC_EOF;
+        
         /* Maybe if the webserver is overloaded, or when there's bad SSL
            support... */
@@ -269 +312 @@
         }
         
+        if( !( req->flags & HTTPC_STREAMING ) )
+        {
+                /* Returns FALSE if we were redirected, in which case we should abort
+                   and not run any callback yet. */
+                if( !http_handle_headers( req ) )
+                        return FALSE;
+        }
+
+cleanup:
+        if( req->ssl )
+                ssl_disconnect( req->ssl );
+        else
+                closesocket( req->fd );
+        
+        if( ( s = get_rfc822_header( req->reply_headers, "Content-Length", 0 ) ) &&
+            sscanf( s, "%zd", &content_length ) == 1 )
+        {
+                if( content_length < req->body_size )
+                {
+                        req->status_code = -1;
+                        g_free( req->status_string );
+                        req->status_string = g_strdup( "Response truncated" );
+                }
+        }
+        g_free( s );
+        
+        if( getenv( "BITLBEE_DEBUG" ) && req )
+                printf( "Finishing HTTP request with status: %s\n",
+                        req->status_string ? req->status_string : "NULL" );
+        
+        req->func( req );
+        http_free( req );
+        return FALSE;
+}
+
+/* Splits headers and body. Checks result code, in case of 300s it'll handle
+   redirects. If this returns FALSE, don't call any callbacks! */
+static gboolean http_handle_headers( struct http_request *req )
+{
+        char *end1, *end2;
+        int evil_server = 0;
+        
         /* Zero termination is very convenient. */
-        req->reply_headers[req->bytes_read] = 0;
+        req->reply_headers[req->bytes_read] = '\0';
         
         /* Find the separation between headers and body, and keep stupid
@@ -289 +374 @@
         {
                 req->status_string = g_strdup( "Malformed HTTP reply" );
-                goto cleanup;
+                return TRUE;
         }
         
@@ -306 +391 @@
         if( ( end1 = strchr( req->reply_headers, ' ' ) ) != NULL )
         {
-                if( sscanf( end1 + 1, "%d", &req->status_code ) != 1 )
+                if( sscanf( end1 + 1, "%hd", &req->status_code ) != 1 )
                 {
                         req->status_string = g_strdup( "Can't parse status code" );
@@ -349 +434 @@
                 {
                         req->status_string = g_strdup( "Can't locate Location: header" );
-                        goto cleanup;
+                        return TRUE;
                 }
                 
@@ -369 +454 @@
                         req->status_string = g_strdup( "Can't handle recursive redirects" );
                         
-                        goto cleanup;
+                        return TRUE;
                 }
                 else
@@ -380 +465 @@
                         s = strstr( loc, "\r\n" );
                         if( s == NULL )
-                                goto cleanup;
+                                return TRUE;
                         
                         url = g_new0( url_t, 1 );
@@ -389 +474 @@
                                 req->status_string = g_strdup( "Malformed redirect URL" );
                                 g_free( url );
-                                goto cleanup;
+                                return TRUE;
                         }
                         
@@ -401 +486 @@
                                 req->status_string = g_strdup( "Error while rebuilding request string" );
                                 g_free( url );
-                                goto cleanup;
+                                return TRUE;
                         }
                         
@@ -467 +552 @@
                         req->status_string = g_strdup( "Connection problem during redirect" );
                         g_free( new_request );
-                        goto cleanup;
+                        return TRUE;
                 }
                 
@@ -480 +565 @@
         }
         
-        /* Assume that a closed connection means we're finished, this indeed
-           breaks with keep-alive connections and faulty connections. */
-        /* req->finished = 1; */
-
-cleanup:
+        return TRUE;
+}
+
+void http_flush_bytes( struct http_request *req, size_t len )
+{
+        if( len <= 0 || len > req->body_size || !( req->flags & HTTPC_STREAMING ) )
+                return;
+        
+        req->reply_body += len;
+        req->body_size -= len;
+        
+        if( req->reply_body - req->sbuf >= 512 )
+        {
+                printf( "Wasting %ld bytes, cleaning up stream buffer\n", req->reply_body - req->sbuf );
+                char *new = g_memdup( req->reply_body, req->body_size + 1 );
+                g_free( req->sbuf );
+                req->reply_body = req->sbuf = new;
+                req->sblen = req->body_size;
+        }
+}
+
+void http_close( struct http_request *req )
+{
+        if( !req )
+                return;
+        
+        if( req->inpa > 0 )
+                b_event_remove( req->inpa );
+        
         if( req->ssl )
                 ssl_disconnect( req->ssl );
@@ -490 +599 @@
                 closesocket( req->fd );
         
-        if( ( s = get_rfc822_header( req->reply_headers, "Content-Length", 0 ) ) &&
-            sscanf( s, "%zd", &content_length ) == 1 )
-        {
-                if( content_length < req->body_size )
-                {
-                        req->status_code = -1;
-                        g_free( req->status_string );
-                        req->status_string = g_strdup( "Response truncated" );
-                }
-        }
-        g_free( s );
-        
-        if( getenv( "BITLBEE_DEBUG" ) && req )
-                printf( "Finishing HTTP request with status: %s\n",
-                        req->status_string ? req->status_string : "NULL" );
-        
-        req->func( req );
         http_free( req );
-        return FALSE;
 }
 
@@ -516 +607 @@
         g_free( req->reply_headers );
         g_free( req->status_string );
+        g_free( req->sbuf );
         g_free( req );
 }
-

lib/http_client.h

@@ -26 +26 @@
 /* http_client allows you to talk (asynchronously, again) to HTTP servers.
    In the "background" it will send the whole query and wait for a complete
-   response to come back. Right now it's only used by the MSN Passport
-   authentication code, but it might be useful for other things too (for
-   example the AIM usericon patch uses this so icons can be stored on
-   webservers instead of the local filesystem).
+   response to come back. Initially written for MS Passport authentication,
+   but used for many other things now like OAuth and Twitter.
    
-   Didn't test this too much, but it seems to work well. Just don't look
-   at the code that handles HTTP 30x redirects. ;-) The function is
-   probably not very useful for downloading lots of data since it keeps 
-   everything in a memory buffer until the download is completed (and
-   can't pass any data or whatever before then). It's very useful for
-   doing quick requests without blocking the whole program, though. */
+   It's very useful for doing quick requests without blocking the whole
+   program. Unfortunately it doesn't support fancy stuff like HTTP keep-
+   alives. */
 
 #include <glib.h>
@@ -42 +37 @@
 
 struct http_request;
+
+typedef enum http_client_flags
+{
+        HTTPC_STREAMING = 1,
+        HTTPC_EOF = 2,
+        
+        /* Let's reserve 0x1000000+ for lib users. */
+} http_client_flags_t;
 
 /* Your callback function should look like this: */
@@ -53 +56 @@
         char *request;          /* The request to send to the server. */
         int request_length;     /* Its size. */
-        int status_code;        /* The numeric HTTP status code. (Or -1
+        short status_code;      /* The numeric HTTP status code. (Or -1
                                    if something really went wrong) */
         char *status_string;    /* The error text. */
@@ -59 +62 @@
         char *reply_body;
         int body_size;          /* The number of bytes in reply_body. */
-        /* int finished;           Set to non-0 if the request was completed
-                                   successfully. */
-        int redir_ttl;          /* You can set it to 0 if you don't want
+        short redir_ttl;        /* You can set it to 0 if you don't want
                                    http_client to follow them. */
+        
+        http_client_flags_t flags;
         
         http_input_function func;
@@ -68 +71 @@
         
         /* Please don't touch the things down here, you shouldn't need them. */
-        
         void *ssl;
         int fd;
@@ -75 +77 @@
         int bytes_written;
         int bytes_read;
+        
+        /* Used in streaming mode. Caller should read from reply_body. */
+        char *sbuf;
+        size_t sblen;
 };
 
@@ -83 +89 @@
 struct http_request *http_dorequest( char *host, int port, int ssl, char *request, http_input_function func, gpointer data );
 struct http_request *http_dorequest_url( char *url_string, http_input_function func, gpointer data );
+
+/* For streaming connections only; flushes len bytes at the start of the buffer. */
+void http_flush_bytes( struct http_request *req, size_t len );
+void http_close( struct http_request *req );

lib/oauth2.c

@@ -4 +4 @@
 *  Simple OAuth client (consumer) implementation.                           *
 *                                                                           *
-*  Copyright 2010-2011 Wilmer van der Gaast <wilmer@gaast.net>              *
+*  Copyright 2010-2012 Wilmer van der Gaast <wilmer@gaast.net>              *
 *                                                                           *
 *  This program is free software; you can redistribute it and/or modify     *
@@ -26 +26 @@
 #include "oauth2.h"
 #include "oauth.h"
+#include "json.h"
 #include "url.h"
 
@@ -44 +45 @@
 };
 
-static char *oauth2_json_dumb_get( const char *json, const char *key );
 static void oauth2_access_token_done( struct http_request *req );
 
@@ -115 +115 @@
         else if( content_type && strstr( content_type, "application/json" ) )
         {
-                atoken = oauth2_json_dumb_get( req->reply_body, "access_token" );
-                rtoken = oauth2_json_dumb_get( req->reply_body, "refresh_token" );
+                json_value *js = json_parse( req->reply_body );
+                if( js && js->type == json_object )
+                {
+                        int i;
+                        
+                        for( i = 0; i < js->u.object.length; i ++ )
+                        {
+                                if( js->u.object.values[i].value->type != json_string )
+                                        continue;
+                                if( strcmp( js->u.object.values[i].name, "access_token" ) == 0 )
+                                        atoken = g_strdup( js->u.object.values[i].value->u.string.ptr );
+                                if( strcmp( js->u.object.values[i].name, "refresh_token" ) == 0 )
+                                        rtoken = g_strdup( js->u.object.values[i].value->u.string.ptr );
+                        }
+                }
+                json_value_free( js );
         }
         else
@@ -137 +151 @@
         g_free( cb_data );
 }
-
-/* Super dumb. I absolutely refuse to use/add a complete json parser library
-   (adding a new dependency to BitlBee for the first time in.. 6 years?) just
-   to parse 100 bytes of data. So I have to do my own parsing because OAuth2
-   dropped support for XML. (GRRR!) This is very dumb and for example won't
-   work for integer values, nor will it strip/handle backslashes. */
-static char *oauth2_json_dumb_get( const char *json, const char *key )
-{
-        int is_key = 0; /* 1 == reading key, 0 == reading value */
-        int found_key = 0;
-                
-        while( json && *json )
-        {
-                /* Grab strings and see if they're what we're looking for. */
-                if( *json == '"' || *json == '\'' )
-                {
-                        char q = *json;
-                        const char *str_start;
-                        json ++;
-                        str_start = json;
-                        
-                        while( *json )
-                        {
-                                /* \' and \" are not string terminators. */
-                                if( *json == '\\' && json[1] == q )
-                                        json ++;
-                                /* But without a \ it is. */
-                                else if( *json == q )
-                                        break;
-                                json ++;
-                        }
-                        if( *json == '\0' )
-                                return NULL;
-                        
-                        if( is_key && strncmp( str_start, key, strlen( key ) ) == 0 )
-                        {
-                                found_key = 1;
-                        }
-                        else if( !is_key && found_key )
-                        {
-                                char *ret = g_memdup( str_start, json - str_start + 1 );
-                                ret[json-str_start] = '\0';
-                                return ret;
-                        }
-                        
-                }
-                else if( *json == '{' || *json == ',' )
-                {
-                        found_key = 0;
-                        is_key = 1;
-                }
-                else if( *json == ':' )
-                        is_key = 0;
-                
-                json ++;
-        }
-        
-        return NULL;
-}

lib/ssl_gnutls.c

@@ -38 +38 @@
 
 static gboolean initialized = FALSE;
-gnutls_certificate_credentials xcred;
+gnutls_certificate_credentials_t xcred;
 
 #include <limits.h>
@@ -60 +60 @@
         gboolean verify;
         
-        gnutls_session session;
+        gnutls_session_t session;
 };
 
@@ -134 +134 @@
         conn->data = data;
         conn->inpa = -1;
-        conn->hostname = hostname;
+        conn->hostname = g_strdup( hostname );
         
         /* For now, SSL verification is globally enabled by setting the cafile
@@ -171 +171 @@
         int verifyret = 0;
         gnutls_x509_crt_t cert;
-        const char *hostname;
-        
-        hostname = gnutls_session_get_ptr( session );
+        struct scd *conn;
+        
+        conn = gnutls_session_get_ptr( session );
 
         gnutlsret = gnutls_certificate_verify_peers2( session, &status );
@@ -211 +211 @@
                 return VERIFY_CERT_ERROR;
 
-        if( !gnutls_x509_crt_check_hostname( cert, hostname ) )
+        if( !gnutls_x509_crt_check_hostname( cert, conn->hostname ) )
         {
                 verifyret |= VERIFY_CERT_INVALID;
@@ -267 +267 @@
         
         gnutls_init( &conn->session, GNUTLS_CLIENT );
-        if( conn->verify )
-                gnutls_session_set_ptr( conn->session, (void *) conn->hostname );
+        gnutls_session_set_ptr( conn->session, (void *) conn );
 #if GNUTLS_VERSION_NUMBER < 0x020c00
         gnutls_transport_set_lowat( conn->session, 0 );
@@ -276 +275 @@
         
         sock_make_nonblocking( conn->fd );
-        gnutls_transport_set_ptr( conn->session, (gnutls_transport_ptr) GNUTLS_STUPID_CAST conn->fd );
+        gnutls_transport_set_ptr( conn->session, (gnutls_transport_ptr_t) GNUTLS_STUPID_CAST conn->fd );
         
         return ssl_handshake( data, source, cond );
@@ -402 +401 @@
         if( conn->session )
                 gnutls_deinit( conn->session );
+        g_free( conn->hostname );
         g_free( conn );
 }

nick.c

@@ -134 +134 @@
                                 chop = fmt[1];
                                 if( chop == '\0' )
+                                {
+                                        g_string_free( ret, TRUE );
                                         return NULL;
+                                }
                                 fmt += 2;
                         }
@@ -187 +190 @@
                         else
                         {
+                                g_string_free( ret, TRUE );
                                 return NULL;
                         }

protocols/msn/msn.c

@@ -53 +53 @@
         
         ic->proto_data = md;
+        ic->flags |= OPT_PONGS | OPT_PONGED;
         
         if( strchr( acc->user, '@' ) == NULL )

protocols/msn/ns.c

@@ -576 +576 @@
                 if( num_parts >= 7 )
                         handler->msglen = atoi( cmd[6] );
+        }
+        else if( strcmp( cmd[0], "QNG" ) == 0 )
+        {
+                ic->flags |= OPT_PONGED;
         }
         else if( isdigit( cmd[0][0] ) )

protocols/nogaim.c

@@ -261 +261 @@
         struct im_connection *ic = d;
         
+        if( ( ic->flags & OPT_PONGS ) && !( ic->flags & OPT_PONGED ) )
+        {
+                /* This protocol is expected to ack keepalives and hasn't
+                   since the last time we were here. */
+                imcb_error( ic, "Connection timeout" );
+                imc_logout( ic, TRUE );
+                return FALSE;
+        }
+        ic->flags &= ~OPT_PONGED;
+        
         if( ic->acc->prpl->keepalive )
                 ic->acc->prpl->keepalive( ic );
         
         return TRUE;
+}
+
+void start_keepalives( struct im_connection *ic, int interval )
+{
+        b_event_remove( ic->keepalive );
+        ic->keepalive = b_timeout_add( interval, send_keepalive, ic );
+        
+        /* Connecting successfully counts as a first successful pong. */
+        if( ic->flags & OPT_PONGS )
+                ic->flags |= OPT_PONGED;
 }
 
@@ -277 +297 @@
         imcb_log( ic, "Logged in" );
         
-        b_event_remove( ic->keepalive );
-        ic->keepalive = b_timeout_add( 60000, send_keepalive, ic );
         ic->flags |= OPT_LOGGED_IN;
+        start_keepalives( ic, 60000 );
         
         /* Necessary to send initial presence status, even if we're not away. */

protocols/nogaim.h

@@ -68 +68 @@
 #define OPT_THINKING    0x00000200 /* about these values... Stupid me! */
 #define OPT_NOOTR       0x00001000 /* protocol not suitable for OTR */
+#define OPT_PONGS       0x00010000 /* Service sends us keep-alives */
+#define OPT_PONGED      0x00020000 /* Received a keep-alive during last interval */
 
 /* ok. now the fun begins. first we create a connection structure */

protocols/twitter/twitter.c

@@ -4 +4 @@
 *  Simple module to facilitate twitter functionality.                       *
 *                                                                           *
-*  Copyright 2009 Geert Mulders <g.c.w.m.mulders@gmail.com>                 *
+*  Copyright 2009-2010 Geert Mulders <g.c.w.m.mulders@gmail.com>            *
+*  Copyright 2010-2012 Wilmer van der Gaast <wilmer@gaast.net>              *
 *                                                                           *
 *  This library is free software; you can redistribute it and/or            *
@@ -29 +30 @@
 #include "url.h"
 
-#define twitter_msg( ic, fmt... ) \
-        do {                                            \
-                struct twitter_data *td = ic->proto_data;   \
-                if( td->timeline_gc )                       \
-                        imcb_chat_log( td->timeline_gc, fmt );  \
-                else                                        \
-                        imcb_log( ic, fmt );                    \
-        } while( 0 );
-
 GSList *twitter_connections = NULL;
-
 /**
  * Main loop function
@@ -62 +53 @@
         struct twitter_data *td = ic->proto_data;
 
+        /* Create the room now that we "logged in". */
+        if (td->flags & TWITTER_MODE_CHAT)
+                twitter_groupchat_init(ic);
+
         imcb_log(ic, "Getting initial statuses");
 
-        // Run this once. After this queue the main loop function.
+        // Run this once. After this queue the main loop function (or open the
+        // stream if available).
         twitter_main_loop(ic, -1, 0);
-
-        // Queue the main_loop
-        // Save the return value, so we can remove the timeout on logout.
-        td->main_loop_id =
-            b_timeout_add(set_getint(&ic->acc->set, "fetch_interval") * 1000, twitter_main_loop, ic);
+        
+        if (set_getbool(&ic->acc->set, "stream")) {
+                /* That fetch was just to get backlog, the stream will give
+                   us the rest. \o/ */
+                twitter_open_stream(ic);
+                
+                /* Stream sends keepalives (empty lines) or actual data at
+                   least twice a minute. Disconnect if this stops. */
+                ic->flags |= OPT_PONGS;
+        } else {
+                /* Not using the streaming API, so keep polling the old-
+                   fashioned way. :-( */
+                td->main_loop_id =
+                    b_timeout_add(set_getint(&ic->acc->set, "fetch_interval") * 1000,
+                                  twitter_main_loop, ic);
+        }
+}
+
+struct groupchat *twitter_groupchat_init(struct im_connection *ic)
+{
+        char *name_hint;
+        struct groupchat *gc;
+        struct twitter_data *td = ic->proto_data;
+        GSList *l;
+
+        if (td->timeline_gc)
+                return td->timeline_gc;
+
+        td->timeline_gc = gc = imcb_chat_new(ic, "twitter/timeline");
+
+        name_hint = g_strdup_printf("%s_%s", td->prefix, ic->acc->user);
+        imcb_chat_name_hint(gc, name_hint);
+        g_free(name_hint);
+
+        for (l = ic->bee->users; l; l = l->next) {
+                bee_user_t *bu = l->data;
+                if (bu->ic == ic)
+                        imcb_chat_add_buddy(gc, bu->handle);
+        }
+        imcb_chat_add_buddy(gc, ic->acc->user);
+        
+        return gc;
 }
 
@@ -83 +116 @@
         if (set_getbool(&ic->acc->set, "oauth") && !td->oauth_info)
                 twitter_oauth_start(ic);
-        else if (g_strcasecmp(set_getstr(&ic->acc->set, "mode"), "one") != 0 &&
-                 !(td->flags & TWITTER_HAVE_FRIENDS)) {
+        else if (!(td->flags & TWITTER_MODE_ONE) &&
+                 !(td->flags & TWITTER_HAVE_FRIENDS)) {
                 imcb_log(ic, "Getting contact list");
                 twitter_get_friends_ids(ic, -1);
-                //twitter_get_statuses_friends(ic, -1);
         } else
                 twitter_main_loop_start(ic);
@@ -187 +219 @@
 }
 
-
-static char *set_eval_mode(set_t * set, char *value)
-{
-        if (g_strcasecmp(value, "one") == 0 ||
-            g_strcasecmp(value, "many") == 0 || g_strcasecmp(value, "chat") == 0)
-                return value;
-        else
-                return NULL;
-}
-
 int twitter_url_len_diff(gchar *msg, unsigned int target_len)
 {
@@ -233 +255 @@
                 return TRUE;
 
-        imcb_error(ic, "Maximum message length exceeded: %d > %d", len, max);
+        twitter_log(ic, "Maximum message length exceeded: %d > %d", len, max);
 
         return FALSE;
+}
+
+static char *set_eval_commands(set_t * set, char *value)
+{
+        if (g_strcasecmp(value, "strict") == 0 )
+                return value;
+        else
+                return set_eval_bool(set, value);
+}
+
+static char *set_eval_mode(set_t * set, char *value)
+{
+        if (g_strcasecmp(value, "one") == 0 ||
+            g_strcasecmp(value, "many") == 0 || g_strcasecmp(value, "chat") == 0)
+                return value;
+        else
+                return NULL;
 }
 
@@ -257 +296 @@
         s->flags |= ACC_SET_OFFLINE_ONLY;
 
-        s = set_add(&acc->set, "commands", "true", set_eval_bool, acc);
+        s = set_add(&acc->set, "commands", "true", set_eval_commands, acc);
 
         s = set_add(&acc->set, "fetch_interval", "60", set_eval_int, acc);
@@ -274 +313 @@
 
         s = set_add(&acc->set, "show_ids", "true", set_eval_bool, acc);
-        s->flags |= ACC_SET_OFFLINE_ONLY;
 
         s = set_add(&acc->set, "show_old_mentions", "20", set_eval_int, acc);
 
         s = set_add(&acc->set, "strip_newlines", "false", set_eval_bool, acc);
+        
+        if (strcmp(acc->prpl->name, "twitter") == 0) {
+                s = set_add(&acc->set, "stream", "true", set_eval_bool, acc);
+                s->flags |= ACC_SET_OFFLINE_ONLY;
+        }
 }
 
 /**
- * Login method. Since the twitter API works with seperate HTTP request we 
+ * Login method. Since the twitter API works with separate HTTP request we
  * only save the user and pass to the twitter_data object.
  */
@@ -298 +341 @@
                 imc_logout(ic, FALSE);
                 return;
+        }
+
+        if (!strstr(url.host, "twitter.com") &&
+            set_getbool(&ic->acc->set, "stream")) {
+                imcb_error(ic, "Warning: The streaming API is only supported by Twitter, "
+                               "and you seem to be connecting to a different service.");
         }
 
@@ -340 +389 @@
         imcb_buddy_status(ic, name, OPT_LOGGED_IN, NULL, NULL);
 
-        if (set_getbool(&acc->set, "show_ids"))
-                td->log = g_new0(struct twitter_log_data, TWITTER_LOG_LENGTH);
+        td->log = g_new0(struct twitter_log_data, TWITTER_LOG_LENGTH);
+        td->log_id = -1;
+        
+        s = set_getstr(&ic->acc->set, "mode");
+        if (g_strcasecmp(s, "one") == 0)
+                td->flags |= TWITTER_MODE_ONE;
+        else if (g_strcasecmp(s, "many") == 0)
+                td->flags |= TWITTER_MODE_MANY;
+        else
+                td->flags |= TWITTER_MODE_CHAT;
 
         twitter_login_finish(ic);
@@ -363 +420 @@
 
         if (td) {
+                http_close(td->stream);
                 oauth_info_free(td->oauth_info);
                 g_free(td->user);
@@ -495 +553 @@
  *  Returns 0 if the user provides garbage.
  */
-static guint64 twitter_message_id_from_command_arg(struct im_connection *ic, struct twitter_data *td, char *arg) {
+static guint64 twitter_message_id_from_command_arg(struct im_connection *ic, char *arg, bee_user_t **bu_) {
+        struct twitter_data *td = ic->proto_data;
         struct twitter_user_data *tud;
-        bee_user_t *bu;
+        bee_user_t *bu = NULL;
         guint64 id = 0;
-        if (g_str_has_prefix(arg, "#") &&
-                sscanf(arg + 1, "%" G_GUINT64_FORMAT, &id) == 1) {
-                if (id < TWITTER_LOG_LENGTH && td->log)
+        
+        if (bu_)
+                *bu_ = NULL;
+        if (!arg || !arg[0])
+                return 0;
+        
+        if (arg[0] != '#' && (bu = bee_user_by_handle(ic->bee, ic, arg))) {
+                if ((tud = bu->data))
+                        id = tud->last_id;
+        } else {
+                if (arg[0] == '#')
+                        arg++;
+                if (sscanf(arg, "%" G_GINT64_MODIFIER "x", &id) == 1 &&
+                    id < TWITTER_LOG_LENGTH) {
+                        bu = td->log[id].bu;
                         id = td->log[id].id;
-        } else if ((bu = bee_user_by_handle(ic->bee, ic, arg)) &&
-                (tud = bu->data) && tud->last_id)
-                id = tud->last_id;
-        else if (sscanf(arg, "%" G_GUINT64_FORMAT, &id) == 1){
-                if (id < TWITTER_LOG_LENGTH && td->log)
-                        id = td->log[id].id;
-        }
+                        /* Beware of dangling pointers! */
+                        if (!g_slist_find(ic->bee->users, bu))
+                                bu = NULL;
+                } else if (sscanf(arg, "%" G_GINT64_MODIFIER "d", &id) == 1) {
+                        /* Allow normal tweet IDs as well; not a very useful
+                           feature but it's always been there. Just ignore
+                           very low IDs to avoid accidents. */
+                        if (id < 1000000)
+                                id = 0;
+                }
+        }
+        if (bu_)
+                *bu_ = bu;
         return id;
 }
@@ -517 +594 @@
         struct twitter_data *td = ic->proto_data;
         char *cmds, **cmd, *new = NULL;
-        guint64 in_reply_to = 0;
+        guint64 in_reply_to = 0, id;
+        gboolean allow_post =
+                g_strcasecmp(set_getstr(&ic->acc->set, "commands"), "strict") != 0;
+        bee_user_t *bu = NULL;
 
         cmds = g_strdup(message);
@@ -523 +603 @@
 
         if (cmd[0] == NULL) {
-                g_free(cmds);
-                return;
-        } else if (!set_getbool(&ic->acc->set, "commands")) {
-                /* Not supporting commands. */
+                goto eof;
+        } else if (!set_getbool(&ic->acc->set, "commands") && allow_post) {
+                /* Not supporting commands if "commands" is set to true/strict. */
         } else if (g_strcasecmp(cmd[0], "undo") == 0) {
-                guint64 id;
-
                 if (cmd[1] == NULL)
                         twitter_status_destroy(ic, td->last_status_id);
-                else if (sscanf(cmd[1], "%" G_GUINT64_FORMAT, &id) == 1) {
-                        if (id < TWITTER_LOG_LENGTH && td->log)
-                                id = td->log[id].id;
-                        
+                else if ((id = twitter_message_id_from_command_arg(ic, cmd[1], NULL)))
                         twitter_status_destroy(ic, id);
-                } else
-                        twitter_msg(ic, "Could not undo last action");
-
-                g_free(cmds);
-                return;
+                else
+                        twitter_log(ic, "Could not undo last action");
+
+                goto eof;
         } else if (g_strcasecmp(cmd[0], "favourite") == 0 && cmd[1]) {
-                guint64 id;
-                if ((id = twitter_message_id_from_command_arg(ic, td, cmd[1]))) {
+                if ((id = twitter_message_id_from_command_arg(ic, cmd[1], NULL))) {
                         twitter_favourite_tweet(ic, id);
                 } else {
-                        twitter_msg(ic, "Please provide a message ID or username.");
+                        twitter_log(ic, "Please provide a message ID or username.");
                 }
-                g_free(cmds);
-                return;
+                goto eof;
         } else if (g_strcasecmp(cmd[0], "follow") == 0 && cmd[1]) {
                 twitter_add_buddy(ic, cmd[1], NULL);
-                g_free(cmds);
-                return;
+                goto eof;
         } else if (g_strcasecmp(cmd[0], "unfollow") == 0 && cmd[1]) {
                 twitter_remove_buddy(ic, cmd[1], NULL);
-                g_free(cmds);
-                return;
+                goto eof;
         } else if ((g_strcasecmp(cmd[0], "report") == 0 ||
                     g_strcasecmp(cmd[0], "spam") == 0) && cmd[1]) {
-                char * screen_name;
-                guint64 id;
-                screen_name = cmd[1];
+                char *screen_name;
+                
                 /* Report nominally works on users but look up the user who
                    posted the given ID if the user wants to do it that way */
-                if (g_str_has_prefix(cmd[1], "#") &&
-                    sscanf(cmd[1] + 1, "%" G_GUINT64_FORMAT, &id) == 1) {
-                        if (id < TWITTER_LOG_LENGTH && td->log) {
-                                if (g_slist_find(ic->bee->users, td->log[id].bu)) {
-                                        screen_name = td->log[id].bu->handle;
-                                }
-                        }
-                }
+                twitter_message_id_from_command_arg(ic, cmd[1], &bu);
+                if (bu)
+                        screen_name = bu->handle;
+                else
+                        screen_name = cmd[1];
+                
                 twitter_report_spam(ic, screen_name);
-                g_free(cmds);
-                return;
+                goto eof;
         } else if (g_strcasecmp(cmd[0], "rt") == 0 && cmd[1]) {
-                guint64 id = twitter_message_id_from_command_arg(ic, td, cmd[1]);
+                id = twitter_message_id_from_command_arg(ic, cmd[1], NULL);
 
                 td->last_status_id = 0;
@@ -584 +649 @@
                         twitter_status_retweet(ic, id);
                 else
-                        twitter_msg(ic, "User `%s' does not exist or didn't "
+                        twitter_log(ic, "User `%s' does not exist or didn't "
                                     "post any statuses recently", cmd[1]);
 
-                g_free(cmds);
-                return;
+                goto eof;
         } else if (g_strcasecmp(cmd[0], "reply") == 0 && cmd[1] && cmd[2]) {
-                struct twitter_user_data *tud;
-                bee_user_t *bu = NULL;
-                guint64 id = 0;
-
-                if (g_str_has_prefix(cmd[1], "#") &&
-                    sscanf(cmd[1] + 1, "%" G_GUINT64_FORMAT, &id) == 1 &&
-                    (id < TWITTER_LOG_LENGTH) && td->log) {
-                        bu = td->log[id].bu;
-                        if (g_slist_find(ic->bee->users, bu))
-                                id = td->log[id].id;
-                        else
-                                bu = NULL;
-                } else if ((bu = bee_user_by_handle(ic->bee, ic, cmd[1])) &&
-                    (tud = bu->data) && tud->last_id) {
-                        id = tud->last_id;
-                } else if (sscanf(cmd[1], "%" G_GUINT64_FORMAT, &id) == 1 &&
-                           (id < TWITTER_LOG_LENGTH) && td->log) {
-                        bu = td->log[id].bu;
-                        if (g_slist_find(ic->bee->users, bu))
-                                id = td->log[id].id;
-                        else
-                                bu = NULL;
-                }
-
+                id = twitter_message_id_from_command_arg(ic, cmd[1], &bu);
                 if (!id || !bu) {
-                        twitter_msg(ic, "User `%s' does not exist or didn't "
+                        twitter_log(ic, "User `%s' does not exist or didn't "
                                     "post any statuses recently", cmd[1]);
-                        g_free(cmds);
-                        return;
+                        goto eof;
                 }
                 message = new = g_strdup_printf("@%s %s", bu->handle, message + (cmd[2] - cmd[0]));
                 in_reply_to = id;
+                allow_post = TRUE;
         } else if (g_strcasecmp(cmd[0], "post") == 0) {
                 message += 5;
-        }
-
-        {
+                allow_post = TRUE;
+        }
+
+        if (allow_post) {
                 char *s;
-                bee_user_t *bu;
-
-                if (!twitter_length_check(ic, message)) {
-                        g_free(new);
-                        g_free(cmds);
-                        return;
-                }
+
+                if (!twitter_length_check(ic, message))
+                        goto eof;
 
                 s = cmd[0] + strlen(cmd[0]) - 1;
@@ -657 +695 @@
                 td->last_status_id = 0;
                 twitter_post_status(ic, message, in_reply_to);
-                g_free(new);
-        }
+        } else {
+                twitter_log(ic, "Unknown command: %s", cmd[0]);
+        }
+eof:
+        g_free(new);
         g_free(cmds);
 }
+
+void twitter_log(struct im_connection *ic, char *format, ... )
+{
+        struct twitter_data *td = ic->proto_data;
+        va_list params;
+        char *text;
+        
+        va_start(params, format);
+        text = g_strdup_vprintf(format, params);
+        va_end(params);
+        
+        if (td->timeline_gc)
+                imcb_chat_log(td->timeline_gc, "%s", text);
+        else
+                imcb_log(ic, "%s", text);
+        
+        g_free(text);
+}
+
 
 void twitter_initmodule()

protocols/twitter/twitter.h

@@ -4 +4 @@
 *  Simple module to facilitate twitter functionality.                       *
 *                                                                           *
-*  Copyright 2009 Geert Mulders <g.c.w.m.mulders@gmail.com>                 *
+*  Copyright 2009-2010 Geert Mulders <g.c.w.m.mulders@gmail.com>            *
+*  Copyright 2010-2012 Wilmer van der Gaast <wilmer@gaast.net>              *
 *                                                                           *
 *  This library is free software; you can redistribute it and/or            *
@@ -35 +36 @@
 typedef enum
 {
-        TWITTER_HAVE_FRIENDS = 1,
+        TWITTER_HAVE_FRIENDS   = 0x00001,
+        TWITTER_MODE_ONE       = 0x00002,
+        TWITTER_MODE_MANY      = 0x00004,
+        TWITTER_MODE_CHAT      = 0x00008,
         TWITTER_DOING_TIMELINE = 0x10000,
-        TWITTER_GOT_TIMELINE = 0x20000,
-        TWITTER_GOT_MENTIONS = 0x40000,
+        TWITTER_GOT_TIMELINE   = 0x20000,
+        TWITTER_GOT_MENTIONS   = 0x40000,
 } twitter_flags_t;
 
@@ -57 +61 @@
         guint64 last_status_id; /* For undo */
         gint main_loop_id;
+        struct http_request *stream;
         struct groupchat *timeline_gc;
         gint http_fails;
@@ -80 +85 @@
 };
 
-#define TWITTER_LOG_LENGTH 100
+#define TWITTER_LOG_LENGTH 256
 struct twitter_log_data
 {
@@ -99 +104 @@
 char *twitter_parse_error( struct http_request *req );
 
+void twitter_log(struct im_connection *ic, char *format, ... );
+struct groupchat *twitter_groupchat_init(struct im_connection *ic);
+
 #endif //_TWITTER_H

protocols/twitter/twitter_http.c

@@ -47 +47 @@
  * This is actually pretty generic function... Perhaps it should move to the lib/http_client.c
  */
-void *twitter_http(struct im_connection *ic, char *url_string, http_input_function func,
-                   gpointer data, int is_post, char **arguments, int arguments_len)
+struct http_request *twitter_http(struct im_connection *ic, char *url_string, http_input_function func,
+                                  gpointer data, int is_post, char **arguments, int arguments_len)
 {
         struct twitter_data *td = ic->proto_data;
@@ -55 +55 @@
         void *ret;
         char *url_arguments;
+        url_t *base_url = NULL;
 
         url_arguments = g_strdup("");
@@ -67 +68 @@
                 }
         }
+        
+        if (strstr(url_string, "://")) {
+                base_url = g_new0(url_t, 1);
+                if (!url_set(base_url, url_string)) {
+                        g_free(base_url);
+                        return NULL;
+                }
+        }
+        
         // Make the request.
         g_string_printf(request, "%s %s%s%s%s HTTP/1.0\r\n"
@@ -72 +82 @@
                         "User-Agent: BitlBee " BITLBEE_VERSION " " ARCH "/" CPU "\r\n",
                         is_post ? "POST" : "GET",
-                        td->url_path, url_string,
-                        is_post ? "" : "?", is_post ? "" : url_arguments, td->url_host);
+                        base_url ? base_url->file : td->url_path,
+                        base_url ? "" : url_string,
+                        is_post ? "" : "?", is_post ? "" : url_arguments,
+                        base_url ? base_url->host : td->url_host);
 
         // If a pass and user are given we append them to the request.
@@ -80 +92 @@
                 char *full_url;
 
-                full_url = g_strconcat(set_getstr(&ic->acc->set, "base_url"), url_string, NULL);
+                if (base_url)
+                        full_url = g_strdup(url_string);
+                else
+                        full_url = g_strconcat(set_getstr(&ic->acc->set, "base_url"), url_string, NULL);
                 full_header = oauth_http_header(td->oauth_info, is_post ? "POST" : "GET",
                                                 full_url, url_arguments);
@@ -109 +124 @@
         }
 
-        ret = http_dorequest(td->url_host, td->url_port, td->url_ssl, request->str, func, data);
+        if (base_url)
+                ret = http_dorequest(base_url->host, base_url->port, base_url->proto == PROTO_HTTPS, request->str, func, data);
+        else
+                ret = http_dorequest(td->url_host, td->url_port, td->url_ssl, request->str, func, data);
 
         g_free(url_arguments);
         g_string_free(request, TRUE);
+        g_free(base_url);
+        return ret;
+}
+
+struct http_request *twitter_http_f(struct im_connection *ic, char *url_string, http_input_function func,
+                                    gpointer data, int is_post, char **arguments, int arguments_len, twitter_http_flags_t flags)
+{
+        struct http_request *ret = twitter_http(ic, url_string, func, data, is_post, arguments, arguments_len);
+        if (ret)
+                ret->flags |= flags;
         return ret;
 }

protocols/twitter/twitter_http.h

@@ -28 +28 @@
 #include "http_client.h"
 
+typedef enum {
+        /* With this set, twitter_http_post() will post a generic confirmation
+           message to the user. */
+        TWITTER_HTTP_USER_ACK = 0x1000000,
+} twitter_http_flags_t;
+
 struct oauth_info;
 
-void *twitter_http(struct im_connection *ic, char *url_string, http_input_function func,
-                   gpointer data, int is_post, char** arguments, int arguments_len);
+struct http_request *twitter_http(struct im_connection *ic, char *url_string, http_input_function func,
+                                  gpointer data, int is_post, char** arguments, int arguments_len);
+struct http_request *twitter_http_f(struct im_connection *ic, char *url_string, http_input_function func,
+                                    gpointer data, int is_post, char** arguments, int arguments_len, twitter_http_flags_t flags);
 
 #endif //_TWITTER_HTTP_H

protocols/twitter/twitter_lib.c

@@ -35 +35 @@
 #include "misc.h"
 #include "base64.h"
-#include "xmltree.h"
 #include "twitter_lib.h"
+#include "json_util.h"
 #include <ctype.h>
 #include <errno.h>
@@ -67 +67 @@
         char *text;
         struct twitter_xml_user *user;
-        guint64 id, reply_to;
+        guint64 id, rt_id; /* Usually equal, with RTs id == *original* id */
+        guint64 reply_to;
 };
-
-static void twitter_groupchat_init(struct im_connection *ic);
 
 /**
@@ -148 +147 @@
         // Check if the buddy is already in the buddy list.
         if (!bee_user_by_handle(ic->bee, ic, name)) {
-                char *mode = set_getstr(&ic->acc->set, "mode");
-
                 // The buddy is not in the list, add the buddy and set the status to logged in.
                 imcb_add_buddy(ic, name, NULL);
                 imcb_rename_buddy(ic, name, fullname);
-                if (g_strcasecmp(mode, "chat") == 0) {
+                if (td->flags & TWITTER_MODE_CHAT) {
                         /* Necessary so that nicks always get translated to the
                            exact Twitter username. */
                         imcb_buddy_nick_hint(ic, name, name);
-                        imcb_chat_add_buddy(td->timeline_gc, name);
-                } else if (g_strcasecmp(mode, "many") == 0)
+                        if (td->timeline_gc)
+                                imcb_chat_add_buddy(td->timeline_gc, name);
+                } else if (td->flags & TWITTER_MODE_MANY)
                         imcb_buddy_status(ic, name, OPT_LOGGED_IN, NULL, NULL);
         }
@@ -168 +166 @@
 {
         static char *ret = NULL;
-        struct xt_node *root, *node, *err;
+        json_value *root, *err;
 
         g_free(ret);
@@ -174 +172 @@
 
         if (req->body_size > 0) {
-                root = xt_from_string(req->reply_body, req->body_size);
-                
-                for (node = root; node; node = node->next)
-                        if ((err = xt_find_node(node->children, "error")) && err->text_len > 0) {
-                                ret = g_strdup_printf("%s (%s)", req->status_string, err->text);
-                                break;
-                        }
-
-                xt_free_node(root);
+                root = json_parse(req->reply_body);
+                err = json_o_get(root, "errors");
+                if (err && err->type == json_array && (err = err->u.array.values[0]) &&
+                    err->type == json_object) {
+                        const char *msg = json_o_str(err, "message");
+                        if (msg)
+                                ret = g_strdup_printf("%s (%s)", req->status_string, msg);
+                }
+                json_value_free(root);
         }
 
@@ -188 +186 @@
 }
 
-static struct xt_node *twitter_parse_response(struct im_connection *ic, struct http_request *req)
+/* WATCH OUT: This function might or might not destroy your connection.
+   Sub-optimal indeed, but just be careful when this returns NULL! */
+static json_value *twitter_parse_response(struct im_connection *ic, struct http_request *req)
 {
         gboolean logging_in = !(ic->flags & OPT_LOGGED_IN);
         gboolean periodic;
         struct twitter_data *td = ic->proto_data;
-        struct xt_node *ret;
+        json_value *ret;
         char path[64] = "", *s;
         
@@ -211 +211 @@
                    throwing 401s so I'll keep treating this one as fatal
                    only during login. */
-                imcb_error(ic, "Authentication failure");
+                imcb_error(ic, "Authentication failure (%s)",
+                               twitter_parse_error(req));
                 imc_logout(ic, FALSE);
                 return NULL;
@@ -217 +218 @@
                 // It didn't go well, output the error and return.
                 if (!periodic || logging_in || ++td->http_fails >= 5)
-                        imcb_error(ic, "Could not retrieve %s: %s",
-                                   path, twitter_parse_error(req));
+                        twitter_log(ic, "Error: Could not retrieve %s: %s",
+                                    path, twitter_parse_error(req));
                 
                 if (logging_in)
@@ -227 +228 @@
         }
 
-        if ((ret = xt_from_string(req->reply_body, req->body_size)) == NULL) {
+        if ((ret = json_parse(req->reply_body)) == NULL) {
                 imcb_error(ic, "Could not retrieve %s: %s",
                            path, "XML parse error");
@@ -251 +252 @@
 
 /**
- * Function to help fill a list.
- */
-static xt_status twitter_xt_next_cursor(struct xt_node *node, struct twitter_xml_list *txl)
-{
-        char *end = NULL;
-
-        if (node->text)
-                txl->next_cursor = g_ascii_strtoll(node->text, &end, 10);
-        if (end == NULL)
-                txl->next_cursor = -1;
-
-        return XT_HANDLED;
-}
-
-/**
  * Fill a list of ids.
  */
-static xt_status twitter_xt_get_friends_id_list(struct xt_node *node, struct twitter_xml_list *txl)
-{
-        struct xt_node *child;
+static gboolean twitter_xt_get_friends_id_list(json_value *node, struct twitter_xml_list *txl)
+{
+        json_value *c;
+        int i;
 
         // Set the list type.
         txl->type = TXL_ID;
 
-        // The root <statuses> node should hold the list of statuses <status>
-        // Walk over the nodes children.
-        for (child = node->children; child; child = child->next) {
-                if (g_strcasecmp("ids", child->name) == 0) {
-                        struct xt_node *idc;
-                        for (idc = child->children; idc; idc = idc->next)
-                                if (g_strcasecmp(idc->name, "id") == 0)
-                                        txl->list = g_slist_prepend(txl->list,
-                                                g_memdup(idc->text, idc->text_len + 1));
-                } else if (g_strcasecmp("next_cursor", child->name) == 0) {
-                        twitter_xt_next_cursor(child, txl);
-                }
-        }
-
-        return XT_HANDLED;
+        c = json_o_get(node, "ids");
+        if (!c || c->type != json_array)
+                return FALSE;
+
+        for (i = 0; i < c->u.array.length; i ++) {
+                if (c->u.array.values[i]->type != json_integer)
+                        continue;
+                
+                txl->list = g_slist_prepend(txl->list,
+                        g_strdup_printf("%lld", c->u.array.values[i]->u.integer));
+        }
+        
+        c = json_o_get(node, "next_cursor");
+        if (c && c->type == json_integer)
+                txl->next_cursor = c->u.integer;
+        else
+                txl->next_cursor = -1;
+        
+        return TRUE;
 }
 
@@ -300 +291 @@
 {
         struct im_connection *ic;
-        struct xt_node *parsed;
+        json_value *parsed;
         struct twitter_xml_list *txl;
         struct twitter_data *td;
@@ -312 +303 @@
         td = ic->proto_data;
 
-        /* Create the room now that we "logged in". */
-        if (!td->timeline_gc && g_strcasecmp(set_getstr(&ic->acc->set, "mode"), "chat") == 0)
-                twitter_groupchat_init(ic);
-
         txl = g_new0(struct twitter_xml_list, 1);
         txl->list = td->follow_ids;
@@ -322 +309 @@
         if (!(parsed = twitter_parse_response(ic, req)))
                 return;
+        
         twitter_xt_get_friends_id_list(parsed, txl);
-        xt_free_node(parsed);
+        json_value_free(parsed);
 
         td->follow_ids = txl->list;
@@ -338 +326 @@
 }
 
-static xt_status twitter_xt_get_users(struct xt_node *node, struct twitter_xml_list *txl);
+static gboolean twitter_xt_get_users(json_value *node, struct twitter_xml_list *txl);
 static void twitter_http_get_users_lookup(struct http_request *req);
 
@@ -379 +367 @@
 {
         struct im_connection *ic = req->data;
-        struct xt_node *parsed;
+        json_value *parsed;
         struct twitter_xml_list *txl;
         GSList *l = NULL;
@@ -395 +383 @@
                 return;
         twitter_xt_get_users(parsed, txl);
-        xt_free_node(parsed);
+        json_value_free(parsed);
 
         // Add the users as buddies.
@@ -409 +397 @@
 }
 
-/**
- * Function to fill a twitter_xml_user struct.
- * It sets:
- *  - the name and
- *  - the screen_name.
- */
-static xt_status twitter_xt_get_user(struct xt_node *node, struct twitter_xml_user *txu)
-{
-        struct xt_node *child;
-
-        // Walk over the nodes children.
-        for (child = node->children; child; child = child->next) {
-                if (g_strcasecmp("name", child->name) == 0) {
-                        txu->name = g_memdup(child->text, child->text_len + 1);
-                } else if (g_strcasecmp("screen_name", child->name) == 0) {
-                        txu->screen_name = g_memdup(child->text, child->text_len + 1);
-                }
-        }
-        return XT_HANDLED;
+struct twitter_xml_user *twitter_xt_get_user(const json_value *node)
+{
+        struct twitter_xml_user *txu;
+        
+        txu = g_new0(struct twitter_xml_user, 1);
+        txu->name = g_strdup(json_o_str(node, "name"));
+        txu->screen_name = g_strdup(json_o_str(node, "screen_name"));
+        
+        return txu;
 }
 
@@ -435 +413 @@
  *  - all <user>s from the <users> element.
  */
-static xt_status twitter_xt_get_users(struct xt_node *node, struct twitter_xml_list *txl)
+static gboolean twitter_xt_get_users(json_value *node, struct twitter_xml_list *txl)
 {
         struct twitter_xml_user *txu;
-        struct xt_node *child;
+        int i;
 
         // Set the type of the list.
         txl->type = TXL_USER;
 
+        if (!node || node->type != json_array)
+                return FALSE;
+
         // The root <users> node should hold the list of users <user>
         // Walk over the nodes children.
-        for (child = node->children; child; child = child->next) {
-                if (g_strcasecmp("user", child->name) == 0) {
-                        txu = g_new0(struct twitter_xml_user, 1);
-                        twitter_xt_get_user(child, txu);
-                        // Put the item in the front of the list.
+        for (i = 0; i < node->u.array.length; i ++) {
+                txu = twitter_xt_get_user(node->u.array.values[i]);
+                if (txu)
                         txl->list = g_slist_prepend(txl->list, txu);
-                }
-        }
-
-        return XT_HANDLED;
+        }
+
+        return TRUE;
 }
 
@@ -462 +440 @@
 #define TWITTER_TIME_FORMAT "%a %b %d %H:%M:%S +0000 %Y"
 #endif
+
+static char* expand_entities(char* text, const json_value *entities);
 
 /**
@@ -471 +451 @@
  *  - the user in a twitter_xml_user struct.
  */
-static xt_status twitter_xt_get_status(struct xt_node *node, struct twitter_xml_status *txs)
-{
-        struct xt_node *child, *rt = NULL;
-
-        // Walk over the nodes children.
-        for (child = node->children; child; child = child->next) {
-                if (g_strcasecmp("text", child->name) == 0) {
-                        txs->text = g_memdup(child->text, child->text_len + 1);
-                } else if (g_strcasecmp("retweeted_status", child->name) == 0) {
-                        rt = child;
-                } else if (g_strcasecmp("created_at", child->name) == 0) {
+static struct twitter_xml_status *twitter_xt_get_status(const json_value *node)
+{
+        struct twitter_xml_status *txs;
+        const json_value *rt = NULL, *entities = NULL;
+        
+        if (node->type != json_object)
+                return FALSE;
+        txs = g_new0(struct twitter_xml_status, 1);
+
+        JSON_O_FOREACH (node, k, v) {
+                if (strcmp("text", k) == 0 && v->type == json_string) {
+                        txs->text = g_memdup(v->u.string.ptr, v->u.string.length + 1);
+                        strip_html(txs->text);
+                } else if (strcmp("retweeted_status", k) == 0 && v->type == json_object) {
+                        rt = v;
+                } else if (strcmp("created_at", k) == 0 && v->type == json_string) {
                         struct tm parsed;
 
@@ -487 +472 @@
                            this field. :-( Also assumes the timezone used
                            is UTC since C time handling functions suck. */
-                        if (strptime(child->text, TWITTER_TIME_FORMAT, &parsed) != NULL)
+                        if (strptime(v->u.string.ptr, TWITTER_TIME_FORMAT, &parsed) != NULL)
                                 txs->created_at = mktime_utc(&parsed);
-                } else if (g_strcasecmp("user", child->name) == 0) {
-                        txs->user = g_new0(struct twitter_xml_user, 1);
-                        twitter_xt_get_user(child, txs->user);
-                } else if (g_strcasecmp("id", child->name) == 0) {
-                        txs->id = g_ascii_strtoull(child->text, NULL, 10);
-                } else if (g_strcasecmp("in_reply_to_status_id", child->name) == 0) {
-                        txs->reply_to = g_ascii_strtoull(child->text, NULL, 10);
+                } else if (strcmp("user", k) == 0 && v->type == json_object) {
+                        txs->user = twitter_xt_get_user(v);
+                } else if (strcmp("id", k) == 0 && v->type == json_integer) {
+                        txs->rt_id = txs->id = v->u.integer;
+                } else if (strcmp("in_reply_to_status_id", k) == 0 && v->type == json_integer) {
+                        txs->reply_to = v->u.integer;
+                } else if (strcmp("entities", k) == 0 && v->type == json_object) {
+                        entities = v;
                 }
         }
@@ -502 +488 @@
            wasn't truncated because it may be lying. */
         if (rt) {
-                struct twitter_xml_status *rtxs = g_new0(struct twitter_xml_status, 1);
-                if (twitter_xt_get_status(rt, rtxs) != XT_HANDLED) {
+                struct twitter_xml_status *rtxs = twitter_xt_get_status(rt);
+                if (rtxs) {
+                        g_free(txs->text);
+                        txs->text = g_strdup_printf("RT @%s: %s", rtxs->user->screen_name, rtxs->text);
+                        txs->id = rtxs->id;
                         txs_free(rtxs);
-                        return XT_HANDLED;
-                }
-
-                g_free(txs->text);
-                txs->text = g_strdup_printf("RT @%s: %s", rtxs->user->screen_name, rtxs->text);
-                txs_free(rtxs);
-        } else {
-                struct xt_node *urls, *url;
+                }
+        } else if (entities) {
+                txs->text = expand_entities(txs->text, entities);
+        }
+
+        if (txs->text && txs->user && txs->id)
+                return txs;
+        
+        txs_free(txs);
+        return NULL;
+}
+
+/**
+ * Function to fill a twitter_xml_status struct (DM variant).
+ */
+static struct twitter_xml_status *twitter_xt_get_dm(const json_value *node)
+{
+        struct twitter_xml_status *txs;
+        const json_value *entities = NULL;
+        
+        if (node->type != json_object)
+                return FALSE;
+        txs = g_new0(struct twitter_xml_status, 1);
+
+        JSON_O_FOREACH (node, k, v) {
+                if (strcmp("text", k) == 0 && v->type == json_string) {
+                        txs->text = g_memdup(v->u.string.ptr, v->u.string.length + 1);
+                        strip_html(txs->text);
+                } else if (strcmp("created_at", k) == 0 && v->type == json_string) {
+                        struct tm parsed;
+
+                        /* Very sensitive to changes to the formatting of
+                           this field. :-( Also assumes the timezone used
+                           is UTC since C time handling functions suck. */
+                        if (strptime(v->u.string.ptr, TWITTER_TIME_FORMAT, &parsed) != NULL)
+                                txs->created_at = mktime_utc(&parsed);
+                } else if (strcmp("sender", k) == 0 && v->type == json_object) {
+                        txs->user = twitter_xt_get_user(v);
+                } else if (strcmp("id", k) == 0 && v->type == json_integer) {
+                        txs->id = v->u.integer;
+                }
+        }
+
+        if (entities) {
+                txs->text = expand_entities(txs->text, entities);
+        }
+
+        if (txs->text && txs->user && txs->id)
+                return txs;
+        
+        txs_free(txs);
+        return NULL;
+}
+
+static char* expand_entities(char* text, const json_value *entities) {
+        JSON_O_FOREACH (entities, k, v) {
+                int i;
                 
-                urls = xt_find_path(node, "entities");
-                if (urls != NULL)
-                        urls = urls->children;
-                for (; urls; urls = urls->next) {
-                        if (strcmp(urls->name, "urls") != 0 && strcmp(urls->name, "media") != 0)
+                if (v->type != json_array)
+                        continue;
+                if (strcmp(k, "urls") != 0 && strcmp(k, "media") != 0)
+                        continue;
+                
+                for (i = 0; i < v->u.array.length; i ++) {
+                        if (v->u.array.values[i]->type != json_object)
                                 continue;
                         
-                        for (url = urls ? urls->children : NULL; url; url = url->next) {
-                                /* "short" is a reserved word. :-P */
-                                struct xt_node *kort = xt_find_node(url->children, "url");
-                                struct xt_node *disp = xt_find_node(url->children, "display_url");
-                                char *pos, *new;
-                                
-                                if (!kort || !kort->text || !disp || !disp->text ||
-                                    !(pos = strstr(txs->text, kort->text)))
-                                        continue;
-                                
-                                *pos = '\0';
-                                new = g_strdup_printf("%s%s &lt;%s&gt;%s", txs->text, kort->text,
-                                                      disp->text, pos + strlen(kort->text));
-                                
-                                g_free(txs->text);
-                                txs->text = new;
-                        }
-                }
-        }
-
-        return XT_HANDLED;
+                        const char *kort = json_o_str(v->u.array.values[i], "url");
+                        const char *disp = json_o_str(v->u.array.values[i], "display_url");
+                        char *pos, *new;
+                        
+                        if (!kort || !disp || !(pos = strstr(text, kort)))
+                                continue;
+                        
+                        *pos = '\0';
+                        new = g_strdup_printf("%s%s <%s>%s", text, kort,
+                                              disp, pos + strlen(kort));
+                        
+                        g_free(text);
+                        text = new;
+                }
+        }
+        
+        return text;
 }
 
@@ -550 +586 @@
  *  - the next_cursor.
  */
-static xt_status twitter_xt_get_status_list(struct im_connection *ic, struct xt_node *node,
-                                            struct twitter_xml_list *txl)
+static gboolean twitter_xt_get_status_list(struct im_connection *ic, const json_value *node,
+                                           struct twitter_xml_list *txl)
 {
         struct twitter_xml_status *txs;
-        struct xt_node *child;
-        bee_user_t *bu;
+        int i;
 
         // Set the type of the list.
         txl->type = TXL_STATUS;
+        
+        if (node->type != json_array)
+                return FALSE;
 
         // The root <statuses> node should hold the list of statuses <status>
         // Walk over the nodes children.
-        for (child = node->children; child; child = child->next) {
-                if (g_strcasecmp("status", child->name) == 0) {
-                        txs = g_new0(struct twitter_xml_status, 1);
-                        twitter_xt_get_status(child, txs);
-                        // Put the item in the front of the list.
-                        txl->list = g_slist_prepend(txl->list, txs);
-
-                        if (txs->user && txs->user->screen_name &&
-                            (bu = bee_user_by_handle(ic->bee, ic, txs->user->screen_name))) {
-                                struct twitter_user_data *tud = bu->data;
-
-                                if (txs->id > tud->last_id) {
-                                        tud->last_id = txs->id;
-                                        tud->last_time = txs->created_at;
-                                }
-                        }
-                } else if (g_strcasecmp("next_cursor", child->name) == 0) {
-                        twitter_xt_next_cursor(child, txl);
-                }
-        }
-
-        return XT_HANDLED;
-}
-
+        for (i = 0; i < node->u.array.length; i ++) {
+                txs = twitter_xt_get_status(node->u.array.values[i]);
+                if (!txs)
+                        continue;
+                
+                txl->list = g_slist_prepend(txl->list, txs);
+        }
+
+        return TRUE;
+}
+
+/* Will log messages either way. Need to keep track of IDs for stream deduping.
+   Plus, show_ids is on by default and I don't see why anyone would disable it. */
 static char *twitter_msg_add_id(struct im_connection *ic,
                                 struct twitter_xml_status *txs, const char *prefix)
 {
         struct twitter_data *td = ic->proto_data;
-        char *ret = NULL;
-
-        if (!set_getbool(&ic->acc->set, "show_ids")) {
+        int reply_to = -1;
+        bee_user_t *bu;
+
+        if (txs->reply_to) {
+                int i;
+                for (i = 0; i < TWITTER_LOG_LENGTH; i++)
+                        if (td->log[i].id == txs->reply_to) {
+                                reply_to = i;
+                                break;
+                        }
+        }
+
+        if (txs->user && txs->user->screen_name &&
+            (bu = bee_user_by_handle(ic->bee, ic, txs->user->screen_name))) {
+                struct twitter_user_data *tud = bu->data;
+
+                if (txs->id > tud->last_id) {
+                        tud->last_id = txs->id;
+                        tud->last_time = txs->created_at;
+                }
+        }
+        
+        td->log_id = (td->log_id + 1) % TWITTER_LOG_LENGTH;
+        td->log[td->log_id].id = txs->id;
+        td->log[td->log_id].bu = bee_user_by_handle(ic->bee, ic, txs->user->screen_name);
+        
+        /* This is all getting hairy. :-( If we RT'ed something ourselves,
+           remember OUR id instead so undo will work. In other cases, the
+           original tweet's id should be remembered for deduplicating. */
+        if (strcmp(txs->user->screen_name, td->user) == 0)
+                td->log[td->log_id].id = txs->rt_id;
+        
+        if (set_getbool(&ic->acc->set, "show_ids")) {
+                if (reply_to != -1)
+                        return g_strdup_printf("\002[\002%02x->%02x\002]\002 %s%s",
+                                               td->log_id, reply_to, prefix, txs->text);
+                else
+                        return g_strdup_printf("\002[\002%02x\002]\002 %s%s",
+                                               td->log_id, prefix, txs->text);
+        } else {
                 if (*prefix)
                         return g_strconcat(prefix, txs->text, NULL);
@@ -598 +662 @@
                         return NULL;
         }
-
-        td->log[td->log_id].id = txs->id;
-        td->log[td->log_id].bu = bee_user_by_handle(ic->bee, ic, txs->user->screen_name);
-        if (txs->reply_to) {
-                int i;
-                for (i = 0; i < TWITTER_LOG_LENGTH; i++)
-                        if (td->log[i].id == txs->reply_to) {
-                                ret = g_strdup_printf("\002[\002%02d->%02d\002]\002 %s%s",
-                                                      td->log_id, i, prefix, txs->text);
-                                break;
-                        }
-        }
-        if (ret == NULL)
-                ret = g_strdup_printf("\002[\002%02d\002]\002 %s%s", td->log_id, prefix, txs->text);
-        td->log_id = (td->log_id + 1) % TWITTER_LOG_LENGTH;
-
-        return ret;
-}
-
-static void twitter_groupchat_init(struct im_connection *ic)
-{
-        char *name_hint;
+}
+
+/**
+ * Function that is called to see the statuses in a groupchat window.
+ */
+static void twitter_status_show_chat(struct im_connection *ic, struct twitter_xml_status *status)
+{
+        struct twitter_data *td = ic->proto_data;
         struct groupchat *gc;
-        struct twitter_data *td = ic->proto_data;
-        GSList *l;
-
-        td->timeline_gc = gc = imcb_chat_new(ic, "twitter/timeline");
-
-        name_hint = g_strdup_printf("%s_%s", td->prefix, ic->acc->user);
-        imcb_chat_name_hint(gc, name_hint);
-        g_free(name_hint);
-
-        for (l = ic->bee->users; l; l = l->next) {
-                bee_user_t *bu = l->data;
-                if (bu->ic == ic)
-                        imcb_chat_add_buddy(td->timeline_gc, bu->handle);
-        }
-}
-
-/**
- * Function that is called to see the statuses in a groupchat window.
- */
-static void twitter_groupchat(struct im_connection *ic, GSList * list)
-{
-        struct twitter_data *td = ic->proto_data;
-        GSList *l = NULL;
-        struct twitter_xml_status *status;
-        struct groupchat *gc;
-        guint64 last_id = 0;
+        gboolean me = g_strcasecmp(td->user, status->user->screen_name) == 0;
+        char *msg;
 
         // Create a new groupchat if it does not exsist.
-        if (!td->timeline_gc)
-                twitter_groupchat_init(ic);
-
-        gc = td->timeline_gc;
-        if (!gc->joined)
-                imcb_chat_add_buddy(gc, ic->acc->user);
-
-        for (l = list; l; l = g_slist_next(l)) {
-                char *msg;
-
-                status = l->data;
-                if (status->user == NULL || status->text == NULL || last_id == status->id)
-                        continue;
-
-                last_id = status->id;
-
-                strip_html(status->text);
-
-                if (set_getbool(&ic->acc->set, "strip_newlines"))
-                        strip_newlines(status->text);
-
-                msg = twitter_msg_add_id(ic, status, "");
-
-                // Say it!
-                if (g_strcasecmp(td->user, status->user->screen_name) == 0) {
-                        imcb_chat_log(gc, "You: %s", msg ? msg : status->text);
-                } else {
-                        twitter_add_buddy(ic, status->user->screen_name, status->user->name);
-
-                        imcb_chat_msg(gc, status->user->screen_name,
-                                      msg ? msg : status->text, 0, status->created_at);
-                }
-
-                g_free(msg);
-
-                // Update the timeline_id to hold the highest id, so that by the next request
-                // we won't pick up the updates already in the list.
-                td->timeline_id = MAX(td->timeline_id, status->id);
-        }
+        gc = twitter_groupchat_init(ic);
+
+        if (!me)
+                /* MUST be done before twitter_msg_add_id() to avoid #872. */
+                twitter_add_buddy(ic, status->user->screen_name, status->user->name);
+        msg = twitter_msg_add_id(ic, status, "");
+        
+        // Say it!
+        if (me) {
+                imcb_chat_log(gc, "You: %s", msg ? msg : status->text);
+        } else {
+                imcb_chat_msg(gc, status->user->screen_name,
+                              msg ? msg : status->text, 0, status->created_at);
+        }
+
+        g_free(msg);
 }
 
@@ -693 +696 @@
  * Function that is called to see statuses as private messages.
  */
-static void twitter_private_message_chat(struct im_connection *ic, GSList * list)
-{
-        struct twitter_data *td = ic->proto_data;
-        GSList *l = NULL;
-        struct twitter_xml_status *status;
-        char from[MAX_STRING];
-        gboolean mode_one;
-        guint64 last_id = 0;
-
-        mode_one = g_strcasecmp(set_getstr(&ic->acc->set, "mode"), "one") == 0;
-
-        if (mode_one) {
+static void twitter_status_show_msg(struct im_connection *ic, struct twitter_xml_status *status)
+{
+        struct twitter_data *td = ic->proto_data;
+        char from[MAX_STRING] = "";
+        char *prefix = NULL, *text = NULL;
+        gboolean me = g_strcasecmp(td->user, status->user->screen_name) == 0;
+
+        if (td->flags & TWITTER_MODE_ONE) {
                 g_snprintf(from, sizeof(from) - 1, "%s_%s", td->prefix, ic->acc->user);
                 from[MAX_STRING - 1] = '\0';
         }
 
-        for (l = list; l; l = g_slist_next(l)) {
-                char *prefix = NULL, *text = NULL;
-
-                status = l->data;
-                if (status->user == NULL || status->text == NULL || last_id == status->id)
-                        continue;
-
-                last_id = status->id;
-
-                strip_html(status->text);
-                if (mode_one)
-                        prefix = g_strdup_printf("\002<\002%s\002>\002 ",
-                                                 status->user->screen_name);
-                else
-                        twitter_add_buddy(ic, status->user->screen_name, status->user->name);
-
-                text = twitter_msg_add_id(ic, status, prefix ? prefix : "");
-
-                imcb_buddy_msg(ic,
-                               mode_one ? from : status->user->screen_name,
-                               text ? text : status->text, 0, status->created_at);
-
-                // Update the timeline_id to hold the highest id, so that by the next request
-                // we won't pick up the updates already in the list.
-                td->timeline_id = MAX(td->timeline_id, status->id);
-
-                g_free(text);
-                g_free(prefix);
-        }
-}
-
-static void twitter_http_get_home_timeline(struct http_request *req);
-static void twitter_http_get_mentions(struct http_request *req);
+        if (td->flags & TWITTER_MODE_ONE)
+                prefix = g_strdup_printf("\002<\002%s\002>\002 ",
+                                         status->user->screen_name);
+        else if (!me)
+                twitter_add_buddy(ic, status->user->screen_name, status->user->name);
+        else
+                prefix = g_strdup("You: ");
+
+        text = twitter_msg_add_id(ic, status, prefix ? prefix : "");
+
+        imcb_buddy_msg(ic,
+                       *from ? from : status->user->screen_name,
+                       text ? text : status->text, 0, status->created_at);
+
+        g_free(text);
+        g_free(prefix);
+}
+
+static void twitter_status_show(struct im_connection *ic, struct twitter_xml_status *status)
+{
+        struct twitter_data *td = ic->proto_data;
+        
+        if (status->user == NULL || status->text == NULL)
+                return;
+        
+        /* Grrrr. Would like to do this during parsing, but can't access
+           settings from there. */
+        if (set_getbool(&ic->acc->set, "strip_newlines"))
+                strip_newlines(status->text);
+        
+        if (td->flags & TWITTER_MODE_CHAT)
+                twitter_status_show_chat(ic, status);
+        else
+                twitter_status_show_msg(ic, status);
+
+        // Update the timeline_id to hold the highest id, so that by the next request
+        // we won't pick up the updates already in the list.
+        td->timeline_id = MAX(td->timeline_id, status->rt_id);
+}
+
+static gboolean twitter_stream_handle_object(struct im_connection *ic, json_value *o);
+
+static void twitter_http_stream(struct http_request *req)
+{
+        struct im_connection *ic = req->data;
+        struct twitter_data *td;
+        json_value *parsed;
+        int len = 0;
+        char c, *nl;
+        
+        if (!g_slist_find(twitter_connections, ic))
+                return;
+        
+        ic->flags |= OPT_PONGED;
+        td = ic->proto_data;
+        
+        if ((req->flags & HTTPC_EOF) || !req->reply_body) {
+                td->stream = NULL;
+                imcb_error(ic, "Stream closed (%s)", req->status_string);
+                imc_logout(ic, TRUE);
+                return;
+        }
+        
+        printf( "%d bytes in stream\n", req->body_size );
+        
+        /* MUST search for CRLF, not just LF:
+           https://dev.twitter.com/docs/streaming-apis/processing#Parsing_responses */
+        nl = strstr(req->reply_body, "\r\n");
+        
+        if (!nl) {
+                printf("Incomplete data\n");
+                return;
+        }
+        
+        len = nl - req->reply_body;
+        if (len > 0) {
+                c = req->reply_body[len];
+                req->reply_body[len] = '\0';
+                
+                printf("JSON: %s\n", req->reply_body);
+                printf("parsed: %p\n", (parsed = json_parse(req->reply_body)));
+                if (parsed) {
+                        twitter_stream_handle_object(ic, parsed);
+                }
+                json_value_free(parsed);
+                req->reply_body[len] = c;
+        }
+        
+        http_flush_bytes(req, len + 2);
+        
+        /* One notification might bring multiple events! */
+        if (req->body_size > 0)
+                twitter_http_stream(req);
+}
+
+static gboolean twitter_stream_handle_event(struct im_connection *ic, json_value *o);
+static gboolean twitter_stream_handle_status(struct im_connection *ic, struct twitter_xml_status *txs);
+
+static gboolean twitter_stream_handle_object(struct im_connection *ic, json_value *o)
+{
+        struct twitter_data *td = ic->proto_data;
+        struct twitter_xml_status *txs;
+        json_value *c;
+        
+        if ((txs = twitter_xt_get_status(o))) {
+                gboolean ret = twitter_stream_handle_status(ic, txs);
+                txs_free(txs);
+                return ret;
+        } else if ((c = json_o_get(o, "direct_message")) &&
+                   (txs = twitter_xt_get_dm(c))) {
+                if (strcmp(txs->user->screen_name, td->user) != 0)
+                        imcb_buddy_msg(ic, txs->user->screen_name,
+                                       txs->text, 0, txs->created_at);
+                txs_free(txs);
+                return TRUE;
+        } else if ((c = json_o_get(o, "event")) && c->type == json_string) {
+                twitter_stream_handle_event(ic, o);
+                return TRUE;
+        } else if ((c = json_o_get(o, "disconnect")) && c->type == json_object) {
+                /* HACK: Because we're inside an event handler, we can't just
+                   disconnect here. Instead, just change the HTTP status string
+                   into a Twitter status string. */
+                char *reason = json_o_strdup(c, "reason");
+                if (reason) {
+                        g_free(td->stream->status_string);
+                        td->stream->status_string = reason;
+                }
+                return TRUE;
+        }
+        return FALSE;
+}
+
+static gboolean twitter_stream_handle_status(struct im_connection *ic, struct twitter_xml_status *txs)
+{
+        struct twitter_data *td = ic->proto_data;
+        int i;
+        
+        for (i = 0; i < TWITTER_LOG_LENGTH; i++) {
+                if (td->log[i].id == txs->id) {
+                        /* Got a duplicate (RT, probably). Drop it. */
+                        return TRUE;
+                }
+        }
+        
+        if (!(strcmp(txs->user->screen_name, td->user) == 0 ||
+              set_getbool(&ic->acc->set, "fetch_mentions") ||
+              bee_user_by_handle(ic->bee, ic, txs->user->screen_name))) {
+                /* Tweet is from an unknown person and the user does not want
+                   to see @mentions, so drop it. twitter_stream_handle_event()
+                   picks up new follows so this simple filter should be safe. */
+                /* TODO: The streaming API seems to do poor @mention matching.
+                   I.e. I'm getting mentions for @WilmerSomething, not just for
+                   @Wilmer. But meh. You want spam, you get spam. */
+                return TRUE;
+        }
+        
+        twitter_status_show(ic, txs);
+        
+        return TRUE;
+}
+
+static gboolean twitter_stream_handle_event(struct im_connection *ic, json_value *o)
+{
+        struct twitter_data *td = ic->proto_data;
+        json_value *source = json_o_get(o, "source");
+        json_value *target = json_o_get(o, "target");
+        const char *type = json_o_str(o, "event");
+        
+        if (!type || !source || source->type != json_object
+                  || !target || target->type != json_object) {
+                return FALSE;
+        }
+        
+        if (strcmp(type, "follow") == 0) {
+                struct twitter_xml_user *us = twitter_xt_get_user(source);
+                struct twitter_xml_user *ut = twitter_xt_get_user(target);
+                if (strcmp(us->screen_name, td->user) == 0) {
+                        twitter_add_buddy(ic, ut->screen_name, ut->name);
+                }
+                txu_free(us);
+                txu_free(ut);
+        }
+        
+        return TRUE;
+}
+
+gboolean twitter_open_stream(struct im_connection *ic)
+{
+        struct twitter_data *td = ic->proto_data;
+        char *args[2] = {"with", "followings"};
+        
+        if ((td->stream = twitter_http(ic, TWITTER_USER_STREAM_URL,
+                                       twitter_http_stream, ic, 0, args, 2))) {
+                /* This flag must be enabled or we'll get no data until EOF
+                   (which err, kind of, defeats the purpose of a streaming API). */
+                td->stream->flags |= HTTPC_STREAMING;
+                return TRUE;
+        }
+        
+        return FALSE;
+}
+
+static void twitter_get_home_timeline(struct im_connection *ic, gint64 next_cursor);
+static void twitter_get_mentions(struct im_connection *ic, gint64 next_cursor);
 
 /**
@@ -778 +948 @@
         struct twitter_xml_list *home_timeline = td->home_timeline_obj;
         struct twitter_xml_list *mentions = td->mentions_obj;
+        guint64 last_id = 0;
         GSList *output = NULL;
         GSList *l;
 
+        imcb_connected(ic);
+        
         if (!(td->flags & TWITTER_GOT_TIMELINE)) {
                 return;
@@ -804 +977 @@
                 }
         }
-        
-        if (!(ic->flags & OPT_LOGGED_IN))
-                imcb_connected(ic);
 
         // See if the user wants to see the messages in a groupchat window or as private messages.
-        if (g_strcasecmp(set_getstr(&ic->acc->set, "mode"), "chat") == 0)
-                twitter_groupchat(ic, output);
-        else
-                twitter_private_message_chat(ic, output);
-
-        g_slist_free(output);
+        while (output) {
+                struct twitter_xml_status *txs = output->data;
+                if (txs->id != last_id)
+                        twitter_status_show(ic, txs);
+                last_id = txs->id;
+                output = g_slist_remove(output, txs);
+        }
 
         txl_free(home_timeline);
@@ -823 +994 @@
 }
 
+static void twitter_http_get_home_timeline(struct http_request *req);
+static void twitter_http_get_mentions(struct http_request *req);
+
 /**
  * Get the timeline.
  */
-void twitter_get_home_timeline(struct im_connection *ic, gint64 next_cursor)
+static void twitter_get_home_timeline(struct im_connection *ic, gint64 next_cursor)
 {
         struct twitter_data *td = ic->proto_data;
@@ -862 +1036 @@
  * Get mentions.
  */
-void twitter_get_mentions(struct im_connection *ic, gint64 next_cursor)
+static void twitter_get_mentions(struct im_connection *ic, gint64 next_cursor)
 {
         struct twitter_data *td = ic->proto_data;
@@ -893 +1067 @@
 
         g_free(args[1]);
-        if (td->timeline_id) {
-                g_free(args[5]);
-        }
+        g_free(args[5]);
 }
 
@@ -905 +1077 @@
         struct im_connection *ic = req->data;
         struct twitter_data *td;
-        struct xt_node *parsed;
+        json_value *parsed;
         struct twitter_xml_list *txl;
 
@@ -921 +1093 @@
                 goto end;
         twitter_xt_get_status_list(ic, parsed, txl);
-        xt_free_node(parsed);
+        json_value_free(parsed);
 
         td->home_timeline_obj = txl;
 
       end:
+        if (!g_slist_find(twitter_connections, ic))
+                return;
+
         td->flags |= TWITTER_GOT_TIMELINE;
 
@@ -938 +1113 @@
         struct im_connection *ic = req->data;
         struct twitter_data *td;
-        struct xt_node *parsed;
+        json_value *parsed;
         struct twitter_xml_list *txl;
 
@@ -954 +1129 @@
                 goto end;
         twitter_xt_get_status_list(ic, parsed, txl);
-        xt_free_node(parsed);
+        json_value_free(parsed);
 
         td->mentions_obj = txl;
 
       end:
+        if (!g_slist_find(twitter_connections, ic))
+                return;
+
         td->flags |= TWITTER_GOT_MENTIONS;
 
@@ -972 +1150 @@
         struct im_connection *ic = req->data;
         struct twitter_data *td;
-        struct xt_node *parsed, *node;
+        json_value *parsed, *id;
 
         // Check if the connection is still active.
@@ -984 +1162 @@
                 return;
         
-        if ((node = xt_find_node(parsed, "status")) &&
-            (node = xt_find_node(node->children, "id")) && node->text)
-                td->last_status_id = g_ascii_strtoull(node->text, NULL, 10);
+        if ((id = json_o_get(parsed, "id")) && id->type == json_integer) {
+                td->last_status_id = id->u.integer;
+        }
+        
+        json_value_free(parsed);
+        
+        if (req->flags & TWITTER_HTTP_USER_ACK)
+                twitter_log(ic, "Command processed successfully");
 }
 
@@ -1032 +1215 @@
         char *url;
         url = g_strdup_printf("%s%llu%s", TWITTER_STATUS_DESTROY_URL,
-                              (unsigned long long) id, ".xml");
-        twitter_http(ic, url, twitter_http_post, ic, 1, NULL, 0);
+                              (unsigned long long) id, ".json");
+        twitter_http_f(ic, url, twitter_http_post, ic, 1, NULL, 0,
+                       TWITTER_HTTP_USER_ACK);
         g_free(url);
 }
@@ -1041 +1225 @@
         char *url;
         url = g_strdup_printf("%s%llu%s", TWITTER_STATUS_RETWEET_URL,
-                              (unsigned long long) id, ".xml");
-        twitter_http(ic, url, twitter_http_post, ic, 1, NULL, 0);
+                              (unsigned long long) id, ".json");
+        twitter_http_f(ic, url, twitter_http_post, ic, 1, NULL, 0,
+                       TWITTER_HTTP_USER_ACK);
         g_free(url);
 }
@@ -1056 +1241 @@
         };
         args[1] = screen_name;
-        twitter_http(ic, TWITTER_REPORT_SPAM_URL, twitter_http_post,
-                     ic, 1, args, 2);
+        twitter_http_f(ic, TWITTER_REPORT_SPAM_URL, twitter_http_post,
+                       ic, 1, args, 2, TWITTER_HTTP_USER_ACK);
 }
 
@@ -1067 +1252 @@
         char *url;
         url = g_strdup_printf("%s%llu%s", TWITTER_FAVORITE_CREATE_URL,
-                              (unsigned long long) id, ".xml");
-        twitter_http(ic, url, twitter_http_post, ic, 1, NULL, 0);
+                              (unsigned long long) id, ".json");
+        twitter_http_f(ic, url, twitter_http_post, ic, 1, NULL, 0,
+                       TWITTER_HTTP_USER_ACK);
         g_free(url);
 }

protocols/twitter/twitter_lib.h

@@ -29 +29 @@
 #include "twitter_http.h"
 
-#define TWITTER_API_URL "http://api.twitter.com/1"
+#define TWITTER_API_URL "http://api.twitter.com/1.1"
 #define IDENTICA_API_URL "https://identi.ca/api"
 
 /* Status URLs */
-#define TWITTER_STATUS_UPDATE_URL "/statuses/update.xml"
+#define TWITTER_STATUS_UPDATE_URL "/statuses/update.json"
 #define TWITTER_STATUS_SHOW_URL "/statuses/show/"
 #define TWITTER_STATUS_DESTROY_URL "/statuses/destroy/"
@@ -39 +39 @@
 
 /* Timeline URLs */
-#define TWITTER_PUBLIC_TIMELINE_URL "/statuses/public_timeline.xml"
-#define TWITTER_FEATURED_USERS_URL "/statuses/featured.xml"
-#define TWITTER_FRIENDS_TIMELINE_URL "/statuses/friends_timeline.xml"
-#define TWITTER_HOME_TIMELINE_URL "/statuses/home_timeline.xml"
-#define TWITTER_MENTIONS_URL "/statuses/mentions.xml"
-#define TWITTER_USER_TIMELINE_URL "/statuses/user_timeline.xml"
+#define TWITTER_PUBLIC_TIMELINE_URL "/statuses/public_timeline.json"
+#define TWITTER_FEATURED_USERS_URL "/statuses/featured.json"
+#define TWITTER_FRIENDS_TIMELINE_URL "/statuses/friends_timeline.json"
+#define TWITTER_HOME_TIMELINE_URL "/statuses/home_timeline.json"
+#define TWITTER_MENTIONS_URL "/statuses/mentions_timeline.json"
+#define TWITTER_USER_TIMELINE_URL "/statuses/user_timeline.json"
 
 /* Users URLs */
-#define TWITTER_USERS_LOOKUP_URL "/users/lookup.xml"
+#define TWITTER_USERS_LOOKUP_URL "/users/lookup.json"
 
 /* Direct messages URLs */
-#define TWITTER_DIRECT_MESSAGES_URL "/direct_messages.xml"
-#define TWITTER_DIRECT_MESSAGES_NEW_URL "/direct_messages/new.xml"
-#define TWITTER_DIRECT_MESSAGES_SENT_URL "/direct_messages/sent.xml"
+#define TWITTER_DIRECT_MESSAGES_URL "/direct_messages.json"
+#define TWITTER_DIRECT_MESSAGES_NEW_URL "/direct_messages/new.json"
+#define TWITTER_DIRECT_MESSAGES_SENT_URL "/direct_messages/sent.json"
 #define TWITTER_DIRECT_MESSAGES_DESTROY_URL "/direct_messages/destroy/"
 
 /* Friendships URLs */
-#define TWITTER_FRIENDSHIPS_CREATE_URL "/friendships/create.xml"
-#define TWITTER_FRIENDSHIPS_DESTROY_URL "/friendships/destroy.xml"
-#define TWITTER_FRIENDSHIPS_SHOW_URL "/friendships/show.xml"
+#define TWITTER_FRIENDSHIPS_CREATE_URL "/friendships/create.json"
+#define TWITTER_FRIENDSHIPS_DESTROY_URL "/friendships/destroy.json"
+#define TWITTER_FRIENDSHIPS_SHOW_URL "/friendships/show.json"
 
 /* Social graphs URLs */
-#define TWITTER_FRIENDS_IDS_URL "/friends/ids.xml"
-#define TWITTER_FOLLOWERS_IDS_URL "/followers/ids.xml"
+#define TWITTER_FRIENDS_IDS_URL "/friends/ids.json"
+#define TWITTER_FOLLOWERS_IDS_URL "/followers/ids.json"
 
 /* Account URLs */
-#define TWITTER_ACCOUNT_RATE_LIMIT_URL "/account/rate_limit_status.xml"
+#define TWITTER_ACCOUNT_RATE_LIMIT_URL "/account/rate_limit_status.json"
 
 /* Favorites URLs */
-#define TWITTER_FAVORITES_GET_URL "/favorites.xml"
+#define TWITTER_FAVORITES_GET_URL "/favorites.json"
 #define TWITTER_FAVORITE_CREATE_URL "/favorites/create/"
 #define TWITTER_FAVORITE_DESTROY_URL "/favorites/destroy/"
@@ -77 +77 @@
 
 /* Report spam */
-#define TWITTER_REPORT_SPAM_URL "/report_spam.xml"
+#define TWITTER_REPORT_SPAM_URL "/report_spam.json"
 
+#define TWITTER_USER_STREAM_URL "https://userstream.twitter.com/1.1/user.json"
+
+gboolean twitter_open_stream(struct im_connection *ic);
 void twitter_get_timeline(struct im_connection *ic, gint64 next_cursor);
 void twitter_get_friends_ids(struct im_connection *ic, gint64 next_cursor);
-void twitter_get_home_timeline(struct im_connection *ic, gint64 next_cursor);
-void twitter_get_mentions(struct im_connection *ic, gint64 next_cursor);
 void twitter_get_statuses_friends(struct im_connection *ic, gint64 next_cursor);