Twitter API using node/javascript

Today I’m going to talk about how to search using the Twitter API search with node/javascript.

why?

Well, I tried messing with this for one reason: sentiment analysis. I’m creating a dictionary that has all major adjectives in portuguese, followed by a number that represents the adjective’s popularity on TwitterAfter I have that, I can start classifying all the adjectives as good/bad/neutral. It’s going to be a basic dictionary, but for that, I need to filter all the adjectives. Portuguese language has about 8k adjectives ( only talking about the masculine form ). Imagine if I had to classify all those manually… It would take ages! That’s why im using this approach to figure out all adjectives that are mentioned on Twitter: to keep it small.

Ok, lez do this!

Right now, our dictionary looks something like this:

à-toa
aacheniano
aalénio
aalênio
aarónico
aarônico
ab-reativo
ab-rogado
ab-rogador
...

It’s still just a list of adjectives. But we are going to change that. First, let’s install the javascript wrapper for twitter:

npm install twitter

This is a great wrapper library, so if you want to learn more about it, I suggest you take a further look at their git repository.

After that, we need to register our app on Twitter. We do this by going to https://apps.twitter.com/ and clicking the button to register a new app.

After you have a new app registered, you can access the following info:

consumer_key
consumer_secret
access_token_key
access_token_secret

This info is necessary to consume the Twitter api. If you need more info on this subject, make sure you check this page.

Cool! Now what?

Now we can start our app:

var Twitter = require('twitter');
 
var client = new Twitter({
consumer_key: 'xxxxxxxx',
consumer_secret: 'xxxxxxxx',
access_token_key: 'xxx-xxxxx',
access_token_secret: 'xxxxxxxx'
});

Make sure that you change all the x’s with your app’s information.

Now that we have that configured, we make the call like this:

client.get('/search/tweets.json', 
          {q:'WhatYouLookinFor'},
          function(error, params, response) {});

So, this function has 3 arguments:

  • The first argument is the api call. You can find more about that here. But basically, thats the REST API call in a string format.
  • The second argument is an object containing the API parameters. That argument is optional, but in this case, we need it.
  • The third argument, is a callback function that it’s going to get called after the API call succeeds ( or fails ).

If you want to access the return object, just watch the params argument inside the callback function. That’s the JSON object that gets returned! Or you can have the raw http response object as well: that’s the third argument of the callback function.

One thing about the Twitter’s search API is that it’s limited for about 180 calls in a 15 minute window, so beware of  how many calls you make or you might get an error ( first argument of the callback function )! Our app is making 1 call each 5 seconds.

So, basically, this is the final code. It takes all words on a file, tracks how much it’s being mentioned on Twitter ( on a scale that goes from 0 to 15 ), and appends it to a new file. It does that every 5 seconds:

var Twitter = require('twitter');
var fs = require('fs');
var client = new Twitter({ consumer_key: 'xxxxxxxx', consumer_secret: 'xxxxxxxx', access_token_key: 'xxx-xxxxx', access_token_secret: 'xxxxxxxx' });

var find = function(i, array) {
    if (array.length > i) {
        client.get('/search/tweets.json', {
            q: array[i]
        }, function(error, params, response) {
            if (error) throw error;
            console.log(array[i] + ':', params.statuses.length);
            if (params.statuses.length !== 0) {
                fs.appendFile('new.txt', array[i] + ':' + params.statuses.length + '\n', function(err) {
                    i++;
                    console.log('Saved to file');
                    setTimeout(function() {
                        console.log('calling twitter');
                        find(i, array);
                    }, 5000);
                });
            } else {
                i++;
                setTimeout(function() {
                    console.log('calling twitter');
                    find(i, array);
                }, 5000);
            }
        });
    }
};

fs.readFile('dicionarioAdjetivo.txt', {
    encoding: 'utf-8'
}, function(err, data) {
    if (err) throw err;
    var i = 0;
    find(i, data.split('\n'));
});
end result

I had to keep this running for about 11 hours ( because of the calling rate restrictions ).

I’ve realized that all those adjectives that were on the original file ( about 8k ) were reduced by 1.5k. That’s not much, and not at all what I’ve expected, but now I can filter this file once more just to get the adjectives that were mentioned at least 15 times on Twitter. To acomplish this, I just created a small program that does that:

var fs = require('fs');

var fifteen = function(i, arr) {


    if (i < arr.length) {
        var quant = parseInt(arr[i].slice(arr[i].indexOf(':') + 1));
        if (quant === 15) {
            fs.appendFile('15.txt', arr[i].slice(0, arr[i].indexOf(':')) + '\n', function(err) {
                i++;
                fifteen(i, arr);

            });
        } else {
            i++;
            fifteen(i, arr);
        }
    }
};

fs.readFile('new.txt', {
    encoding: 'utf-8'
}, function(err, data) {
    if (err) throw err;
    var i = 0;
    fifteen(i, data.split('\n'));
});

and the end result is:

à-toa:15
abaetê:15
abafado:15
abafador:15
abaixado:15
abalado:15
...
considerations

Well, I have a couple observations that I would like to share about the whole Twitter API javascript experience:

  1. About the seed dictionary: It’s not that easy to find an all adjectives compilation of a determined language. Or maybe I just didn’t do my research very well. What I’ve done was, I created a crawler to go through a webpage called something like “allportugueseadjectives.com.br”, and had to gather all those adjectives in a file. For that I used a simplified http request library called ( yes, you guessed it ) request. I also used some regex and string manipulation to crawl stuff from the website.
  2. About the 0 to 15 mention scale: It happens that Twitter only displays 15 results at once for a query. If you want to see the other results, you might have to make another query using some sort of API pagination. The intent for this experience is to mess a little bit with the Twitter API using javascript, and I believe it wasn’t that necessary to get the accurate mention number. If it’s mentioned at least 15 times, it’s good enough to keep on the final dictionary, don’t you think so? I do.

When I finish classifying all these adjectives, I’ll be sharing the result on github, if anyone’s interested. Cya, guys!

2 Replies to “Twitter API using node/javascript”

  1. Hi! I love this post , I am trying to do something similar. I am trying to showcase the timelines of my users on a Chrome Web App. I can only use JS. I’m a bit lost with how I am supposed to place the code invthe directory do you think you could help?

    1. Hi! Thanks for liking my post! Actually I would recommend that you take a look at twitter`s API directly, since a lot of stuff has changed, I guess! 😀

Leave a Reply

Your email address will not be published. Required fields are marked *