Articles

Navigation ( archives ›› 2011 ›› January ›› 16 ›› Facebook Friend Export Chrome Extension Internals )

Facebook Friend Export Chrome Extension Internals

Posted on January 16, 2011, 11:21 pm EST

There has been a lot of buzz regarding why Facebook doesn't allow you to export your friends out of Faceboook. Well, the first question many people have stated "Why can't you use Facebook's API to do this". The answer is quite simple but not obvious, Facebook does not allow their API (Application Programming Interface) to export your friends Email addresses that they made public for you to see. Hopefully in this post, I will explain the internals how this Google Chrome Extension was constructed.

Table of Contents

Introduction

You can download the extension from Chrome Web Store and the source code is hosted on GitHub (feel free to fork, send patches, and help me improve it)

Within the media, it has been concluded that Facebook owns your friends most valuable data, their email address. In my opinion, the reason why Facebook became so popular is because of "us", we allowed them to export our friends and connect to them by extracting our friends from Hotmail, MSN, Gmail, Twitter, etc. But the reverse was not possible, there was no way for us to export them back. Absolutely no way! I invited many, many, people to Facebook through this process because I thought it was a neat concept and I believe in their vision. Facebook has grown so fast, that most of my in real life friends are on Facebook and not in my contact app (Outlook, Google Contacts). There is no way for me to connect back to my friends unless I do that within Facebook. It feels like Facebook is controlling the data that I have provided for them for free. That quite bothered me, hence the creation of this extension.

Without the original work from an anonymous person who hosted fb-exporter on Google Code (which is now blocked), I have used the source code, changed it dramatically, and made it easier, cleaner, faster, and safer to use. So whats new?

  • A different approach to fetch your friends by directly communicating to Facebook's JavaScript Objects, and retrieving your initial friends information quickly such as Name, Image, and URL.
  • Cached results, since it takes so much time to export your friends, some people might accidently close their browser, or want to continue the process some other time. That is now possible via HTML5 Web Storage.
  • Not only emails are exported, it exports your contacts phone numbers, Google Talk, Yahoo, ICQ, Skype, QQ, and MSN (Windows Live Messenger) aliases. jQuery is beautiful when it comes to simplicity.
  • It uses real time error handling, if something goes wrong, it will automatically figure out what that issue is and properly handle it.
  • New User Interface, you can see a grid of all your friends with their profile photos. When you start the lengthy process, it tells you what is CACHED, READY, STARTED, PROCESSED, SUCCESS and FAILED! These status messages help the user know what is happening in real time. Thanks to Chrome Extensions Message Passing.
  • The code is mostly refactored to make it cleaner and easier to use. As you noticed, the UI differs too. It is commented fully and restructured to make it easier on the external contributions.

Communications between Facebook and the Extension process

In Google Chrome extension, the only way to communicate to the DOM, is through Content Scripts. But there is only one problem, they have a couple of limitations that we need. Content Scripts cannot:

  • Use variables or functions defined by their extension's pages
  • Use variables or functions defined by web pages or by other content scripts
  • Make cross-site XMLHttpRequests

Communication from the Facebook World to the Chrome Extension World

We need to access our main extension page because we want to transfer our data to Google Gmail Contacts through cross-site XMLHttpRequests. No fear, we can use Chrome's Message Passing to send messages back and fourth to our extensions background page, which is just an HTML page that runs in the extension process. It exists for the lifetime of your extension and only one instance of it at a time is active. Once we reach the extension process, we can send data back and fourth to Gmail as we please (of course, with the proper permissions set in the manifest. We want our users to know what the extension has access to.)

In fb-exporter, we actually use Chrome's Message Passing to communicate between both worlds, since they are isolated. It is really simple to setup. Within the Content Script we add add a extension listener, chrome.extension.onRequest.addListener, which in this case gets fired when a request is sent from the extension process (reduced snippet below):

chrome.extension.onRequest.addListener(function(request, sender, sendResponse) {
  if (request.getFriendsMap) {
    sendResponse({data: friendsMap});
  }
});

Communication from the Extension process back to Facebook

In the extension world's ( background page ), we maintain the Facebook's tab ID when the user first clicks on the "Export Friends" button on Facebook's navigation. When the "Export Friends" button is clicked, it sends a request to our extension process to tell it to open the main controller (that controls the steps for exporting your friends) window. We use chrome.extension.sendRequest to send the request to the extension process from the content script:

chrome.extension.sendRequest({switchToWorkerTab: 1});

So how does the background page listen for content script requests that comes from Facebook? The exact same way we discussed before with the content script:

chrome.extension.onRequest.addListener(function(request, sender, sendResponse) {
  if (request.switchToWorkerTab) {
    facebook_id = sender.tab.id;
    chrome.tabs.create({url: 'controller.html', selected: true}, function(tab) {
      worker_id = tab.id;
    });
  }
});

What the above snippet states, when a request is coming in from the Facebook page via Content Script, we keep track of the ID's of those pages. We need those ID's so that we can send information back and fourth between different pages directly. It becomes easier to handle.

Then we can use chrome.tabs.sendRequest, to send a request back to Facebook Content Script and even back to the extension process:

chrome.tabs.sendRequest(facebook_id, { facebookError: 1 });
chrome.tabs.sendRequest(worker_id, { facebookError: 1 });

Using content scripts to read the internal JavaScript directly from Facebook

The traditional way of grabbing information is by screen scraping, but we are not doing that. If you used Chrome Web Inspector, you will notice an internal JavaScript variable hosted in Facebook is running the show. But there are some problems and limitations, as we discussed before. We cannot access that variable directly from our Extension and even our Content Script!

The only way the our extension can communicate to Facebook is through Content Scripts, and the only way Content Scripts can communicate with the embedding page is through the DOM. The DOM is what shared between both worlds, since they are completely isolated from each other.

To do this, we inject the Facebook page with a 'script' tag that creates a custom event. The custom event dispatches when friends object has been retrieved. We use custom events because everything is asynchronous. We want to inform the Content Script that we have successfully grabbed the object. If you are familiar with how Java does drag and drop, the concept of Transfer nodes are similar in this case (kinda).

// JS script injection to the facebook's World. This function can only be read
// in the Facebook world, it has absolutely no access to the content script
// nor our extension process.
var postFriendMap = function() {
  // Use events to notify the content script. Replicate the event the content
  // script has, so we can pass this event to that world.
  var exportEvent = document.createEvent('Event');
  exportEvent.initEvent('friendExported', true, true);
  
  // Create a transfer node DOM, since that is the only way two worlds can
  // communicate with each other.
  var transferDOM = document.getElementById('fb-transfer-dom-area');
  transferDOM.innerText = JSON.stringify(FriendSearchPane._data);
    
  // Inform our content script that we have received the object from Facebook.
  window.dispatchEvent(exportEvent);
};
 
// Create a dummy textarea DOM.
var transferDOM = document.createElement('div');
$(transferDOM).attr('id', 'fb-transfer-dom-area')
              .hide()
              .appendTo($(document.body));
  
// Start injecting the JS script.
var script = document.createElement('script');
script.setAttribute('id', 'fb-inject-area');
script.appendChild(document.createTextNode('(' + postFriendMap + ')();'));
document.body.appendChild(script); 

Our content script listens on that custom event so it will know, asynchronously, when we can start processing our list of friends. And the only way to transfer data through these two worlds is by the DOM as shown in the snippet above. A dummy transfer DIV is created on the DOM that both worlds understand.

// Listen on the real DOM requests to check if the friends list has been exported.
window.addEventListener('friendExported', function() {
  // Save the map to this content script world, so our extension can read it.
  var transferDOM = $('#fb-transfer-dom-area');
  friendsMap = JSON.parse(transferDOM.text()); // Global
  
  // Clean up since we no longer need this.
  $(transferDOM).remove();
  $('#fb-inject-area').remove();
  
  // Count the number of friends.
  var i = Objects.keys(friendsMap).length;
  
  // Tell the worker tab that we are set!
  chrome.extension.sendRequest({friendListReceived: 1, count: i});
}); 

The listener reads the contents stored in the transfer node, and deserializes it back so we can store it. It does all the cleanup afterwards since we no longer need it. Since we are in the content script, it asynchronously notifies the extension process that we have received the friends list.

Voila, we now have read the internal data Facebook has stored.

Caching of Friends using Web SQL Database

Caching friends is really important for any lengthy process. If computer restarted, tab closed, error happened, or anything strange happens, we don't want to redo the whole process again. The obvious way is to continue where we left off.

There are two ways to do data caching for a Web Application:

  • HTML5 Local Storage
  • HTML5 Web SQL Database (no longer active, IndexDB currently in Standards)

localStorage was initially going to be chosen, but the 5MB (2.5MB UTF) limitation was my reason of not choosing it. localStorage is just a key[value] synchronous storage mechanism, which is super easy to use. Unfortunately, 2.5MB is not enough to store all your friends.

Web SQL Database has the limitation as well, but you can override that limitation by specifying in the Chrome Extension manifest unlimitedStorage It is an asynchronous API that uses SQL syntax. Take a look at the database.js file in the codebase on GitHub. The following snippet is how getting a freind from the cache is done.

FriendDB.prototype.getFriend = function(id, response) {
  this.db.transaction(function(tx) {
    tx.executeSql('SELECT data FROM friend WHERE id = ?', [id],
        function (tx, rs) {
          if (rs.rows.length != 0) {
            response({status: true, data: JSON.parse(rs.rows.item(0).data)});
          }
          else {
            response({status: false});
          }
        }, this.onError
    );
  });
};

In the controller page, once the friends list is requested (so I can render my friends in the page), I iterate through all the friends, and check if each user is in the cache, if they are in the cache, I inform the Facebook page that a cache exists. A separate cache map is stored so when it comes to a time to process each friend, it skips that friend because the data is already in the cached map.

chrome.tabs.sendRequest(bkg.facebook_id, {getFriendsMap: 1}, function(response) {
  $.each(response.data, function(key, value) {
    bkg.db.getFriend(key, function(result) {
      if (result.status) {
        chrome.tabs.sendRequest(bkg.facebook_id, ({cached: true, data: result.data}));
      }
    })
  });
});

The beauty of having caching, it will result in a neat progress bar :) It isn't instant, because I am making an SQL call for each user. Sure, I can make one big SQL query that checks each ID, but this is cooler since it shows to the user something is happening :)- But it is on my TODO list.

Sending the data to Gmail Contacts

The extension uses OAuth (which is an open protocol to allow secure API authorization), Google Contacts API (yes its public, and you can import and export), and Google Data API (to transfer the data in a nice format)

I wont explain much here, all that is done here is saving the data in Gmail Contacts. We use the APIs mentioned above to do this securely. Of course, there is absolutely no way I can know your authorization, it uses OAuth for secure Authorization. The first thing that happens, it creates a group called "Exported from Facebook". There is some smart logic there that checks for duplicate friends, and only adds friends that are not there. You can later log on Gmail Contacts and organize your friends in more groups and merge contacts together (very neat UI).

Conclusion

Well, I am not hating on Facebook. My country, Canada, due to privacy laws, we own our data. Our data is private information, we cannot let Facebook control our data and own it. I like everything to be in the clear, and that is why I created this blog post, to show everyone how its done. Sure I spent numerous hours designing, developing, and debugging it (I have spent many hours behind my computer to make sure its right). But I am happy of the end result. All my contacts are safely in Gmail and I can happily contact my friends through other means, and synchronize my friends with my phones and computers.

I hope this post helped you understand the technology behind creating this extension, and will help you create your own in the near future. The Chrome Extension API is very very very amusing to work with :)

About this Article:

Comments (7) - Add yours, or View the replies

Categoy (Software)

Views (14218)

Digg it: Digg this article