So here comes “Sign in with twitter”. Twitter has been moving to OAuth to provide an open platform for authentication, meaning that sites like www.dirtmind.com can use twitter as a way of logging in users, even brand new ones, in just two clicks! Sounds awesome doesn’t it?

This is just the button, it doesn't do a login here!

There were a few problems with getting this all to work… phpbb has an open architecture and a mechanism for you to create your own authentication plugins, so rather than use the default database authentication mechanism you can use LDAP or Apache or write you own. Unfortunately, despite my detailed searching it seems no one has written an auth_twitter plugin for phpbb3. On the other hand there are some good php twitter oauth examples out there so I thought it shouldn’t be too hard to write my own.

Some requirements:

1. New users must be able to use twitter oauth to create and login to new dirtmind accounts seamlessly.

2. Existing dirtmind members must be able to login using their normal username/password combination and twitter oauth interchangeably.

3. No twitter user should be able to hijack an existing dirtmind account

4. A twitter user needs to be able to claim a dirtmind account as theirs as part of the twitter oath flow

5. A dirtmind user needs to be able to specify a twitter account to log in with, that isn’t necessarily the same name

6. An account created by twitter oauth needs to be a standard normal phpbb user account, the user should be able to change their profile and even set a password that will then work as part of normal phpbb db login

7. All logins should honour phpbb security, inactive accounts and banned accounts settings

Easy right?

The big challenge

The main problem in achieving this was that there is some architectural impedence between twitter oauth and phpbb login. This is probably why I couldn’t find anything out there that I could just pick up and use.

Twitter oauth is a two stage process, you request an authorisation url, you go to it and enter account details on the twitter site (or use your current logged in status) and then it calls back to the original applicaiton.

Phpbb login is a single process, you call login with a username/password and the user is either logged in or not.

The solution

Yes, I got it to work, you can see the results on www.dirtmind.com (any part of the site, forum, games, image gallery, whatever – it’s side wide awesomeness). I’m not going to post all of the code for how login works on my website for obvious security reasons but here’s the solution and some snippets to help you put it all together.

So I did write an auth_twitter plugin for phpbb3 however this practically no twitter oauth flow in it at all. It’s basically the same as the standard auth_db plugin with some notable exceptions:

• If the source of the login attempt is twitter then allow null passwords
• If the source of the login is twitter then auto-create a phpbb account if one doesn’t exist
• During auto-creation of accounts if the target already exists than redirect to a mapping page that allows the user to claim the target dirtmind account or alternatively specify an arbitrary name for their new dirtmind account

How to create a new phpbb users (from auth_twitter)

 $row = array(
 'username' => $username,
 'user_twitter' => $db->sql_escape(utf8_clean_string($username)),
 'user_password' => phpbb_hash(''),
 'user_email' => 'none@dirtmind.com',
 'group_id' => (int) $grow['group_id'],
 'user_type' => USER_NORMAL,
 'user_ip' => getenv('REMOTE_ADDR'), 
 'user_timezone' => '1.00',
 'user_dst' => 0,
 'user_lang' => 'en',
 'user_actkey' => '',
 'user_dateformat' => 'd M Y H:i', 
 'user_style' => 1,
 'user_regdate' => time(),
 ); 

Normally you’d then call:

$phpbb_user_id = user_add($row); 

To add the user to the database, but since you’re inside the login mechanism you just pass the row back as part of the return data of the login_twitter function

I then simply call the login_twitter method with username and no password from the twitter oauth flow, which is built into my basic page template For simple php oauth I strongly recommend Abraham’s twitter oauth. Obviously there are some issues to deal with like not allowing arbitrary account linkage, preventing simple db account hacking based on the blank passwords and other anti-hacking things to consider, but those are all fairly standard solutions to fairly standard problems. The results are a really sweet login experience for people

I can now run TweetMind for hours without memory, USER Objects, GDI Objects or anything else getting too out of hand! Huzzah! This should make TweetMind scalable enough for even the most hardcore of users

Technically I did this by multithreading.I execute all of the main twitter lookups (friends timeline, replies, DMs) on throttled threads dumping all their data into the cache, the UI then reads the cache when it wants to which is derived from when you’re looking at it. Underneath the UI the “engine” of the system periodically, depending on things like API limit usage, looks up info on Twitter in parallel. This means that the user experience is quite quick for TweetMind vs. some other twitter clients

The aim was to provide a nice friendly intuitive twitter client. I basically wanted twhirl but with more features like drag and drop groups, better following/unfolllowing stuff etc.

Scalability, Caching and Performance

Twitter gives you a lot of data if you’re following a lot of people and I want TweetMind to work for noobs and hardcore twitter users alike. So, with this in mind I set about thinking about scalability and implemented a number of cunning techniques to keep memory usage down:

- I cache all the date twitter gives TweetMind for seaches, quick user lookups etc.

- I wrote my own cache, that includes memory management so if it starts getting too big it dumps out old stuff that may not be needed anymore

- I cache tweets, user information and avatars

- The UI is seperate from the TweetMind Cache so I use virtual display techniques. There aren’t any tweet displays in existence outside of the immediately visible tab and it’s visible area. Everything else pops in and out of existence as you scroll or tab about. Unfortunately this makes scrolling a little slow on my system but it’s well worth it!

Garbage Collection my arse!

During development I noticed that if I left TweetMind running for a while it would run out of memory. That was fairly easy to solve as I simply hadn’t implemented my cache memory management very well. Once that was sorted I noticed I started getting “Unable to create window handle” exceptions all over the place and everything would die. A bit of googline about and I found some guidance saying that I should add the “USER Objects” column to my Windows Task Manager and have a look at what was going on.

Lo and behold I was quickly cranking up to 5000ish of these things and crashing horribly. It seems that applications have a max load of 10,000 and I wasn’t approaching it fast enough to crash. Although I had virtualised the tweet display the window objects I was creating to display a tweet weren’t being disposed of properly and the Garbage Collector was never going anywhere near these things. In the end I implemented a manual dispose method, explicitly called and suddenly life is grand.

If you add the “USER Objects” column to your task manager and fire up TweetMind you can see this working yourself. Just flip about between tabs and you’ll see USER Objects spike up, especially when going back to the home tab if you’ve been recieving a lot of tweets. But almost as soon as it spikes up it should settle back down to around 800 – 1000 depending on many things (don’t worry if it’s a bit more or less so long as it’s staying roughly stable);

I can now run TweetMind for hours without memory, USER Objects, GDI Objects or anything else getting too out of hand! Huzzah! This should make TweetMind scalable enough for even the most hardcore of users

What order do things happen in?

Normally when programming things happen one after another. This is fairly simple and deterministic therefore when there’s bugs it’s reasonably simple to track them down and fix them. However, looking up information on the internet is fairly slow and when TweetMind starts it needs to lookup who you are, your friends tweets (equivalent of your “home” page on twitter), your replies and your direct messages. Imagine doing all that in your browser, it would take a while. Image that long waiting for a program to start and become responsive. The quality of user experience would be total shit. Therefore I decided to do it all at the same time, and in the background. This means the main TweetMind window can happily respond to your actions while stuff is loading.

Technically I did this by multithreading.I execute all of the main twitter lookups (friends timeline, replies, DMs) on throttled threads dumping all their data into the cache, the UI then reads the cache when it wants to which is derived from when you’re looking at it. Underneath the UI the “engine” of the system periodically, depending on things like API limit usage, looks up info on Twitter in parallel. This means that the user experience is quite quick for TweetMind vs. some other twitter clients

Oh yeah, TweetMind public beta is now open

I got fed up with twitter clients not being able to do what I wanted! I like the simplicity of twirl but need groups and more advanced features like tweetdeck, plus multiple accounts, a customisable interface, font size, drag and drop for grouping people, and all sorts of other goodies. Twhirl doesn’t do enough, Tweetdeck is too limiting, Seesmic Desktop suck monkey balls. So I decided to write my own, twitter client and so TweetMind was born.

So far it’s in early development and isn’t available for public use. I plan to start limited alpha testing in the next few weeks followed by public beta after that. In the meantime, here’s a screenshot of the development version – bear in mind that visual details will change, I’ve not finished adding nice buttons and sorting out the bottom of my tweet forms yet!

TweetMind Alpha 0.9 screenshot

Click the pic for a full-size verison.  Oh and of course like all DirtMind stuff this will be free and there will be no ads!

Here’s an updated screenshot from May 27th:

For a long time I’ve wanted to make uploading and using images on forums simpler. FireFox gives us the capabilities to do this since when writing a firefox extension you’ve got access to a bit more functionality than you normally do from javascript as it’s trusted code.

The mission, upload an image by right clicking on it, generating all necessary BBCode.

The solution:

1) Write a nice XUL user interface for adding right click options to the firefox browser, selecting a local picture, establishing selection of a remote picture and providing nice upload feedback

2) Using the firefox extension encode the local image, or download a remote image and encode it (without saving a local file – that would be cheating)

3) Upload to the image host, decode the image data back into a picture.

After that it’s easy  because we’re just into the normal serverside image processing stuff that the image host does.

Here’s some screenshots of it in action:

Selecting an image to upload

The image uploaded

Because of the other stuff I’ve blogged about before in terms of image captioning and deep site integration with phpBB you can now upload an image, caption it and post it extremely easily. Check out this HQ video of the whole process!

Or watch in in lower quality on youtubte:

In the end this has had a lot of advantages as the Image captioner is far more specific to my design Server-side it’s pretty simple as I already had some image processing going on to watermark images with my site logo. Rather than make a stand alone image captioner I decided to integrate the functionality throughout the image gallery – that means all uploaded images are captionable. If you’re the owner of an image you can overwrite it with a captioned version, otherwise you can just save it as a new image (of course owners can do this too).

Of course some people will want a stand alone captioner, or at least a landing page so I created one for these people at www.dirtmind.com/captioner.php but this is really just a single file upload box that goes to the normal gallery pages

Play with it

As with everything at DirtMind it’s free and has no ads, if you want to try it out just look at any image on DirtMind! Here’s an example to play with (just click the pic).

You need to have a registered DM account to save your pic, or upload new ones, but you can play about with the captions without registering!

Technically the most challenging thing about doing this was the client side javascript. It seems that if you write something that works in FireFox then it pretty much works in Safari and Chrome. IE however has it’s own plans, and needs entirely different code. I spent a lot of time getting it to work in each platform. If you use another browser I’d appreciate you giving it a test run!

Reviews are all important for firefox extension developers as without a number (minumum 3 decent) reviews you can’t get an extension out of the sandbox and into the public site. That’s important because in the sandbox you have to have a log in to get at an extension on the main public area anyone can download and use anonymously, so clearly there’s a barrier to normal users for sandbox extensions. This is for a good reason of course, good reviews mean that an extension isn’t just spam or complete shit, it’s actually useful.

1) If you have a log in to AMO then please leave reviews for the extensions you like, even if they’re already public

2) Please leave a review for my extensions

3) If you’re just sponging off the way extension developers make cool stuff for free help them out by leaving a review, it’s not like it costs anything more than 5 mins of your time!

The obvious answer is to get your users to do leave reviews. The problem is it can be hard to get users if you don’t have reviews. I’ve gone about this by examining the target user base for my addon, and then telling them about it. My latest extension is about generating bbcode for forum users so naturally forum users are my target audience, I started off with my own forum and then let other forums know about it. If an extension is properly labelled then interested people will find it, of course many of them use google rather than AMO so you need to think about getting a website or page up that shows off your extension that people will find by search engines. Be careful with forums though, they are notoriously twitchy about percieved spam!

Do you personally know people/internet peeps that would be interested in using your addon? Then invite them to leave a review and make sure you explain how to!

List it on AMO. People do naturally find extensions by searching AMO, I know because they’ve found mine!

Blog it! Blogs are a great way to direct search engine traffic to something of interest.

Tweet it! If you have a twitter following then let your tweeps know about your new found extension awesomeness.  It’s bound to be relevant to some of them, or at least as relevant as knowing that you’ve woken up.

Get external links! If your target audience has a community or reference website then ask the webmasters if they’d be interested in linking to your extension in their resources section. Remember, everyone hates a spammer but people like contributors so you’ll often find that if you word your communication properly you’ll get a positive response. This will drive users to your extension, and in turn will drive up the number of reviews.

Pimp it out! Whatever mediums/networks you’re a part of should know about your extension. Post it to Facebook, twitter, yammer, bebo, blinko, linkedin, hi5, plaxo, friendfeed, plurj, rejaw, meemi etc. But only let them know once, otherwise you’ll be just another spammer and ignored as such.

Finally, get an AMO account and leave review for other people. If I see reviews for one of my extensions I’ll lookup the username and leave a review for their extension. A bit of quid quo pro works well. So leave me a review and I’ll leave you one, there are many others that have the same philosophy, if you’re one of them then why not leave a comment here gently spamming your firefox extension.

Do you have any other tips?

@dirtythoughts is a twitter account that anyone can post anything to, anonymously in realtime. There’s no filtering or censoring, there’s not even a moderation queue so as soon as you type in your dirty thought it’s posted on twitter, it’s also posted on the dirtythoughts website so that users can discuss. You don’t have to be a twitter user to join in so tell the world your dirty thought now!

This was inspired by secrettweet which is good but censored, moderated and very slow, therefore, improvements

This irresponsible scaremongering Jeni Barnett was originally brought up by Bad Science blogger Dr Ben Goldacre (someone actually qualified to comment) who posted the original audio. He got slapped with a cease and desist but the audio is spreading like wildfire. So here it is for your listening displeasure.

Perhaps her brain would be less addled if she ate less pies?

Comment anonymously here

        $username = 'dirtythoughts';
        $password = 'yeahlikeimdumbenoughtoincludethathere';
        // The message you want to send
        $tweet = 'message that has been filtered for html and stuff';
        // The twitter API address
        $url = 'http://twitter.com/statuses/update.xml';
        // Set up and execute the curl process
        $curl_handle = curl_init();
        curl_setopt($curl_handle, CURLOPT_URL, "$url");
        curl_setopt($curl_handle, CURLOPT_CONNECTTIMEOUT, 2);
        curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($curl_handle, CURLOPT_POST, 1);
        curl_setopt($curl_handle, CURLOPT_POSTFIELDS, "status=$tweet");
        curl_setopt($curl_handle, CURLOPT_USERPWD, "$username:$password");
        $buffer = curl_exec($curl_handle);
        curl_close($curl_handle);
        // check for success or failure
        if ((!empty($buffer))  && (!$buffer))
        {
            $error[] = "There was an error while trying to talk to twitter";
            $error[] = "Twitter said: $buffer";
            $error[] = "Try again later!";
        }
	else if ($buffer)
	{
		$p = xml_parser_create();
		xml_parse_into_struct($p, $buffer, $vals, $index);
		$tweetid = $vals[3]["value"];
	}

This last bit about the  xml parser reads the return xml from twitter, when there hasn’t been an error and gets the tweetid. This is used to link to individual tweets on a twitter account like so “http://twitter.com/dirtythoughts/status/ + tweetid” so that once a user had tweeted using the DirtyThoughts anonymous account I could give them a link to their tweet.

        $poll = $uid = $bitfield = $options = '';   
        generate_text_for_storage($twitter_status, $uid, $bitfield, $options, true, true, true);

        $data = array(
           'forum_id'      => 11, //the subforum that my DirtyThoughts thread is in
           'icon_id'      => false,
           'topic_id'      => 435, //the thread to reply to
           'poster_id'      => 146, //the id of the DirtyThoughts user              

           'enable_bbcode'   => true,
           'enable_smilies'   => true,
           'enable_urls'      => true,
           'enable_sig'      => true,

           'message'      => $twitter_status,
           'message_md5'   => md5($twitter_status),

           'bbcode_bitfield'   => $bitfield,
           'bbcode_uid'      => $uid,

           'post_edit_locked'   => 0,
           'topic_title'      => '',
           'notify_set'      => false,
           'notify'         => false,
           'post_time'       => 0,
           'forum_name'      => '',
           'enable_indexing'   => true,          
        );          
        //store current user
        $old_user_id=$user->data['user_id'];
        $old_username=$user->data['username'];       
        $old_usercol=$user->data['user_colour'];       
        //change to bot
        $user->data['username']='DirtyThoughts';
        $user->data['user_id']= 146;
        $user->data['user_colour']= '';
        //make post
        submit_post('reply', 'Dirty Thought', 'DirtyThoughts', POST_NORMAL, $poll, $data, '');    
        //restore user
        $user->data['username']=$old_username;
        $user->data['user_id']= $old_user_id;
        $user->data['user_colour'] = $old_usercol;

Interestingly the submit_post function always uses the current user object to make a post, regardless of the settings in $data. To get around this I store the current userid, name and colour in variables  and switch to the bot values to make the post before returning the current users settings to their normal values.