Challenges in Developing a Twitter Client
June 23, 2009 at 6:48 pm | In Blog, TweetMind | Leave a CommentTags: alpha, architecture, beta, cache, client, deterministic, free, garbage collection, memory, multi-threading, new, release, twitter, virtual
The aim was to provide a nice friendly intuitive twitter client. I basically wanted twhirl but with more features like drag and drop groups, better following/unfolllowing stuff etc.
Scalability, Caching and Performance
Twitter gives you a lot of data if you’re following a lot of people and I want TweetMind to work for noobs and hardcore twitter users alike. So, with this in mind I set about thinking about scalability and implemented a number of cunning techniques to keep memory usage down:
- I cache all the date twitter gives TweetMind for seaches, quick user lookups etc.
- I wrote my own cache, that includes memory management so if it starts getting too big it dumps out old stuff that may not be needed anymore
- I cache tweets, user information and avatars
- The UI is seperate from the TweetMind Cache so I use virtual display techniques. There aren’t any tweet displays in existence outside of the immediately visible tab and it’s visible area. Everything else pops in and out of existence as you scroll or tab about. Unfortunately this makes scrolling a little slow on my system but it’s well worth it!
Garbage Collection my arse!
During development I noticed that if I left TweetMind running for a while it would run out of memory. That was fairly easy to solve as I simply hadn’t implemented my cache memory management very well. Once that was sorted I noticed I started getting “Unable to create window handle” exceptions all over the place and everything would die. A bit of googline about and I found some guidance saying that I should add the “USER Objects” column to my Windows Task Manager and have a look at what was going on.
Lo and behold I was quickly cranking up to 5000ish of these things and crashing horribly. It seems that applications have a max load of 10,000 and I wasn’t approaching it fast enough to crash. Although I had virtualised the tweet display the window objects I was creating to display a tweet weren’t being disposed of properly and the Garbage Collector was never going anywhere near these things. In the end I implemented a manual dispose method, explicitly called and suddenly life is grand.
If you add the “USER Objects” column to your task manager and fire up TweetMind you can see this working yourself. Just flip about between tabs and you’ll see USER Objects spike up, especially when going back to the home tab if you’ve been recieving a lot of tweets. But almost as soon as it spikes up it should settle back down to around 800 – 1000 depending on many things (don’t worry if it’s a bit more or less so long as it’s staying roughly stable);
I can now run TweetMind for hours without memory, USER Objects, GDI Objects or anything else getting too out of hand! Huzzah! This should make TweetMind scalable enough for even the most hardcore of users
What order do things happen in?
Normally when programming things happen one after another. This is fairly simple and deterministic therefore when there’s bugs it’s reasonably simple to track them down and fix them. However, looking up information on the internet is fairly slow and when TweetMind starts it needs to lookup who you are, your friends tweets (equivalent of your “home” page on twitter), your replies and your direct messages. Imagine doing all that in your browser, it would take a while. Image that long waiting for a program to start and become responsive. The quality of user experience would be total shit. Therefore I decided to do it all at the same time, and in the background. This means the main TweetMind window can happily respond to your actions while stuff is loading.
Technically I did this by multithreading.I execute all of the main twitter lookups (friends timeline, replies, DMs) on throttled threads dumping all their data into the cache, the UI then reads the cache when it wants to which is derived from when you’re looking at it. Underneath the UI the “engine” of the system periodically, depending on things like API limit usage, looks up info on Twitter in parallel. This means that the user experience is quite quick for TweetMind vs. some other twitter clients
Oh yeah, TweetMind public beta is now open
Coming soon… TweetMind
May 27, 2009 at 10:49 am | In Blog, TweetMind | 2 CommentsTags: alpha, beta, dirtmind, free, groups, screenshot, seesmic desktop, software, tweetdeck, tweetmind, twhirl, twitter
Coming soon to an intrawebz near you is the ultimate twitter client: TweetMind!
I got fed up with twitter clients not being able to do what I wanted! I like the simplicity of twirl but need groups and more advanced features like tweetdeck, plus multiple accounts, a customisable interface, font size, drag and drop for grouping people, and all sorts of other goodies. Twhirl doesn’t do enough, Tweetdeck is too limiting, Seesmic Desktop suck monkey balls. So I decided to write my own, twitter client and so TweetMind was born.
So far it’s in early development and isn’t available for public use. I plan to start limited alpha testing in the next few weeks followed by public beta after that. In the meantime, here’s a screenshot of the development version – bear in mind that visual details will change, I’ve not finished adding nice buttons and sorting out the bottom of my tweet forms yet!
Click the pic for a full-size verison. Oh and of course like all DirtMind stuff this will be free and there will be no ads!
Here’s an updated screenshot from May 27th:
Uploading images with a firefox extension
May 6, 2009 at 10:34 am | In FireFox Extension, Website Development | Leave a CommentTags: addon, bbcode, extension, firefox, forum, free, howto, image, lol, lolcat
I’ve blogged before about some of the technology behind the free DirtMind image gallery so here’s some more…
For a long time I’ve wanted to make uploading and using images on forums simpler. FireFox gives us the capabilities to do this since when writing a firefox extension you’ve got access to a bit more functionality than you normally do from javascript as it’s trusted code.
The mission, upload an image by right clicking on it, generating all necessary BBCode.
The solution:
1) Write a nice XUL user interface for adding right click options to the firefox browser, selecting a local picture, establishing selection of a remote picture and providing nice upload feedback
2) Using the firefox extension encode the local image, or download a remote image and encode it (without saving a local file – that would be cheating)
3) Upload to the image host, decode the image data back into a picture.
After that it’s easy because we’re just into the normal serverside image processing stuff that the image host does.
Here’s some screenshots of it in action:
Because of the other stuff I’ve blogged about before in terms of image captioning and deep site integration with phpBB you can now upload an image, caption it and post it extremely easily. Check out this HQ video of the whole process!
Or watch in in lower quality on youtubte:
Make your own lolcats, rofls and captions online
May 1, 2009 at 1:27 pm | In Blog, Website Development | Leave a CommentTags: caption, chrome, firefox, free, hosting, ie, image, javascript, lol, lolcats, online, rofl, safari
I decided I wanted to allow DirtMind site members to be able to add captions to images easily so I set about thinking how to add an Image Captioner to the free DirtMind image hosting service. I intially talked to the developer of a well known online image captioner about getting his code to work as part of my site. Unfortunately though the backend technologies of his site and DirtMind didn’t mesh so I set about writing my own.
In the end this has had a lot of advantages as the Image captioner is far more specific to my design
Server-side it’s pretty simple as I already had some image processing going on to watermark images with my site logo. Rather than make a stand alone image captioner I decided to integrate the functionality throughout the image gallery – that means all uploaded images are captionable. If you’re the owner of an image you can overwrite it with a captioned version, otherwise you can just save it as a new image (of course owners can do this too).
Of course some people will want a stand alone captioner, or at least a landing page so I created one for these people at www.dirtmind.com/captioner.php but this is really just a single file upload box that goes to the normal gallery pages
Play with it
As with everything at DirtMind it’s free and has no ads, if you want to try it out just look at any image on DirtMind! Here’s an example to play with (just click the pic).

You need to have a registered DM account to save your pic, or upload new ones, but you can play about with the captions without registering!
Technically the most challenging thing about doing this was the client side javascript. It seems that if you write something that works in FireFox then it pretty much works in Safari and Chrome. IE however has it’s own plans, and needs entirely different code. I spent a lot of time getting it to work in each platform. If you use another browser I’d appreciate you giving it a test run!
How to get reviews for FireFox extensions
February 20, 2009 at 2:06 am | In Blog, FireFox Extension | Leave a CommentTags: addon, bbcode, blogging, extension, facebook, firefox, forum, hints, how to, linkedin, mozilla, review, tips, tricks, twitter, yammer
I posted this guidance recently based on the response I got from a comment on an AMO (addons.mozilla.org) editor’s blog. Basically it’s instructions on how to leave a review but not how to get a review. I’ve released several firefox addons/extensions under various different IDs so here’s my advice on actually getting reviews…
Reviews are all important for firefox extension developers as without a number (minumum 3 decent) reviews you can’t get an extension out of the sandbox and into the public site. That’s important because in the sandbox you have to have a log in to get at an extension on the main public area anyone can download and use anonymously, so clearly there’s a barrier to normal users for sandbox extensions. This is for a good reason of course, good reviews mean that an extension isn’t just spam or complete shit, it’s actually useful.
So, 3 things:
1) If you have a log in to AMO then please leave reviews for the extensions you like, even if they’re already public
2) Please leave a review for my extensions
3) If you’re just sponging off the way extension developers make cool stuff for free help them out by leaving a review, it’s not like it costs anything more than 5 mins of your time!
If you’re an extension developer, then you already know all this. So the critical question is how do you get reviews for your extension?
The obvious answer is to get your users to do leave reviews. The problem is it can be hard to get users if you don’t have reviews. I’ve gone about this by examining the target user base for my addon, and then telling them about it. My latest extension is about generating bbcode for forum users so naturally forum users are my target audience, I started off with my own forum and then let other forums know about it. If an extension is properly labelled then interested people will find it, of course many of them use google rather than AMO so you need to think about getting a website or page up that shows off your extension that people will find by search engines. Be careful with forums though, they are notoriously twitchy about percieved spam!
Do you personally know people/internet peeps that would be interested in using your addon? Then invite them to leave a review and make sure you explain how to!
List it on AMO. People do naturally find extensions by searching AMO, I know because they’ve found mine!
Blog it! Blogs are a great way to direct search engine traffic to something of interest.
Tweet it! If you have a twitter following then let your tweeps know about your new found extension awesomeness. It’s bound to be relevant to some of them, or at least as relevant as knowing that you’ve woken up.
Get external links! If your target audience has a community or reference website then ask the webmasters if they’d be interested in linking to your extension in their resources section. Remember, everyone hates a spammer but people like contributors so you’ll often find that if you word your communication properly you’ll get a positive response. This will drive users to your extension, and in turn will drive up the number of reviews.
Pimp it out! Whatever mediums/networks you’re a part of should know about your extension. Post it to Facebook, twitter, yammer, bebo, blinko, linkedin, hi5, plaxo, friendfeed, plurj, rejaw, meemi etc. But only let them know once, otherwise you’ll be just another spammer and ignored as such.
Finally, get an AMO account and leave review for other people. If I see reviews for one of my extensions I’ll lookup the username and leave a review for their extension. A bit of quid quo pro works well. So leave me a review and I’ll leave you one, there are many others that have the same philosophy, if you’re one of them then why not leave a comment here gently spamming your firefox extension.
Do you have any other tips?
Why @dirtythoughts are better than secrettweets
February 12, 2009 at 1:56 pm | In Blog, FireFox Extension | Leave a CommentTags: anonymous, dirty, dirtythoughts, secret, twitter, uncensored
Want to read people’s innermost secret dirty thoughts?
@dirtythoughts is a twitter account that anyone can post anything to, anonymously in realtime. There’s no filtering or censoring, there’s not even a moderation queue so as soon as you type in your dirty thought it’s posted on twitter, it’s also posted on the dirtythoughts website so that users can discuss. You don’t have to be a twitter user to join in so tell the world your dirty thought now!
This was inspired by secrettweet which is good but censored, moderated and very slow, therefore, improvements
How to sent a message to twitter using php
February 9, 2009 at 11:51 am | In Blog, Website Development | 3 CommentsTags: anonymous, code, parser, php, twitter, xml
As part of the Integration of DirtMind with twitter I needed to send tweets to twitter from php to the anonymous @dirtythoughts account. Here’s how I did it
$username = 'dirtythoughts';
$password = 'yeahlikeimdumbenoughtoincludethathere';
// The message you want to send
$tweet = 'message that has been filtered for html and stuff';
// The twitter API address
$url = 'http://twitter.com/statuses/update.xml';
// Set up and execute the curl process
$curl_handle = curl_init();
curl_setopt($curl_handle, CURLOPT_URL, "$url");
curl_setopt($curl_handle, CURLOPT_CONNECTTIMEOUT, 2);
curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl_handle, CURLOPT_POST, 1);
curl_setopt($curl_handle, CURLOPT_POSTFIELDS, "status=$tweet");
curl_setopt($curl_handle, CURLOPT_USERPWD, "$username:$password");
$buffer = curl_exec($curl_handle);
curl_close($curl_handle);
// check for success or failure
if ((!empty($buffer)) && (!$buffer))
{
$error[] = "There was an error while trying to talk to twitter";
$error[] = "Twitter said: $buffer";
$error[] = "Try again later!";
}
else if ($buffer)
{
$p = xml_parser_create();
xml_parse_into_struct($p, $buffer, $vals, $index);
$tweetid = $vals[3]["value"];
}
This last bit about the xml parser reads the return xml from twitter, when there hasn’t been an error and gets the tweetid. This is used to link to individual tweets on a twitter account like so “http://twitter.com/dirtythoughts/status/ + tweetid” so that once a user had tweeted using the DirtyThoughts anonymous account I could give them a link to their tweet.
How to submit a post to phpbb3 by a different user
February 9, 2009 at 11:51 am | In Blog, Website Development | Leave a CommentTags: anonymous, code, php, phpbb3
As part of the Integration of DirtMind with twitter I needed to be able to submit posts into my phpbb3 forum using a bot account to protect anonymity of my users when they’re tweeting their DirtyThoughts anonymously, and I didn’t want to use the anonymous/guest account because I wanted to generate hit counts etc. Here’s how I did it
$poll = $uid = $bitfield = $options = '';
generate_text_for_storage($twitter_status, $uid, $bitfield, $options, true, true, true);
$data = array(
'forum_id' => 11, //the subforum that my DirtyThoughts thread is in
'icon_id' => false,
'topic_id' => 435, //the thread to reply to
'poster_id' => 146, //the id of the DirtyThoughts user
'enable_bbcode' => true,
'enable_smilies' => true,
'enable_urls' => true,
'enable_sig' => true,
'message' => $twitter_status,
'message_md5' => md5($twitter_status),
'bbcode_bitfield' => $bitfield,
'bbcode_uid' => $uid,
'post_edit_locked' => 0,
'topic_title' => '',
'notify_set' => false,
'notify' => false,
'post_time' => 0,
'forum_name' => '',
'enable_indexing' => true,
);
//store current user
$old_user_id=$user->data['user_id'];
$old_username=$user->data['username'];
$old_usercol=$user->data['user_colour'];
//change to bot
$user->data['username']='DirtyThoughts';
$user->data['user_id']= 146;
$user->data['user_colour']= '';
//make post
submit_post('reply', 'Dirty Thought', 'DirtyThoughts', POST_NORMAL, $poll, $data, '');
//restore user
$user->data['username']=$old_username;
$user->data['user_id']= $old_user_id;
$user->data['user_colour'] = $old_usercol;
Interestingly the submit_post function always uses the current user object to make a post, regardless of the settings in $data. To get around this I store the current userid, name and colour in variables and switch to the bot values to make the post before returning the current users settings to their normal values.
Integrating a website with twitter
February 9, 2009 at 11:51 am | In Blog, Website Development | 2 CommentsTags: anonymous, api, Blog, dirtythoughts, forum, icq, integration, modification, msn, php, phpbb, phpbb3, rss, secrets, twitter, twitterfeed, uncensored
As part of the continuing campaign to take over the internet I’ve been looking at twitter integration for DirtMind.com. I’m a big fan of twitter (follow me @dirtmind) so I’ve been wanting to do something with it for a while.
There are a number of natural touchpoints with twitter that I looked at:
- Tweeting DirtMind top stories
- Allowing registered members to list their twitter account in their profile
- DirtyThoughts – anonymous, uncencensored tweeting to the @dirtythoughts account
I’ve now achieved all of these. Tweeting top stories wa the easiest and is done using twitterfeed which lets you tweet from an RSS feed. Using twitter feed I can send out both new entries on this blog (or at least those tagged with the “blog” tag like this one) and also the top stories from the DirtMind RSS feed (a custom feed I built on my top stories database entries).
Allowing registered members to list their twitter account was on the surface quite easy. I wanted to let people enter their twitter username in their profile info and then show up a little blue “t” next to their posts and in their public profile view that links to their twitter feed. Just the way the site already does for msn, icq etc. Once I started looking into this though it quickly became apparent that it was quite a bit of work, so in normal style I thought that someone else must have already done this fairly obvious site modification. The advantage of using a popular system as your base architecture is that you can make use of the cool modifications everyone else has done. I found and implemented RMcGirr83’s phpbb3 twitter mod with no modifications (other than changing the icon) which is a first for me as I normally modify modifications and then modify those until I’m recursing into a black hole. Anyway, it’s excellent and does exactly what I wanted. You can see an example on my public profile listing at DirtMind.
Finally, DirtyThoughts… I liked the idea of anonymous tweeting as a way of people getting something off their chest, expressing themselves or saying something they wouldn’t normally say, and the ability to comment on these dirty thoughts and discuss them. Well forum software is built for commenting and discussing so a cool thing to do would be to have a setup whereby people could enter in a comment and it would get sent to both the forum and twitter. Using the twitter API was simple enough, so I designed a twitter page for @dirtythoughts, then created a DirtyThoughts user on the DirtMind site. Now I just needed a way to show the latest thoughts. To do this I butchered the standard viewtopic page (sorted by latest first) and customised it to have the dirtythoughts posting form at the top. It works really well
I also edited the forum and thread navigation pages to automatically redirect to this special viewing and posting page rather than the normal phpbb3 pages. You can see the result here. It works really well and is already attracting some interesting and disgusting anonymous tweets! Please stop by and add one yourself
The hardest part of all of this was getting the visual confirmation stuff to work on my new page as I’d not gone anywhere near that stuff before. Some of the interesting bits I came accross were:
Blog at WordPress.com. | Theme: Pool by Borja Fernandez.
Entries and comments feeds.



