mindspill

Which is the most popular social network site in your country?

Tue, 18 Dec 2007 01:14:28 GMT

The numbers speak for themselves – Social Networking Sites are popular all over the world.

In 90 (79%) countries a major social networking site features in the top 10 sites of that country.
In 19 of these countries, the social networking site is the highest ranking site in the country – ranking higher than any search engine.
From a sample of 116 countries only 2 (Taiwan and Vietnam) didn’t include a popular social networking site in the list of the top 100 websites.

The popularity of social networking sites is no surprise, and several statistics (1, 2, 3) have been published about the major social networking sites like facebook and myspace. There are however few reports on the use of these sites by geographic region. The only geographic distributions I came across were from Comscore, the Social Network Sites paper published in JCMC, and ValleyWag .

Using the Many Eyes platform I created three different visualizations of the most popular SN sites used in each country. The data used for determining the country popularity was collected from Alexa ratings. For more information on how the data was extracted see – how to collect geographic website rankings from the internet.

The world map is a colour coded map with each social networking site represented in a different colour. Where data wasn’t available, the country border is not displayed. If you click on a site from the list on the right, the countries that use that site are highlighted.

This second display shows a rectangular table display (treemap) of the data divided either by social networking platform, or by country. To alter between the displays reorder the treemap hierarchy by dragging the ordering on top of the visualization display.

The third visualization shows the ranking of the social network sites, and the number of internet users in each region. In the darker coloured regions, social networking sites ranked higher than other websites. The size of each rectangle is proportional to the number of internet users in the country, the bigger the rectangle, the more users there are.

Which visual representation of the data set do you prefer, and why? Do you think that one of the displays is superior to the others? Can you think of other different ways to present this data graphically? The aim of this exercise is to display some interesting data using Many Eyes and stimulate discussions on the different visualizations and data presented.

If you’d like to voice your comments, comment on the specific visualization by clicking the comment link in the respective visualization. The data used to generate the results is freely accessible on the Many Eyes site. You can use the uploaded data to create other visualizations in Many Eyes. After all, if you reply with a picture it’s like you’re writing a thousand words, isn’t it.

How to collect geographic website rankings from the internet?

Tue, 18 Dec 2007 01:01:16 GMT

The data for the social network sites by country visualisations was collected from the Alexa country rankings site. This is the only known available free source which provides this information for a reasonable number of countries. I wrote some perl scripts to automate the process and allow the same data to be collected in the future for comparison. (I can send the scripts to anyone who’s interested, just drop me an email). This is a brief description on how the data was collected:-

1. Extract the top websites in each country
The main ranking page for each country was first saved. The site ranking, site name and site description were extracted from the source html.

2. Identify the social networking sites
Most websites in Alexa have a brief description about the website. All the sites that had the word social in the text were saved in the social network definition file.

This process proved to be a good first guess at generating a list of social network sites. Some of the identified social networking sites (like badoo, yonja and perfspot) weren’t in the wikipedia list of social networking sites. As expected, the process also identified sites with the word social that aren’t SN sites. These sites were manually deleted.

After running the SN extract process and the results were analysed, some countries, especially non English speaking countries, had very low scores for the social networking sites that were originally identified. This was because these countries were in fact using social networking sites in their own language. This was most evident in Russian speaking countries using the website Vkontakte, which is very popular in Russian speaking countries.

Finally a few popular sites that weren’t identified from the above process were manually inserted from the Wikipedia list of social networking websites.

Why are QQ.com and Flickr not there?
The list of selected SN sites is by no means exhaustive and there are some popular sites that have social networking features embedded in them which make them almost SN sites. Qq.com the popular Chinese site, and Flikr are two prime examples. These sites were excluded from the list because they are not purely Social Networking sites, but they evolved from another website form (instant messaging and photo sharing in the case of Qq and flickr) into social networking sites.

This is the final list of sites that were used in the section process.

3.Retrieve the highest ranking website from the country rankings
The final extraction process selected the highest ranking SN site from the list along with the ranking and site name. The website address was used to match the websites to avoid any language problems with different character sets. The original idea was to include only SN sites that have a ranking between 1-20, however in order to populate the world map visualisation better this ranking restriction was omitted. In some cases there’s more than 1 SN site listed in the top 10 list which suggests that the distinction between the SN sites used in the country is not clear cut. Unfortunately in the world map visualisation it was difficult to split the country in 2 to show two different SN sites, therefore only the first ranking site in each case was considered.

4. Adding data from other sources
To help with the visualisation of the data collected data some additional data that wasn’t available in the original data source was required. The additional data was continent data, and the number of internet users in each country. At first I tried to look for this data from the new data search engine graphwise however the results returned were far from satisfactory. A google search later pointed me to the data sources that were finally used to get this information.
Continent Data
Internet Users Data

Mysql Login / Run script / Export CSV

Wed, 08 Mar 2006 21:42:17 GMT

Login: mysql [database name] -u [user] -p
E.g. mysql baseball -u mike -p
Run Script: mysql [database name] -f -u [user] -p < file_name.sql
E.g. mysql baseball -f -u mike -p < insert_teams.sql (use -f to ignore errors)
Export data to CSV: SELECT * INTO OUTFILE '/somefile.csv' FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' FROM table WHERE column='value'";

Perl Search and Replace One Liner

Wed, 08 Mar 2006 21:35:31 GMT

perl -p -e s/think/tank/g < input_file.txt > output_file.txt

Software Quality Assurance Testing Articles

Sun, 12 Feb 2006 23:41:32 GMT

Software Quality Assurance Testing Articles - Articles, white papers, and more on important software testing topics.

Good Reads

Fri, 09 Dec 2005 22:33:52 GMT

I'm not posting too much nowadays and the excuse I'm using with my writing soul is that I'm focusing on doing things rather than writing about doing them. Today I've managed to read longish posts and it was time well spent, so I'll share:-

Saying No - Rands in Repose

Choosing a major - Philip Greenspun

Reading List of Software Management Training Program - Joel Spolsky
The thing about New Yorkers (Seth did you read this!?)

On broken windows

Wed, 09 Nov 2005 04:57:23 GMT

One of the many beneficial side effects of the conference was that I learned about the Pragmatic Set of Books. I managed to get "The Pragmatic Programmer" and "Pragmatic Unit Testing" and I thought that anybody who reads this blog should know about the broken window concept.

"In inner cities, some buildings are beautiful and clean, while others are rotting hulks. Why? Researchers in the field of crime and urban decay discovered a fascinating trigger mechanism, one that very quickly turns a clean, intact, inhabited building into a smashed and abandoned derelict.

A broken window.

One broken window, left unrepaired for any substantial length of time, instills in the inhabitants of the building a sense of abandonmenta sense that the powers that be don't care about the building. So another window gets broken. People start littering. Graffiti appears. Serious structural damage begins. In a relatively short space of time, the building becomes damaged beyond the owner's desire to fix it, and the sense of abandonment becomes reality."

Source: The pragmatic programmer - A Hunt, D Thomas - Chapter 1 - Software Entropy

I think that the concept of having a broken window ruining a whole project and setting an open path for disaster is very true and applicable to both software and real life. I can recognize that I'm very susceptible to fall in the trap. Apart from code where I've seen it happen in my projects it also happens in daily life - I try to diet and keep it up until I break the first window by eating a piece of cake, I try to stop swearing until the first time somebody pisses me of and sets me on my usual route, and the list goes on and on...

I can also identify with the cleaning up the glass. If say I tidy up a project by refactoring, updating documentation etc. I have a feeling that the project is back up to scratch once again and is alive. The same happens with my room when I tidy up and everything is in order I start liking to live in the room again, until I break the first window again by leave it uncleaned for a day.

Word: pragmatic
Meaning:

archaic - Busy, Officious, Opinionated
relating to matters of fact or practical affairs often to the exclusion of intellectual or artistic matters : practical as opposed to idealistic
relating to or being in accordance with philosophical pragmatism

Usage: pragmatic men of power have had no time or inclination to deal with... social morality -- K. B. Clark

Report from New York

Mon, 07 Nov 2005 04:18:08 GMT

Ah, New York. Visiting the big apple was one of my 10 things to do before I die and I'm glad I made it. Getting here wasn't easy by any standard because all the flights out of my country were full and I even I got offloaded from the plane, after getting physically on it, because the pilot didn't want me in the cabin jump seat even though I had all the necessary authorizations. The alternatives to get to London were very bleak because all the British were returning back home after the midterm holidays. At last I got lucky because some passengers no showed on the midday BA flight to Gatwick and I got on instead. From here onwards everything was plain sailing - got the National Express from Gatwick to Heathrow (circa 1 hour 15 minutes) and then the 6pm flight to JFK. The flight landed at around 930pm and by 1030 I was in the Roosevelt Hotel. I need not say that I missed the ice hockey game between the NY Rangers and Montreal Canadians that I had already purchased the tickets for; $53 down the drain cause of a bastard pilot.

The hotel room wasn't as sophisticated as I expected it to be for $280 a day and there wasn't a playstation in the room as they claim on their website. The rest of the hotel though was very glamorous with high ceilings full of chandeliers and expensive looking décor. The Software Test and Performance Conference was held on the Mezzanine Level or for us non Americans 'The First Floor'. This floor is a dedicated conference area with about 10 different suites to host more than 450 visitors that attended the conference. This said, there wasn't space to host a buffet on the last day of the conference as they usually do so I got no free food at all.

As they say you compare things to others you're familiar with, and the closest anything comes to New York for me is London. The difference is that NY has got all those huge skyscrapers and characteristic doses of America piped into it. Anyone who's ever played battleships should easily be able to work out how to navigate the streets of Manhattan because the road network is a pure grid. To make things easier both the avenues and the streets are sequentially ordered (at least where I was walking) so getting around is easy.

The reputation New York has for shopping is well deserved because there're lots and lots of shops around here. I didn't find a huge number of department stores of the type where they sell anything, in fact the two only department stores I visited where Virgin and Best Buy (which is an electronics shop). Talking of department stores I might tend to put the Toys R' Us store in Times Square with the list also. Grandiose and deserves a visit even if you're too old for toys (who is?)

In the end after a three day conference with only two effective shopping evenings including a 1 hour midday break I brought home:-

Levis Jeans ($38)
Sports Jacket ($20)
Brand new swatch (the first one the Times Square shop sold) for my missus.
Grey Hooded Sweater
Another sport Jacket ($57)
$200 worth of books from B&N (only 3 of them were computer books)
3 packets 3x5 index cards and a 15 pack colored sticky notes
Hiking boots ($130)
Ralph Lauren Eau de Parfum from Saks (another present)

Now if I tell you that my dad looked pretty amazed when I returned back with 4 bags instead of 2 you're likely to believe me. And also if I tell you that I'm broke to the bones you'll not laugh at me.

With the unbelievable amount of restaurants (see this view from Google Earth) and delis around I couldn't resist the temptation to spoil myself in an American indulgence no matter that I wasn't hungry. In the afternoon I went to a deli/self service and got a couple of Chinese food and noodles in a plastic container which was then weighed and cost $10.21 with a berry juice drink rather cheap considering that there were at least two full plates in my choice. For dinner which I didn't plan to take, I was lured into a sports grill diner with a sign for lobster at $12.75 but then I ended up taking a Times Square burger and a pint of Guinness to feel more American (total cost with tips $25) while watching ice hockey on the big screen. The following day I resisted the temptation to eat to spend more time shopping and on the last day I tried to redeem the conference voucher at a snack place called Cosi, but the $15 voucher didn't validate and I had to pay for a sandwich, bitter iced tea and a chocolate cake.

On a more serious note - the conference was fantastic. On the first day we had a full day tutorial from BJ Robinson, a Microsoft Technical Trainer, about Testing Techniques. Apart from bashing Microsoft we learned the functional testing techniques of boundary values testing, equivalence class partitioning and pairwise testing, and the structural techniques behind code coverage. I think the single most important think that sank down in my brain was that testing is not just randomly bashing at the keyboard trying to find bugs. There actually exist techniques that help you execute exercise the code in a way to potential uncover errors without spending eternity and a half testing. The lecture was very insightful, beginner friendly, informal and very practical - just the thing I was looking for. I worked with my bench mate a lady working at Boeing and we went along quite fine, I wasn't a social beast.

By far the best lecturer of the conference was Robert Sabourin. This hyperactive human has his own very particular way of teaching by shouting out loud, showing us fun videos, spending ½ the lesson telling us a story seemingly unrelated to the subject but very relevant to the whole point of the lesson, pacing wildly in the room and being very informal. The first class was about bug prioritization and the central concept was that - The priority and severity of bugs is CONTEXT DEPENDENT. Priority specifies when a bug is going to be solved and severity specifies how dangerous it is to keep the bug in the system.

I ended up choosing an additional two classes of Rob after being enthralled by his first premier. In this class as we had the opportunity to do some Unit Testing in a team using a Test Driven Development (TDD) approach. Using a post office scenario we had to first create the test cases for the different package types then with the help of some black felt, Velcro and flowchart symbols design the system flowchart from the test cases. Amongst other things I realized that it's not trivial to move to a TDD mindset after using traditional approaches to development for so long.

The rest of the classes I attended to where:-

Recruiting, Hiring and Motivating top testing talent (Feldstein) - Excellent lecturer who shared some great tips on the role of testing in an organization and how testing should be looked at from employers. If everybody thought like this I would already have a job in testing because I met most criteria he specified. Btw, testers must be developers .
Exploiting Web Application Code - SQL Injection (Bodkin) - Informative and enlightening lecture on the importance of application security and the real threats found in web applications. Easy and effective.
Automate Database Testing - Testing and using stored procedures (Sweeney) - I already knew most stuff about stored procedures but the idea of using stored procedures for automated testing was worth the 90 minutes.
How to optimize your web testing strategy (Nguyen) - Nothing to do with actual web strategy and more focused toward management.
Software Endgames - How to finish what you've started (Galen) - Again too much focused on management and the importance of defining goals to attain them.

I will hopefully post my personal incoherent course notes either tomorrow or the day after.

Update:The course notes are here: Testing Conference Notes

Looking for company

Mon, 10 Oct 2005 07:03:29 GMT

If anyone is going to attend the software test and performance conference in NY next month send me a mail. My company is poor enough to send me alone and I wouldn't mind knowing somebody before the conference.

In the mean time these links are going to be useful for the rest of the week.
Pavel's Online Presence - Pavel's software engineering oriented blog
Testing Reflections - The title says it all

Oracle Database Shutdown

Mon, 05 Sep 2005 17:21:54 GMT

Did you ever forget to write "immediate" after shutdown when trying to stop a database? When this happens Oracle waits for all connections to terminate before shutting down which usually takes ages. Oracle obviously disallows the creation of new processes so if you try to shutdown the database again using the immediate keyword you will receive the "Not connected to Oracle" message. To recover from this situation you have to open another session and startup force the database and then shut it down again using shutdown immediate.

Linux Linux Everywhere

Fri, 17 Jun 2005 04:50:23 GMT

At the moment I'm having Linux for breakfast, lunch and dinner in a desperate attempt to learn anything about Linux in preparation for the RHCE course next week. The more I learn the more I realise how little I know, and how unlikely I'm going to pass this exam. At least though its fun learning, and I'm gradually uncovering the huge community of Linux Users which seems bigger and bigger every day. These are some interesting sites I came across these few days.

FrozenTech: Has got loads of Linux live CD's. Live CD's are Linux distributions that run only from a single CD without touching anything of your current OS. I've downloaded a Kill Bill edition of SLAX and Kanotix both of which were very easy to install and use.

Distrowatch: With all these distribution you need something to help you choose. Distrowatch is the answer.
LUG Radio: The site of the most popular Linux Radio show around. Worth downloading some podcasts there's much to learn from hear (including some swear words).
The Linux User Show: Home of the best podcast I've listened to till now. This was the first show but I'm eager to hear what Jon has to offer. Highly recommended
linuxiso.org: Download any Linux Distribution from here
Really Linux: For those who're still new to the Linux world this is a friendly introductory site to this new world.

Re-engineering stale old processes

Thu, 12 May 2005 04:46:26 GMT

Do you want to find flaws in a process in it in your organisation? The solution to this problem is simple try to computerise the system. When you are computerising a system you have to study the current system in such detail and depth that very few other studies can hope to achieve. This week we started the rollout of the Leave Management Information system we've implemented in the last couple of months and we're facing the baffled users of the system who don't know what hit them.

At face value the system serves as a mere paper reducer because it replaces the old leave application forms, however at a deeper level it is being used to enforce the streamlining of a leave process in the organisation. Throughout all these years the leave application and authorisation process has been moulded into dozens of different formats with different rules applied to different departments without any control and rational. Now the system is being used to find out and eliminate the exceptions and loopholes in the process which have now become the norm.

There are several benefits you can gain from introducing an information system, but one which is not commonly acknowledged and adequately quantified is; how much savings does a system bring with by reducing abuses to the old system. Despite the fact that the system was implemented for all the wrong reasons, we're now recognizing some of the more subtle yet more beneficial advantages the system will bring. This is also the first information system that is going to be used throughout the whole company for about 2000 personnel. I'm pleased with the achievement we've obtained as it was no easy task for only two people to undertake in such a short time.