New Appstore User Review Flaw Emerges

Apple has recently made major improvements to the user review system for apps, making it generally much fairer and more useful to customers.

Reviews are now labeled with the date on which they were written, the version of app being reviewed, and there is a graphical view of the distribution of ratings, making it much easier to see the overall response of users to the app.

But there are still some ways in which the system can be abused or gamed, as I have only just discovered myself, after noticing that despite Oz Weather’s typically high user review scores, the 3 reviews shown to all users on their first view of the app information (in the Australian app store) happened to be 1 and 2 star reviews, with correspondingly negative sentiments.😦

Oz Weather Reviews

The reason that these unrepresentative reviews are being shown persistently is because Apple considers them to be the “most helpful” ones according to the algorithm that they apply to their peer review system ie. where users rate other users’ reviews.

But closer examination showed that those particular reviews had received feedback from just one other user ie. they were marked with the statement “1 out of 1 customers found this review helpful“. And looking down the list of 31 reviews of the current version it was apparent that the low ranking reviews had all been found “helpful” by 1 out of 1 users, whereas all the high ranking reviews (many more of them of course!) had all been found helpful by 0 out of 1 users (ie. found “unhelpful”).

Given this pattern, what’s to bet that all this was the work of just one person?

Most disappointingly, a review I left myself was similarly marked as unhelpful, thus making it virtually invisible to most users. I had left this for the purpose of trying to help those users who had app problems but not realised they could just email me for support. And in it, I clearly identified myself as the developer. (As I had to leave a star ranking, it had to be 5 stars – what would user’s have thought of the app if it’s own maker had ranked it less?!)

Own Review

This problem with user reviews of reviews has been there since day one of the app store, but due to the fact that numbers of users, their reviews, and their reviews of other reviews was always growing, it wasn’t a big issue for long. Once enough users had been acquired, the chance that a single user’s feedback could affect things this dramatically became negligible.

But with Apple’s new review system in place, whenever any app update is issued, the reviews for the current version only start at a count of zero, and the prospect of this distortion and/or possibility of gaming the review system by just one (or very few people working together) arises again, at least until the number of user reviews has grown enough to drown it out again. Given that many app developers do release updates relatively frequently, this makes it into a real issue – one that happens to have been made much worse as a consequence of the new review system.

So how could this be solved? Well Apple has already solved a similar issue in relation to the average star rating ie. they no longer give a star rating at all until a sufficient number of reviews have been received. So perhaps the solution is simply to ignore the helpful/unhelpful ratings until a sufficient number have accrued – at least several, preferably more.

Another approach would be to limit the number of helpful/unhelpful ratings that any one user can give – thus preventing them from sullying the entire complement of existing reviews in one go, as appears to have happened to Oz Weather in this case.

Yet another would be to look for patterns of user response eg. if a user consistently rated all low star reviews as helpful, and high star reviews as unhelpful (or vice versa), then their ratings are simply ignored. Of course this might require some advanced algorithms to make it work, which may be unrealistic for now, but it doesn’t hurt to think about them!

Appeal

Because a solution to this issue is not going to be implemented or appear overnight (despite Apple’s many other talents!), I would like to appeal to any existing Oz Weather users to go into the Oz Weather app store entry in iTunes (you can use this link) and use the helpful/unhelpful feature to rate some of the existing other reviews. I would ask you to do this with integrity and honesty ie. please don’t just do it for the sake of panning the bad reviews or glorifying the good ones – rather do what feels right and appropriate to make the review rating more realistically representative of your own opinion of the app.

Postscript

By the way, those bad reviews on the front page are no-doubt quite honest. The app can (rarely) start crashing, and the only solution is to delete then re-install the app, and I do believe that it is very important to listen to complaints, and think carefully about what you could do to avoid similar complaints in future.

To this end, I have recently rooted out the likely cause of any crashing, and also put in a crash recovery mechanism just for those 1 in 10,000 type scenarios. Hopefully this will mean that eventually there will be no 1 star reviews at all – and then it wouldn’t be possible for anyone to game the helpfulness of reviews either! What? Me? An idealist? Well maybe.🙂

1 thought on “New Appstore User Review Flaw Emerges”

  1. I did notice those reviews myself and did wonder why a 5four star app had those reviews at the top.

    I also wondered why they wer complaining about. Never mised a beat for me.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s