New Appstore User Review Flaw Emerges

Apple has recently made major improvements to the user review system for apps, making it generally much fairer and more useful to customers.

Reviews are now labeled with the date on which they were written, the version of app being reviewed, and there is a graphical view of the distribution of ratings, making it much easier to see the overall response of users to the app.

But there are still some ways in which the system can be abused or gamed, as I have only just discovered myself, after noticing that despite Oz Weather’s typically high user review scores, the 3 reviews shown to all users on their first view of the app information (in the Australian app store) happened to be 1 and 2 star reviews, with correspondingly negative sentiments. ūüė¶

Oz Weather Reviews

The reason that these unrepresentative reviews are being shown persistently is because Apple considers them to be the “most helpful” ones according to the algorithm that they apply to their peer review system ie. where users rate other users’ reviews.

But closer examination showed that those particular reviews had received feedback from just one other user ie. they were marked with the statement “1 out of 1 customers found this review helpful“. And looking down the list of 31 reviews of the current version it was apparent that the low ranking reviews had all been found “helpful” by 1 out of 1 users, whereas all the high ranking reviews (many more of them of course!) had all been found helpful by 0 out of 1 users (ie. found “unhelpful”).

Given this pattern, what’s to bet that all this was the work of just one person?

Most disappointingly, a review I left myself was similarly marked as unhelpful, thus making it virtually invisible to most users. I had left this for the purpose of trying to help those users who had app problems but not realised they could just email me for support. And in it, I clearly identified myself as the developer. (As I had to leave a star ranking, it had to be 5 stars – what would user’s have thought of the app if it’s own maker had ranked it less?!)

Own Review

This problem with user reviews of reviews has been there since day one of the app store, but due to the fact that numbers of users, their reviews, and their reviews of other reviews was always growing, it wasn’t a big issue for long. Once enough users had been acquired, the chance that a single user’s feedback could affect things this dramatically became negligible.

But with Apple’s new review system in place, whenever any app update is issued, the reviews for the current version only start at a count of zero, and the prospect of this distortion and/or possibility of gaming the review system by just one (or very few people working together) arises again, at least until the number of user reviews has grown enough to drown it out again. Given that many app developers do release updates relatively frequently, this makes it into a real issue – one that happens to have been made much worse as a consequence of the new review system.

So how could this be solved? Well Apple has already solved a similar issue in relation to the average star rating ie. they no longer give a star rating at all until a sufficient number of reviews have been received. So perhaps the solution is simply to ignore the helpful/unhelpful ratings until a sufficient number have accrued – at least several, preferably more.

Another approach would be to limit the number of helpful/unhelpful ratings that any one user can give – thus preventing them from sullying the entire complement of existing reviews in one go, as appears to have happened to Oz Weather in this case.

Yet another would be to look for patterns of user response eg. if a user consistently rated all low star reviews as helpful, and high star reviews as unhelpful (or vice versa), then their ratings are simply ignored. Of course this might require some advanced algorithms to make it work, which may be unrealistic for now, but it doesn’t hurt to think about them!

Appeal

Because a solution to this issue is not going to be implemented or appear overnight (despite Apple’s many other talents!), I would like to appeal to any existing Oz Weather users to go into the Oz Weather app store entry in iTunes (you can use this link) and use the helpful/unhelpful feature to rate some of the existing other reviews. I would ask you to do this with integrity and honesty ie. please don’t just do it for the sake of panning the bad reviews or glorifying the good ones – rather do what feels right and appropriate to make the review rating more realistically representative of your own opinion of the app.

Postscript

By the way, those bad reviews on the front page are no-doubt quite honest. The app can (rarely) start crashing, and the only solution is to delete then re-install the app, and I do believe that it is very important to listen to complaints, and think carefully about what you could do to avoid similar complaints in future.

To this end, I have recently rooted out the likely cause of any crashing, and also put in a crash recovery mechanism just for those 1 in 10,000 type scenarios. Hopefully this will mean that eventually there will be no 1 star reviews at all – and then it wouldn’t be possible for anyone to game the helpfulness of reviews either! What? Me? An idealist? Well maybe. ūüôā

Advertisements

Estimating App Sales From Rankings – Part 2

In an earlier post I showed a graph of Oz Weather’s daily sales levels versus its ranking in the Australian iTunes app store, and then extrapolated the findings to get a very rough estimate of global sales versus iTunes US app store rankings.

Australian Sales vs Rankings

As Oz Weather has now been selling for more than 3 months, and has had an extended period ranked #1 in the Australian store, so I am now in a position to make an enhanced analysis. But note that there were some anomalous sales figures over the Christmas period (24th Dec to 31st Dec) due to three unusual circumstances:-

  • the surge in overall Christmas sales
  • the fact that I discounted the app over that period
  • thirdly due to a major loss of sales when the app store broke down temporarily

The graph below shows the¬†entire history of sales, minus that week’s worth of ¬†“anomalous” sales data.

Sales Rankings Australia

I tried fitting logarithmic, exponential and power curves to the data – the power curve gave by far the best fit, with the approximate equation

Daily Sales = 425 * Ranking ^ -0.5

This equation can be used to give a reasonable estimate of sales of paid apps in the Australian iTunes app store using the overall ranking (not the ranking in any individual category such as entertainment, lifestyle, weather etc). However note that there is a wide range of variation about this mean trend line on a day to day basis. For example while Oz Weather was ranked #1 sales varied between about 260 and 620, and at rank #2 from 190 to 420.

World Sales versus Rankings

We now have a good idea of the apponomics of the Australian store, but what about world-wide sales? To begin to answer this, I have gathered as much data as I could find  giving US store app ranking versus daily sales. The main sources I found were

Using data from these sources Рsome real sales figures and others inferred and estimated, I constructed the following graph. 

Sales Ranking World

The main assumption here is that US rankings reflect Worldwide rankings reasonably well, which although not necessarily true for specific apps, is probably true on average. Also note that, to clarify the picture, I did exclude some of the higher figures given for #1 ranking – up to 50,000 or more supposedly. I suspect that sales at the #1 spot will be much more variable than lower ranked slots, so it seemed sensible to include only the lowest figures that resulted in a #1 ranking, for this purpose.

As before I tried fitting logarithmic, exponential and power curves to the data Рand again the power curve gave by far the best fit, with the approximate equation

Daily Sales = 15000 * Ranking ^ -0.75

This equation can probably be used to give a reasonable estimate of sales of paid apps in the US iTunes app store using the overall ranking. It roughly confirms my earlier guess that world sales were typically 30 times more than Australian sales. In fact, for the given equations, the World/Australia sales ratio varies from 35 (#1 ranking) to 13 (#50 ranking).

Given that there is no reason to suspect any major difference in sales dynamics, one might have expected the ratio to remain similar at all rankings (ie. the power factor to be the same in each equation). However, note that the Australian curve is based on data only going down to #17 ranking, whereas the US curve has data down to #100 ranking. I therefore suspect that the US curve power factor is more realistic for the long tail, and that the drop-off in sales levels for Australia well below the #20 ranking might be somewhat less than that predicted by the given equation.

So there you have it – the best that I can do with the available data. As more data becomes available I will add it in to the mix, to see how much further this can be refined.

Oz Weather Apponomics – Part 3

Following on from Apponomics Part 1 after the first two weeks of sales and Apponomics Part 2 after one month of sales,  Oz Weather has now had just over three full months of sales. Here is a complete history of the daily unit sales numbers, up until 5th Feb 2009.

Sales to 5th Feb 2009

The red and yellow bars show Australian and overseas daily unit sales at AUD$2.49 (US$1.99), and the blue and orange bars are at the temporary sale price of AUD$1.19 (US$0.99).

I’ve annotated the graph with some significant events which did (or might have been expected to) affect sales volumes. These events were as follows.

  • Release of v1.1 Update – saw an immediate jump in sales and rise in ranking from #6 to #3
  • Release of v1.2 Update – sales flat
  • Xmas Sale – decided to try a brief sale over Christmas period due to falling sales & ranking having dropped to #9
  • App Store Broken – a couple of days when customers were unable to purchase Oz Weather
  • Major Heatwave in Parts of Australia – consistently high sales level, bringing ranking to #2
  • Release of v1.3 Update – major app enhancements and an eye-catching new icon pushed the Australian iTunes paid app ranking to #1 – where it remains at time of writing

Summary of Sales & Revenue

  • Total units sold:¬†23,800
  • Average sales per day: 245
  • Approximate revenue:¬†AUD$35,000 (US$22,500)

Upgrade Statistics

  • Total units sold prior to v1.3:¬†19,800
  • Total upgrades to v1.3 in 9 days since release:¬†13,350
  • Upgrade ratio:¬†67.4%

Costs

Although I indicated in a previous blog entry that I had broken even on 19th December, I did invest considerable extra development time and effort into the latest app update, and also into further professional design work . Consequently I estimate my total nominal costs so far to be around the AUD$25,000 (US$16,000) mark. This leaves a nominal profit to date of AUD$10,000 (US$6,400).

Thoughts and Comments

Although Oz Weather has been substantially profitable (for a sole developer) over this 3 month period, the apponomics depend to a large degree on how much ongoing additional development effort is required to maintain its level of sales. So far, Oz Weather sales levels have not dropped off in a way that seems to be typical of many other apps. It remains an open question as to how much of this is due to the additional development effort I have made versus what would have happened without this frequency and depth of app updates.

Competition is of course another important factor. Whilst Oz Weather has taken the major share of the Australian Weather apps market since its launch, the visibility of its success is an encouragement to competitors. The developers of Pocket Weather have made no secret of the fact that they would very much like to regain the position they had in the app store before Oz Weather jumped in, and appear to be investing considerable time and effort into fighting back with their own major updates. There also remains the possiblity if not likelihood that some other major player in the Australian weather scene will want to launch their own app, backed with pre-existing infrastructure which will give them some competitive advantages.

So despite all the stories you might hear about overnight successes in the app store, overnight falls from grace are probably just as frequent. In reality, it is just a reflection of life as a whole. Hard work and listening carefully to your users will maximize your chances, but the only real certainty is uncertainty! Stay tuned for the next instalment.

Oz Weather v1.3 Released – Hits #1 Ranking

iTunes Rankings

A couple of days ago the latest update to Oz Weather – v1.3 – was approved for sale by Apple. There are some major enhancements. A lot of work went into this one!

Shortly after release of the new version, Oz Weather finally jumped into #1 spot in paid app rankings in the Australian iTunes store. I’ll cover the effects of this on sales figures in a future blog post. ūüôā

From the appearance angle, there is a striking new icon (emphasizing the Australia-only weather focus of the app) and a range of fun new background color themes to choose from – just in case anyone tires of seeing the same app colour scheme day after day – as you might only in app that typically gets daily use.

Oz Weather v1.3

As with previous improvements to the app appearance, I’ve got Peter Fellows to thank for this great design work.

From the functionality point of view, there are several big advances too. Although I’d covered most of the available Bureau of Meteorology forecast locations in previous versions, this version covers all available forecast locations, and for each one will display current weather from up to the 6 nearest observations locations in each case – accessible via the famous side-swipe action that is so often admired in the iPhone’s much more basic built-in weather app.

There is also a new “Capital Cities” item which allows you to see forecasts for the 7 main capital cities all on one page, and to side-swipe through the latest observations for each one, or tap on a city’s forecast line to jump straight to its current observations.

Capital Cities Adelaide Observations

There are also a range of other new features and enhancements like

  • Sunrise and sunset times for today
  • Forecast min and max temperatures displayed on current observations page
  • Dates added to day name on forecast page

You can choose whether or not to enable these new features by going to your iPhone’s home page, selecting the “Settings” icon, and then scrolling down to the “Oz Weather” entry in the list of application settings.

I’ve actually had a lot of fun building this, and the jump is rankings and sales is very gratifying. But it remains to be seen how durable this will be. There is currently an extreme heatwave hitting Melbourne and Adelaide and this could account for a general increase in weather app sales. In fact competing apps have also done very well in the rankings at the same time despite having had no recent updates or icon changes, which does seem to suggest that the unusual/extreme weather is affecting things. But I’ve also had some very positive feedback about the new design. WDYT?

Broken App Store – Update

Since my last post about app store re-pricing problems over Christmas, I’ve found evidence that the problem has affected quite a few others too – some publishers losing sales of many thousands of dollars.

There is a thread in the Apple Developer Forums (only accessible via login to registered developers) in which many publishers are venting their frustrations – with one quantifying their losses so far at over $10,000.

As for Oz Weather, the strategy of reverting the app price back to the discounted level seems to have worked. Sales started again almost immediately Рpurchases were no longer being blocked.  Yesterday 415 units were sold, up from the low of 17 units after I attempted to change the price, and the app ranking rose back to #2 paid app in the Australian app store, up from its low of #23. But of course now the app price is stuck at the discounted level Рif I want to have any sales at all that is!

Sales ranking

 

Key dates on this graph are as follows.

  • 23rd December – ranking #9 – after gradual decline over past week, ¬†introduced short-term sale pricing
  • 28th December – ranking #2 – ¬†short-term sale finished, reverted to normal pricing
  • 30th December – ranking #23 – after numerous customer complaints about being unable to purchase, re-introduced sale pricing
  • 1st January – ranking #2 – but stuck with sale pricing

It seems likely to me that the reason users were being blocked from buying was due to mismatch between the price cached on the user’s machine/iphone and the price on the purchase server. So when I re-lowered the price, the mismatch no longer occured. It seems that a major de-synchronisation has occured between different components in Apple’s app store system.

Right now, the Oz Weather app description that users are seeing is still the one I had with an Xmas Sale message in it. Once this decription gets updated to the correct, more recent one, I might be willing to risk raising the price again. Also, the number of reviews has been static for about 10 days, despite high sales over Xmas. So watching for when the number of reviews suddenly jumps up with all those missing reviews may be another sign to look for to show that the problem is being resolved.

My advice to other publishers? Don’t make any app changes via iTunes Connect until this problem is resolved – unless it is perhaps to try some harmless subtle text change to your app’s description so that you can test whether or not the update actually filters through the system to the app store.

Oz Weather Xmas Sales – Is the App Store Broken?

You may already have heard from some other sources that iPhone app sales were very strong over the Christmas period, possibly enhanced by many new iPod Touch devices being delivered from Santa’s sleigh.

I can certainly confirm that it was a strong sales period for Oz Weather Рalthough the picture is complicated by the fact that I temporarily dropped the price for the Christmas period. Here are the daily sales figures to date Рfrom 1st November to 29th December.

salesgraph_2008_12

The red and yellow colors show units sold at the normal price of AUD$2.49 (US$1.99). The purple and orange colors show units sold at the discounted price of AUD$1.19 (US$0.99) – a sale which started on 24th December (day 54) and finished on 28th December (day 58).

Key Points

  • The¬†break-even point for the Oz Weather project was reached on¬†19th December (day 49) – ie. cumulative net revenue of¬†AUD$15,500 (US$10,850).
  • After ranking as high as #2 in Paid Apps category of the Australian app store from 5th to 9th December, ranking fell gradually to #9 on 23rd December. Hence my decision to run a price sale over the Christmas period.
  • Christmas Day had the¬†highest ever¬†sales of¬†451¬†units – followed by similar levels on 26th and 27th December. The app ranking rose to #2 again. This is the celebratory part of this article. ūüôā
  • The last item on the graph (item 59 = 29th December) shows daily sales of just 17 units. This is not an error, and not just a partial day’s sales. It was the actual full day’s sales! Ranking has fallen to less than #10 – but to exactly what level is not possible to ascertain, due to problems in the app store. This is the disappointing part of this article. ūüė¶

Major App Store Technical Problems

Did people suddenly stop wanting to buy Oz Weather because the price discount had ended? Did Oz Weather suddenly become unpopular? Did a competitor suddenly release a much better app?

The answer to all these question is no. Many people are still wanting to buy Oz Weather, but are unable to do so, due to problems in the app store. On the 29th and 30th December I received 7 emails from potential customers wanting to know why they couldn’t purchase it. This is very likely the tip of the iceberg. The vast majority of customers simply wouldn’t bother trying to contact the app publisher when they receive a message like the following (sent to me by a customer).

itunesna

So what is going on here, Apple? I have tried contacting Apple support, and so far have only managed to get an automated reply saying:

Please plan for a delayed response time from the iTunes Connect Support team from December 23rd through January 5th as we will be short-staffed over the holiday break.

It is apparent that the app store problems go beyond just the Oz Weather app. The Australian store paid app rankings were “stuck” for several days, and showed widely different results depending on exactly where you look at the rankings.

  • iTunes home page – showed Oz Weather #6 – this morning dropped off the table
  • iTunes app store main page – showed Oz Weather #4 – this morning dropped off the table
  • iTunes top 100 paid app rankings – showed Oz Weather #2 this morning – this afternoon had dropped to #23
  • iPhone top 25 rankings – showed Oz Weather #6 – still now at #6

I am not generally one to jump on the hindsight bandwagon – but in this case I am very tempted to say to Apple that “you should have realised that Christmas was going to be your busiest period, and employed extra staff instead of sending them all on vacation”.¬†Now I am in a position where, due to this Apple glitch, I am probably losing several hundred dollars per day in sales, and my app ranking has artificially fallen from #2 to way off the radar¬†– and I can’t find any way to get any response from Apple.

My questions to other app developers

  • Am I the only one having such serious app store problems?
  • If not, then do these problems affect only apps which are undergoing price changes?
  • Do these problems affect only the Australian app store?
In the absence of any way to get immediate support from Apple, I’m really hoping that someone who reads this blog can provide some useful feedback!

Late Addition (31 Dec): The following link to a PCWorld article which indicates that there have also been other app store/iTunes problems this Xmas.

Late Addition (1 Jan): I am not alone. The app store IS broken. See the comment from VPool.

Oz Weather Progress

I’ll save exact sales figures for a later post, but Oz Weather has moved gradually up the paid apps ranking in the Australian iTunes app store to 3rd spot.

Oz Weather App Rankings

As icing on the cake, Oz Weather also appeared for the first time today in the “What’s Hot” rankings in 8th spot. This means that Oz Weather now appears twice on the front page of the Australian iTunes store. It seems clear that this can only result in a sales boost – as it is being put right in everyone’s face, so to speak.

iTunes Front Page

I have read in some postings on the Apple iPhone developer forums that it seems a bit unfair that top ranking apps by sales may also appear in the what’s hot rankings. As a beneficiary of this policy, I’m obviously not going to complain about it, but I do tend to agree anyway. I’m sure customers would prefer not to see the same app more than once on the front page.

So it is all great exposure and advertising – and yet all this without even a cent of advertising expenditure on my part. I guess that’s what makes the 30% Apple cut seem so very reasonable – especially when you compare it with the time and energy you need to invest when you are starting up cold with a new app or website in the “real world”. I’m betting you’d need to give at least a 50% cut to any marketing/sales partner you joined forces with.

The other benefit of the app store model is that developers don’t need to have a dual focus on development and marketing/sales. Instead they can just focus on what they presumably do best ie. develop good quality applications. I’m not saying that some additional advertising and promotion won’t help your sales, but if that’s not your expertise, using the app store to sell your products may allow you to achieve some success without it.

PS – Almost forgot to mention it, but Oz Weather v1.1 appeared in the app store on 20th November – about 4 days after it was submitted to Apple for approval (in contrast to 2 days for the first version). This updated version is in response to user requests for more radar coverage, and a number of other smaller tweaks and enhancements. In fact there are now around 150 different radar locations and ranges covered in all. It makes for a bit of a hit on the Oz Weather server, but user feedback has been very positive, so well worth the extra time and effort it took.

Brisbane Radar Radar Options