Over the past weekend the online shop of SpaceCat stopped working.

That was the result of some unfortunate decisions and the fortunate situation of being featured. Anyway, that should have never happened, but I re-learned a few lessons.

Problem #1: The shop is our own implementation

I always defend to not reinvent the wheel, but in this case we did. Should we had used a 3rd system such TapJoy or ScoreLoop -which are already integrated-, this should have never happened, partly because their code is more tested, partly because it relies on their servers.

Why did we did it?

Because we wanted to have a shop that was not bounded to just Android Market to be able to distribute the game over other channels such as Amazon AppStore. We integrated PayPal for that. And we had to implement tracking of purchases per user, which is done for you when you use Android Market IAB .

ScoreLoop did not support it. We could have done it with TapJoy, but the initial release did not include that library, since this was originaly designed for Chalk Ball.

In summary, that was the result of legacy decisions that could have been changed later on but we did not want to throw out work away.

Was it worth it?

Hell no! Paypal is a nightmare from the accounting point of view (not entering other discussions about PayPal here) and at the end we did not delivered SpaceCat through other channels, so 100% not worth it.

Problem #2: The shop was hosted on a shared server

Yes, we just put it in on a server we used for hosting other websites. At the time it looked like a good idea and it was not like we were making too many requests. After all, users don’t open the shop that often, they do play, and that does not require the server.

Why did we did it?

Because we did not have a dedicated server out there and setting it up for this sounded like an overkill.

Was it worth it?

Yes, until the moment SpaceCat was featured. We did spare the cost of a dedicated server for 6 months, while the game was not popular enough.

We should have prepared the migration at the very moment SpaceCat was featured, but honestly I did not see this comming.

Problem #3: The URL of the API was hardcoded

Yes it was. What else do you want me to say?

It required an update of the game to get it fixed. Should this happened to an iOS game we could have been unable to fix it for days.

Why did we did it?

Good question. Because I did not think of it at the moment. No excuse.

Was it worth it?

No, it could have been a much worst scenario. I am glad it just required to do an update and that the continuous integration system was properly setup so we could do it quickly.

What happened exactly?

Once SpaceCat got featured, we started having around 50,000 new users a day, 50% of them opened the shop. So after a week, that database had grown from 50,000 to 300,000 rows.

In addition to that, we started having more than 4 requests per second hitting that table. While this should not be that bad, it was too much for a shared server and the account was suspended.

The fix

The solution was to set up a dedicated hosting so we can have total control and maybe install other tools that are not mySQL in the future if it is required.

Once the host was delivered (and the company did a very good job by completing it in less than 5 hours) installing the code of the shop and migrate the data was done in less than an hour.

Then, we had to reconfigure SpaceCat and publish an update on Android Market. I’m glad to have jenkins in place.

All in all the system was down only for 12 hours, which is not that bad.

Even more

In addition to that, once the dedicated server was up, I noticed that the table was being slow even if the server was not heavily loaded. I looked again at everything and I noticed that it was missing one index.

How did that happen? I don’t know, I was sure to have it in place. Probably because we lacked a proper deploy method and that index was only on the dev environment. Probably because of the lack of stress tests; until 300.000 entries & 4 request per second the table was not performing that bad and we never noticed it. This is, again, a problem of implementing your own solution.

Now the system is good and working, but it has given me too many headaches over the past days so, again, if you are going to create an app that uses a service and it has the potential to be used by many people, use a 3rd party solution if it is available.

This lack of good apps for this game may be astonishing at first. Magic is a popular game and Android is a popular platform. You should expect a few good mtg apps. WotC (Wizards of the coast, the makers of the game) is going to release an official app soon, but that has not been the case for the past years, so why there are no good free mtg apps?

Well, the fact is that there were good Android mtg apps, not to long ago M:tG Tracker was a free app and you could upgrade to the paid version. The app was as good as to get 3rd place on the Best Reference category of Best App Ever 

And that was not the only one. There were a few more, like MtG Tutor with a lot of features including a very good advanced search or MtG Depot with a very nice interface. Also, on the paid side there was Deck Forge, which had a lot of features and, I’ll admit it, took the lead of innovation in many aspects before I did.

But all those apps are no longer there, and the reason is that WotC have sent copyright infringement notices to google, asking  to remove them. Why now? The only reason I can think of is that they are going to release their own app and they want to wipe out competition.

Why making a better product if you can have “the only alternative”?

Let me clarify that.

Don’t take me wrong, there were material from WotC and is their right to claim the suspension of apps, but up to now this apps were made with good will, and were adding value to their product. Also, there was never the intention to talk, none of the developers of those apps were warned or given a chance to remove the copyrighted material from their apps.

Also, I’ve seen the screenshots and the app looks promising, so I don’t think they are going to do a bad product. But why should you risk your market share anyway? Maybe for the sake of more than 200.000 people that were using apps that are no longer available while the official alternative has not been released yet.

Make no mistake, WotC is not releasing the official app to help players, it is going to do it to make money. That is the only conclusion I can reach from their actions.

At this point: Why is M:tG Tracker (formerly pro version) still available?

When I got the free version suspended, I went through the app and I removed all the copyrighted material, the fonts, the mana icons, the tap icons, etc. with the hope that it will prevent that one from being suspended… it has worked so far.

I am trying to clarify what exactly can be put and what can not, and also asking for the procedure of licensing the material so I don’t have to worry about it in the future. While this is not settled, I can not afford to continue with the features I had in mind, risking that all that work may be trashed at anytime.

Thanks to all the users that have been supporting me and the other developers. If someone asks you “Why are no good mtg free apps for Android?” you can now tell them why.

I approach any Android book with low expectations. The ecosystem is changing too quickly and too often that is impossible for a traditional media like printed books to catch up properly. However, I gave this one a try.

Of course, there is no mention to the Compatibility Library, the Action Bar or Fragments, which are Honeycomb / ICS specific. I should have been shocked if there were, because they are very new, but I consider them myself the starting point of Android development as of today.

On the other hand, development up to Gingerbread is very well structured, starting with the basics of Activities and Layouts to keep going quite deep onto Services and Content Providers. They go even further, talking about Aidl and NDK, which I had no need to use yet, but it is good to have them referenced.

There are two variants of introductory technical books: the ones that teach you general concepts, and the ones that are structured around an example. I personally prefer the second type, it helps me seeing real applications of what I am learning and is easier to focus on the content. This book is one of those, which I think is one of its best points.

I recommend the book to get an overview of Android or as your first Android book. It will serve you for quite long in the learning process -I even learned a couple of new tricks myself-. But if you want to be a really good android developer you want to complement the lessons of the book with the Compatibility Library, the Action Bar and Fragments.

These are the slides from the talk “The importance of Visibility” which I delivered during Appsterdam lunchtime lectures and DroidCon.NL

I was expecting them to be available on Droidcon, but since that has not happened yet, and people asked for them, I am putting them here.

These are the slides of the talk I did earlier today. They are not the same without the spoken part, but the video will be available at the Appsterdam site eventually.

Thanks to everyone that attended, I had a great time today.