Azure CPU spike all Sunday afternoon and evening

The Monster Cache server is hosted on Microsoft Windows Azure.  I made that decision for a few reasons:

1. Initially I was just going to use my existing Lanitek server, but I have another important web site running there, GPRA Upload.  Once I had an issue on Monster Cache where I wasn't closing some database connections and it brought down not only Monster Cache, but GPRA Upload too.  That convinced me to get Monster Cache its own server.
2. At the time, I had just recently attended a seminar on Windows Azure, so I guess I was curious to try it out.  And they were offering a 90 day free-trial (or maybe it was 180 day?), so that was good since we were several months away from being ready to launch.
3. I had, and maybe still do have, a hope that if Google or Apple ever promote the game then there will be a spike in traffic, and then I could easily increase the server capabilities on a short-term basis, if needed.

I haven't been super happy with Windows Azure for these reasons:

1. Its more complicated to use than a normal web server.  Some things that took extra time that I can think of off-hand are web Sessions, file uploading, and making backups.
2. They keep changing the features. When I signed up, they had pretty decent tech support.  Now you have to pay extra for tech support, with the minimum plan being an extra $29 / month
3. It hasn't worked perfectly, I've had a couple instances of long down time that I consider their fault.

One instance of down time was yesterday.  I noticed in the afternoon that my Monster Cache app wouldn't log in, but I thought it was my phone's issue.  When I tried the web site at 9 pm that night, I found the web site wasn't even coming up, but just failing to respond.  I started looking for support options, and that's when I found that  I no longer had tech support, just billing support.  So then I looked for a way to "reboot" the app, since there isn't really a lone server to reboot.  That's when I found the CPU monitoring tool.  It showed that Monster Cache had been using 86% CPU since mid-afternoon.  Here's their chart:

That was very strange since I know that our traffic is VERY low, like only dozens to hundreds of hits per day.  There was an option to reboot an instance of a role.  I only have one instance, so I rebooted it at about 10 pm.  You can see that the CPU went back down to 0% and its been working fine so far today.

Monster Cache European Launch

Once Apple approved the latest Monster Cache version on iOS, we were ready to launch in Europe!  I'd already generated all the monsters a couple months ago or so.  I tried to use about the same monsters / square mile proportions as the U.S..  But when using the monster viewer, it still seemed like there were too many empty patches, so I added a lot more level 1 monsters than my initial calculations (And around the same time I generated another million level 1 monsters for the U.S., to try to reduce empty spots in U.S. as well).

To do the launch, I just went into the app configuration areas in both Android and iOS and selected all the new countries in which I wanted to release.  It was pretty much that easy, after all the other preparations had been made in the application itself.  Oh yeah, I also had to add the new countries to our "Supported Countries" table in the database so that they would show up on the registration form for people to indicate where they are from.

Next was promoting it.  I only had two leads there.

1. When we first made the beta and the Vimeo video a year ago, a German blogger had found Monster Cache and blogged about it.   Unfortunately, Monster Cache wasn't ready for Germany at that time.  Now that it was, I tried to comment on that blog post to let them know, but the comment never got approved.
2. As a result of that post, 3 German people had signed up a year ago during that beta.  I had had to email them to tell them we weren't ready for Germany.  So now I was able to email them again and let them know it was ready!  But I didn't hear back from any, nor did it seem like any started playing.

Even though both of those ideas seemed to fail, we started getting sign-ups from Germany and the UK.  As it stands now, we have 9 German players on the Legends board, and one from the UK.  Current Legend board total is 172 players with 2693 monsters found.  And we got our first German In App Purchase too.

Here is the full list of European countries for which I generated monsters and released the app, along with their approximate square miles, which I think I got from Wikipedia.  The countries I chose were essentially based on wanting to get the biggest Western European countries and then including other countries that fell within the large box that bounded all of those.  Not sure why Apple did not provide the capability to release in Boznia and Herzegovina.

UK	94060
Ireland	32595
Spain	194897
Portugal	35655
France	211209
Germany	137847
Belgium	13120
Netherlands	16485
Switzerland	15943
Austria	32383
Italy	116304
Slovenia	7819
Croatia	21829
Czech Republic	30365
Boznia and Herzegovina	19741	Android only

Android 4.4 and iOS 7

The European launch got sidetracked by Android 4.4, aka Kit Kat, and iOS7.  It all started when....November 12th I did a presentation for the Google Developers Group here in Kansas City.

A few people in the audience downloaded the app.  Since these were avid Google users, a couple of them had the new Android 4.4 installed on their phones.  They quickly learned, and showed me, that the app crashed on the registration screen right when you tried to open the dropdown menu that listed state names.  I decided I better get that fixed before I opened the app up for wider distribution.  Since it was an OS-specific issue, I figured it might be fixed by updating PhoneGap.  I stopped updating PhoneGap when I realized it was going to break my In App Purchasing, so that wasn't a great option.  Instead, I just put in a hack so that Android 4.4 replaces the drop down list with a type-in text field :-)

I made one other small bug fix to update a user's view range right after binoculars are purchased, then I released that version to Android.  I then built it on Apple and submitted that version for iOS.  I soon got an email from Apple that said "Missing recommended icon file - The bundle does not contain an app icon for iPhone / iPod Touch of exactly '120x120' pixels, in .png format".  And there were a couple other sizes.  Turns out they are all new sizes for iOS 7.  We've already created sooooo many different size icons for Android and iOS, it was frustrating to have to make more. But our designer, Kent, got them to me pretty quickly, then all I had to do was get them in the bundle file.  I couldn't remember how to do that, but I found this post on Stack Overflow, which helped.  I resubmitted this afternoon, and no email yet about missing icons, so knock on wood, I think it worked.  Made it just before Apple's December 21st cutoff for submitting apps before they take a week break of not allowing new submissions.  That's from their email that said "iTunes Connect will be temporarily unavailable from Saturday, December 21, to Friday, December 27, 2013."

So now I'll wait until the iOS app is approved, and then on to Germany!  Current Legends count is 158 with Whopper on top with 743 monsters!

Preparing for European Launch

Our ultimate promotional goal is still to get a featured spot in the Google Play store or Apple App Store.  When/if that happens, then hopefully we'll see a big spike in number of downloads and signups.  that number will, of course, be proportional to the number of people who see the promotion.  So in order to make the largest number of people see that promotion when/if it happens, we want Monster Cache to be available in as many countries as possible at that time.

The only reason we haven't made Monster Cache available outside of the US up until now is because I was worried about that adding a lot more monsters to the database would slow down the performance.  We already have 21 million monsters in the database for the US.  The queries are pretty quick, I have indexes built for the main search parameters.  Still, I wasn't sure how many more millions it would take before slowing down.  So I moved the monster searching query to a stored procedure to try to speed it up.  Its hard to tell how much it helps since the query takes less than a second, but I don't think it can hurt.

A lesser consideration is the space needed in the database. In SQL Azure, we pay by the GB, so adding another few million monsters might make it an extra $5-$10 / month.

And last, since we can't really test out the new monsters I'll be placing in Europe, I wanted a way to be able to view the placements of new monsters.  So I built a new monster viewing administration tool.  Its almost the same Google Map code lifted straight from the game, but I can move anywhere I want, zoom however I want, and see monsters of any level.

Now that the monster viewer and stored procedure are in place, I'm almost ready to generate the monsters!  Its a bummer that the timing has worked out such that its now middle of Fall.  Not sure how many people will want to roam around looking for monsters if its cold and snowy soon.  But at least its ready so I can generate them at any time now.

The signups from the first round of emails to geocachers seem to have stalled now.  We're at 151 users on the Legends board from about 493 total signups (31% of signups find one or more monsters).

Latest Purchase Statistics

Our campaign to geocachers combined with our decision to increase the size of the coin packages is paying off!  In the last week, we've had two people purchase our Unlimited Nets package for $14.99 and a few more purchases at $11.99 and $3.99.  With server costs of $30 a month, it looks like we will be profitable this month!  We have to take 30% out of that for Google and Apple, and one person bought the exact same $11.99 package twice in a row and may ask for a refund if that was a mistake, but even then we should still be close to profitable.

Some stats so far....
473 signed up players
$136 total purchases

after 30% to Google and Apple that leaves

$95.73 for us, which is
$0.20 per signed up player

That's not bad if we can get our registered users into the tens of thousands.

Feedback from Geocachers

Its been exciting for Monster Cache the last few days, after we sent emails to lots of geocachers.  We've had about 86 sign ups so far, and 25 of those have found a monster so far.  That's about 30%, which is about average, maybe a bit more.  So 136 on the Legends board now.

One critical thing an iOS user found right away was in the Store.  Since our in-app product of "Unlimited Nets" was rejected in iOS, I had just added more and bigger coin options.  I didn't realize that made the list so long that it scrolled off of the bottom of the screen on iPhones.  So the "CLOSE" button was not visible, so if you decided not to buy coins, you couldn't close that window or go to another tab, so you were stuck and had to exit the app.  Ideally, I would like to keep all the coin options and have that window scroll, but that means a new app version.  So for now I just disabled some purchase options.  Turns out there was the same issue on Android for my small phone, so I did the same thing on Android.

I also noticed that referral emails were coming from my kcwebprogrammers.com address, which wasn't good.  Google wasn't letting me authenticate with lanitek.com, and monstercache.com was still forwarding from GoDaddy.  So I had to set up a new email address on GoDaddy and do SMTP through them.  Worked fine, just took a while to figure out what was going on.

For the email fix, and to add an answer about binoculars to the FAQ, I had to do a web release.  Naturally, Windows Azure decided my release credentials had expired or something.  I'm really not sure why it stopped, or what credentials I had before.  But luckily I figured out how to create a certificate and found a good comment about where to upload it, so I got past it.

Last interesting thing (for now) was I got two nearly identical emails from two different people that were disappointed that they couldn't play forever for free.  Here is one, and the response I sent to both:

"This is going to be a very short lived game if the ONLY way to earn coins without spending real money is referring friends. What happens if your friends don't want to play or or you run out of friends to get signed up? The game HAS to be self sustaining. People will quit playing if they have to spend real money."

My Response:
"Hello,

Thank you for your feedback on Monster Cache.  Almost all phone applications try to generate revenue in some way -- either by advertising, or an up-front purchase price, or in-application purchasing.  For Monster Cache, we have chosen in-application purchasing.  We want people to evaluate the game for free, and hope that they will enjoy it enough to purchase more coins, or refer friends. If you feel that you did not have enough time to evaluate the game, then we are happy to give you some more coins so that you can play a while longer.  We have added another 100 coins to your account.  Also, if you reach Level 4, then there are some Treasure Chests with small amounts of coins that you can find as well.

Again, we appreciate the feedback.  Best of luck in your monster hunting!

Monster Cache Team"

Emails to Geocachers

After I got back from my fishing trip on September 8th and we wanted to send emails to geocachers that weekend of the 13th.  Somehow, we didn't get around to it, but we did try one new promotion that weekend.  We put up a poster at Kansas Coffee Company in Olathe that had a picture of a Cyclops Bunny and said "There is a Cyclops Bunny here", with a short description of the game and some QR codes for download links.  Very nice of our friends there to put the poster up.  It has its own referral code, but I haven't seen anyone sign up with it yet.

Anyway, we thought about sending the emails on the Sunday of the next weekend, the 22nd, but realized we'd rather send them out on a Thursday so that people might use right away on the weekend without forgetting about it.  So finally, we started sending out emails on the 26th.  Kathleen had a list of 230 or so emails she got from web sites.  We sent them in batches of about 30 at a time and sent all 230!

Soon, we got rejection notices from Google saying that all of the email but one had been rejected as potential junk mail, i.e. spam.  I guess technically you could say they were spam, in that they were unsolicited emails trying to promote something to a large number of people we don't know.  But on the other hand, they were very specifically targeted to people who put their emails online because they weer interested in geocaching.  We do think they'll like it and they can try it for free to decide for themselves, so we don't feel too bad about sending the emails.

Since only 30 got out that first night, Kathleen tried sending more the next day and only got another 10 or so out before getting the message again.  And then maybe 10 more the next day.  That resulted in about 8 sign-ups, and 2 more people on the leader board, consistent with our ratio of about 75% of signups never finding a monster.  And interestingly, one Like of Monster Cache on Facebook, so that was nice.  But still not quite yet the impact we were hoping for.

There were 111 on the Legends board last week, and 113 now after the first 50 emails.