iamallama

Devblog 91: The SW writeup

Recommended Posts

Hey all! Sorry for the delay this month (and last). I've been having some crazy fun at work and home and have been WAY too busy to do much recently. Some stuff got flipped upside down at work and now I apparently am running our entire project. At half staffing too. I'm now the project manager of a pretty large project, so promotion, even though it wasn't in a good way (but I can't talk about that). Needless to say, it's been kind of hectic recently, but things are starting to normalize like they usually do. I have lots of upcoming travel planned and more work than I have time for. And by the time I get home now, I usually don't even want to look at a computer.

Supporter Weekend writeup
If you don't want to listen to me rant about the issues we had, skip everything after the first sentence or two.

As for the game, I didn't really comment much last month, but the supporter weekend happened. I had a bunch of issues for the first day or two, but we still had a decent turnout and stuff was done. Starting out we had an issue with the database. This started because I was missing a column in the production database for something I added like a day before (this is why feature freezes are necessary). And because I wanted to do things as complicated as possible, I thought it would be a good idea to move the player saving-to-database stuff to a background thread. Unity is single threaded and so queries on the main thread will end up blocking it until they are done. No bueno, as could be seen later on (it's coming). The issue here though is that when not using the main thread, Unity doesn't provide any sort of standard logging or error reporting. So I didn't even know that the process saving players was failing as I was getting no error reports. And since the issue wasn't happening on every query, some queries would work, some would fail. This was what caused issues like duplicate attributes or things in inventory disappearing on login (they existed while you were in memory in game, after logout they weren't saved to the database, on next login it didn't find them). So the first fix was to move queries off the background thread into the main thread. This showed there were errors which I was able to fix pretty quickly. However, that introduced issue number two. The queries were slow. Which is exactly why I was using a non-blocking background thread. They were so slow in fact that it would take about 2 seconds to save a single player. And this wasn't because the queries were really that slow, but more that there are quite a lot of queries that need to be run for every single player. Every single slot in your inventory is another row in the database. Every single attribute on your player is another row. Every single item you wear is another row. Every single item in your bank is another row. And there isn't a simple magic bullet fix that can help here. Players simply use a lot of information. The player save process usually runs every 30 seconds. When testing this there were about 12 people online. So the entire game would freeze up running queries for about 20-25 seconds. You could play for 5-10 seconds and then everything freezes again as it kicks off the process again. Usually you would enable transactions which is a way to tell the database "I'm going to make a lot of changes, queue them up in memory and I'll tell you when I'm done so you can save to disk." But the queries already use transactions. Or so I thought. Turns out it wasn't, or really that it was done poorly (my fault). So the first thing I did here was to properly implement transactions. This helped a lot, got the queries down from around two seconds per player to around 300ms (0.3 seconds) per player. Really though, this is still unacceptable. Every 30 seconds the game would freeze for a few seconds and the freeze time would grow pretty steady the more players that were online. Now I could be content with this and just move the few seconds of process back to a background thread, but that wouldn't last for very long before it was an issue again. 100 players online and I would run into the same issue of it taking 30 seconds to save in total and the process kicking off again before the previous one started. So, what's the issue now? Seems as though we are still running WAY too many queries. So let's combine multiple queries into one. So now I went through and combined all the inventory queries into a single query. Now when saving a player, instead of running one query for each player inventory slot, I run one query total that just has all the inventory items. Update the game and that gets the total query time down to around 100ms. Now I can support up to approximately 300 players online using a background thread. But I'm still running around 12 queries per player online. One for each table that makes up the player's data. So what can be done now? Try to combine queries per table across multiple characters. So this was the big update I ended up doing when I had to rewrite most of the player save process. Now it loops over all the players and saves all their data into a massive list, one list for each table. After processing all the players, it chops up that data into massive chunks. Each of these chunks represents all the data for one table across several players. It builds the necessary queries and runs them, saving a hundred or so rows to the database at a time. Now the entire save process for roughly 10 players was taking around 50ms (0.05 seconds). And when a few more players were online, the time to save increased by around 10ms because the number of queries needed didn't really increase by much. So this is where I ended up. After that, I left the player save process in the main thread to monitor it and it never really jumped too high. After that there were a few small issues with the server freezing (seemed to be a memory leak somewhere, I assume it was a change I have made) but a quick reset and it would be online again for a few more hours.

Changes

After all that, things ran pretty smooth. I hope you all got at least a few minutes to run around and attack things. I created the thread for the "Big list of issues" and got some responses. I have spent what little time I have going through and fixing a few things. Then I was waiting for an update of the framework I use that revamped the skills system to make it more flexible. Now I am working on converting some code over to the new version and trying to change a few things.

Next Supporter Weekend

I know you all just want to play and I'm trying to open up the game more. There isn't a date set yet, but I am aiming for near the end of March or early April. This will mostly be the same, just more stability and testing changes. I'm not planning on any major features in the near future, just the bug fixes from the recent supporter weekend and maybe some skills as I play around with that.

Also, I'll update the roadmap in next month.

Share this post


Link to post
Share on other sites

Congratulations on the promotion, but I'm going to need you to quit your job, leave the life you now know and focus on Waves of Time full-time.

Haha, just kidding. Congratulations and keep up the good work!

 

Llamas < Alpacas

Share this post


Link to post
Share on other sites

I like reading about the details of the problems you unwittingly run into and how you end up finding and nailing them down. Just always kinda interested in what sorta pains go into the making of an MMO, but not brave/interested enough to try to jump into that ocean myself. Oh one request pls indent your writeup at least two (2) times :P next time

Share this post


Link to post
Share on other sites
13 hours ago, TypicalWhiteAsianBoy said:

OMG IS THE FULL GAME RELEASE REALLY NEXT MONTH?!?!

 

 

Nah jk, not really, for those who didn't read it and just looking at comments. ;P

Lol, you literally made me throw my phone across the room.  The camera fell off the phone lol.

Share this post


Link to post
Share on other sites
On ‎2‎/‎7‎/‎2018 at 9:31 AM, iamallama said:

I'm now the project manager of a pretty large project, so promotion, even though it wasn't in a good way (but I can't talk about that).

Llama's in charge of some big NSA project? Hmm, I wonder who got assassinated at the office...

Maybe he hired a hitman with all our donation money! IT'S ALL A FRONT BOYS. GET DOWN!

Share this post


Link to post
Share on other sites

I use paypal only because it is easy, popular and accepts pretty much everything. You can pay using a credit card via paypal without needing a paypal account. However that does mean entering your credit card information to paypal. I don't have another 3rd party payment processor available. Maybe in the future when the game is live I will end up finding something that integrates into Unity and can process all sorts of payments through there. But for now, paypal is it.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now