It’s easy to get made at a game developer for bugs and issues with their game. Too often you’ll see players take to social media to voice their outrage, wondering things like “what’s going on over there?” and “does QA even do their jobs?” The truth is, game development is an extraordinarily complex beast, and games like Destiny 2 perhaps even more so when you consider how many systems impact other systems. After hitting a particularly nasty inventory deletion bug for the second time in two weeks, Bungie opened up about why the issue happened, what they did to fix it, and how they plan to prevent it in the future. It’s a fascinating look into just how complicated game development can be, particularly with live service games that have countless moving parts all interacting with one another.
In this particular instance, bugs at reset on January 28, 2020, and again on February 11, 2020, caused players to lose certain items from their inventories. In both instances, it required Bungie to issue account rollbacks to server backups from earlier each morning, the first time Bungie’s ever needed to do such a rollback in Destiny history. The bug involved a complicated interaction between how quests were coded for inventory management and Destiny 2’s unique server configurations. Warning: it’s about to get a little bit technical.
How did the Destiny 2 Inventory Deletion Bug Happen?
Every item in Destiny 2 is coded with a timestamp of when it was first obtained by the player. This timestamp allows for the sorting of quests, but is applied to all inventory items. When Bungie fixed a different bug a few months ago, related to quest-sorting behaviors, by disabling the timestamp reset that occurred on login. This fix created a cascading effect that eventually bled over into inventory caps and caused the game to delete items above an incorrectly calculated inventory cap. Bungie did initially catch the bug, but misattributed it to “a tooling failure in debug workflows” and not an issue in production. This is the basics of how the bug made it through to production the first time, but how did it end up happening again?
Bungie noted that, while it tries to stress test, there’s nothing like millions of real-world players logging into the servers, so in addition to pre-stress testing ahead of release, they closely monitor things post-release of any new update. For Shadowkeep’s launch (along with the launch of Destiny 2 New Light as a free-to-play version, and adding Stadia to the mix), Bungie added more Worldservers than it had ever used before. On the back end, Bungie says that many servers did have some small side-effects and issues, but most were generally invisible on the player side of things.
“For example, one issue was that a small percentage (less than 1 percent) of these servers would crash on start-up due to the volume of servers overwhelming one of the backing databases.” Bungie would simply restart the crashed servers, and at the time, there was no discernible impact to players.
Fixing the Destiny 2 Inventory Deletion Bug
On January 28, when the first bug happened, Bungie found the issue and was able to deploy a fix on the server side, instead of needing to make players download a client update. It was the servers processing inventory caps and quantities, and that was what had caused some items to be deleted. Fast forward to February 11, when the Crimson Days update rolled out. As had happened before, a small subset of the worldservers crashed, and Bungie simply restarted them as they had done before. This time, however, the restarted servers didn’t have the inventory deletion issue fixed, so the small percentage of players who logged into those worldservers found themselves faced with the issue once again.
Though Bungie has verification systems in place to check for mismatches between the servers, the crashes and restarts had rendered these particular servers immune to that process so the bug slipped through. “Prior to this morning, we had believed skipping these overrides and verifications to be impossible.”
Though QA had tested this prior to release, none of the test accounts had gotten into the crashed servers, so it went undetected on Bungie’s end until players began reporting missing materials. Any player who had accessed their account (whether in-game or through a third-party application) and got through to one of the old worldservers running corrupt code had inventory deleted thanks to the same issue that cropped up on January 28.
How to Stop the Destiny 2 Inventory Deletion Bug From Happening Again
Now that Bungie fully understands not only the inventory deletion bug, but additional circumstances that caused certain worldservers to not be current, the studio is taking steps to mitigate the issues and make sure they don’t happen again.
- Additional measures were taken to ensure servers start up with the proper and current version applied
- Issues causing worldservers to crash on startup are being fixed with the next major update
- The inventory deletion/corruption bug will be fixed on the client side
- Bungie is looking at ways to optimize rollback/account recovery should it need to in the future
- Change in development workflow and methods to catch critical errors like this earlier
That’s just a high-level overview of what was occurring and why, not to mention why it was so difficult to find and fix. And this isn’t the first time that Bungie has highlighted the extreme complexity of game development through seemingly simple bug fixes. Back in Destiny 1, there was a major heavy ammo deletion bug that was so baked into the core code of the world load-in process, it ended up being an enormous undertaking to tear apart and fix, also ensuring that the fix didn’t break anything else in the process.
Likewise, other developers have shared stories of small bugs that had their roots embedded in improbable code. Check out this legendary thread on a bug that caused companions to climb up infinite ladders in The Outer Worlds and then fall to their inexplicable and unknown deaths.
We released patch 1.2 of #TheOuterWorlds today, and it includes a fix for the dreaded “the game thinks my companions are dead” bug, which I believe I spent more time investigating than I have for any other individual bug in my career (1/18)
— Taylor Swope (@_taylorswope) December 12, 2019
Those are just small windows into the hundreds and hundreds of bugs that game developers are looking at constantly. Video games are complex machines, each one developed in unique and interesting ways that can cause countless unintended interactions within the code. Add constantly updated live service games to the mix, and you create a whole new layer of potential and probable issues that can occur. Even fixes for the simplest of things need to be relentlessly tested, lest they cause additional unforeseen problems down the line.
Bungie giving players a peek behind the scenes here helps offer some perspective as to what the team is working with every day. game development isn’t all just fun and games, and something that takes players 30 seconds to go online and complain about could require hundreds of dev hours to identify, troubleshoot, and fix.
Check out the even more in-depth and technical version of the above issue on Bungie’s site.
