ArcheAge Unchained – EU-Belstrom Online Again After 72 Hours

Source

Key points.  Lag-induced item duping was still present in ArcheAge’s code (seriously, this is 2019 and XLGAME still has improper server side validation for some of their game systems; I say “has” instead of “had” because who knows just how hardened this part of their game code is).  The lag in this case wasn’t specifically network lag but an issue with the auction house not successfully cancelling.  The point is that transaction based processing, has to be fail safe when it comes to validation of that data.

So from what I can surmise, they manually went through the data (resulting in the bulk of the downtime), found just 9 accounts abusing the cancelling function, and permanently banned those.  There’s a point early on (when you are facing a mountain of data to have to go through) where it would have been “easier” to roll back to a point before the original payout issue manifest itself.

MMO’s are supposed to have fault tolerant database designs with extensive logging that can be snapshot so that it can be rolled back without a lot/any loss.  That is of course dependent on how often the databases are snapshot (and it seems that they only do this weekly during maintenance which means a lot transaction processing would need to take place to bring it to a point closer to the start of when the issues began).

While they are working on a compensation plan for Belstrom players, they have yet to come up with a resolution for the Diligence coin and labor recharger grants (and we’re already at the end of the second week, and will likely be heading into week 3 without anything).