Hey guys, sure has been a crazy day hasn't it?
So, I know we have that megathread over there, and guess what my weekend gets to be (It's reading that and every other thread.) but I wanted to make a new post here just so that information was in a good, easy place to see.
Let's just get right into it:
This is some profiling output from a fun little tool we made, which analyzes the server and finds each thread that is running and times how long it takes for the server to tick. It does a lot more than that, but that's what you need to know to understand this:
Total Duration 765.60 MS 1.31 FPS
StructureTurretBaseBP_C : 7977 234.98 MS
StructureTurretPlant_C : 9722 119.4 MS
This is turrets. BaseBP is normal metal turrets, Plant is SpeciesX.
Now, with this information, you need to have some other information:
Since about October or so, we saw a MASSIVE spike in the size of stalls and poor performance on our dedicated servers, but only some of them, most of them Ragnarok Servers.
So we investigated, and we found that an interesting change in the meta for base building had occured: People were making new, super-dense bases with insane turret counts, far beyond anything we'd ever seen before. The bases of these types were on all of our servers, but primarily belonged to Alpha Tribes who had set up bases on the Ragnarok servers. Similar setups caused equally bad performance on Center and Island servers in the rare cases we found them.
I want you to understand what all of this means to us, so I'll try to be brief.
Turrets, were very suddenly being used in a configuration that was atypical, and up to 7x the number of turrets we would see on any other server, even if it was a highly populated, well trafficked server. There were definitely other cases of densely packed bases, but this was higher by many magnitudes.
Let me emphasize now, that for the last two years, turrets have been a pain in our butt, but we have not encountered people making monstrosities like this. Every alpha tribe in the game has existed just fine, and gotten along well enough without these insanely dense crazy bases.
So, we looked at what we could do about it. Let me wrap this all up in context, these things I'm going to put here are facts:
Our turrets are highly optimized for what they do with respect to gameplay. They use a very fast bitmasked octree overlap to return all of specific types of actors (excluding irrelivant actors) in the aoe that they defend, and then act according to the settings you configure, early-outing from any extra calculations or wasted instructions if their shot is invalid. We could make that area they search smaller, which would be faster, but then they would need correspondingly smaller ranges.
We could make them search for less things in their overlap check, but then you'd have turrets that didn't fire on some things.
We could make them acquire targets less frequently, but then they'd be much less accurate and in many cases would not pick up things like rocket projectiles that entered their range between scans.
We could make them scan at different ranges for different things, but then their functionality would be weirdly ambiguous and what things would we reduce scan range for anyways? Anything that a turret should be shooting at is important.
Additionally, even if we did one or all of these things, the per-frame cost of each turret would not be greatly reduced (fractions of milliseconds) and 15k+ turrets would still cripple servers.
Another truth is that in the worst case, servers were ticking at as low as 1 frame per second. That means that no matter what you're seeing in game, the server can only update the whole game state one frame per one second. That is the worst our servers have ever run, and is completely unplayable, in every way.
No matter who you are, I think you can agree that it's not enjoyable to play the game like that.
As such, we are left with a dillema:
We can reduce the number of turrets that people can place, and attempt to recover some of the performance loss that happens because of them, or we can not, or we can try and find some medium where we save as much performance as we can, while still allowing players to defend themselves.
On a server with 70 people playing on it, after reducing turret density, this is the new framerate on the server:
Total Duration 222.86 MS 4.5 FPS
Now, this is a worst case scenario, but the difference is absolutely clear to us, and one thing that we cannot under any circumstances forget, is the impact that servers running at 1 fps has on all of the players on the server. 4-5 fps is actually playable-- it can be a little bit choppy, but it's not infuriatingly awful.
Okay, so, if you're still with me, you understand what the raw numbers are that went into this, and I'll provide a to-now TLDR:
A certain method of building bases with turrets caused server fps to plummet to historic lows. The culprit is undeniably turrets: in some cases costing between 400 and 500ms per frame on the server tick. Our turrets are already extremely well optimized, but players are placing tens of thousands of them. That's too many. Our changes showed a change of 1.1 -> 4.5 fps. Reducing their numbers was now not a question of "if" but of "how much".
So what do we actually do?
Well, we try to solve it. And we try to talk about how best to solve it. Are there going to be some edge cases we have to deal with? Sure. Will we maybe have to adjust the balance of some things? Probably. We're letting you know weeks in advance as part of a method of opening up that communication channel. So that you can make decisions about what to do well ahead of time. Crunch numbers. Adjust strategies. Yell at us, and let us know what some of the things that are important to you guys are that we might not be thinking of-- hundreds of thousands of people play this game in all sorts of different ways every single day, we simply can't know what all of those ways are.
Now you know the technical issues we face. There is no other place to "save" 500ms of server tick time and let you just keep turrets how they are. There's no way to make turrets faster in any meaningful way, without fundamentally changing turrets and how they work.
As an important note, this change will also make all of the turrets you DO have much more accurate and reliable, because the low server fps made them inaccurate and miss often. Kind of a compounding problem.
This is a graph of the tick times of our worst servers, right off our backend. A delta time of 500 simply means that the server is locked at 2.5fps (it can actually run slower than that, but that's as low as our recording goes.)
PC servers are far, far worse off than console servers, due to their increased caps on everything, and each and every one of these servers suffers enormously from turret-induced slowdown. PvE doesn't even come close. Out of the top 150 slowest servers we have, only 15 of them in fact are PvE servers. This is a pvp issue, and it's really bad. There is no truth to the claim that PvE runs worse. It is a myth.
So let's talk about it. I am going to post this, and then I'm going to sit down and try to have dinner, and when I come back from dinner, I'm going to read more of what people have had to say. We want you to still be able to defend your bases and all of the stuff you've spent enormous amounts of time to make. We also want you to be able to play the game at higher than 1 fps.
- The Right Hand