TheRightHand

Turret Changes: A Technical Talk about why.

Jerryn

The conversation continues, including feedback from TheRightHand on comments and concerns, in this thread:

 

Message added by Jerryn

Recommended Posts

Hey guys, sure has been a crazy day hasn't it?

So, I know we have that megathread over there, and guess what my weekend gets to be (It's reading that and every other thread.) but I wanted to make a new post here just so that information was in a good, easy place to see.

Let's just get right into it:

This is some profiling output from a fun little tool we made, which analyzes the server and finds each thread that is running and times how long it takes for the server to tick. It does a lot more than that, but that's what you need to know to understand this:

Total Duration 765.60 MS 1.31 FPS

StructureTurretBaseBP_C :     7977 234.98 MS
StructureTurretPlant_C :     9722 119.4 MS

This is turrets. BaseBP is normal metal turrets, Plant is SpeciesX.

Now, with this information, you need to have some other information:

Since about October or so, we saw a MASSIVE spike in the size of stalls and poor performance on our dedicated servers, but only some of them, most of them Ragnarok Servers.

So we investigated, and we found that an interesting change in the meta for base building had occured: People were making new, super-dense bases with insane turret counts, far beyond anything we'd ever seen before. The bases of these types were on all of our servers, but primarily belonged to Alpha Tribes who had set up bases on the Ragnarok servers. Similar setups caused equally bad performance on Center and Island servers in the rare cases we found them.

I want you to understand what all of this means to us, so I'll try to be brief.

Turrets, were very suddenly being used in a configuration that was atypical, and up to 7x the number of turrets we would see on any other server, even if it was a highly populated, well trafficked server. There were definitely other cases of densely packed bases, but this was higher by many magnitudes.

Let me emphasize now, that for the last two years, turrets have been a pain in our butt, but we have not encountered people making monstrosities like this. Every alpha tribe in the game has existed just fine, and gotten along well enough without these insanely dense crazy bases.

So, we looked at what we could do about it. Let me wrap this all up in context, these things I'm going to put here are facts:

Our turrets are highly optimized for what they do with respect to gameplay. They use a very fast bitmasked octree overlap to return all of specific types of actors (excluding irrelivant actors) in the aoe that they defend, and then act according to the settings you configure, early-outing from any extra calculations or wasted instructions if their shot is invalid. We could make that area they search smaller, which would be faster, but then they would need correspondingly smaller ranges.

We could make them search for less things in their overlap check, but then you'd have turrets that didn't fire on some things.

We could make them acquire targets less frequently, but then they'd be much less accurate and in many cases would not pick up things like rocket projectiles that entered their range between scans.

We could make them scan at different ranges for different things, but then their functionality would be weirdly ambiguous and what things would we reduce scan range for anyways? Anything that a turret should be shooting at is important.

Additionally, even if we did one or all of these things, the per-frame cost of each turret would not be greatly reduced (fractions of milliseconds) and 15k+ turrets would still cripple servers.

Another truth is that in the worst case, servers were ticking at as low as 1 frame per second. That means that no matter what you're seeing in game, the server can only update the whole game state one frame per one second. That is the worst our servers have ever run, and is completely unplayable, in every way.

No matter who you are, I think you can agree that it's not enjoyable to play the game like that.

As such, we are left with a dillema:

We can reduce the number of turrets that people can place, and attempt to recover some of the performance loss that happens because of them, or we can not, or we can try and find some medium where we save as much performance as we can, while still allowing players to defend themselves.

On a server with 70 people playing on it, after reducing turret density, this is the new framerate on the server:

Total Duration 222.86 MS 4.5 FPS

Now, this is a worst case scenario, but the difference is absolutely clear to us, and one thing that we cannot under any circumstances forget, is the impact that servers running at 1 fps has on all of the players on the server. 4-5 fps is actually playable-- it can be a little bit choppy, but it's not infuriatingly awful.

Okay, so, if you're still with me, you understand what the raw numbers are that went into this, and I'll provide a to-now TLDR:

A certain method of building bases with turrets caused server fps to plummet to historic lows. The culprit is undeniably turrets: in some cases costing between 400 and 500ms per frame on the server tick. Our turrets are already extremely well optimized, but players are placing tens of thousands of them. That's too many. Our changes showed a change of 1.1 -> 4.5 fps. Reducing their numbers was now not a question of "if" but of "how much".

So what do we actually do?

Well, we try to solve it. And we try to talk about how best to solve it. Are there going to be some edge cases we have to deal with? Sure. Will we maybe have to adjust the balance of some things? Probably. We're letting you know weeks in advance as part of a method of opening up that communication channel. So that you can make decisions about what to do well ahead of time. Crunch numbers. Adjust strategies. Yell at us, and let us know what some of the things that are important to you guys are that we might not be thinking of-- hundreds of thousands of people play this game in all sorts of different ways every single day, we simply can't know what all of those ways are.

Now you know the technical issues we face. There is no other place to "save" 500ms of server tick time and let you just keep turrets how they are. There's no way to make turrets faster in any meaningful way, without fundamentally changing turrets and how they work.

As an important note, this change will also make all of the turrets you DO have much more accurate and reliable, because the low server fps made them inaccurate and miss often. Kind of a compounding problem.

https://gyazo.com/e307c4f0324ee5fc239862ace0365c38

This is a graph of the tick times of our worst servers, right off our backend. A delta time of 500 simply means that the server is locked at 2.5fps (it can actually run slower than that, but that's as low as our recording goes.)

PC servers are far, far worse off than console servers, due to their increased caps on everything, and each and every one of these servers suffers enormously from turret-induced slowdown. PvE doesn't even come close. Out of the top 150 slowest servers we have, only 15 of them in fact are PvE servers. This is a pvp issue, and it's really bad. There is no truth to the claim that PvE runs worse. It is a myth.

So let's talk about it. I am going to post this, and then I'm going to sit down and try to have dinner, and when I come back from dinner, I'm going to read more of what people have had to say. We want you to still be able to defend your bases and all of the stuff you've spent enormous amounts of time to make. We also want you to be able to play the game at higher than 1 fps.

- The Right Hand

  • Like 6
  • Thanks 9
  • Facepalm 3

Share this post


Link to post
Share on other sites
Quote

Turrets, were very suddenly being used in a configuration that was atypical, and up to 7x the number of turrets

Are you really saying that 100 turrets (tek, auto, psx) in a 10k unit square is typical? These configurations are incredibly sparse. We're forced to reduce our turret count by ~80% in some cases on the center. Increasing the damage is a good option (although 20% is a bit lax), but increasing the inventory is effectively pointless. Bringing more cakes is not difficult, we need equivalent dps not damage over a greater duration.

Perhaps introduce a new turret with more expensive ammo that hits substantially harder? Re-use the same turret code, but just adjust values?

Edited by Drenith
provided suggestion
  • Like 1

Share this post


Link to post
Share on other sites

So, if your above post is honest, why would you not increase the damage output of a single turret to 5x to 7x current rate? This would be in line with the number of turrets being removed and "still allowing players to defend themselves" -- the entire issue your playerbase has on this change is that you are reducing the overall capability of a tribe to defend their base from an overwhelming attack by a large number of bullet sponge tames.

Because you are barely increasing the damage output of turrets but decreasing the number so drastically, it means you have other reasons (beyond technical) for this decision. If so, be honest about what that reasoning is so players can decide whether to move on since we cannot have any serious base defense in ARK. If you say the decision is completely based on reducing the effects the number of turrets are having on the servers...then put the damage output to be actually proportionate to the drastic decrease in turret placement density.

Share this post


Link to post
Share on other sites
Quote

why would you not increase the damage output of a single turret to 5x to 7x current rate

The issue here is cost to damage ratio. If you increase the damage of a single turret by 5x to 7x you're making individual turrets way more powerful. This makes configurations that were previously under the 100 turret cap far more powerful than they were before. I think we'd need more expensive turrets with more expensive ammo (and higher health. Just an overall 'super turret') to maintain the current damage to cost ratio.

Edited by Drenith

Share this post


Link to post
Share on other sites
2 minutes ago, Drenith said:

The issue here is cost to damage. If you increase the damage of a single turret by 5x to 7x you're making individual turrets way more powerful. This makes configurations that were previously under the 100 turret cap far more powerful than they were before. I think we'd need more expensive turrets with more expensive ammo to maintain the current damage to cost ratio.

Yes, they should introduce a number of new turrets with higher damage and higher resource cost and higher ammo usage. Then the bigger inventory would make sense.

Keep the normal turrets and introduce 2x 4x 6x and maybe 8x turrets.

  • Like 1

Share this post


Link to post
Share on other sites
Quote
1 hour ago, Drenith said:

Bringing more cakes is not difficult, we need equivalent dps not damage over a greater duration.

 

This. The new stego meta along with large masses of bullet sponges will rip bases wide open. You take just ONE of the current near 8k base hp stegos and put them against 100 turrets on a wall and 100% of the time the base gets dropped. A drastic change such as what you guys propose would require a MASSIVE rework of almost every dino in the game along with the turret dps rework itself. And honestly, most of us know that this change come the 5th, will be RIP for alot of peoples bases. Even so numbers do not lie, but as most of us experienced, Rag servers were terrible lag from the release on new servers, with or without the turrets. I don't think majority of the PVP player base would ever agree to such a drastic change to turrets like this. Calling this off and letting people know you are actually working to find a better solution to solve lag problems, be it new server hosting/possible turret limits (not to this extent)/even figuring out why this problem is majority on Rag servers rather than island/center/SE would be the better business choice honestly. @TheRightHand

Edited by JakeBoxBoxBox
  • Thanks 3

Share this post


Link to post
Share on other sites
3 minutes ago, Drenith said:

The issue here is cost to damage. If you increase the damage of a single turret by 5x to 7x you're making individual turrets way more powerful. This makes configurations that were previously under the 100 turret cap far more powerful than they were before. I think we'd need more expensive turrets with more expensive ammo to maintain the current damage to cost ratio.

Cost to damage? Sure. Make the ammo more expensive. It won't be hard to grind. At least everyone will still have the tames to go farm this way. Seems like a fair trade given that we'll be able to maintain the defenses we are accustomed to and have so much better server performance, right?

The issue is the change as proposed is creating a massive shortfall in terms of being able to defend bases. I'm sure everyone will agree on this on December 6th once every tribe with anything more than trike gets completely wiped by small strike teams of flak soldiers with a stack of C4. And maybe a single steggo, too, just to make sure the raid is a success.

Share this post


Link to post
Share on other sites

I think you should reduce the amount of turrets able to be placed on one structure. People are able to put like 15 turrets on just one side of a giant metal hatch frame and that’s why your seeing these bases with denser turret pops. If u reduce it to 1 turret per structure you will still be able to defend your base using catwalks again and it won’t completely solve your lag problem but it will help it a lot!

  • Thanks 1

Share this post


Link to post
Share on other sites

If anything, buff the turrets way more than u think, and scale them back afterwards if its too op. Don't let everybody's stuff get lost just for haphazard testing purposes. You can always change turret damage later but you can't replace any fallen bases.

Edited by waterKeeper
  • Thanks 4

Share this post


Link to post
Share on other sites

Hey @TheRightHand, thanks for demystifying the PvE myth since I saw it threw around and I knew it was bullpoop. I mentioned it in the MegaThread but I thought I would suggest it here too. How about adding a setting to the turrets that allows to configure the "power" of them. For example, you set it to power 3, and it will costs 3 bullets every time it shoots for 3 times the damage. Caculations/ray-tracing are done once but this allows turrets to be stronger at a fair bullet costs. I think this would be a fair balance that optimize server performance while also allowing players to be better protected. Think of it of having 100 turrets that are equals to 300 but only thinks for 100.

  • Like 2

Share this post


Link to post
Share on other sites

This is Absolutely ridiculous, I agree putting a Limit on Turrets ect will help but not like this.. please Try to play PvP for a while like most of us do you will see why we are saying this won’t work... I’d rather have Low frames than low Defense.. Even than this won’t work, speaking from having. Small experience in a mega Tribe people will just expand Turret coverage so Weakend Tames won’t be able to get to bases, eventually maps will become fully covered by Turret Pill box’s An you will be Kill upon joining a server... I’m trying to say people will treat this range thing the same as Enemy foundations they will expand Turrets so it’s just barely outside this 10k range. 

 

Solution- Get rid of Dino Tail Swing. Reduce Environmental Graphics ect such as This Aid Rain that you cant see through. 

But doing this Turret thing for certain ranges especially at 100 counting Plant X also is going to Kill your game An put a even worse name on yoir company.. You really can’t expect people to be cool using 20 places x An 80 Turrets ect. I for one will tell you this Mega Tribes that are already untouchable An have Members on like 24/7 will be least effected, the maps they base on will be covered in Turrets On damn near every spawn. Thanks for reading this.. Find a better way to go about doing this.. make the Turret count more than 100 or individually count Turrets an Plant X, something. Thanks for reading. :) 

  • Thanks 1

Share this post


Link to post
Share on other sites

I am a Alpha on a Xbox ragnorak server. I appreciate you for showing us the technical side it explains a lot but what you are about to do is going to kill the game. We had the same exact layout in the legacy severers except we had more plant x on the legacy server. There are plenty of other ways to help with lag. Make the tails stop moving on the creatures. Don’t make them move period if they are just standing there parked. If you cut down on the number of turrets allowed in a certain space at least allow more then what you’re currently planning no one will be able to defend their bases even with this “advanced notice”(which not everyone has). We had A LOT more turrets on legacy, the turrets are a definite issue but i think You should delay the update. no one will be mad. Delaying the update will allow more time to think about it and figure it out. in the planned update time you could try experimenting in other ways to decrease lag(plant x and tames). It wouldn’t hurt to try other things first before you destroy what have spent so much time on. Ark is my hobby please don’t ruin it. 

Edited by XxPackeyXx
  • Like 2
  • Thanks 1

Share this post


Link to post
Share on other sites
9 minutes ago, invincibleqc said:

@TheRightHandHow about adding a setting to the turrets that allows to configure the "power" of them. For example, you set it to power 3, and it will costs 3 bullets every time it shoots for 3 times the damage.

While I'd prefer to see different tiers of turrets (fewer turrets with the same health is still going to be an issue) the above proposal would potentially be the easiest for WC to implement. That said we'd definitely need more than the 50% inventory increase then.

Would we be able to tell the power level via spyglass or would it be a bit of a game of chance when attempting to raid as sometimes turrets will barely tickle and other times you'll get straight demolished?

Edited by Drenith

Share this post


Link to post
Share on other sites

Hello right hand. If you sort out the meta of raiding you will fix this issue. The damage from the turret damage is an ovious one. Riding a tanking Dino into battle placing C4 whilst not getting shot by player only turrets needs to be addressed. ASAP. Brontos with crazy health soaking up bullets. There are your issues. If you get raided by a large tribe with a dozen stegosaurus you need the excessive defences. Look forward to your next nerf or buff.

Share this post


Link to post
Share on other sites
1 minute ago, XxPackeyXx said:

There are plenty of other ways to help with lag. Make the tails stop moving on the creatures. Don’t make them move period if they are just standing there parked.

Pretty sure their animations are completely handled client-side when creatures are idling. Also, I would like to point out that there is already a limit of creatures on the servers because of how costly they are so limiting the turrets as well seems logical, no? I means, currently there can be 10,500 turrets per render-range, calculate this all over the world geometry and you get millions of turrets and with the data provided above it just doesn't make any sense that it could be handled. Maybe NASA could handle them, but NASA servers are expensive... are you ready to pay 60k for a copy of the game instead of $60? I don't think so. Limiting it to 100, you basically limit to 0.95% of what it was allowed meaning that you optimize quite a lot.

  • Like 1

Share this post


Link to post
Share on other sites
5 minutes ago, XxPackeyXx said:

I am a Alpha on a Xbox ragnorak server. I appreciate you for showing us the technical side it explains a lot but what you are about to do is going to kill the game. We had the same exact layout in the legacy severers except we had more plant x on the legacy server. There are plenty of other ways to help with lag. Make the tails stop moving on the creatures. Don’t make them move period if they are just standing there parked. If you cut down on the turret at least allow more then what you’re currently planning no one will be able to defend their bases even with this “advanced notice”(which not everyone has). We had A LOT more turrets on legacy, the turrets are a definite issue but i think You should delay the update. no one will be mad. Delaying the update will allow more time to think about it and figure it out. in the planned update time you could try experimenting in other ways to decrease lag(plant x and takes). It wouldn’t hurt to try other things first before you destroy what have spent so much time on. Ark is my hobby please don’t ruin it 

Agreed on Legacy (xbox) we had 1000's of turrets and when we were being raided at least 200+ were firing at once and it DC'd everyone in the base over and over .... We could not even join the game in an effort to defend which was quite disappointing since you always prepare for the inevitable attack... 

I think the new higher tiers of turrets that can eventually get to 5 and 6 times the damage of the current turrets would allow bases to be defended ...

IF they do go through with this December 5th will be an ARK Apocalypse ... So many tribes will be wiped

Share this post


Link to post
Share on other sites

I like that you are actually explaining things in a sensible manner, something that might have helped when patch was announced, instead of muddying the water with a 100,000 units or whatever number that meant nothing to most players.

I like that your explaining things and the reasoning behind them.

Honestly, I still think you are overlooking  the capabilities of alpha tribes and  you will likely find the incidents of offline raiding will massively increase with the hugely weakened base defenses, The reason we are using mass thousands of turrets on our bases is because we have to. the dinosaurs and player numbers alpha tribes fight mean we Need all those turrets, bullets and defenses just to give us a chance of waking up before they get inside allowing us to mount some kind of defense.

 

At its base level, the reason turrets are being mass spammed is they probably arent powerful enough and are too cheap, a limit is a good idea. but you need more than one kind of autoturret, give alphas the ability to make auto cannons or other larger turrets so we do not need to have so many thousands of bullet firehoses that lag the server so bad.

but please, still let us defend our bases. we need to sleep to. that would be really nice.

  • Like 2

Share this post


Link to post
Share on other sites
3 minutes ago, Wedgbury said:

Will private servers be affected?  If I'm paying for my own server then I should be able to not apply the nerf 

If you check the Patch Notes, you can see that it can be enabled/disabled using the bLimitTurretsInRange parameter so you are more than free to configure your server how to wish to.

Share this post


Link to post
Share on other sites
Guest
This topic is now closed to further replies.