Anyone else seeing high CPU spikes (and server tick drops) on Valguero?

[Thalana]

Since adding Valguero we've been seeing pretty high CPU usage, much higher than I can recall on any other map in recent months. With 20-30 people online I'm seeing 6-14 server fps. Usually we're around 20 on the other maps once we hit that many players.

At this point I'm terrified the server is going to start tanking once people start their breeding operations.

[Toni]

6 hours ago, Thalana said:

Since adding Valguero we've been seeing pretty high CPU usage, much higher than I can recall on any other map in recent months. With 20-30 people online I'm seeing 6-14 server fps. Usually we're around 20 on the other maps once we hit that many players.

At this point I'm terrified the server is going to start tanking once people start their breeding operations.

20players @30fps (including 12 mods). So no, there isn't such a problem, must be one of your installed mods (or wrong server settings like too much dinos and things like that).

[Thalana]

9 hours ago, Toni said:

20players @30fps (including 12 mods). So no, there isn't such a problem, must be one of your installed mods (or wrong server settings like too much dinos and things like that).

Hm, possibly a mod then. The odd thing is, with the same mods/settings on Ragnarok we aren't see a server tick rate that low with peak players. Figuring out what mod it is is going to be a pita. Guessing whatever mod is doing it is conflicting with Val.

[Freedom1987]

We have the same problem. Server rate dropping drom 65 to 12. There are only 10 players online. 

 

12 core 64gb ram and 2x 1000mb up and down.

only have 4 mods on it.

S+

clasic flyers

superspyglas

meatspoiler

[Thalana]

20 hours ago, Freedom1987 said:

We have the same problem. Server rate dropping drom 65 to 12. There are only 10 players online. 

 

12 core 64gb ram and 2x 1000mb up and down.

only have 4 mods on it.

S+

clasic flyers

superspyglas

meatspoiler

Sorry to hear. I've spoken to some other server owners and they're reporting the same, though not all are so clearly there's more to it. It seems the issue isn't so bad with less than 10 players online but once 15-20+ log on there's a significant rise in CPU usage which in turn causes the server rate to plummet. Here's what we've tried so far on Valguero (with little to no success):

- Stripped eco mods and many other mods to the point where all we have are lightweight QoL mods and S+
- Reduced wild dino count from 36k to 24k
- Fixed all overspawns
- Significantly reduced the size of our Game.ini and GUS.ini files
- Disabled Ice Worms as it was reported they are falling beneath the mesh
- Combed the entire map, including under the mesh in hopes of finding an overspawn
- Cleared all beehives in case they were stacking
- Clamped resources and enabled optimized harvesting health
- Set individual harvesting rates (2-3x) as opposed to using the global harvest multiplier
- Tested without using our flyer mod (Naj)
- Disabled all logging settings
- Reduced flyer speeds to see if that was lagging the server
- Implemented a rule against Bronto and Rock Elemental harvesting as we read these can lag a server out
- Implemented a rule regarding too many light sources
- Investigated for lag boxes

There's probably more. In comparison to our Ragnarok server, this server is running the barebones and sees significantly lower memory usage but terrible CPU usage. Our Ragnarok server has double the mods and big castles all over the place and CPU, while not great, doesn't come close to what Val experiences.

When observing the server fps rate and CPU, it seems it's directly tied to player activity but there could be something going on behind the scenes which merely makes player activity the catalyst for this terrible performance. At idle CPU can drop as low as 4% but once 5 players login it jumps to 70%. At 10 players it's around 90-100% and 15-20+ it's spiking to the point where the server fps is around 5. Something is definitely off. I'm sure someone can argue it's mods or whatever, but we've done over 30 hours of tests at this point. Our Ragnarok server is on another level in terms of things built, things being bred and the mod load is huge over there. It's just not even a close comparison. With 20 people on Ragnarok we may hit high CPU but server fps is typically in the 15-20 range. At this point I'm chalking it up to bad optimization. I'm not sure if you use Linux but it's possible there's an issue with the Linux binary causing this intense stress. At this point I'm giving up and hoping WC looks into it. I'll update if I find anything but I'm doubting it at this point.

[Thalana]

Update:

We did a stress test with 20 players last night on a fresh copy of the map with zero mods and saw the exact same CPU spikes. At this point I've spoken to several people who are experiencing the same. I can provide data of player activity and usage just to show it's not BS. Everyone I've spoken to who is experiencing the same is using the Linux server build so clearly there's an issue here. Even with 6 players the map is spiking horribly. While Ragnarok has a reputation and is often called Lagnarok, Valguero is easily worse.

[PigZilla]

Same for my cluster server, already had 5 maps, and now with Valguero we got spike lags all the time, even if the others maps still fine. For me the spikes starts with 30ppl and just get worst until reach 60.
I've tried the old method of destroying the beehives of the map, some maps had the problem once, with stacking beehives.