[A:SA] Dedicated Server Exchange about Performance and Cluster

[pleinx]

Hrello together! 🙂
I want to start a exchange about server hosting from server owners. I'm currently trying everything to get the Server-FPS more stable while the players scaling with any success.

Actually i'm coming from the linux world and changed to windows because of A:SA (the docker/wine/proton solution wasn't even better). So what i have done for now is to create a windows10-pro x64 VM (proxmox/qemu) machine to play a little bit with A:SA. The VM is assigned to the host CPU with all cores, no limits and RAID-0 NVME storages.

Hardware Specs

• Intel, 9900K 8c/16t - Single-Performance 5GHz, rest of the cores has 4.7GHz Turbo-Boost
• Memory, 128GB DDR4 2667 MHz
• Storage, 2x 1TB NVMe in RAID 0
• Netwrok, NIC 1 Gbit  Intel I219-LM

 

Server Specs

• Hx1.5 / Tx2.5 / Bx6, rest mostly like official
• No mods

 

Currently running 4 instances of A:SA Servers (1 is public, rest just for testing performance things, but its doesn't matter if i stop them or not). On the public server the server-fps are really unstable:

• 1-3: no problems, around 30 FPS
• 4-6: around 20-25 FPS
• 7-9: around 15-18 FPS
• 10+: 10-15 FPS

This seems not normal for a host-system like mine but whats your experience in this case?

 

What i've tried to optimize on my windows server until now...

• disabled designs, beauties etc.
• set max performance mode (in control panel)
• disabled unnecessary services
• disabled hiberation
• disabled swap (pagesys)
• enabled high performance mode
• ofc latest driver and updates installed

Currently i'm playing with CPU priority and affinity. I assigned 4 cores to one instance. It really doesn't matter if i assigne 4 or all 16 cores to the instance. The server-fps still bad. < 4 cores you can see a descrease of server-fps when more than 6-8 at the same time online.

 

My goal is to handle more players. Is that not possible with my hardware or there are other tweaks. which i missed? Thanks in advance for any help.

 

So collecting my questions again

• Whats your experience in terms of server-fps / players?
• Whats your using hardware and tweaks?
• Also the question: whats your plans about hosting a full cluster (12 maps) with the current RAM issue?

Edited November 22, 2023 by pleinx

[pleinx]

Really no one like to share his experience to get more performance for hosting servers? Its mystery?

[Larkfields]

I don't measure/record fps, so didn't feel that I could contribute, and I was waiting to see what other's experiences were 🙂
 

[pleinx]

Ah okay 🙂 I'm really fighting with my self... i feel a bit stupid that at seems "hopeless" to improve the performance of the asa-servers. But i cannot give up and spend night for night more ours in this project. When i read other topics of this forum where owners have above 25-70 players at the same time online with a old CPU like i7 7700 or what ever, i really cannot believe that or i'm doing it totally wrong.

So, every tip is helpful to me. Currently i'm thinking about to upgrade (again and again) the host machine to a Intel 12900K or a AMD Ryzen 9 3900/Ryzen 7 3700X to see how this CPU tackles the ASA-Server.

[Larkfields]

Like you, I struggle to see how servers can cope with high numbers of players.
The Ark server performance is dependent on CPU core speed.  It doesn't use multi-cores, and doesn't use the GPU for any calculations.

I've been seeing between 15 and 25 online players, with ok server performance on an 19-13900k
ASA's Island was using 17GB RAM with 32 people online a couple of hours ago.

Windows 11 pro - bare metal.  ASE and ASA alone don't need to be in VMs.  Individual maps might crash occasionally, but they don't take each other down.

[pleinx]

Good Morning.

Quote

Like you, I struggle to see how servers can cope with high numbers of players.

Same for me. I already asked 2 times in the WC DC #unofficial-setup-help chat to find players for a 5-10min session on our server to test the performance with higher counts of players. No chance... I'm thinking about to start a open Discord, where players can join and server owners can ask for a 5-10 time slot where people can join to check the performance. Idk if players will do. 

Quote

The Ark server performance is dependent on CPU core speed.  It doesn't use multi-cores, and doesn't use the GPU for any calculations.

Yepp, that i already know from A:SE. There it was 2c per Instance. For A:SA the wiki says 4c/instance. Thats what i'm currently trying (affinity) i assigned 4 cores to the island server. But one of the cores already hits 90% usage with 7-8 players. Assigning another core doesn't really help, only when the server is saving the world it uses more cores than 4 i could observe.

But it wasn't worthy. I test it on a benchmark server with a 210MB savegame. This server has also 4cores and he needs 1,8s to save the world. When i assigned 6cores it needs 1,65-1,7s. So isn't worth to assign more cores than 4.

Quote

ASA's Island was using 17GB RAM with 32 people online a couple of hours ago.

Thats not cool... that means the servers memory will strongly increase with more players? I knew that one player takes around 150mb while he is online/active on the server but that in general it sounds like to much if you say 17GB for just the island... what happend then on gen2/ragnarok/lost island? 🤯

 

10 hours ago, Larkfields said:

Windows 11 pro - bare metal.  ASE and ASA alone don't need to be in VMs.  Individual maps might crash occasionally, but they don't take each other down.

I'm using currently proxmox to get rid of the old A:SE cluster. In the community we dediced to take it down because no one wants - after the A:SA release - continue the old game (makes sense). So in this time i decided to rent a new server (the 9900k one) and install proxmox on it to have:

• one VM with ubuntu: for the A:SE cluster (only with 3 cores assigned, because only 1-3 players will render there base until the end 1/1/2024 is the shutdown of this cluster) - Fun Fact: it works with 3 cores
• one VM with windows: because A:SA has no linux binaries for now. The docker/wine/proton solution i doesn't like, thats why i chosed a windows VM. Proxmox says only 1-3% losing of performance vs. a bare metal. What do you think? Is that maybe my issue? 

 

In general: what we awesome if you can observe a bit your server FPS and report it here. One thing what i could observe yesterday was: alle the 10 players currently really close together on the map. The new render-distance for A:SA is insane (positiv). My idea was that this could be maybe also the problem because every player can look the other base because the base is 90% rendered, also the dinos etc. Then a group of 4-5 players leaving their base and going into a cave. In this time the server fps was going to 28-30 (3-4 players still in the base, 4-5 in the cave). 

Is it maybe a problem with the new parameter?

-AlwaysTickDedicatedSkeletalMeshes

The wiki says:

Quote

Disables optimization to dynamically not tick idle/inactive creature animations on the dedicated server based on server frame rate (only becomes active when server is low FPS). Note: side-effects can include inaccurate collisions for idle creatures on the client versus the server (such as inaccurate jumping position while standing on an inactive/idle Brontosaurus and jumping on its tail as it sits in your base).

So, that means by default (without the parameter) is the optimization active or only if the parameter is set? I guess the first one, i dont need to add this start parameter, to get the benefit of more stable server fps, right?

[pleinx]

Happy New Year! Better performance on 2024? No! 😧

But i've done many more tests over the last weeks:

• RAMDisk
• Different CPUs/Setups
• Different Hard Drives
• Playing with CPU Affinity and Reserved Cores
• Playing with Windows Services (saving processes, threads, handles)

 

I will try to make it short to not bore you. Not sure if you are interest or not but i cannot good sleep with such performance issues.

 

RAMDisk ❌

Has no impact on my tests. Tried to outsource only the savegame itself (TheIsland_WP fodler) and one another test i put the whole ark-server on RAMDisk just to make sure, there is no dependency.

 

Different CPUs/Setups ✔️

I've test:
# AMD Ryzen 7 3700X (8c/16t) -  NVME SSD
# AMD Epyc 7401P (24c/48t) - NVME SSD Gen3 U.2.
# Intel i9 9900K (8c/16t) - NVME Gen3
# Intel i5 12500 (6c/12t) - NVME Gen4
# AMD Ryzen 9 3900 (12c/24t) - NVMe SSD Gen3 U.2
# AMD 5950X (16c/32t) - NVME Gen3
# Intel i9 12900K (16c (8p+8e) / 24t) - NVME Gen4

All systems had ddr4 2667 or 3200, 128gb, same network-card and RAID-0 (Debian12 with proxmox 😎

The winner is (i have also rent it finally since 1 month) the i9 12900K with Gen4. Such a beast also with the e-cores. I was thinking the 5950x is faster tbh. Really bad performance (the SSD and low freq is the problem) has the ryzen7 and epyc 7401P.

Best Price/Performance shows the Ryzen9 even with the NVME SSD. That was really crazy result. In terms of saving the world the AMD R9 had no change compared to the Gen4 NVME (0,80s vs 0,53s) but the Server-FPS was mostly really stable around AVG(27) (the i9 12900 has constantly 29-30).

All Tests are done with the same player amount, player position, savegame, dayoftime.

 

Result of this "small" benchmark (have now only less options to test more, see bottom text): 
With less instances you are really good to go with the 9900K (or 3700X if you get real NVMe devices).
Best Price/Performance has the R9 3900 is in my eyes the winner. Try go get real NVMe devices if you have the chance.

Like it said i wanna the maxValue/price and thats the i9-12900K for now which runs in general really good BUT...see next text at the bottom #goal

 

Different Hard Drives ✔️

Like i mention: RAMDisk had really zero impact. Please if you do same test and get better result, let me know!

Best so far i could test was the Gen4 NVME (Samsung). That means get the fastest devices you can, put into RAID-0. Don't forget your backups!

 

CPU Affinity and Reserved CPUs ✔️

Yes, the documentation recommended 4 logical cores per instance. This isn't completely true. Observe your worldsave with 4, 5, 6 cores (more makes no difference). But between 4 and 5 cores there is a benefit for worldsave (NOT SERVER-FPS) of around 100ms. Fun Fact: if you assign 6 cores, it will use it when saving. If you assign more than 6 they never get a load. Wildcard can you please clarify this vs. your docs? I/O uses two threads more than documented.

 

Playing with Windows Services (saving processes, threads, handles)

Here i'm not completely done with tests. My test was on windows server-2022. I reduced the OS from around 100 processes to 60. Also the handles and threads getting less around 25%. All looks so good but has no impact on the worldsave for now. I'll continue cleaning up the OS and observe is there somewhere a benefit to get from I/O. At least: its always good to keep it running smooth.

 

Finally...🏁

So, the i9 12000K doing a good job BUT when over 20 players (around 3,5k active dinos) the server-fps drops to around 10. Only one instance is running currently on the machine, so the server has the resoures completely for himself. No other I/O processes, no limitation of anything. Everything is setup to the highest clock/boost. That makes me really unhappy.

What's your XP with server-fps depends on players/active dinos?
I think only the active dinos are the problem. If you have more players around the map, you will have more active dinos. When the server is thitting the 2,5k the server-fps slightly drops. All the last "optimization" patches HAS NO IMPACT ON ANYTHING... cpu usage, server-fps or RAM usage... thats so sad.

Next setup i wanna test: DDR5 i9 13900 with Gen4 NVME.

 

If wildcard is reading this:
Please do more optimizations on the active dinos and RAM usage. The old known stasis parameters are not longer working in ASA, so we cannot optimize by ourself. How is that working on official servers? Or are the unoffial servers limit in terms of performance? 

 

Questions

• How much Server-FPS you guys have (pls note also players and active NPC)
• How long take your worldsave if you do it manually 3 times (one after another)

Edited January 9 by pleinx

[JDawg35]

I appreciate you sharing your experience and the steps you've tried. 
Based on what I've read, I think most people are in the same boat. The only times I've seen people posting lower than usual tick rates (even with 1-0 players) is if their hardware was much older.

I'm only running a Ryzen 9 5950X 16Cores @4.2Ghz and I'm seeing similar results with server tick rate (server fps) scaling down from 30FPS (cap, i believe) with 1-3 players and drop to about ~15FPS with 8-10. 

I also tried overclocking to see if performance improved. I didn't see significant changes so I kept it at clock I already know is stable (the original 4.2Ghz). 

I've seen others that took similar steps and jump through all kinds of hoops with little to no luck.

I'm afraid we need to wait for more optimization updates from WC. Some changes they've made have already reduced RAM significantly and I'm sure there's more coming. 
When more maps are released, players will tend to spread out to more maps (and CPU cores lol). While more optimization is definitely needed in general, I see a lot of this alleviating when more maps are released. As you know, everyone is crammed into one server and breeding like crazy with tons of dinos etc. which is worse case scenario atm. 
 

[pleinx]

Thanks for your replay. And "happy" to see that also others have troubles to get stable server-fps.

 

Overclocking is a good idea (just for testing those things ofc) and see how the A:SA-Server is react to it.

Higher clock-rates always nice2have - no doubt. But where is the end? If modern and high-end hardware like ours not able to handle a simple ark-server then we need a NASA computer in the future. Just joking.

However: i have now replaced my own PC with a 14700K to see how the newest CPU generation handles ARK-server. Ups, i did it again to try another (none-NASA) hardware. My system is running on DDR5-6000 (my roots have DDR4-3200) but also Gen4 NVMe.

For now what i could see is that the worldsaves are improved by around 200ms. Which is in general cool, but i could imagine thats my faster NMVe compared to the dedicated server. 
 

Next-Steps: i'll ask in my community if we find 15-20 people they wanna join this test-server with the i7 14700K and see what 5.6 GHz p-cores can you instead of 5.2 GHz (from my dedicated server with i9 12900K) in terms of server-fps. Will let you know if i find time.

 

Quote

I'm afraid we need to wait for more optimization updates from WC. Some changes they've made have already reduced RAM significantly and I'm sure there's more coming.

At the end you're right. If all have those problems then we have to wait for WC. And it looks like they heard us now:

Quote

1. Server Performance: We hear you loud and clear on server performance. We're relentlessly optimizing the server infrastructure to ensure silky-smooth gameplay. Expect continued improvements throughout January, and know that we're constantly monitoring and tweaking to make the Official Network stable.

Source: https://survivetheark.com/index.php?/articles.html/community-crunch-392-development-update-cross-platform-mod-spotlight-and-more-r2228/

 

 

About the Memory Issue:
Its true, they could already find 2-3GB and reduced it. But we have still a long way to come back to the ASE. The Island tooks there (on linux) around 4,5GB (empty map) and ends up after 3 years (my experience) at around 7GB. Which is fine.

 

I'm still thinking what i wanna do in the future in terms of clusters:
If they cannot reduce more RAM in the future, all owners with 128GB RAM (which i guess have 90% of the hosted unofficial servers) have a problem. When i started to host ARK-servers i had two rotation servers for maps which are not popular. In this case i was able to host a cluster with only 64GB ram dedicated server. But someday there was a good offer for a machine with 128GB so i switched to a full 12-maps cluster.

With A:SA is the problem: 
Nobody will know what the players now prefers. The next coming map is The Center which was in A:SE not really popular. So i cannot make a decision >now< should i add this map to rotation or not. And swapping afterwards from long-time server to map-rotation-server is not nice for the players they already built there.

So looks like to spend more money into another dedicated-server. Currently my option b) is to rent another 128GB server, but with weaker CPU to keep the costs on this host a bit lower. This machine should then handle the servers/maps which seems not getting popular.

[pleinx]

FYI: The latest Updates related to the "Dino Tracker" polling seems helpful and stablized a bit the server-fps. 

Ours went from, if 20 players are connected, from 9-12 to 12-16.

That's a good direction, but i hope that's not all.... They annoucend to made big performance updates in january... I really hope that wasn't everything

 

Seems they are with implementing stuff like the custom cosmetics and transfer...