Linux Server - Performance question

[ToeiRei]

Hi guys,

I run a small virtual machine (6 vCPUs, 8 GB RAM, 100 GB hdd) to play ark on. OS is a debian 9 stripped down to the bare minimum (no X, no Mailserver, just SMB and backup software).
The ShooterGameServer process spawns a 15ish threads, whereas only 2-3 do real work on 2 CPU cores. RAM usage is at 5.5GB on the Island, 3 players.

As far as I understand, the server isn't even doing much disk activity, but "it could perform better" (tm) - There are really huge lag spikes on a player joining. Any ideas on getting things to run a bit more smooth?

Cheers
Rei

[Hodlin]

See disk activiity with 'iotop'

apt install iotop

Always check 'dmesg' or syslog for performance warnings.  If there are any server side, it will let you know.  You can tune it to show verbosity on your network interfaces too.

'iftop' is a good tool to see realtime connection status.

Get the IP of your client ( netstat -a | grep udp ), and dump their connection information to a file (open/analyze it with wireshark):

Outgoing packets:

tcpdump -n dst host x.x.x.x -w client_out.pcap

Incoming:

tcpdump -n src host x.x.x.x -w client_in.pcap

Both:

tcpdump -n host x.x.x.x -w client.pcap

 

In general, anything in red (by default theme) on the wireshark interface should be looked over.  See what step in the packet transer it fails at, and it will reveal the culprit.  Get the expected result from a good connection, and compare.

 

Tuning:

What's the output of all the following commands, with all your clients connected:

free

ulimit -n

Get the PID of the Ark process:

pidof ShooterGameServer

then:

lsof -a -p PID | sed 's/myusername//g'

(The sed pipe is to remove the username you run the process as ... you shouldn't post that output here)

 

How fast is the disk able to write?

dd if=/dev/zero of=/tmp/disktest.img bs=1G count=1 oflag=dsync && rm /tmp/disktest.img

(The last line should tell you XXXMbps)

[ToeiRei]

I am well aware of kernel and system logs, but they didn't show anything suspicious. As I had my share on linux kernel development, I started to dig deeper and did some tweaks:

- CONFIG_HZ on the debian kernel used was quite very low. Recompiled the kernel using 1kHz.
- Disk Scheduler was CFQ which is bullpoop on a VM where the host system below should take care of that (noop or deadline!)
- Profiling and debugging stuff enabled per default
- MTU of the machine set to be a bit lower than what my ISP offers (1500 is a nice default, but isn't really what they offer)

et voila - we are running on much better speed now - but still it seems like the unreal engine being used just uses 2 cores - one for the engine, one for physics (guessing here)

[ToeiRei]

Memory:

ark@ark:~$ free
              total        used        free      shared  buff/cache   available
Mem:        8181528     5688152     1042576        9132     1450800     2178320
Swap:       6809596           0     6809596

open-file limit is set correctly:

ark@ark:~$ ulimit -n
1000000

For the user, I have to admit, I don't care as it's a home server. Nothing too funky.

ark@ark:~$ lsof -a -p 6222
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF      NODE NAME
sh      6222  ark  cwd    DIR    8,1       88  12934030 /home/ark/serverfiles/ShooterGame/Binaries/Linux
sh      6222  ark  rtd    DIR    8,1     4096       128 /
sh      6222  ark  txt    REG    8,1   117208 134218012 /bin/dash
sh      6222  ark  mem    REG    8,1  1689360   1201547 /lib/x86_64-linux-gnu/libc-2.24.so
sh      6222  ark  mem    REG    8,1   153288    373439 /lib/x86_64-linux-gnu/ld-2.24.so
sh      6222  ark    0u   CHR  136,0      0t0         3 /dev/pts/0
sh      6222  ark    1u   CHR  136,0      0t0         3 /dev/pts/0
sh      6222  ark    2u   CHR  136,0      0t0         3 /dev/pts/0

Disk performance is at 350-370 MB/s on a simple XFS volume

root@ark:~# hdparm -Tt /dev/sda

/dev/sda:
 Timing cached reads:   9582 MB in  1.99 seconds = 4804.04 MB/sec
 Timing buffered disk reads: 1124 MB in  3.00 seconds = 374.24 MB/sec

root@ark:~# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output
10240+0 records in
10240+0 records out
83886080 bytes (84 MB, 80 MiB) copied, 0.23532 s, 356 MB/s

 

[kusty974]

mechanical or ssd hard drive ?

[ToeiRei]

On 12/2/2018 at 2:54 PM, kusty974 said:

mechanical or ssd hard drive ?

multi tiered storage through virtualization layer.

[ToeiRei]

I did a few changes at the kernel again. Mostly disabling debug stuff. As far as my debugging is able to dig in, most stuff seems to be poor programming at the server side which is mostly resolved by CONFIG_HZ set to 1000. Dropping debug stuff and removing some cruft from the kernel does slightly improve the situation, but I'd consider that part not worth the effort as it's completely bonkers for the average sysadmin to dive that deep on linux kernels.

My next try to do some tweaking here is at the TCP layer - advanced congestion controls. Common Linux server usually use RENO or CUBIC (feel free to check using 

sysctl net.ipv4.tcp_available_congestion_control

It should return the two options mentioned. As I already rebuilt my kernel, it's all just a matter of tweaking etc/sysctl.conf to try the new scheduler kid of the block: BBR by google:

net.core.default_qdisc=fq
net.ipv4.tcp_congestion_control=bbr

Lesson learned:
Never let a nerd debug such a problem as it's a big can 'o worms...

[ToeiRei]

I've set up RRD Graphs monitoring disk I/O. The graph shows clearly that reading happens on server restarts (which the graph actually has a few) and that writing, savegames isn't much either. 

[shiny]

I did battle with vm's and ark server for a while. Tried a few different providers and so on, tried any tweaks i could imagine.

Had almost the same problems - lag when joining, small lag when people press E to ride a dino, or again when they dismount.

I can't say i am a linux expert, i've been only working with it for 18 years, so after i lost my patience, i just went to a dedicated server hosting

Guess what happened ?

Not trying to discourage you here, all i wanted was to play, not continue doing my job when i was supposed to play ark.

[kusty974]

It's only a guess but maybe the access to the hardware ressources from emulated devices on vm introduce latency in some way... I run it on a dedicated machine so I can't tell much.

[ToeiRei]

It's the vanilla debian kernel. Since I tweaked my kernel to run at 1000 Hz lag is pretty much gone.

[kusty974]

15 hours ago, ToeiRei said:

It's the vanilla debian kernel. Since I tweaked my kernel to run at 1000 Hz lag is pretty much gone.

Hm I'm pretty curious to know the "how much" is the improvement... Did you try to bench it yourself ? I mean did you try to reproduce any situation that may lag occur ? Fly to the rendering area of a big base, travel an area with a lot of dinos/tames, mana is also a good test when traveling accross the map, teleport etc.

[ToeiRei]

We use this on our servers here and it was a big improvement compared to the 'unpatched' kernel in terms of you could move in our base whereas unpatched took about 5 minutes to load in so you could actually see where you were going.