High memory usage and client problems possibly after 345.23 update

[Reboot4X]

We started seeing a problem where no players could log in after this update.   

Client users get a screen with Snapshot_9 loading for a long time, then a Host Connection Timeout (Lost/timedout connection to host) 

Once the Snapshot_9 page "hangs" on the client for a while, I notice the server memory jump up to 24-28 GB in seconds from around 6GB.    Then we see a server crash with a GenericPlatformMemory.cpp (line 51) error.   Seems like a bad memory leak since it is consuming way more than is needed for the game. 

 

Anyone start seeing problems on their clients connecting to the server lately.  

 

Running Genesis 1

Server has 32 GB RAM and runs on Win 10. 

[Larkfields]

Gen 2 looks ok on my server...

[Reboot4X]

I've never seen it go so high, but only after the Snapshot_9 and Host Connection timeout.   Happens on more than one client, so it seems like it is something wrong with the server, but I have no clue what could be causing it.  

[Larkfields]

What is "Reboot4x-Genesis1" ?  Did you rename the exe file ?

[invincibleqc]

1 hour ago, Reboot4X said:

Happens on more than one client

If it happens for some, but not everyone, then it is likely related to a specific location on the map. Last time I saw similar issues, it was caused by extremely fast breeding rates where thousands of babies were in the same spot. Not sure if this could be the issue here, since babies are no longer wandering on birth. Launch your server with the following option:

-parseservertojson

And have a look at the console how many actors were parsed as well as the generated ../Logs/AuditLogs/*.json file to see if anything seems out of the ordinary. First thing you should try though, is to launch with the following option:

-forcerespawndinos

So that dinos are re-serialized and respawned by the hibernation manager. In any case, please upload your save somewhere.

1 hour ago, Larkfields said:

What is "Reboot4x-Genesis1" ?  Did you rename the exe file ?

Pretty sure it's an ASM's feature where they generate symlinks to launch the servers instead of direct executable paths.

[Reboot4X]

Yep, Reboot4x-Genesis1 is just my Profile name in ASM.   I'll try those ideas, Invicibleqc.   Thanks! 

[Reboot4X]

Here's the out of memory error with the trace.   Seems like the allocation fails on LoadTributePlayerDatas() when trying to populate a structure like a hashmap or something.   But it gobbles up 20GB in a matter of seconds when I watch it from TaskManager.  

The number of dinos(actors) was fairly small in the reported console startup output - like 147 or so.  

The json audit log was a good idea, but I didn't see anything that jumped out as odd.  I even pulled the json into a browser to do a parsing test, but it was fine.   Examined the players in the file and nothing looked out of the ordinary.  

 

Any other ideas or diagnostic techniques?

[invincibleqc]

14 minutes ago, Reboot4X said:

Seems like the allocation fails on LoadTributePlayerDatas() when trying to populate a structure like a hashmap or something.

That function is reading the .arktributetribe files for uploaded players that join the server so that they can select a bed from that tribe before actually downloading their profile from the cluster files. By the look of it, it tries to allocate ~12gb which likely indicate these files for some players are corrupted. Try to delete all tribute files and add players back into their tribe manually using admin commands.

[Reboot4X]

Ah, I think you are onto something.   It looks like all the tribe and profile files are 1KB and they just have the filename as the content, which is really weird.    I'll see if I can find them from an earlier save. 

[Reboot4X]

Yep, that did it.  I restored just the 1KB files (.arkprofile, .arktributetribe) from a couple weeks ago and was able to connect in with a player.   

Thanks to Invincibleqc!