Jump to content

Does anyone have a windows server with -automanagedmods working?


Recommended Posts

I have been wrestling with this function unsuccessfully for a long time. I cannot seem to get it to work. I have the -automanagedmods command line argument in, and I have the [modinstaller] with the mod IDS=xxxx listed out in the game.ini as well. This server was working 100% for months before trying to work with this function again. If I take out the command line, and remove the lines from the ini, and copy the mod files manually from a client install, everything loads up fine again, so its definitely this function that is the problem. If I do get the function to download and install mods, it will never copy all the files into the mod folder correctly. Lots of missing files, the .mod file will be missing from half or more. I have scoured the internet for solutions but have only seen a few helpful posts for linux servers. I am running the server on windows server 2016 in a poweredge r410 with 128 gigs of memory. Any help would be appreciated.

Link to comment
Share on other sites

Yup, all fine here.  On Windows Pro 10.

Game.ini
[ModInstaller]
ModIDS=1364327869
ModIDS=731604991

GUS.ini
[ServerSettings]
ActiveMods=1404697612,731604991


1. Check that the mods are still available.  Quite a few have been removed from the steam workshop.
2. Scan your drives for errors.
3. On your server, open an administrator CMD prompt and       sfc /scannow

Link to comment
Share on other sites

I have already ran SFC with no errors showing.

The mods definitely all exist in the workshop still, as I am copying them from my client install on my PC after wiping them all out and reinstalling.

Maybe I have a permissions issue? Something with windows server 2016?

Maybe it's a file path problem? I was pretty sure I installed with the default file path, but I'm not positive. 

Link to comment
Share on other sites

2 hours ago, Larkfields said:

Is the Ark server running under a different user ID to you, when you login and successfully copy the files?

I'm not up on server 2016...  would Ark need any network permissions?  Is it allowed through the firewall?

 

I really do not know about any permissions it may need. Only have the 1 administrator account and that's it.

 

1 hour ago, invincibleqc said:

My first guess would rather be a cache issue. Try to remove all the mods files, and delete (or rename) your ../steamapps/ directory.

I have deleted all the mods and attempted to have the automanagedmods redownload them multiple times.

Which /steamapps/ directory? I have seen several folders with that name and I wanna make sure I understand 100%

Link to comment
Share on other sites

  • Volunteer Moderator
30 minutes ago, Markusius said:

Which /steamapps/ directory? I have seen several folders with that name and I wanna make sure I understand 100%

The following one:

../Engine/Binaries/ThirdParty/SteamCMD/Win64/steamapps/

Mods are first downloaded there before being extracted to the mods directory. Actually, try the following instead:

  • Shutdown your server.
  • Rename ../Engine/Binaries/ThirdParty/SteamCMD/Win64/ to Win64.old.
  • Create a new Win64 directory.
  • Copy steamcmd.exe from Win64.old to Win64.
  • Restart your server.

Also, check the log files under the /logs/ directory which may give you some insights. In any case, a permission issue is very unlikely -- all files would fail to be written, not just random ones.

Link to comment
Share on other sites

17 hours ago, invincibleqc said:

The following one:

../Engine/Binaries/ThirdParty/SteamCMD/Win64/steamapps/

Mods are first downloaded there before being extracted to the mods directory. Actually, try the following instead:

  • Shutdown your server.
  • Rename ../Engine/Binaries/ThirdParty/SteamCMD/Win64/ to Win64.old.
  • Create a new Win64 directory.
  • Copy steamcmd.exe from Win64.old to Win64.
  • Restart your server.

Also, check the log files under the /logs/ directory which may give you some insights. In any case, a permission issue is very unlikely -- all files would fail to be written, not just random ones.

Deleted all mod files on the server, did all of the above. When I restarted the server, steamcmd opened and updated its files first, and then it appeared to be downloading/installed the mod files. SteamCMD closed, shootergame server opened and crashed.

Upon inspecting the mod folder for the server, 6 of the .mod files are missing. Most of the mod folders appear to be intact.

I looked through the logs and don't see much in the way of errors, although I may just be missing them. I will attach them for your viewing pleasure if you are so inclined, along with the crashstack. It looks to me like the server is crashing due to missing mod files, again. 😑

I sincerely appreciate the continued help. Please bare with my ignorance!

 

03.10-10.14.21.crashstack appinfo_log.txt bootstrap_log.txt configstore_log.txt connection_log.txt content_log.txt workshop_log.txt

Link to comment
Share on other sites

There's an "access denied" message in the workshop_log.txt file :-

[2023-03-10 10:04:12] [AppID 346110] Download item 2185399633 result : OK
[2023-03-10 10:04:12] [AppID 346110] Finished Workshop download job : No Error
[2023-03-10 10:04:13] [AppID 346110] Updating workshop item details failed with Access Denied
[2023-03-10 10:04:13] [AppID 346110] Download item 731604991 requested by app

bootstrap_log.txt also reports some errors.

The crash stack needs someone with internal knowledge of Ark to look at.


VERSION: 357.4 
ShooterGameServer.exe!ULinkerLoad::Preload() (0x00007ff76f9200ba) + 0 bytes [f:\build\lostisland\engine\source\runtime\coreuobject\private\uobject\linkerload.cpp:2929]
ShooterGameServer.exe!UStructProperty::LinkInternal() (0x00007ff76f949543) + 0 bytes 

 

 

Link to comment
Share on other sites

On 3/10/2023 at 2:00 PM, Larkfields said:

There's an "access denied" message in the workshop_log.txt file :-

[2023-03-10 10:04:12] [AppID 346110] Download item 2185399633 result : OK
[2023-03-10 10:04:12] [AppID 346110] Finished Workshop download job : No Error
[2023-03-10 10:04:13] [AppID 346110] Updating workshop item details failed with Access Denied
[2023-03-10 10:04:13] [AppID 346110] Download item 731604991 requested by app

bootstrap_log.txt also reports some errors.

The crash stack needs someone with internal knowledge of Ark to look at.


VERSION: 357.4 
ShooterGameServer.exe!ULinkerLoad::Preload() (0x00007ff76f9200ba) + 0 bytes [f:\build\lostisland\engine\source\runtime\coreuobject\private\uobject\linkerload.cpp:2929]
ShooterGameServer.exe!UStructProperty::LinkInternal() (0x00007ff76f949543) + 0 bytes 

 

 

Yeah I saw that, but what access is it denying, and by what authority. I have no idea where to even start looking.

Maybe I should just do a full reinstall of SteamCMD

Link to comment
Share on other sites

  • Volunteer Moderator
19 minutes ago, Markusius said:

Yeah I saw that, but what access is it denying, and by what authority. I have no idea where to even start looking.

I would not be too worried about that one. It means it failed to write to the following file:

../steamapps/workshop/appworkshop_346110.acf

Likely because it didn't exist yet, or the asynchronous writer was not done with it. Worst that will happens, is it will validate that specific mod again on next launch.

On 3/10/2023 at 5:00 PM, Larkfields said:

bootstrap_log.txt also reports some errors.

Looks like it fails to download the following manifest:

http://media.steampowered.com/client/steam_cmd_win32

Meaning the required packages are not being downloaded. Perhaps his server blocks non-secure requests (port 80).

26 minutes ago, Markusius said:

Maybe I should just do a full reinstall of SteamCMD

What's the content of the following file?

../Engine/Binaries/ThirdParty/SteamCMD/Win64/config/config.vdf

 

Link to comment
Share on other sites

  • Volunteer Moderator
On 3/12/2023 at 4:30 PM, Markusius said:

There is no recent failures logged. Are the missing files properly downloaded/extracted but not moved to the mods folder? For instance, are they into the following directory?

../Engine/Binaries/ThirdParty/SteamCMD/Win64/steamapps/workshop/content/346110/*

The following thread appears to report the same issues as you are experiencing:

And claim that they narrowed it down to their download speed. For science, I tested by clamping my speed to 10gbs and all that happened was that it timed out after exactly 300 seconds but didn't result into missing files as the entire update was aborted.

In any case, are you launching more than one servers from the same directory all auto managing mods? I could certainly see this result into a race condition where files are purged/moved concurrently causing random issues like that.

Edited by invincibleqc
Link to comment
Share on other sites

On 3/16/2023 at 2:42 AM, invincibleqc said:

There is no recent failures logged. Are the missing files properly downloaded/extracted but not moved to the mods folder? For instance, are they into the following directory?

../Engine/Binaries/ThirdParty/SteamCMD/Win64/steamapps/workshop/content/346110/*

The following thread appears to report the same issues as you are experiencing:

And claim that they narrowed it down to their download speed. For science, I tested by clamping my speed to 10gbs and all that happened was that it timed out after exactly 300 seconds but didn't result into missing files as the entire update was aborted.

In any case, are you launching more than one servers from the same directory all auto managing mods? I could certainly see this result into a race condition where files are purged/moved concurrently causing random issues like that.

The Mod files are definitely downloading into the content/346110 folder you listed. In that folder, the mod files appear to be complete, but just like the other thread you linked, the ones in the servers mod files are missing tons of the files. 

My internet speed is very unlikely to be a factor. Any speedtest I run to any server hits 125Mbps on the lowest end.

I intend to run more than 1 server with these settings but I have only been launching a single one through all of this testing/troubleshooting.

Just for clarity; every time I am attempting a new fix/idea for this issue, I am deleting all the mods in the server mod files, and all the mods in the content/346110 folder as well, and attempting to have the server reinstall it all from scratch. I then attempt to boot the server several more times, along with computer restarts, before calling it a fail. I am not sure if there's more I should be doing.

Should I just be using ASM?

Link to comment
Share on other sites

  • Volunteer Moderator
3 hours ago, Markusius said:

The Mod files are definitely downloading into the content/346110 folder you listed. In that folder, the mod files appear to be complete, but just like the other thread you linked, the ones in the servers mod files are missing tons of the files.

Into the directory of a mod that failed to properly update (e.g. ../ShooterGame/Content/Mods/<id>) or any of its subdirectories, do you have any file named SaveCompressed.tmp?

 

Link to comment
Share on other sites

16 hours ago, invincibleqc said:

Into the directory of a mod that failed to properly update (e.g. ../ShooterGame/Content/Mods/<id>) or any of its subdirectories, do you have any file named SaveCompressed.tmp?

 

There are not. I checked multiple subdirectories as well as having windows search the entire folder.

 

I actually received my new server computer yesterday and just decided to set it up. This is a totally different machine, different OS (server 2012 vs 2016), fresh operating system install with only microsoft updates and chrome installed. Brand new SteamCMD install, new ARK server install, fresh ark data, brand new save folder. I attempted to boot this brand new setup with only a single mod (Structures plus) and it produced the exact same result. I am now wondering if I have a setting issue that is obvious and I am missing. I am including my server boot .bat and the game.ini I used.

Game.ini Start_Fjordur.bat

Link to comment
Share on other sites

23 minutes ago, Larkfields said:

Not related (I don't think), but I don't believe the Ark Server likes drive letters  in paths.

-ClusterDirOverride=C:\Cluster    might need to be...
-ClusterDirOverride=\Cluster

That cluster setting has been working for a long time, I am not inclined to change it unless it's causing a problem. I could give it a shot and see though.

Link to comment
Share on other sites

  • Volunteer Moderator
3 hours ago, Markusius said:

There are not. I checked multiple subdirectories as well as having windows search the entire folder.

The reason I asked about this specific file is because I was able to reproduce the issue, or at least its symptoms, and this was the one at fault in most cases but it could be any. Basically, I wrote a simple program that listen for file creation events in the mods directory and keep their handle hostage to simulate "sharing violations". I tested with one of your mod (2185399633) and this resulted into the following ../Mods/2185399633/ folder:

SQIqWkv.png

Caused by the following violations (captured by procmon) :

8vgjtaB.png

Which I believe is your issue; a third-party process (e.g. an antivirus that scan new files for threats, a deduplication service, a remote client, etc.) is accessing new files with an exclusive mode and is not releasing them quick enough causing missing files.

Edited by invincibleqc
Link to comment
Share on other sites

2 hours ago, invincibleqc said:

The reason I asked about this specific file is because I was able to reproduce the issue, or at least its symptoms, and this was the one at fault in most cases but it could be any. Basically, I wrote a simple program that listen for file creation events in the mods directory and keep their handle hostage to simulate "sharing violations". I tested with one of your mod (2185399633) and this resulted into the following ../Mods/2185399633/ folder:

SQIqWkv.png

Caused by the following violations (captured by procmon) :

8vgjtaB.png

Which I believe is your issue; a third-party process (e.g. an antivirus that scan new files for threats, a deduplication service, a remote client, etc.) is accessing new files with an exclusive mode and is not releasing them quick enough causing missing files.

I don't understand what could be doing that. There is no windows defender on this version of windows, there is no antivirus software of any kind installed. The only programs installed at all are SteamCMD, Chrome, and Notepad++.

Maybe there is some rabbit hole advanced setting in server 2012/2016 that is causing it? I have no idea what to try at this point.

Link to comment
Share on other sites

  • Volunteer Moderator
25 minutes ago, Markusius said:

I have no idea what to try at this point.

Delete all non-builtin mod files from the /Mods/ folder, download procmon and run it, start a capture with CTRL+E, launch your server and look at the logs what process/error are reported about your mods.

Link to comment
Share on other sites

  • 3 weeks later...
On 3/18/2023 at 8:49 PM, invincibleqc said:

Delete all non-builtin mod files from the /Mods/ folder, download procmon and run it, start a capture with CTRL+E, launch your server and look at the logs what process/error are reported about your mods.

Sorry for the long time between, I had work trips that made me stop working on this for a few weeks.

So I did exactly what you said above. I am only attempting to have it do 1 mod to simplify the process and troubleshooting for the moment. The mod is structures plus 731604991. Same result as before of course, mod files definitely download, but several of them do not copy over from the Binaries/ThirdParty/SteamCMD/Win64/steamapps/workshop/content/346110/ directory into the content/mods directory in the server files.

I pored over the log file created with procmon. I have found no sharing violations, no access denied, no failures of any sort.

The process seems to move files alphabetically, but it stops after it finishes "platforms". It goes from "crafting" through "platforms" but skips everything after that alphabetically. It also doesn't get any of the files that begin with _ , such as _Animations and so on. 

I can see in the procmon log that at 9:38:29.8328077 AM, it tries to create the 731604991.mod file, which is one that also is not being copied correctly, with a result of "name not found"

Maybe there are errors that I'm not seeing, and I also just probably don't understand this log well enough to comprehend exactly what is going on. I will include a drive link to the logfile since it is massive and can't upload directly.

https://drive.google.com/file/d/1HsofkRggE0TZbYlCUCS0t5b42SE3h962/view?usp=sharing

Link to comment
Share on other sites

  • Volunteer Moderator
10 hours ago, Markusius said:

It goes from "crafting" through "platforms" but skips everything after that alphabetically.

Actually, it fails at the following file and break the whole loop:

..\Platforms\Tree\Wedge\Sloped_Gate\Glass\PrimalItemStructure_Tree_Wedge_Sloped_Gate_Glass.uasset

The reason for this is that the entire path:

C:\SteamCMD\steamapps\common\ARK Survival Evolved Dedicated Server\Engine\Binaries\ThirdParty\SteamCMD\Win64\steamapps\workshop\content\346110\731604991\WindowsNoEditor\Platforms\Tree\Wedge\Sloped_Gate\Glass\PrimalItemStructure_Tree_Wedge_Sloped_Gate_Glass.uasset

Exceeds the MAX_PATH limit of 260 characters (with a length of 263). Shorten your base path and it will work as intended.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...