Admin / Trusted Player Tool - Game Save Visualiser

[Zargh]

@MirageUK Yes, that corresponds with the amount of RAM our VPS needs to load 892, aproxx 22 GB.

That is the reason why we are hoping so bad for a version of your program to clean up this monster.

 

Looks like you made it?

 

Greetings from Germany

 

[darkradeon]

1 hour ago, MirageUK said:

@Alpha Prime using official backups or your own server?  Is the file size for .ark larger than 2GB?  Do you have a copy I can download to investigate?

Still loading "NA-PVE-Official-TheIsland892" on my machine and no issues so far - other than the length of time and amount of RAM it is using. 22GB and counting.

I am sorry you had to load it I know, it's a big save (and my tribe is one of the biggest guilty XD). Hint: if you are able to make a connection with a client I observed the memory usage going down a bit.

Yeah I suspected there was some kind of limitation about memory but I confess, I am not so into C# and .net programming XD

we are using that save for primary real tests in the meantime we are getting ready to launch our private small cluster (we are also dealing to reduce some buildings and storage usages before final saves).

For a lot of unofficial that will use one or more official 2023 saves your amazing tool would be really precious to quickly find and wipe players data that will not join our cluster

Edited September 1, 2023 by darkradeon

[Alpha Prime]

2 hours ago, MirageUK said:

@Alpha Prime using official backups or your own server?  Is the file size for .ark larger than 2GB?  Do you have a copy I can download to investigate?

Still loading "NA-PVE-Official-TheIsland892" on my machine and no issues so far - other than the length of time and amount of RAM it is using. 22GB and counting.

Hi, I´m using my private server backup file from Nitrado

[MirageUK]

@Alpha Prime if it's less than 2GB can I have a copy so I can investigate?

Trying to trim the fat on these officials to a reasonable size atm so it doesn't take an hour each time I want to try read it into ASV to investigate further.

[Zargh]

Under 2 GB before loaded or after? Official Fjordur Server 1205 Save File is less than 2 GB and will extract to 12 GB aprox...

 

Maybe this helps?

 

https://arkfiles.com/pc/2023-08-30/nitradoeu321.newofficialfjordur1205.zip

 

[MirageUK]

@Zargh - 45 minutes to load up but got there in the end with a few minor modifications to my code.

Shutting down for the night now but will test it against the larger Island file from earlier today and if it works on that too I'll put together a release.

64k tames is quite a few lol.

Edited September 1, 2023 by MirageUK

[Zargh]

@MirageUK Nice. Thank you so much for your time and work, i really appreciate it. 

Pressing thumbs that it will work with our 892 save file also then. Wanted to give you a Thumb up, but can't, maybe my rank is too low here...

Please keep up your excellent work, have a nice evening.

 

[SpenceFoxtrot]

On 8/31/2023 at 5:29 PM, MirageUK said:

4.6.0.3 now available for download:

https://drive.google.com/file/d/1RpMh-56LKhQpFGnT7Q33f5bANUoli3Zo/view?usp=drive_link

Features

• Support for V11 format from official servers.. and any unofficial using "-newsaveformat -usestore" command line options. @darkradeon @gingai
• Changed header in Structures tab to more accurately read "Ally in Range".
• Changed Tribe data so it uses the latest date of either last player active or .arktribe file timestamp.

 

 

Working good so.

But, it's not possible to avoid "ally in range" and only use "last active tribe member" (We don't care about ally team... it's false data)

And for tribe date, integrate this ? (if all players are gone on cluster, and there is no logs when they refresh, ally will virtullaly refresh time of structures).


Edit : I forgot 1 thing...
The tribelogrelay plugin can grab borns...
Thats means the tribe log can be updated without any activity... like pvp with an offline raid.

The date of structure by refresh by tribe only will be the best for real "last activity".

Edited September 2, 2023 by SpenceFoxtrot

[darkradeon]

I noticed on those huge official saves that after server is running if one players connectes the memory mused goes down a lil bit.

[MultiKore]

Idk what im doing wrong but it refuses to open the Gen2 923 .ark file, I've tried so many different things but it fails to load....kinda annoying with a not very helpful error.

Edited September 3, 2023 by MultiKore

[Zargh]

@MultiKoreYou don't do anything wrong. Just wait patiently until @MirageUKhas done his tests and hopefully created a version that works with those large save files. 

 

If u read a few postings above u see where the problem is...

Edited September 3, 2023 by Zargh

[MirageUK]

Apologies for the delay.

I actually took a couple days off after the disheartening result of waiting 1.5 hours for The Island 892 to finally fail without pointing at anything obvious I had done wrong.

Back to work today but will try get back to it this evening and hopefully make some more progress.

[darkradeon]

@MirageUK do you think it would be possible to re-extract the .arkprofile files from the worldsave since it is using -usestore ? WC did the dumb decision to remove all .arkprofile files from the saves because they contained the last IP used by players.. Except this broke an easy migration to solo / local host since those profiles are required to get back the char..

 

edit: are there any updated sources for this project?

Edited September 5, 2023 by darkradeon

[MirageUK]

@darkradeon I might be able to read some of this data but I definitely won't be abe to write any of it. 

I honestly do not understand the ARK file format and changes I make to the reading of raw data are purely exploratory and not with any knowledge of how it is truly defined.  This is why I have so many issues and it takes me so long to come up with fixes when the file format itself changes and the toolkit I use no longer reads them in.   

I'm not one to quit though and still doing what I can to work out a fix.  It just takes so long to load everything so not getting very far each evevning in what time I have spare.  Even trying to load the Islan892 file up in single player to verify my findings is horrid.  15-20 minute load time then it's like "turn based" ARK.  I move forward a few steps.. then wait 30 seconds to move any further lol.

I did find that there are additional bytes in the Hibernation section that I thought I had worked around after spotting there was always a certain byte before the "count" of the hibernation entries started.  Turns out with Island892 there are multiple occurrences of this byte before the count so it's throwing everything out.  I really need to do some investigation and see if I can work out what the additional bytes are and how they lead into the actual hibernation entry data to fix this.

There is also a variance in how many bytes are saved in a cryopod - so might be more work around reading them in once I've solved the hibernation change.

...edit..

Sources for what project? ASV? - all on my Github. https://github.com/miragedmuk/ASV

If you mean the toolkit I use - there are a few forks but the main one that's active is the one used by ARK Smark Breeding and the team.  They are also actively looking into how to fix the reads for v11.

Edited September 6, 2023 by MirageUK

[darkradeon]

As for cryopods: there were multiple updates in the past to add gender and color regions data showing when looking at them, and to make them to work you had to uncryo and cryo again the dino. While 2 years (?) has passed I know some people still have very very old cryopod never "updated" to show such info. Not sure if there were in the past updates related to saddles to keep the colours on them (or maybe I am remembering wrong).

I will look at the source. I know 892 is a big save,probably one of the biggest of official pve.

[Oli4]

5 hours ago, darkradeon said:

As for cryopods: there were multiple updates in the past to add gender and color regions data showing when looking at them, and to make them to work you had to uncryo and cryo again the dino. While 2 years (?) has passed I know some people still have very very old cryopod never "updated" to show such info. Not sure if there were in the past updates related to saddles to keep the colours on them (or maybe I am remembering wrong).

I will look at the source. I know 892 is a big save,probably one of the biggest of official pve.

 

I was curious to see which saves were the biggest so I made a list:

https://gist.github.com/ark-automated/2dd592eb7295c36f36c209f839f4765b

 

 

[darkradeon]

@MirageUK would be possible for some "corrupted pods data" to cause your issues: in the past it was possible to transfer stuff like chalk golem and ice golem outside valguero, as well it was possible to keep undeployable zoomdodo pods.

[MirageUK]

Quote

5.1G ./PvP/nitradoeu74.newofficialvalguero544/Valguero_P.ark

5.1GB... guess I should stop moaning I'm having to debug a 2.8GB save then 🤣

[darkradeon]

22 minutes ago, MirageUK said:

5.1GB... guess I should stop moaning I'm having to debug a 2.8GB save then 🤣

my save - 3 years old tribe merged with other 2 big and other small - worth like 350MB of those 2.8GB :V

But I am grinding a gazillion of stuff and planning the destruction of "some" buiildings XD

I would not be surprised if in old saves there is a small portion of "corrupted data" or better something that should be not there and that could trigger crashes as well issues on reading the file...

Edited September 6, 2023 by darkradeon

[Oli4]

On 9/1/2023 at 1:00 PM, MirageUK said:

@Alpha Prime if it's less than 2GB can I have a copy so I can investigate?

Trying to trim the fat on these officials to a reasonable size atm so it doesn't take an hour each time I want to try read it into ASV to investigate further.

I can share a bunch of lower sized save files that still run into issues with ASV if that would help?

nitradouswest136.newofficialaberration266/Aberration_P.ark is 421MB and can't be loaded (at least for me) on ASV 4.6.0.3

I also DMed you on Discord(at least I hope I DMed the correct person) to offer some assistance if you are interested. If you prefer to use STA/no help that is also perfectly fine of course

[METALGOD]

I get this error "content failed to load. Please check settings or refresh download to try again."

officialtheisland763

and

officialragnarok210

[MroczneZakamarki]

This error occurred when I turned on the new saving format - newsaveformat on my unoficial server (of course I have the latest version of ASV that allows to read them). I also tried -usestore -converttostore -BackupTransferPlayerDatas, but it didn't help, it only increased the save file.

I would prefer to leave the player files separate from the map, which is sometimes useful, for example when I delete old players and their tribe data. Is it possible to use the new save format but still be able to use player files separately and read them as before in ASV?

@MirageUK  BTW I love your app. It makes working with the server incredibly easy. Do you have an option somewhere to make a donation for a coffee for you for your commitment and hard work?

[MirageUK]

@METALGOD - Work in progress, changes to the file format have led to me spending a number of hours working through byte by byte to see what has changed and if possible re-write the toolkit code to read in the new format. I'm working through debugging against Island PVE 892 and will update you all once I know more and hopefully have a fix to release.

@MroczneZakamarki - Regardless of any fix I come up with ASV will always still attempt to parse .arktribe and .arkprofile data if it is available along with the .ark save file itself if that's what you mean?  I can't even recall what is used from the .arkprofile data in ASV tbh.  Also - I wasn't aware that my app was actually used by so many but have never accepted any donation.  It started as something to benefit myself and my small tribe on our own private server, if it benefits others I'm happy to continue supporting it in what free time I have.

Hopefully getting somewhere though - I managed to painfully reduce the .ark file size of 892 from 2.8GB to 1.5GB and got it all loading in ASV but when I tried the original it still errors out so I'm now working on that again to see why and what is causing the issues.

 

..edit..

FYI THIS has been my "bedtime reading" for the last week after work in my spare time - not the usual well defined "specification" I'm used to working with on the day job.

Edited September 9, 2023 by MirageUK

[MirageUK]

Quick upate. I think we finally got there!!

Between myself, Alex and Cadei who help with the ASB team it appears we  can now finally read in Island892.

I'll re-test all the other maps mentioned and a couple of older ones I have just to make sure but if no issues are found after work with them I'll put together a release this evening.

Over 110k tames in cryo.. total of 117,259 tames!! No wonder it was still going after I left it 5 hours this morning.  Thankfully I got a fix.. this backup now loads on my machine in approx. 15 minutes.

... edit ..

Tested and working with:

• The Island 892
• Aberration 266
• Ragnarok 210
• The Island 763

Still working on fixes to read in Valguero 544 - the stored creatures appear to be split between two portions of the file with it being more than 4GB in size.

Edited September 11, 2023 by MirageUK

[darkradeon]

and 892 is only 3 years and a half old :V