Jump to content

ARK Tools v0.6.4 - tools for reading and manipulating ark savegame files


Qowyn

Recommended Posts

15 minutes ago, Toni said:

You anyway have to read each json file for players and tribes. So you can read the file creation date and changing date there. It isn't needed to include this in the json files.

 

 

 

 

Actually I would need to read it on the actual arkprofile and tribeprofile files, since the timestamps on the jason files is based on when do you the extracts with the java package.

Would be kind of a time waste when op is already reading the files for creating the extracts

Link to comment
Share on other sites

  • Replies 369
  • Created
  • Last Reply
3 minutes ago, FatDude said:

Actually I would need to read it on the actual arkprofile and tribeprofile files, since the timestamps on the jason files is based on when do you the extracts with the java package.

Would be kind of a time waste when op is already reading the files for creating the extracts

I have edited my post while you was answering. Please read again the above post

Link to comment
Share on other sites

40 minutes ago, Toni said:

For data/informations like active players last week, its enough to refresh the data once per 24 hours.

For the other informations (tribe / players), its needed to refresh it after each server save.

So including this "only once per 24 hours needed data" in Qowyns library, is overhead. It reads this data each time even if its not needed at this time. So adding this would increase the generation time of the .json files a bit.

 

What you requesting here can be done for example this way (Only once per 24 hours):

Powershell example to get amount of acitve players last month:

$filepath3 = "D:\arkdedicated\ark\ShooterGame\Saved\SavedArks"
$filetype3 = "*.arkprofile"
[array]$activeprofiles = get-childitem $filepath3 $filetype3 | where-object {$_.LastWriteTime -ge (get-Date).AddDays(-31)}
$activeprofilescount = $activeprofiles.count


So you know the amount of players active last month (count)... and you have a list of the active players (array).

 


Generating the tribe datas already now takes 26.5 seconds for me  (310MB savefile / 221 tribes).  Servers with less performance and not a Samsung 950 ssd will need much much longer for that.

Only my 50 cents

 

Actually I'm gonna import all data every 5 minutes so to staging tables, then gonna model slowly changing dims and import all facts with map keys  ;-)

Plan is among others to use the tribe logs to generate events, pvp ranking system and lots of other cool stuff :-)

Adding the file timestamps to the jason files should not increase the extraction time much, or else I will just generate myself but it will create some overhead in my load process

 

Did a small test on my dev system (nothing fancy)

players: Total time 15048 ms
(3034 arkprofiles)

Loading tribes and writing info finished after 2261 ms
(876 tribeprofiles)

All Dinos
Total time 1335 ms
(300mb world save)

Link to comment
Share on other sites

7 minutes ago, FatDude said:

Did a small test on my dev system (nothing fancy)

players: Total time 15048 ms
(3034 arkprofiles)

Loading tribes and writing info finished after 2261 ms
(876 tribeprofiles)

All Dinos
Total time 1335 ms
(300mb world save)

 

So your not using all the capabilities of the library

Add this at the end of your tribe generation command -->     -p -s --pretty-printing --items --creatures --structures --bases

Whats your result with that? I think its something in the range of 60 to 170 seconds according to your other results which i compared with mine.

Link to comment
Share on other sites

14 minutes ago, Toni said:

 

So your not using all the capabilities of the library

Add this at the end of your tribe generation command -->     -p -s --pretty-printing --items --creatures --structures --bases

Whats your result with that? I think its something in the range of 60 to 170 seconds according to your other results which i compared with mine.

Dont currently see the need for tagging bases, creatures also not sure sine I already extracted all dinos with tamed by info. Can just join to players and tribe in the db

Please enlighten me if its useful for anything besides giving a rough overview of items within range of the tagged base. What would the scenario be ? to figure out if xx tribe got 3 or 5 rocket launchers ?? or xxx blueprint ??

On another note I can extract players and tribes with a modified PHP-ArkData-master php script and it only takes a few seconds for the above data set :-)

Would image the extract with ark tools would be faster if all player info was added in one big file instead of 3000 (lots of reading and writing of files there, first for generating and later for reading while importing to db)

Anyway for now I will work on importing players, tribes and dinos every 5-10 minutes to generate some handy stats with history tracking. Tribe logs will be coverted to single line events with history for events and pvp ranking.

Once Im done with the DB model, I will use whichever tool is the fastest for generating the data I need ;-)

Link to comment
Share on other sites

The --bases option is mainly for tribes with multiple bases, if put on a website they could see how much of everything is at which base. And also where they parked that damn argy.

Marking their base could be used to show it on a map (or it would be possible if I didn't forget to output the location of the base).

And forgot to mention that i will add IDs to dinos.

Link to comment
Share on other sites

25 minutes ago, FatDude said:

Dont currently see the need for tagging bases, creatures also not sure sine I already extracted all dinos with tamed by info. Can just join to players and tribe in the db

You can create a map with all bases displayed. (have not done it till now myself).

"tamed by" is not available if --> tribesettings not set to "personal owned dinos". But yeah, you can get the amount of tribe-dinos by counting the rows of your tamed dinos table. This function is not really needed in the library.

 

29 minutes ago, FatDude said:

Please enlighten me if its useful for anything besides giving a rough overview of items within range of the tagged base. What would the scenario be ? to figure out if xx tribe got 3 or 5 rocket launchers ?? or xxx blueprint ??

 

I used this data to give tribemembers the possibility to view all the tribe owned items and tribe owned structures.

33 minutes ago, FatDude said:

Would image the extract with ark tools would be faster if all player info was added in one big file instead of 3000 (lots of reading and writing of files there, first for generating and later for reading while importing to db)
 

The sense of .json files is the possiblity to POST it to another server.  If all player infos would be added to one file, then they couldn't be used for http POSTS. I think thats the reason.

 

 

Link to comment
Share on other sites

42 minutes ago, Qowyn said:

I'll add timestamps to the next version. And a new command that does everything (creatures, players, tribes) in one go, that should help with the total extraction time.

I'm also curious if you measured the first run or a run after that, subsequent runs might be much faster because of OS file caching.

Sounds nice

No differences between first and following runs on a windows system (Win10). The file reading isn't the bottleneck part  (max usage 4%). CPU usage is 100% for 0.5 seconds..and then 20% for the rest of the runtime.

Link to comment
Share on other sites

54 minutes ago, Qowyn said:

The --bases option is mainly for tribes with multiple bases, if put on a website they could see how much of everything is at which base. And also where they parked that damn argy.

Marking their base could be used to show it on a map (or it would be possible if I didn't forget to output the location of the base).

And forgot to mention that i will add IDs to dinos.

Great mate, an option to combine all players (and same for tribes) in 1 single file instead of multiple files would also be handy for me atleast ;-)

Having over 8k player profiles + 1.5k tribes on one of my servers, its just more handy with only a few jason files instead of 10k + :-) (and probably also faster when I do the import to my database)

Link to comment
Share on other sites

While analyzing the tamed dinos data it seems to include dead dinos or perhaps there is some data issues in the extract ?

TP'ed to the location on the centermap and the tamed dinos was not there nor other dinos of that type (dodo)

Im using the creatures to export all dinos.

 

Another funny thing the somewhat bugged spawns on the Center map assuming the extract is just somewhat correct.

I knew the mosa spawns was bugged (overspawning), but coel and trilo ?

type    nr
C:\ark-tools\dinos\Trilobite_Character_C.json    19235
C:\ark-tools\dinos\Coel_Character_BP_C.json    8093
C:\ark-tools\dinos\Mosa_Character_BP_C.json    3627
C:\ark-tools\dinos\Dragonfly_Character_BP_C.json    3626

Link to comment
Share on other sites

Hey, thanks to Qowyn for keeping busy with the tool!
I lost track of the versions a bit, downloaded the latest ark-tools but the java commands i used to use won't work

What i have now is the ark-tools map with:

Ark-tools:
Ark_data.json
ark-tools.exe
ark-tools.jar
ark-tools.sh
TheCenter.ark

Shift+right click: command prompt: 

How do i get the text files with tamed dinos again?

Thanks in advance

Link to comment
Share on other sites

Hi !

Very nice work on this tool ! Surprisingly it's hard to find with google, but I was very happy to find it as it seems very powerful. I have a problem using it though. I can't seem to import a tamed creature data from one saved ark to another. I use your tool this way :

ark-tools.exe export TheIsland_src.ark --creature "Foo" creature.json
ark-tools.exe import TheIsland_dest.ark creature.json TheIsland_edited.ark

However, in TheIsland_edited.ark, I can't see the creature I have imported. Neither in game or in the list of tamed creature your tool can build or if I try to export the creature. Did I made a mistake ? Also, the export command displays stack traces non-deterministically (roughly half the time on the same command) when used with --verbose :
 

sept. 03, 2016 1:47:36 PM com.lmax.disruptor.FatalExceptionHandler handleEventException
GRAVE: Exception processing: 0 qowyn.ark.disruptor.JsonObjectEvent@3a1bbf6a
javax.json.stream.JsonGenerationException: Illegal method during JSON generation, not valid in current context IN_NONE
        at org.glassfish.json.JsonGeneratorImpl.write(JsonGeneratorImpl.java:270)
        at org.glassfish.json.JsonPrettyGeneratorImpl.write(JsonPrettyGeneratorImpl.java:53)
        at qowyn.ark.ArkSavegame.lambda$writeJson$17(ArkSavegame.java:510)
        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129)
        at java.lang.Thread.run(Unknown Source)

Exception in thread "pool-1-thread-1" java.lang.RuntimeException: javax.json.stream.JsonGenerationException: Illegal method during JSON generation, not valid in current context IN_NONE
        at com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
        at com.lmax.disruptor.dsl.ExceptionHandlerWrapper.handleEventException(ExceptionHandlerWrapper.java:18)
        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:148)
        at java.lang.Thread.run(Unknown Source)
Caused by: javax.json.stream.JsonGenerationException: Illegal method during JSON generation, not valid in current context IN_NONE
        at org.glassfish.json.JsonGeneratorImpl.write(JsonGeneratorImpl.java:270)
        at org.glassfish.json.JsonPrettyGeneratorImpl.write(JsonPrettyGeneratorImpl.java:53)
        at qowyn.ark.ArkSavegame.lambda$writeJson$17(ArkSavegame.java:510)
        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129)
        ... 1 more

 

Link to comment
Share on other sites

You could try to run both commands with --async=no . I stared at the code for about an hour now and have no explanation for the exception yet. Might need to take a look at that save.

A word of warning though: this stuff is kind of experimental. While i already did manage to move dinos betweens saves this way, i had to delete structures from platform saddles in the past. And so far no one beside me seems to use those commands.

Link to comment
Share on other sites

Amazing ! The new version fixed the problem I had with import. It now works perfectly !

A few days ago, a tamed dinosaur on my server disappeared without death message. It seems it was removed from the game due to some kind of bug - falling through terrain or flying above height limit ? Your tool was showing that it was actually removed from the ark world. With its export/import feature, I was able to get the dinosaur data from the last saved world and restore it to the last saved world. This is a very convenient way to address game bugs with very little effort. (from the server admin)

Thank you very much for your work, for developing this tool and for the very quick reply and fix on its issue. The players on my server are forever in your dept (and, it seems, that some of them are in love with you).

Link to comment
Share on other sites

Thanks for the last update, really appreciated.

Really nice with the dino ids. Already used it to track some lost dino between different world saves. Man its really helpfull

 

Some stuff to look at when you have time.

1/3 of my ark profiles says possible corrupted and is not exported it seems.

(On a few days old server running Scorched Earth)

 

 

 

 

 

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.


×
×
  • Create New...