Jump to content

ARK Tools v0.6.4 - tools for reading and manipulating ark savegame files


Qowyn

Recommended Posts

Lost a dino and you can't find it? Want to see if it is still alive at all? Or maybe you are a server admin and want your players to be able to look for this info themselves, maybe even give them a list of all wild dinos including the relevant stats for breeding?

Then you've come to the right place.

Current State

ARK Tools is currently available as a platform-independent console program, with a launcher for Windows and a shell script for other OSes. Run ark-tools.exe or ark-tools.sh to get a list of possible commands and options.

A GUI version is planned.

The tools themselves as well as the library they use are written in Java (with probably an overuse of Lambdas and Streams and a lack of JavaDoc) and the sources can be found here:

Roadmap

  • 0.7.0
    • Breaking change of xToJson formats (again), improved support for native array properties (properties which may or may not be ArrayProperty but still contain multiple values)
    • Implement missing stuff to make the database system useful (e.G. json driver should be able to write data to multiple files, skip refreshing files without changes, implement missing support for cluster directory)
    • Add more things to the modify command
    • Add scripting support based on JSR-223 (Java includes javascript by default, other options are Lua, Ruby, Groovy, Python...)

Documentation

A tiny bit of documentation is now available here: https://github.com/Qowyn/ark-tools/wiki

A general documentation about the file formats is available here: https://github.com/Qowyn/ark-savegame-toolkit/wiki

Downloads

Downloads are available as releases on GitHub here: https://github.com/Qowyn/ark-tools/releases/latest

The data required for Class to human readable name translation can be obtained by using the updateData command or downloading them here:https://ark-tools.seen-von-ragan.de/data-download/

Changelog

  • 0.6.4
    • Fixed broken players command
    • players command can now detect LocalPlayer.arkprofile
    • Hibernation support for debug commands
  • 0.6.3
    • Important fixes for cluster and LocalProfile files, can affect map-saves too (I made a typo about 4 months ago, causing some properties to be written as Int8Property instead of ByteProperty)
    • Command export can now select objects by class (e.G. PrimalItemConsumable_Egg_Wyvern_Fertilized_Poison_C for green wyvern eggs)
    • Added switches --without-references and --without-components to export command (use --without-references for items, unless you want the whole base too)
    • Command import is now able to import items into cluster files and into the LocalProfile (with caveats, items will lose durability if you exported them from a map)
    • Command modify is now able to mass-delete objects in maps, cluster and LocalProfile files
    • All creature commands, the tribe command and db command are now aware of hibernation
    • Improved detection of all object types, no longer based on their class name
  • 0.6.2
    • Added support for dropped items, making --non-player actually useful
    • Renamed --non-player to --non-players (the old name will still work)
    • Id of main inventory is now myInventoryComponent for all object types
  • 0.6.1
    • Added missing DinoAncestor writers
    • Added names of ancestors
    • Added Polar Bear, Ragnarok Fire Wyvern and Ice Wyvern to the list of creatures (auto-detection fail)
    • Added --non-player switch to tribes command
    • Fixed lat and lon values for creatures, players and structures
  • 0.6.0
    • Added (still incomplete) Database driver system
    • Replaced javax.json with Jackson, massive performance improvements for mapToJson command (other commands benefit too)
    • Adjustments to the names of some fields
    • Increased amount of gathered data for all commands
    • Allow parallel processing for players, tribes and cluster commands
    • Amount of parallelism can no be controlled via --thread-count, defaults to the number of cores available to the JVM
    • Full support for version 7 to 9 savegames, version 4 local player profile and version 4 cluster storage
    • New options for some commands, "--write-all-fields" can now be used almost everywhere
    • New option "--allow-broken-file" for xToJson  commands will try to salvage as much data from broken files as possible
  • 0.5.4
    • Read-only support for new savegame format versions 8 & 9
  • 0.5.3
    • Experimental support for new savegame format version 7
  • 0.5.2
    • Fixed NPE in modify command
    • Fixed handling of high (>32767) itemStatValues for all item printing commands
  • 0.5.1
    • Added support for new v3 cluster data format
    • Added LatLong data for Ragnarok and TheVolcano
    • Added type name for uploaded dinos
  • 0.5.0
    • Many changes to ark-savegame-toolkit to improve stability and useability
    • Breaks compatibility with old json formats (thingToJson and jsonToThing commands)
    • Added support for previously unknown field in map saves (crash fix)
    • Added support for the new PlayerLocalData.arkprofile Version 3
    • Added updateData command: checks for and downloads newer ark_data.json files
    • Added version command: outputs the installed version of ark-tools
    • Fixed output of unsigned 8 bit values: attributes and colors of creatures
    • Removed ark_data.json from release
  • 0.4.2
    • Added compatibility for changes to .arkprofile files (fix contributed by Kakoen on GitHub)
    • Improved error message for broken installations (missing ark_data.json)
  • 0.4.0
    • Added reading and writing support for sav files (except for text properties)
    • Added --write-all-fields switch to creature commands, should write (almost) all fields
    • Added team, playerId, and ownerName to creature commands
    • Added --tribeless switch to tribe data extraction, this will list everything not belonging to a tribe in tribeless.json
    • Added GPS data for PGAs
    • Added --max-age option to ignore unchanged player profiles
    • Added --positions switch to player command, this will load the map to get the current position of the player
    • Improved player data extraction time by lazy loading tribe data
    • Improved crash protection against broken files by catching all unexpected errors for player profiles, tribe data and cluster data
    • Improved detection of tamed creatures
    • Improved detection of array of struct properties
    • Fixed tribe command ignoring equipped items
    • Fixed crash while processing saves from Volcano map
  • 0.3.0
    • New cluster command which extracts data from files in the clusters directory - uploaded items and creatures
    • New commands to convert PlayerLocalData.arkprofile to json and back
    • New command to modify existing .ark* and cluster files, allows converting Valhalla Beavers to normal beavers
    • Extract colors from creatures
    • Extract stats and colors from eggs
    • Create items from scratch
  • 0.2.7
    • Really fixed tamed creature detection
    • Fixed releasing new versions while half asleep
  • 0.2.6
    • Added ShigoIslands latlon data
    • ark_data.json is now loaded from the same directory as the jar
    • Fixed detection of tamed creatures that have been transfered
    • Added cloudToJson/c2j and jsonToCloud/j2c commands
  • 0.2.5
    • Fixed import command. Again.
  • 0.2.4
    • Fixed level output for creatures with base/wild level 1
    • Added --inventory option to players command
    • --inventory and --items now accept an argument: summary or long, summary will just count items, long will output each item stack with extended information
  • 0.2.3
    • Added GPS values for Valhalla and Mortem Tupiu
    • Added Rafts to creatures/wild/tamed and tribes --creatures commands
    • Fixed import command
    • Added location of bases to --bases option
    • Added id field to creatures/wild/tamed command, needs testing to see if it is stable
    • Updated included ark_data.json to 246.2
  • 0.2.2
    • Added --with-names and --without-dupes options to classes command
    • Changed source of ark data to a new tool, the new data is directly extracted from the game
    • Added name mapping for structures
    • Added global --lang option to output localized names
    • Fixed potential bug in export and import commands
    • Tribes command now ignores items held by sleeping players
    • Tribes command now includes items lying on the floor
  • 0.2.1
    • improved help for syntax errors
    • skip and report potentially broken .arkprofile and .arktribe files
    • new option --verbose to print a stacktrace for broken files
    • new option --bases for the tribes list, for details see above
  • 0.2.0
    • pretty printing of generated JSON files is now optional and disabled by default
    • renamed animals command to creatures
    • added more information to creature list
    • statistic block is now optional and disabled by default
    • conversion commands for .arkprofile and .arktribe
    • hide general options in specific command help
    • include data about items and creatures for the basegame, extracted from the wiki with help from https://github.com/DerekRies/arkpy
    • use extracted data to output creature and item names
    • new command to generate player list
    • new command to generate tribe list
    • new command to extract included list of maps for LatLng calculations
    • automatically import LatLng values on startup if external file is created (so you can add support for different maps)
    • display commands in groups with their descriptions
Link to comment
Share on other sites

  • Replies 369
  • Created
  • Last Reply

It's actually supposed to work exactly like the animals export.

E.g.: java -jar ark-tools.jar players /path/to/save.ark /path/to/output/directory

The main reason for providing a file name is consistency, the players command does not even use it. Instead it will look for .arkprofile files in the same directory.

The tribes command on the other hand may use the provided savegame to gather more info about the tribes if needed, e.g. if using --structures or similiar options.

Edit: I'll probably rename DIRECTORY to OUTPUT_DIRECTORY for the next version to make things clearer.

Link to comment
Share on other sites

12 minutes ago, Qowyn said:

It's actually supposed to work exactly like the animals export.

E.g.: java -jar ark-tools.jar players /path/to/save.ark /path/to/output/directory

The main reason for providing a file name is consistency, the players command does not even use it. Instead it will look for .arkprofile files in the same directory.

The tribes command on the other hand may use the provided savegame to gather more info about the tribes if needed, e.g. if using --structures or similiar options.

Edit: I'll probably rename DIRECTORY to OUTPUT_DIRECTORY for the next version to make things clearer.

C:\Scripts\ark\ark_tools_v2>java -Xmx8G -jar ark-tools.jar players C:\arkdedicated\ark\ShooterGame\Saved\SavedArks\TheIsland.ark C:\Scripts\ark\ark_tools_v2\output\players\ -p -s
java.lang.NullPointerException
        at qowyn.ark.tools.PlayerListCommands.players(PlayerListCommands.java:70)
        at qowyn.ark.tools.App.main(App.java:98)

C:\Scripts\ark\ark_tools_v2>java -Xmx8G -jar ark-tools.jar tribes C:\arkdedicated\ark\ShooterGame\Saved\SavedArks\TheIsland.ark C:\Scripts\ark\ark_tools_v2\output\tribes\ -p -s
java.lang.NullPointerException
        at qowyn.ark.tools.PlayerListCommands.tribes(PlayerListCommands.java:226)
        at qowyn.ark.tools.App.main(App.java:98)

 

And some working command to show that my path's and so on are correct:

C:\Scripts\ark\ark_tools_v2>java -Xmx8G -jar ark-tools.jar tamed C:\arkdedicated\ark\ShooterGame\Saved\SavedArks\TheIsland.ark C:\Scripts\ark\ark_tools_v2\output\tamed\ -p -s
Reading finished after 1668 ms
Dumping finished after 137 ms
Total time 1805 ms

 

Link to comment
Share on other sites

7 hours ago, Qowyn said:

For those interested, all fileformats which ark-savegame-toolkit can read are now documented here:

https://github.com/Qowyn/ark-savegame-toolkit/wiki

Thanks a lot for the documentation! I had already figured out a lot about the file format from spending hours looking at it in a hex editor, but then you came along with a working library! So, I referenced your code to fill in the missing pieces and started writing a C# library for reading/writing these files (only reading is implemented so far, but it works well). If there was a document like this back when I started, I would have been done a long time ago. So, thank you for this service you have done for the community!

I hope that people will use this information to make helpful tools for server admins. I have been working on some C# libraries for various Ark things: server queries, rcon sessions, and now save file reading and editing. I hope to eventually build a nice remote server management tool out of all of it, but it will take a long time since I am only working on it a couple hours per week. Hopefully people with more free time will beat me to it

Link to comment
Share on other sites

@Crystal Thats my hope too. Sorry for the late reply, sometimes i'm just lazy. Especially happy that it's helping you since you seem to help everyone here in the Server Administration section all forums.

 

In other news: 0.2.2 is out, with support for most translated languages. Translation files are available here:

https://ark-data.seen-von-ragan.de/data-download/ark-tools/

Translations are used to get readable/normal names for creatures, structures and items in your language.

Link to comment
Share on other sites

New latLonCalculator.json now available at https://ark-data.seen-von-ragan.de/data-download/ark-tools/latLonCalculator.json

Includes extracted values for correct Latitude and Longitude calculation for TheCenter, Valhalla and Mortem Tupiu. Also added missing language files for languages which only have .locres files.

Link to comment
Share on other sites

Im such a java noob, but is there any easy way to export to mysql or mssql databse ?

What format is the output files for dinos and structures ?

Basically I want to build a web interface around the whole data set, updated after each world save including history tracking for easy back tracking of tamed dinos etc ;-)

I know how to model the database part including type 2 tracking + web part, but I just suck at java ;-/

Link to comment
Share on other sites

16 hours ago, GreatApe said:

The files are exported in json format I believe so should be easy to use with any language.  What language are you using?  I want to do the same (time, always need more time) using php and mysql.

Most likely SSIS packages for ETL, sql server 2016 and php for webpart

Should be able to do a poc within a few days, lets see

Link to comment
Share on other sites

ARK (and UE4 in general) writes things to a temporary file, deletes or renames the old file, and then renames the temporary file. And my tools open files in read-only mode (except for manipulation commands of course), so they are unable to corrupt anything. Worst case would be that ARK is unable to delete the old file, but this can be prevented by using file watching APIs to start data extraction as soon as the temporary file gets renamed.

Link to comment
Share on other sites

On 29.8.2016 at 10:00 PM, FatDude said:

Im such a java noob, but is there any easy way to export to mysql or mssql databse ?

What format is the output files for dinos and structures ?

Basically I want to build a web interface around the whole data set, updated after each world save including history tracking for easy back tracking of tamed dinos etc ;-)

I know how to model the database part including type 2 tracking + web part, but I just suck at java ;-/

I had the same problem ^^

Use a powershell script. Read the file contents and POST it to your webserver where you save it to your database. Surely not the most performant way to do it, but it works

 

What is type 2 tracking?

I do not have a history till now because it would be too much data if i create a new database entry for each update. 

Link to comment
Share on other sites

17 hours ago, Toni said:

I had the same problem ^^

Use a powershell script. Read the file contents and POST it to your webserver where you save it to your database. Surely not the most performant way to do it, but it works

 

What is type 2 tracking?

I do not have a history till now because it would be too much data if i create a new database entry for each update. 

Using MS sql sever and SSIS packages for importing data atm, might convert to mysql once poc is done.

Will do slowly changing dimension type 2 on most relevant columns in order to track history, like when did a player change tribe or when did a dino disapear or was tamed etc.

As for size of DB, just use compression

 

Some feature requests for the op

Add created and lastchanged timestamps to tribes and ark profiles, like below

(Great for filtering on active players past week etc)

          public 'FileCreated'    (length=15)
          public 'FileUpdated'    (length=15)

 

Add some kind of unique identify to each dino. It would make it easier to track. I assume there must be some kind of object id or similar that does not change in the save files

Link to comment
Share on other sites

31 minutes ago, FatDude said:

Some feature requests for the op

Add created and lastchanged timestamps to tribes and ark profiles, like below

(Great for filtering on active players past week etc)


          public 'FileCreated'    (length=15)
          public 'FileUpdated'    (length=15)

 

Add some kind of unique identify to each dino. It would make it easier to track. I assume there must be some kind of object id or similar that does not change in the save files

For data/informations like active players last week, its enough to refresh the data once per 24 hours.

For the other informations (tribe / players), its needed to refresh it after each server save.

So including this "only once per 24 hours needed data" in Qowyns library, is overhead. It reads this data each time even if its not needed at this time. So adding this would increase the generation time of the .json files a bit.

 

What you requesting here can be done for example this way (Only once per 24 hours):

Powershell example to get amount of acitve players last month:

$filepath3 = "D:\arkdedicated\ark\ShooterGame\Saved\SavedArks"
$filetype3 = "*.arkprofile"
[array]$activeprofiles = get-childitem $filepath3 $filetype3 | where-object {$_.LastWriteTime -ge (get-Date).AddDays(-31)}
$activeprofilescount = $activeprofiles.count


So you know the amount of players active last month (count)... and you have a list of the active players (array).

 


Generating the tribe datas already now takes 26.5 seconds for me  (310MB savefile / 221 tribes).  Servers with less performance and not a Samsung 950 ssd will need much much longer for that.

Only my 50 cents

 

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.


×
×
  • Create New...