Qowyn Posted August 8, 2016 Share Posted August 8, 2016 Lost a dino and you can't find it? Want to see if it is still alive at all? Or maybe you are a server admin and want your players to be able to look for this info themselves, maybe even give them a list of all wild dinos including the relevant stats for breeding? Then you've come to the right place. Current State ARK Tools is currently available as a platform-independent console program, with a launcher for Windows and a shell script for other OSes. Run ark-tools.exe or ark-tools.sh to get a list of possible commands and options. A GUI version is planned. The tools themselves as well as the library they use are written in Java (with probably an overuse of Lambdas and Streams and a lack of JavaDoc) and the sources can be found here: https://github.com/Qowyn/ark-tools - Source of ark-tools itself https://github.com/Qowyn/ark-tools-web - WIP webinterface for creatures data, made with Polymer https://github.com/Qowyn/ark-savegame-toolkit - Library for low-level reading and writing of .ark, .arktribe and .arkprofile files Roadmap 0.7.0 Breaking change of xToJson formats (again), improved support for native array properties (properties which may or may not be ArrayProperty but still contain multiple values) Implement missing stuff to make the database system useful (e.G. json driver should be able to write data to multiple files, skip refreshing files without changes, implement missing support for cluster directory) Add more things to the modify command Add scripting support based on JSR-223 (Java includes javascript by default, other options are Lua, Ruby, Groovy, Python...) Documentation A tiny bit of documentation is now available here: https://github.com/Qowyn/ark-tools/wiki A general documentation about the file formats is available here: https://github.com/Qowyn/ark-savegame-toolkit/wiki Downloads Downloads are available as releases on GitHub here: https://github.com/Qowyn/ark-tools/releases/latest The data required for Class to human readable name translation can be obtained by using the updateData command or downloading them here:https://ark-tools.seen-von-ragan.de/data-download/ Changelog 0.6.4 Fixed broken players command players command can now detect LocalPlayer.arkprofile Hibernation support for debug commands 0.6.3 Important fixes for cluster and LocalProfile files, can affect map-saves too (I made a typo about 4 months ago, causing some properties to be written as Int8Property instead of ByteProperty) Command export can now select objects by class (e.G. PrimalItemConsumable_Egg_Wyvern_Fertilized_Poison_C for green wyvern eggs) Added switches --without-references and --without-components to export command (use --without-references for items, unless you want the whole base too) Command import is now able to import items into cluster files and into the LocalProfile (with caveats, items will lose durability if you exported them from a map) Command modify is now able to mass-delete objects in maps, cluster and LocalProfile files All creature commands, the tribe command and db command are now aware of hibernation Improved detection of all object types, no longer based on their class name 0.6.2 Added support for dropped items, making --non-player actually useful Renamed --non-player to --non-players (the old name will still work) Id of main inventory is now myInventoryComponent for all object types 0.6.1 Added missing DinoAncestor writers Added names of ancestors Added Polar Bear, Ragnarok Fire Wyvern and Ice Wyvern to the list of creatures (auto-detection fail) Added --non-player switch to tribes command Fixed lat and lon values for creatures, players and structures 0.6.0 Added (still incomplete) Database driver system Replaced javax.json with Jackson, massive performance improvements for mapToJson command (other commands benefit too) Adjustments to the names of some fields Increased amount of gathered data for all commands Allow parallel processing for players, tribes and cluster commands Amount of parallelism can no be controlled via --thread-count, defaults to the number of cores available to the JVM Full support for version 7 to 9 savegames, version 4 local player profile and version 4 cluster storage New options for some commands, "--write-all-fields" can now be used almost everywhere New option "--allow-broken-file" for xToJson commands will try to salvage as much data from broken files as possible 0.5.4 Read-only support for new savegame format versions 8 & 9 0.5.3 Experimental support for new savegame format version 7 0.5.2 Fixed NPE in modify command Fixed handling of high (>32767) itemStatValues for all item printing commands 0.5.1 Added support for new v3 cluster data format Added LatLong data for Ragnarok and TheVolcano Added type name for uploaded dinos 0.5.0 Many changes to ark-savegame-toolkit to improve stability and useability Breaks compatibility with old json formats (thingToJson and jsonToThing commands) Added support for previously unknown field in map saves (crash fix) Added support for the new PlayerLocalData.arkprofile Version 3 Added updateData command: checks for and downloads newer ark_data.json files Added version command: outputs the installed version of ark-tools Fixed output of unsigned 8 bit values: attributes and colors of creatures Removed ark_data.json from release 0.4.2 Added compatibility for changes to .arkprofile files (fix contributed by Kakoen on GitHub) Improved error message for broken installations (missing ark_data.json) 0.4.0 Added reading and writing support for sav files (except for text properties) Added --write-all-fields switch to creature commands, should write (almost) all fields Added team, playerId, and ownerName to creature commands Added --tribeless switch to tribe data extraction, this will list everything not belonging to a tribe in tribeless.json Added GPS data for PGAs Added --max-age option to ignore unchanged player profiles Added --positions switch to player command, this will load the map to get the current position of the player Improved player data extraction time by lazy loading tribe data Improved crash protection against broken files by catching all unexpected errors for player profiles, tribe data and cluster data Improved detection of tamed creatures Improved detection of array of struct properties Fixed tribe command ignoring equipped items Fixed crash while processing saves from Volcano map 0.3.0 New cluster command which extracts data from files in the clusters directory - uploaded items and creatures New commands to convert PlayerLocalData.arkprofile to json and back New command to modify existing .ark* and cluster files, allows converting Valhalla Beavers to normal beavers Extract colors from creatures Extract stats and colors from eggs Create items from scratch 0.2.7 Really fixed tamed creature detection Fixed releasing new versions while half asleep 0.2.6 Added ShigoIslands latlon data ark_data.json is now loaded from the same directory as the jar Fixed detection of tamed creatures that have been transfered Added cloudToJson/c2j and jsonToCloud/j2c commands 0.2.5 Fixed import command. Again. 0.2.4 Fixed level output for creatures with base/wild level 1 Added --inventory option to players command --inventory and --items now accept an argument: summary or long, summary will just count items, long will output each item stack with extended information 0.2.3 Added GPS values for Valhalla and Mortem Tupiu Added Rafts to creatures/wild/tamed and tribes --creatures commands Fixed import command Added location of bases to --bases option Added id field to creatures/wild/tamed command, needs testing to see if it is stable Updated included ark_data.json to 246.2 0.2.2 Added --with-names and --without-dupes options to classes command Changed source of ark data to a new tool, the new data is directly extracted from the game Added name mapping for structures Added global --lang option to output localized names Fixed potential bug in export and import commands Tribes command now ignores items held by sleeping players Tribes command now includes items lying on the floor 0.2.1 improved help for syntax errors skip and report potentially broken .arkprofile and .arktribe files new option --verbose to print a stacktrace for broken files new option --bases for the tribes list, for details see above 0.2.0 pretty printing of generated JSON files is now optional and disabled by default renamed animals command to creatures added more information to creature list statistic block is now optional and disabled by default conversion commands for .arkprofile and .arktribe hide general options in specific command help include data about items and creatures for the basegame, extracted from the wiki with help from https://github.com/DerekRies/arkpy use extracted data to output creature and item names new command to generate player list new command to generate tribe list new command to extract included list of maps for LatLng calculations automatically import LatLng values on startup if external file is created (so you can add support for different maps) display commands in groups with their descriptions Link to comment Share on other sites More sharing options...
Qowyn Posted August 10, 2016 Author Share Posted August 10, 2016 The savegame has no information about the ingame names for the classes in it, but it might be possible to include that from the ARK SDK in the future. I'll try to include more fields for the dinos in the next release and maybe reports on players. Link to comment Share on other sites More sharing options...
gantoris Posted August 11, 2016 Share Posted August 11, 2016 great work on this tool, this has helped me find some missing dinos. one thing i did notice is it has wild entries for the alpha raptor, carno, and mosa. but it's missing the alpha rex or mega rex or whatever the game calls it lol Link to comment Share on other sites More sharing options...
Toni Posted August 11, 2016 Share Posted August 11, 2016 Very nice work, thank you ! Will you add this for players and tribes too? Link to comment Share on other sites More sharing options...
Qowyn Posted August 12, 2016 Author Share Posted August 12, 2016 Support for listing players and tribes will be in the next verison. Link to comment Share on other sites More sharing options...
Qowyn Posted August 14, 2016 Author Share Posted August 14, 2016 Found something interesting while coding the tribe lists: Apparently ark likes to dupe structures when writing the savegame. Edit: new version released. Link to comment Share on other sites More sharing options...
Toni Posted August 15, 2016 Share Posted August 15, 2016 Removed Link to comment Share on other sites More sharing options...
Toni Posted August 15, 2016 Share Posted August 15, 2016 REmoved Link to comment Share on other sites More sharing options...
Qowyn Posted August 15, 2016 Author Share Posted August 15, 2016 It's actually supposed to work exactly like the animals export. E.g.: java -jar ark-tools.jar players /path/to/save.ark /path/to/output/directory The main reason for providing a file name is consistency, the players command does not even use it. Instead it will look for .arkprofile files in the same directory. The tribes command on the other hand may use the provided savegame to gather more info about the tribes if needed, e.g. if using --structures or similiar options. Edit: I'll probably rename DIRECTORY to OUTPUT_DIRECTORY for the next version to make things clearer. Link to comment Share on other sites More sharing options...
Toni Posted August 15, 2016 Share Posted August 15, 2016 12 minutes ago, Qowyn said: It's actually supposed to work exactly like the animals export. E.g.: java -jar ark-tools.jar players /path/to/save.ark /path/to/output/directory The main reason for providing a file name is consistency, the players command does not even use it. Instead it will look for .arkprofile files in the same directory. The tribes command on the other hand may use the provided savegame to gather more info about the tribes if needed, e.g. if using --structures or similiar options. Edit: I'll probably rename DIRECTORY to OUTPUT_DIRECTORY for the next version to make things clearer. C:\Scripts\ark\ark_tools_v2>java -Xmx8G -jar ark-tools.jar players C:\arkdedicated\ark\ShooterGame\Saved\SavedArks\TheIsland.ark C:\Scripts\ark\ark_tools_v2\output\players\ -p -s java.lang.NullPointerException at qowyn.ark.tools.PlayerListCommands.players(PlayerListCommands.java:70) at qowyn.ark.tools.App.main(App.java:98) C:\Scripts\ark\ark_tools_v2>java -Xmx8G -jar ark-tools.jar tribes C:\arkdedicated\ark\ShooterGame\Saved\SavedArks\TheIsland.ark C:\Scripts\ark\ark_tools_v2\output\tribes\ -p -s java.lang.NullPointerException at qowyn.ark.tools.PlayerListCommands.tribes(PlayerListCommands.java:226) at qowyn.ark.tools.App.main(App.java:98) And some working command to show that my path's and so on are correct: C:\Scripts\ark\ark_tools_v2>java -Xmx8G -jar ark-tools.jar tamed C:\arkdedicated\ark\ShooterGame\Saved\SavedArks\TheIsland.ark C:\Scripts\ark\ark_tools_v2\output\tamed\ -p -s Reading finished after 1668 ms Dumping finished after 137 ms Total time 1805 ms Link to comment Share on other sites More sharing options...
Qowyn Posted August 15, 2016 Author Share Posted August 15, 2016 I expected potential crashes with empty tribes but not tribes without an ID. Thanks for the report. Link to comment Share on other sites More sharing options...
Toni Posted August 15, 2016 Share Posted August 15, 2016 Thanks Let me know when there is a new version to test. Link to comment Share on other sites More sharing options...
Qowyn Posted August 15, 2016 Author Share Posted August 15, 2016 0.2.1 out now with fixes and a new feature. Link to comment Share on other sites More sharing options...
Qowyn Posted August 17, 2016 Author Share Posted August 17, 2016 For those interested, all fileformats which ark-savegame-toolkit can read are now documented here: https://github.com/Qowyn/ark-savegame-toolkit/wiki Link to comment Share on other sites More sharing options...
Crystal Posted August 17, 2016 Share Posted August 17, 2016 7 hours ago, Qowyn said: For those interested, all fileformats which ark-savegame-toolkit can read are now documented here: https://github.com/Qowyn/ark-savegame-toolkit/wiki Thanks a lot for the documentation! I had already figured out a lot about the file format from spending hours looking at it in a hex editor, but then you came along with a working library! So, I referenced your code to fill in the missing pieces and started writing a C# library for reading/writing these files (only reading is implemented so far, but it works well). If there was a document like this back when I started, I would have been done a long time ago. So, thank you for this service you have done for the community! I hope that people will use this information to make helpful tools for server admins. I have been working on some C# libraries for various Ark things: server queries, rcon sessions, and now save file reading and editing. I hope to eventually build a nice remote server management tool out of all of it, but it will take a long time since I am only working on it a couple hours per week. Hopefully people with more free time will beat me to it Link to comment Share on other sites More sharing options...
Qowyn Posted August 24, 2016 Author Share Posted August 24, 2016 @Crystal Thats my hope too. Sorry for the late reply, sometimes i'm just lazy. Especially happy that it's helping you since you seem to help everyone here in the Server Administration section all forums. In other news: 0.2.2 is out, with support for most translated languages. Translation files are available here: https://ark-data.seen-von-ragan.de/data-download/ark-tools/ Translations are used to get readable/normal names for creatures, structures and items in your language. Link to comment Share on other sites More sharing options...
Qowyn Posted August 27, 2016 Author Share Posted August 27, 2016 Are all files empty? Running any other mods? And I probably need to have a look at that save. Link to comment Share on other sites More sharing options...
Qowyn Posted August 27, 2016 Author Share Posted August 27, 2016 New latLonCalculator.json now available at https://ark-data.seen-von-ragan.de/data-download/ark-tools/latLonCalculator.json Includes extracted values for correct Latitude and Longitude calculation for TheCenter, Valhalla and Mortem Tupiu. Also added missing language files for languages which only have .locres files. Link to comment Share on other sites More sharing options...
FatDude Posted August 29, 2016 Share Posted August 29, 2016 Im such a java noob, but is there any easy way to export to mysql or mssql databse ? What format is the output files for dinos and structures ? Basically I want to build a web interface around the whole data set, updated after each world save including history tracking for easy back tracking of tamed dinos etc ;-) I know how to model the database part including type 2 tracking + web part, but I just suck at java ;-/ Link to comment Share on other sites More sharing options...
Qowyn Posted August 30, 2016 Author Share Posted August 30, 2016 As @GreatApe mentioned everything my tool writes is in json format, because what you want to do is excatly what i made it for. And if anyone needs more data: suggest something and i might be able to provide it. Any feedback appreciated. Link to comment Share on other sites More sharing options...
FatDude Posted August 30, 2016 Share Posted August 30, 2016 16 hours ago, GreatApe said: The files are exported in json format I believe so should be easy to use with any language. What language are you using? I want to do the same (time, always need more time) using php and mysql. Most likely SSIS packages for ETL, sql server 2016 and php for webpart Should be able to do a poc within a few days, lets see Link to comment Share on other sites More sharing options...
Qowyn Posted August 30, 2016 Author Share Posted August 30, 2016 ARK (and UE4 in general) writes things to a temporary file, deletes or renames the old file, and then renames the temporary file. And my tools open files in read-only mode (except for manipulation commands of course), so they are unable to corrupt anything. Worst case would be that ARK is unable to delete the old file, but this can be prevented by using file watching APIs to start data extraction as soon as the temporary file gets renamed. Link to comment Share on other sites More sharing options...
Toni Posted August 31, 2016 Share Posted August 31, 2016 On 29.8.2016 at 10:00 PM, FatDude said: Im such a java noob, but is there any easy way to export to mysql or mssql databse ? What format is the output files for dinos and structures ? Basically I want to build a web interface around the whole data set, updated after each world save including history tracking for easy back tracking of tamed dinos etc ;-) I know how to model the database part including type 2 tracking + web part, but I just suck at java ;-/ I had the same problem ^^ Use a powershell script. Read the file contents and POST it to your webserver where you save it to your database. Surely not the most performant way to do it, but it works What is type 2 tracking? I do not have a history till now because it would be too much data if i create a new database entry for each update. Link to comment Share on other sites More sharing options...
FatDude Posted September 1, 2016 Share Posted September 1, 2016 17 hours ago, Toni said: I had the same problem ^^ Use a powershell script. Read the file contents and POST it to your webserver where you save it to your database. Surely not the most performant way to do it, but it works What is type 2 tracking? I do not have a history till now because it would be too much data if i create a new database entry for each update. Using MS sql sever and SSIS packages for importing data atm, might convert to mysql once poc is done. Will do slowly changing dimension type 2 on most relevant columns in order to track history, like when did a player change tribe or when did a dino disapear or was tamed etc. As for size of DB, just use compression Some feature requests for the op Add created and lastchanged timestamps to tribes and ark profiles, like below (Great for filtering on active players past week etc) public 'FileCreated' (length=15) public 'FileUpdated' (length=15) Add some kind of unique identify to each dino. It would make it easier to track. I assume there must be some kind of object id or similar that does not change in the save files Link to comment Share on other sites More sharing options...
Toni Posted September 1, 2016 Share Posted September 1, 2016 31 minutes ago, FatDude said: Some feature requests for the op Add created and lastchanged timestamps to tribes and ark profiles, like below (Great for filtering on active players past week etc) public 'FileCreated' (length=15) public 'FileUpdated' (length=15) Add some kind of unique identify to each dino. It would make it easier to track. I assume there must be some kind of object id or similar that does not change in the save files For data/informations like active players last week, its enough to refresh the data once per 24 hours. For the other informations (tribe / players), its needed to refresh it after each server save. So including this "only once per 24 hours needed data" in Qowyns library, is overhead. It reads this data each time even if its not needed at this time. So adding this would increase the generation time of the .json files a bit. What you requesting here can be done for example this way (Only once per 24 hours): Powershell example to get amount of acitve players last month: $filepath3 = "D:\arkdedicated\ark\ShooterGame\Saved\SavedArks" $filetype3 = "*.arkprofile" [array]$activeprofiles = get-childitem $filepath3 $filetype3 | where-object {$_.LastWriteTime -ge (get-Date).AddDays(-31)} $activeprofilescount = $activeprofiles.count So you know the amount of players active last month (count)... and you have a list of the active players (array). Generating the tribe datas already now takes 26.5 seconds for me (310MB savefile / 221 tribes). Servers with less performance and not a Samsung 950 ssd will need much much longer for that. Only my 50 cents Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.