Jump to content

ARK Tools v0.6.4 - tools for reading and manipulating ark savegame files


Qowyn

Recommended Posts

  • Replies 369
  • Created
  • Last Reply

I've been trying to figure out the Modification File, but I assume the error messages I'm getting is due to "Unknown Cloud Inventory Version 4". I thought I'd fiddle around with c2j and that is where the version error is popping up. Anywho, if it was simply changing the number from 7 to 8 for the save-files to work, would it be just as easy for cloud/cluster files? No idea how I compile stuff, perhaps someone else has run into the same problem and compiled a working version?

For reference (paths redacted due to cluster-ID):
./ark-tools.sh modify INPUT test1.json OUTPUT --file-format cluster

rjava.lang.IllegalArgumentException: No enum constant qowyn.ark.tools.FileFormat.cluster
        at java.lang.Enum.valueOf(Enum.java:238)
        at qowyn.ark.tools.FileFormat.valueOf(FileFormat.java:9)
        at qowyn.ark.tools.EditingCommands.modify(EditingCommands.java:545)
        at qowyn.ark.tools.App.main(App.java:117)


./ark-tools.sh c2j CLOUD test2.json

java.lang.UnsupportedOperationException: Unknown Cloud Inventory Version 4
        at qowyn.ark.ArkCloudInventory.readBinary(ArkCloudInventory.java:65)
        at qowyn.ark.ArkCloudInventory.<init>(ArkCloudInventory.java:53)
        at qowyn.ark.ArkCloudInventory.<init>(ArkCloudInventory.java:31)
        at qowyn.ark.tools.ConvertingCommands.cloudToJson(ConvertingCommands.java:188)
        at qowyn.ark.tools.App.main(App.java:117)

 

Link to comment
Share on other sites

On 8/31/2017 at 3:04 AM, MaeL said:

Hi,

don't know when Qowyn will do it officialy, so here is a patched version of the .jar for v267, I've also inclued a -max-age parameter for the tribes command if you need it (same usage as players -max-age parameter, less tribelogs, huge gain)

https://ufile.io/4d75l

hey @MaeL so apparently now they are using database version 9.  Could you possibly update your fix again?  I would Love you long time!

Link to comment
Share on other sites

Been very busy again with work. And that might happen again. Anyway: New regular and preview versions are available.

https://github.com/Qowyn/ark-tools/releases/tag/0.5.4

0.5.4 Changelog:

  • Contributions by ZokRadonh from GitHub: Added tamedIneffectivenessModifier and DinoAncestor to tamed creature output
  • read-only support for V8 and V9 map-saves

https://github.com/Qowyn/ark-tools/releases/tag/0.6.0-preview2

0.6.0-preview2 Changelog:

  • read-only support for V8 and V9 map-saves
  • Some new options: --allow-broken-file for mapToJson command, tries to get as much data as possible from broken map-saves
  • Improved DataCollector performance, allow reading files in parallel

Known broken stuff:

  • Version 7 to 9 map-saves are read-only
  • Potentially missing some dinos for singleplayer and non-dedicated hosts
  • Clusterfiles aka CloudInventory is now on Version 4

Fun fact:

  • ark-tools m2j Z:\path\to\DnL\Saved\SavedDnLLocal\DNL_ALL.dnl Z:\path\to\DnL\Saved\SavedDnLLocal\DNL_ALL.json --pretty

    Above command works with Dark and Light saves, converting them to the usual .json representation. They are using the (almost) ancient Version 5 format.

Link to comment
Share on other sites

I've tried a number of things to get a server's worldfile working, as it doesn't appear necessarily corrupt, but won't boot in singleplayer or through our hosthavoc server. It boots without mods but exceeds alotted memory and crashes in short order, though not before running long enough to allow people to load in - though I imagine this is only possible because all the S+ structures are gone and no longer burdening the server's memory. It still consumes more than double a fresh map, despite trying dozens of savedarks going back to the 25th of August. After exhausting a number of resources all that seems to be left is your tool, and the json editing I saw mentioned earlier in the thread - but the tool exits itself a moment after I attempt to open it. I've no doubt I'm doing something wrong, but I can't understand what - running as administrator through the exe just gives me a flash of a cmd menu before it simply closes out.

Is there any chance I could get advice, or send you a copy of the worldfile? We're desperate to try and save it.

Link to comment
Share on other sites

@TeaKettle first off it's a command line tool so you have to run it in an already opened command prompt, secondly the software is unfortunately not currently updated to be able to write to the newest savefile formats.

I did however have something that sounds like the exact same issue, what fixed it for me was starting the server with -ForceRespawnDinos in the launch parameters of the server to remove any wild dinos stuck in geometry. Might not be the same issue, but had the same symptoms so I would try it ^^

Link to comment
Share on other sites

Just did the first successful conversion of a Version 9 save to json and back to binary. The next version will also contain support for the Version 4 Local Profile and Version 4 Cloud Inventory format. I'm now in the process of ditching the current json library and will replace it with Jackson. I had already done that on a local branch in the past, which caused the mapToJson command to be up to 2 orders of magnitude faster than before. Still need to write new code to read the JSON back in.

@TeaKettle If you send me the file then I'll take a look.

Link to comment
Share on other sites

0.6.0 Release!

Changelog:

  • Added (still incomplete) Database driver system
  • Replaced javax.json with Jackson, massive performance improvements for mapToJson command (other commands benefit too)
  • Adjustments to the names of some fields
  • Increased amount of gathered data for all commands
  • Allow parallel processing for players, tribes and cluster commands
  • Amount of parallelism can no be controlled via --thread-count, defaults to the number of cores available to the JVM
  • Full support for version 7 to 9 savegames, version 4 local player profile and version 4 cluster storage
  • New options for some commands, "--write-all-fields" can now be used almost everywhere
  • New option "--allow-broken-file" for xToJson  commands will try to salvage as much data from broken files as possible

 

Link to comment
Share on other sites

1 minute ago, Qowyn said:

0.6.0 Release!

Changelog:

  • Added (still incomplete) Database driver system
  • Replaced javax.json with Jackson, massive performance improvements for mapToJson command (other commands benefit too)
  • Adjustments to the names of some fields
  • Increased amount of gathered data for all commands
  • Allow parallel processing for players, tribes and cluster commands
  • Amount of parallelism can no be controlled via --thread-count, defaults to the number of cores available to the JVM
  • Full support for version 7 to 9 savegames, version 4 local player profile and version 4 cluster storage
  • New options for some commands, "--write-all-fields" can now be used almost everywhere
  • New option "--allow-broken-file" for xToJson  commands will try to salvage as much data from broken files as possible

 

Thank you!!

Link to comment
Share on other sites

Next planned updates/Roadmap:

  • 0.6.1
    • Fix for missing ancestor writer, include names of ancestors
  • 0.7.0
    • Breaking change of xToJson formats (again), improved support for native array properties (properties which may or may not be ArrayProperty but still contain multiple values)
    • Fix for objects in the hibernation section being ignored by all commands
    • Implement missing stuff to make the database system useful (e.G. json driver should be able to write data to multiple files, skip refreshing files without changes, implement missing support for cluster directory)
    • Add more things to the modify command
    • Improved detection of creatures
Link to comment
Share on other sites

Quick (but tricky) question: do you guys think these tools could be used to remove all instances of a particular mod objects (or dinos) from a savegame?

To be more specific, I think NPC Bush People is not allowing my server to start, but removing it is tricky, you cant just tell the server not to load the mod. I would like to remove all "Bushmen" from the map in order to allow the server to start.

(it is not a matter of corrupt save file, since I cannot start the server even using previous savegames - I tested 6 or 7 of them)

Link to comment
Share on other sites

On 9/13/2017 at 9:09 PM, Darkblood said:

Quick (but tricky) question: do you guys think these tools could be used to remove all instances of a particular mod objects (or dinos) from a savegame?

To be more specific, I think NPC Bush People is not allowing my server to start, but removing it is tricky, you cant just tell the server not to load the mod. I would like to remove all "Bushmen" from the map in order to allow the server to start.

(it is not a matter of corrupt save file, since I cannot start the server even using previous savegames - I tested 6 or 7 of them)

Ok, I´m trying my hand at it, and I have more objective questions:

Is there a command to delete all objects with a given className from an .ark file?

Link to comment
Share on other sites

Double Release:

  • 0.6.1
    • Added missing DinoAncestor writers
    • Added names of ancestors
    • Added Polar Bear, Ragnarok Fire Wyvern and Ice Wyvern to the list of creatures (auto-detection fail)
    • Added --non-player switch to tribes command
    • Fixed lat and lon values for creatures, players and structures
  • 0.6.2
    • Added support for dropped items, making --non-player actually useful
    • Renamed --non-player to --non-players (the old name will still work)
    • Id of main inventory is now myInventoryComponent for all object types (only useful if you are using the db command)

@Darkblood I was going to comment on that... right when WC was applying updates to this site. Then some problem with the 0.6.1 release came up and i forgot to implement the stuff needed for your problem. Which means it's going to come with 0.6.3. So yeah, while the modify command is intended to mass delete stuff by class/type, it's not yet implemented.

Link to comment
Share on other sites

1 hour ago, Qowyn said:

 

@Darkblood I was going to comment on that... right when WC was applying updates to this site. Then some problem with the 0.6.1 release came up and i forgot to implement the stuff needed for your problem. Which means it's going to come with 0.6.3. So yeah, while the modify command is intended to mass delete stuff by class/type, it's not yet implemented.

Great to hear that, I don't mind waiting, thanks.

Link to comment
Share on other sites

Anyone have a working tutorial on how to add/modify cluster items? Using the example from Modification File I only get errors. Was hoping I'd get a working example and then test from there... but to no avail... on Ubuntu if that matters...

Using modify command:

Expected remapDinoClassNames to be OBJECT but found MISSING
Expected remapItemArchetypes to be OBJECT but found MISSING
Expected removeItems to be ARRAY but found MISSING
Expected addToDefaultInventory to be OBJECT but found MISSING
Expected addToInventory to be OBJECT but found MISSING
java.lang.IllegalArgumentException: No enum constant qowyn.ark.tools.FileFormat.cluster
        at java.lang.Enum.valueOf(Enum.java:238)
        at qowyn.ark.tools.FileFormat.valueOf(FileFormat.java:1)
        at qowyn.ark.tools.EditingCommands.modify(EditingCommands.java:542)
        at qowyn.ark.tools.App.main(App.java:125)

 

Using import command:

java.lang.IllegalArgumentException: No enum constant qowyn.ark.tools.FileFormat.cluster
        at java.lang.Enum.valueOf(Enum.java:238)
        at qowyn.ark.tools.FileFormat.valueOf(FileFormat.java:1)
        at qowyn.ark.tools.EditingCommands.importThing(EditingCommands.java:240)
        at qowyn.ark.tools.App.main(App.java:125)

 

Link to comment
Share on other sites

0.6.3 is out

Changelog

  • Important fixes for cluster and LocalProfile files, can affect map-saves too (I made a typo about 4 months ago, causing some properties to be written as Int8Property instead of ByteProperty)
  • Command export can now select objects by class (e.G. PrimalItemConsumable_Egg_Wyvern_Fertilized_Poison_C for green wyvern eggs)
  • Added switches --without-references and --without-components to export command (use --without-references for items, unless you want the whole base too)
  • Command import is now able to import items into cluster files and into the LocalProfile (with caveats, items will lose durability if you exported them from a map)
  • Command modify is now able to mass-delete objects in maps, cluster and LocalProfile files
  • All creature commands, the tribe command and db command are now aware of hibernation
  • Improved detection of all object types, no longer based on their class name

@Darkblood It's now possible to delete things by using the modify command with a modification file. Documentation on the format can be found here: https://github.com/Qowyn/ark-tools/wiki/Modification-File

@Arksanity I think you managed to trip every bug i had in there. Warnings should be gone with the new version and --file-format is no longer case sensitive.

@Mavrick Thanks to your suggestion it's now possible, eggs can be exported with the export command (one class at a time for now), then imported into a cluster file. Note that they will be at 99% of hatching already done, because the map saves the durability (which is used for the progress) as a percentage, while the cluster saves the absolute value. It's (not yet) possible to convert from percentage to durability because that would require a database of max durability values and other things.

Link to comment
Share on other sites

2 hours ago, Qowyn said:

0.6.3 is out

Changelog

  • Important fixes for cluster and LocalProfile files, can affect map-saves too (I made a typo about 4 months ago, causing some properties to be written as Int8Property instead of ByteProperty)
  • Command export can now select objects by class (e.G. PrimalItemConsumable_Egg_Wyvern_Fertilized_Poison_C for green wyvern eggs)
  • Added switches --without-references and --without-components to export command (use --without-references for items, unless you want the whole base too)
  • Command import is now able to import items into cluster files and into the LocalProfile (with caveats, items will lose durability if you exported them from a map)
  • Command modify is now able to mass-delete objects in maps, cluster and LocalProfile files
  • All creature commands, the tribe command and db command are now aware of hibernation
  • Improved detection of all object types, no longer based on their class name

@Darkblood It's now possible to delete things by using the modify command with a modification file. Documentation on the format can be found here: https://github.com/Qowyn/ark-tools/wiki/Modification-File

@Arksanity I think you managed to trip every bug i had in there. Warnings should be gone with the new version and --file-format is no longer case sensitive.

@Mavrick Thanks to your suggestion it's now possible, eggs can be exported with the export command (one class at a time for now), then imported into a cluster file. Note that they will be at 99% of hatching already done, because the map saves the durability (which is used for the progress) as a percentage, while the cluster saves the absolute value. It's (not yet) possible to convert from percentage to durability because that would require a database of max durability values and other things.

I am not sure what the 'classes' are for export, i tried 'Consumable_Egg_Argent_Fertilized_C' but that didn't work.  If you have an example of a few classes i could work out the rest.

* EDIT : Haha, i looked in the json file and found this, that worked. PrimalItemConsumable_Egg_Ankylo_Fertilized_C

 

Link to comment
Share on other sites

Here is what I tried and didn't work.  Not sure what I did wrong.

g:\ARK>ark-tools import ARK1\TheIsland.ark PrimalItemConsumable_Egg_Ankylo_Fertilized_C.json 76561197971580571 --file-format cluster
java.lang.UnsupportedOperationException: Unknown Cloud Inventory Version 1498021894
        at qowyn.ark.ArkCloudInventory.readBinary(ArkCloudInventory.java:53)
        at qowyn.ark.FileFormatBase.readBinary(FileFormatBase.java:30)
        at qowyn.ark.ArkCloudInventory.<init>(ArkCloudInventory.java:37)
        at qowyn.ark.tools.EditingCommands.importThing(EditingCommands.java:270)
        at qowyn.ark.tools.App.main(App.java:125)

* UPDATE: I was able to get it to export to a local profile and found random fertilized eggs as well as canteens, raw meat, soups, etc.  Not sure what happened but it kinda worked. :D

Link to comment
Share on other sites

0.6.4 Release because i broke the players command

Changelog

  • Fixed broken players command
  • players command can now detect LocalPlayer.arkprofile
  • Hibernation support for debug commands

@Arksanity It was supposed to work with lowercase too, but... yeah.

@Mavrick You can get a list of classes from the save by using the classes command (with "--pretty" option if you don't want to get a wall of text). And i recommend using "--without-references" with the export command, otherwise you will get the container of the item too... and everything connected to it. This could export a whole base if any item is contained in e.g. a refridgerator.

Link to comment
Share on other sites

2 hours ago, Qowyn said:

0.6.4 Release because i broke the players command

Changelog

  • Fixed broken players command
  • players command can now detect LocalPlayer.arkprofile
  • Hibernation support for debug commands

@Arksanity It was supposed to work with lowercase too, but... yeah.

@Mavrick You can get a list of classes from the save by using the classes command (with "--pretty" option if you don't want to get a wall of text). And i recommend using "--without-references" with the export command, otherwise you will get the container of the item too... and everything connected to it. This could export a whole base if any item is contained in e.g. a refridgerator.

Using --without-references issues the following error.

g:\ARK>ark-tools.exe export ARK1\TheIsland.ark PrimalItemConsumable_Egg_Ptero_Fertilized_C.json --without-references --class PrimalItemConsumable_Egg_Ptero_Fertilized_C
java.lang.NullPointerException
        at qowyn.ark.tools.ObjectCollector.doRemap(ObjectCollector.java:296)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
        at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
        at java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
        at java.util.concurrent.CountedCompleter.exec(Unknown Source)
        at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
        at java.util.concurrent.ForkJoinTask.doInvoke(Unknown Source)
        at java.util.concurrent.ForkJoinTask.invoke(Unknown Source)
        at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(Unknown Source)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(Unknown Source)
        at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
        at java.util.stream.ReferencePipeline.forEach(Unknown Source)
        at java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
        at qowyn.ark.tools.ObjectCollector.remap(ObjectCollector.java:223)
        at qowyn.ark.tools.EditingCommands.exportThing(EditingCommands.java:210)
        at qowyn.ark.tools.App.main(App.java:125)

If i leave out the --without-references, it works, but i get all the canteens and other things.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.


×
×
  • Create New...