Jump to content

ARK Tools v0.6.4 - tools for reading and manipulating ark savegame files


Qowyn

Recommended Posts

Hi, Thanks for this great tool!

I am having an issue exporting and importing dinos to a cluster file (havent tried other things).
This a feature that is currently working or maybe I am using it wrong? :P
Many more errors like the one shown here while exporting, but they are the same just different location in the file.
 

./ark-tools.sh export restore/2017-01-08_17.30.10/TheIsland.ark snav/argent.json --dinoid 1147157684391425702 --pretty-printing
Warning: Unknown Struct Type SPLightSettingsStruct at 39dd5f9 trying to read as StructPropertyList

./ark-tools.sh import snav/76561198156293811 snav/argent.json snav/76561198156293811_new
Warning: Unknown Struct Type ArkInventoryData at a9 trying to read as StructPropertyList
java.lang.NullPointerException
        at qowyn.ark.tools.EditingCommands.importIntoClusterData(EditingCommands.java:421)
        at qowyn.ark.tools.EditingCommands.importThing(EditingCommands.java:273)
        at qowyn.ark.tools.App.main(App.java:114)

./ark-tools.sh import snav/76561198156293811 snav/argent.json snav/76561198156293811_new --file-format cluster
java.lang.IllegalArgumentException: No enum constant qowyn.ark.tools.FileFormat.cluster
        at java.lang.Enum.valueOf(Enum.java:238)
        at qowyn.ark.tools.FileFormat.valueOf(FileFormat.java:9)
        at qowyn.ark.tools.EditingCommands.importThing(EditingCommands.java:237)
        at qowyn.ark.tools.App.main(App.java:114)

 

Link to comment
Share on other sites

  • Replies 369
  • Created
  • Last Reply

Is there a way to use this tool and extract dataset from cluster information when a player uploads a dino in the terminal to move from map to another? As part of the clustering and allowing up/downloads of dinos, you can specify the share cluster directory.

I have some ideas to use this process and dino stats.

I tried using this an example:

C:\ARK Survival Evovle Stuff\ark-tools>ark-tools.exe c2j clusterPlayerFilenameHere PlayerFilename.json
Warning: Unknown Struct Type ArkInventoryData at a8 trying to read as StructPropertyList

I did get majority of the dataset, but I think its having issue reading some parts and not getting dino colorset, tribe name ... etc?

Link to comment
Share on other sites

On 24.12.2016 at 6:17 PM, Qowyn said:

That will come with the next version. And I'm developing a mod to enhance the features of ark-tools, this mod will allow you to get the ingame names of item classes and other information too. And it will be possible to extract most information right from the running game and post it to some HTTP(S?) endpoint.

Sounds nice :)  Keep up your nice work :)

 

Link to comment
Share on other sites

21 hours ago, Toni said:

Sounds nice :)  Keep up your nice work :)

 


I use players error can not be used,But there's no problem with Tribes

 

[root@localhost ~]# java -jar ark-tools.jar players /root/Saved/TheIsland.ark /root/bak/
java.lang.NullPointerException
        at org.glassfish.json.UnicodeDetectingInputStream.fillBuf(UnicodeDetectingInputStream.java:90)
        at org.glassfish.json.UnicodeDetectingInputStream.detectEncoding(UnicodeDetectingInputStream.java:129)
        at org.glassfish.json.UnicodeDetectingInputStream.<init>(UnicodeDetectingInputStream.java:76)
        at org.glassfish.json.JsonParserImpl.<init>(JsonParserImpl.java:76)
        at org.glassfish.json.JsonReaderImpl.<init>(JsonReaderImpl.java:68)
        at org.glassfish.json.JsonProviderImpl.createReader(JsonProviderImpl.java:131)
        at javax.json.Json.createReader(Json.java:220)
        at qowyn.ark.tools.CommonFunctions.readJson(CommonFunctions.java:110)
        at qowyn.ark.tools.CommonFunctions.readJsonRelative(CommonFunctions.java:123)
        at qowyn.ark.tools.DataManager.loadData(DataManager.java:33)
        at qowyn.ark.tools.PlayerListCommands.players(PlayerListCommands.java:81)
        at qowyn.ark.tools.App.main(App.java:114)
[root@localhost ~]# 
[root@localhost ~]# java -jar ark-tools.jar cluster /root/Saved/TheIsland.ark /root/bak
java.lang.NullPointerException
        at org.glassfish.json.UnicodeDetectingInputStream.fillBuf(UnicodeDetectingInputStream.java:90)
        at org.glassfish.json.UnicodeDetectingInputStream.detectEncoding(UnicodeDetectingInputStream.java:129)
        at org.glassfish.json.UnicodeDetectingInputStream.<init>(UnicodeDetectingInputStream.java:76)
        at org.glassfish.json.JsonParserImpl.<init>(JsonParserImpl.java:76)
        at org.glassfish.json.JsonReaderImpl.<init>(JsonReaderImpl.java:68)
        at org.glassfish.json.JsonProviderImpl.createReader(JsonProviderImpl.java:131)
        at javax.json.Json.createReader(Json.java:220)
        at qowyn.ark.tools.CommonFunctions.readJson(CommonFunctions.java:110)
        at qowyn.ark.tools.CommonFunctions.readJsonRelative(CommonFunctions.java:123)
        at qowyn.ark.tools.DataManager.loadData(DataManager.java:33)
        at qowyn.ark.tools.PlayerListCommands.cluster(PlayerListCommands.java:725)
        at qowyn.ark.tools.App.main(App.java:114)
[root@localhost ~]# java -jar ark-tools.jar tribes /root/Saved/TheIsland.ark /root/bak
[root@localhost ~]# 

 

Link to comment
Share on other sites

  • 2 weeks later...

im trying to import a player JSON file but cant seem to get it to work, i get the below error:

 

>ark-tools.exe j2p 76561198xxx900789.json 76561198xxx900789.arkprofile

java.lang.NullPointerException
        at org.glassfish.json.JsonObjectBuilderImpl$JsonObjectImpl.getInt(JsonOb
jectBuilderImpl.java:213)
        at qowyn.ark.ArkProfile.readJson(ArkProfile.java:131)
        at qowyn.ark.ArkProfile.<init>(ArkProfile.java:58)
        at qowyn.ark.tools.ConvertingCommands.jsonToProfile(ConvertingCommands.j
ava:114)
        at qowyn.ark.tools.App.main(App.java:114)

i have tried using ark-tools.exe and using relative and absolute locations of the files, generates the same error.

Link to comment
Share on other sites

Unfortunately my Export for Tribes and Dinos isn't working anymore for my The Center Server. Do you maybe know whats causing this?

java.lang.UnsupportedOperationException: Found unexpected Value 2 at 2132
         at qowyn.ark.ArkSavegame.readBinary(ArkSavegame.java:145)
         at qowyn.ark.ArkSavegame.<init>(ArkSavegame.java:78)
         at qowyn.ark.tools.PlayerListCommands.tribes(PlayerListCommands.java:343)
         at qowyn.ark.tools.App.main(App.java:114)

 

EDIT:
I'm sure you put the Error-Checking in "ArkSavegame.java" on purpose, but for me this was the showstopper. And that's why I commented the part out and now everything seems to work again:

 

  public void readBinary(ArkArchive archive, ReadingOptions options) {
    readBinaryHeader(archive);

    if (saveVersion == 6) {
      // Name table is located after the objects block, but will be needed to read the objects block
      readBinaryNameTable(archive);
    }

    readBinaryDataFiles(archive, options);
    readBinaryEmbeddedData(archive, options);

    int unknownValue = archive.getInt();
    if (unknownValue != 0) {
      // McBane: This was my showstopper
      //if (unknownValue > 1) {
        //throw new UnsupportedOperationException("Found unexpected Value " + unknownValue + " at " + (archive.position() - 4));
      //}

      for (int n = 0; n < unknownValue; n++) {
        int unknownFlags = archive.getInt();
        int objectCount = archive.getInt();
        String name = archive.getString();
      }
    }

    readBinaryObjects(archive, options);
    readBinaryObjectProperties(archive, options);
  }

 

Link to comment
Share on other sites

  • 1 month later...

A plea to the php/db gods.

 

Hi chaps, I have managed to sort out the retrieval of wild dino data, via this fantastic tool.

 

I have at the moment, got the json files loading into a mysql db and then this is published on my website.

I feel this is a pretty dirty way around of doing it, but I'll be honest i dont know a better way to do this.

I have scanned the ww and just get lost in all the information out there.

 

I know a few of you have managed to turn this data into a thing of beauty and I know i will never get to that with my skills, but would love it, if you could throw me some pointers.

Stack overflow is a great resource, but i feel i am not properly asking what I require to know over there.

 

Any pointers guys?

 

Thanks in advance.

Link to comment
Share on other sites

@StriderUk I am no God with regards to php/db or html for that matter, but I managed to piece together something both visually pleasing and usable using mr Qowyns own ark-tools-web as a starting point. This uses the raw jsons that the tool provides as a datasource.

Best of luck! ;)

Regards
Steven.

edit: seems I am not able to post URL's, but just go to Qowyns github and you'll find it.

Link to comment
Share on other sites

On 18/3/2017 at 8:02 PM, StriderUk said:

A plea to the php/db gods.

 

Hi chaps, I have managed to sort out the retrieval of wild dino data, via this fantastic tool.

 

I have at the moment, got the json files loading into a mysql db and then this is published on my website.

I feel this is a pretty dirty way around of doing it, but I'll be honest i dont know a better way to do this.

I have scanned the ww and just get lost in all the information out there.

 

I know a few of you have managed to turn this data into a thing of beauty and I know i will never get to that with my skills, but would love it, if you could throw me some pointers.

Stack overflow is a great resource, but i feel i am not properly asking what I require to know over there.

 

Any pointers guys?

 

Thanks in advance.

Sure what do you need help with ?

I've created below for my players:

http://ark.sharpplay.com:81

 

I'm using MS SQL and etl packages, but the same can be done in mysql

Any special task you need help with ?

 

Link to comment
Share on other sites

Hi mate,

Im using mysql, i know not ideal and with the free dev edition of ms sql, maybe I should be looking to change it.

 

I am using a dirty tool called json to my sql to get the files into my db, via batch scripts, that load them in every 5 mins.

I feel there must be a better way, every so often, my files retrieve the wrong column information in the etl process, but quickly correct theirselves.

 

I have scoured the web, but come across so much conflicting information.

I'd love to build something as you have done, but I really dont have the time, with work commitments.

 

So am quite happy with a simple table based site, as i have done here, (its not pretty trust me).

 

http;/warpspeed.zapto.org:1080/joomla/

 

Thanks in advance for any help at all you can give me, if only pointing me in the right direction :D

 

 

Link to comment
Share on other sites

On 21/3/2017 at 8:27 PM, StriderUk said:

Hi mate,

Im using mysql, i know not ideal and with the free dev edition of ms sql, maybe I should be looking to change it.

 

I am using a dirty tool called json to my sql to get the files into my db, via batch scripts, that load them in every 5 mins.

I feel there must be a better way, every so often, my files retrieve the wrong column information in the etl process, but quickly correct theirselves.

 

I have scoured the web, but come across so much conflicting information.

I'd love to build something as you have done, but I really dont have the time, with work commitments.

 

So am quite happy with a simple table based site, as i have done here, (its not pretty trust me).

 

http;/warpspeed.zapto.org:1080/joomla/

 

Thanks in advance for any help at all you can give me, if only pointing me in the right direction :D

 

 

Mysql is fine for the purpose, since we are talking low data volume anyway.

Have you tried any of the opensource php etl frameworks that support json files ?

A quick search gave me this one, though I never used it: https://github.com/leomarquine/php-etl

Also the dino info is coming from the world save which usually saves every 20-30min depending on settings. You can make a trigger php script that checks the world save timestamp and only starts your etl scripts once it's been updated. That will save you etl time.

If you source in the players and tribes to own tables, you can start joining the data together to show ex all dino for a tribe.

I parse the tribe data into a separate table with the tribeid as key, so I can join it on the tribes

For tribes and players, I also source in the filename (removing the extensions) and use that as key

See image:

Since I use slowly changing dimensions (have history on players, tribes and dinos) I sourced into staging tables first (src.*) and then load the dim.* and fact.* tables

You dont really need that, you can just load directly into you "normal" tables and do you reports on those of those.

Link to comment
Share on other sites

  • 2 weeks later...

Is it possible to find the Unicorn?

If I use this cmd line:

ark-tools.exe wild c:\at\theisland.ark c:\at\output\

I cannot find it.. 

Edit:
I can only find the once I tamed by using this:

ark-tools.exe tamed c:\at\theisland.ark c:\at\output\

My ACM says, there should be a wild one spawned.. Gonna test if I can see one, if I kill my tamed Unicorns.. :P

Edit:
When I killed my Unicorn and after a couple of wild dino wipes, the Unicorn spawned and I could find it via this program :-)

Link to comment
Share on other sites

Hey mate.

I dont know if you still work on this project but as I'm stream lining my import and moving it to azure could I have a few feature requests.

Add playerid_fileid (filename) to player output file

Add tribeid_fileid (filename) to tribeout output file

Both are needed in order to join correctly (as tribe names can change, but the file id will not. Way better and also easy to track history on tribes and players)

 

command -SingleFile (outputs all players/tribes to single file instead of multiple files. 1 Single file for players and 1 single file for tribes. Above id's would need to be added)

This would give an huge performance gain when importing to a DB (also make importing a lot easier)

 

Link to comment
Share on other sites

On 12.1.2017 at 5:59 PM, StevenB said:

Hi, Thanks for this great tool!

I am having an issue exporting and importing dinos to a cluster file (havent tried other things).
This a feature that is currently working or maybe I am using it wrong? :P
Many more errors like the one shown here while exporting, but they are the same just different location in the file.
 


./ark-tools.sh export restore/2017-01-08_17.30.10/TheIsland.ark snav/argent.json --dinoid 1147157684391425702 --pretty-printing
Warning: Unknown Struct Type SPLightSettingsStruct at 39dd5f9 trying to read as StructPropertyList

./ark-tools.sh import snav/76561198156293811 snav/argent.json snav/76561198156293811_new
Warning: Unknown Struct Type ArkInventoryData at a9 trying to read as StructPropertyList
java.lang.NullPointerException
        at qowyn.ark.tools.EditingCommands.importIntoClusterData(EditingCommands.java:421)
        at qowyn.ark.tools.EditingCommands.importThing(EditingCommands.java:273)
        at qowyn.ark.tools.App.main(App.java:114)

./ark-tools.sh import snav/76561198156293811 snav/argent.json snav/76561198156293811_new --file-format cluster
java.lang.IllegalArgumentException: No enum constant qowyn.ark.tools.FileFormat.cluster
        at java.lang.Enum.valueOf(Enum.java:238)
        at qowyn.ark.tools.FileFormat.valueOf(FileFormat.java:9)
        at qowyn.ark.tools.EditingCommands.importThing(EditingCommands.java:237)
        at qowyn.ark.tools.App.main(App.java:114)

 

Same here. I tried to alter a mammoth but I also get a NPE

$ ./ark-tools.sh import centerPVE/export/76561198026916269 centerPVE/export/random200.json centerPVE/export/76561198026916269_new
java.lang.NullPointerException
        at qowyn.ark.PropertyContainer.getPropertyValue(PropertyContainer.java:63)
        at qowyn.ark.PropertyContainer.getPropertyValue(PropertyContainer.java:58)
        at qowyn.ark.tools.EditingCommands.importIntoClusterData(EditingCommands.java:403)
        at qowyn.ark.tools.EditingCommands.importThing(EditingCommands.java:273)
        at qowyn.ark.tools.App.main(App.java:114)


And also the cluster format does not work well.

$ ./ark-tools.sh import
Missing argument SAVE.
Imports all objects from JSON into SAVE.
Usage: ark-tools import SAVE JSON NEW_SAVE [OPTIONS]

Option                                   Description
------                                   -----------
--file-format                            Select format of input and output
  <cluster|localprofile|map|profile|tribe>    files.

$ ./ark-tools.sh import centerPVE/export/76561198026916269 centerPVE/export/random200.json centerPVE/export/76561198026916269_new --file-format=cluster
java.lang.IllegalArgumentException: No enum constant qowyn.ark.tools.FileFormat.cluster
        at java.lang.Enum.valueOf(Enum.java:238)
        at qowyn.ark.tools.FileFormat.valueOf(FileFormat.java:9)
        at qowyn.ark.tools.EditingCommands.importThing(EditingCommands.java:237)
        at qowyn.ark.tools.App.main(App.java:114)

On the other hand, I was able to c2j a file, modify it (its a bit tough to understand the structure) and then j2c it back into the cluster.

 

Link to comment
Share on other sites

@FatDude

I'm really interest actually how you got this to load in ssis. Have you used one of the free tools, or have you managed to cook something yourself that loads the json into a staging table?

 

I have finally got around to getting sql 2016 onto my server and will probably end up going down this route for loading my data.

I have tried to load the data, using the new sql commands built into 2016, would be interested to know how you got this working if you dont mind?

 

cheers

Phill

Link to comment
Share on other sites

5 hours ago, StriderUk said:

@FatDude

I'm really interest actually how you got this to load in ssis. Have you used one of the free tools, or have you managed to cook something yourself that loads the json into a staging table?

I have finally got around to getting sql 2016 onto my server and will probably end up going down this route for loading my data.

I have tried to load the data, using the new sql commands built into 2016, would be interested to know how you got this working if you dont mind?

cheers

Phill

Im currently using one of the free opensouce json ssis compoments, If works fine but is a bit weird to setup and need to loop though all the files in a loop

https://jsonsource.codeplex.com/

 

But Im playing with the json support in the new sql server which would probably be a better and easier way..

 

Some sample code you can look at below ( using a blob storage, but can easily be changed to read from a local file(s) instead)

Could write some stored procs to load the data to source tables

SELECT BulkColumn
 FROM OPENROWSET (BULK '1022680475.json', DATA_SOURCE = 'MyAzureBlobStorageTribes', SINGLE_CLOB) as j
 CROSS APPLY OPENJSON(BulkColumn)
	
SELECT players.*
 FROM OPENROWSET (BULK '76561197960815112.json', DATA_SOURCE = 'MyAzureBlobStoragePlayers', SINGLE_CLOB) as j
 CROSS APPLY OPENJSON(BulkColumn)
 WITH( id int, playerName nvarchar(100),name nvarchar(100), level int,
  tribeId int,tribeName nvarchar(100)) AS players

 

Some screen dumps of the the ssis setup:

 

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.


×
×
  • Create New...