Consolidating .tre files

duffstone

New Member
Sep 13, 2013
179
0
0
#1
Ok, before I start adding guides on how to add this or that I need to explain something that's simplifies all my modding and development. That being the way I handle my .tre files themselves...

First we know that SWGEMU as it is today uses some 53 different .tre files to load data into the Core3 server and well... work properly. Thing is, it doesn't HAVE to be 53 different .tre's. I think anyone who's done any server work at all gets tired of bouncing around from .tre to .tre looking for the latest version of a particular object or file. Since the .tre files are loaded in a specific order, it can be assumed that newer assets overwrite older assets. knowing this you can consolidate the .tre files into easier to work with files. All my development is based upon consolidated .tre files using this method:

first, pull up your swgemu_live.cfg file AND your core3 conf file. Both will show the proper order in which the client and core load the 53 different .tre files. I used the Core3 list.

1) in treexplorer open the first .tre file (should be bottom.tre). Select all the folders and choose export. For me I exported to a folder call alltres.

2) once the export is done, open up the next .tre file in the list.
a. select all the files / nodes, export to the same alltres directory, and if prompted overwrite.
b. rinse / repeat until you've added ALL the .tre files to your alltres directory, always overwriting.

3) so now you'll want to repack all this data in to one single .tre file right? well kind of. I'm not sure if it's a size limitation or what but the trepacker app won't pack more than about 2gb of data or 127K data items (those are the maxes I've run into). So you'll need to split your alltres directory in two, I made my split at the object folder, but use your best judgement to get the files as close to equal as possible both in size and item counts. Could take some trial and error.

4) Once you get your folders split, repack them into .tre files (will take awhile) and you're done. you should now have two .tre files that contain all the needed data using all the latest assets and info. This works perfectly with the core and client, although load times are increased on slower machines so be aware. Make backups of your swgemu_live.cfg and core3 conf, change each config file to use the two new .tre files and you're good to go.

5) swgemu_live example:
maxSearchPriority is always total number of files to load. searchTree_00_x= always begins with zero and counts up to the newest file.
[SharedFile]
maxSearchPriority=3
searchTree_00_2=mytre.tre
searchTree_00_1=alltre02.tre
searchTree_00_0=alltre01.tre

Now that you've gotten this far it's time for the next step which is optional if you're using the Core strait up with 14.1 with no new expansion content. However, if you want to use the Expansion content this is by far the easiest way to implement it:

1) First yo need to obtain the last sku0_data.TOC (clarify file name) that SOE released. I don't have the file handy ATM so I'll edit to clarify date/time stamps and such.

2) using the same procedure outlined above for my alltre's, Open the .toc file using treexplorer and export the entire contents to a folder, I used fulltoc.

3) once the export is complete you'll need to organize your fulltoc directory into useable chunks of data. So for instance the appearance folder alone contains more files than trepacker will work with so you need to divide it up. Same with some of the other data groups as well. I ended up needing to make 6 sub folders (fulltoc01-fulltoc06) in order to keep the data groups small enough for trepacker to work without stack overflows.

4) pack each of the sub folders into their own .tre file.

5) modify your swgemu_live.cfg & core3 conf files to load the new .tre files in addition to the 14.1 consolidated .tre files we made above, here is the order I use. BE AWARE THAT THE 14.1 ASSETS ARE WHAT THE CORE NEEDS, so you must set your order such that the 14.1 assets overwrite the TOC assets. Otherwise it's buggy central.

Here's my order:
[SharedFile]
maxSearchPriority=9
searchTree_00_8=mytre.tre
searchTree_00_7=alltre02.tre
searchTree_00_6=alltre01.tre
searchTree_00_5=fulltoc06.tre
searchTree_00_4=fulltoc05.tre
searchTree_00_3=fulltoc04.tre
searchTree_00_2=fulltoc03.tre
searchTree_00_1=fulltoc02.tre
searchTree_00_0=fulltoc01.tre

Use this load order for the core too.

Now, if you've managed to consolidate the assets as I have you can add objects at will, only adding what you need to change / modify to your own personal .tre file (mytre.tre in this example). If you don't need to modify an asset for it work properly, then there's no reason to include it in it's own .tre file. let it load it from the consolidated asset data.

And there you have it. Hopefully this helps some to understand how my other guides and mods work.

-Duff
 

duffstone

New Member
Sep 13, 2013
179
0
0
#2
I need to qualify here that while consolidating the original 14.1 .tre files (all ~53 of them) works just fine with zero issues in all my testing, Consolidating the .TOC's into the 6 additional .tre files CAN cause some issues, especially in non-test, full load production environment.

Example: The TOC data does something funky in and around Restuss, thus you'll get a Crash To Desktop if you happen to wander NE of Restus. I'm guessing this is because restuss was destroyed later in the life of the game and replaced with models that that don't render properly in the 14.1 client without some work.

There are some other things along these lines that don't work either. HOWEVER, if you're needing to do some quick work using post 14.1 data, and you're not looking to go into a production server environment with your results, then the above method works great.

Just use caution.

-Duff
 

Lasko

Moderator
Staff member
Moderator
Feb 13, 2012
249
2
18
#3
Tried this with the pre CU tre's and it works perfectly.

Just having some trouble with anything post that. I loaded the CU weapons from the Lua's that I created but they wont load in game. Seems the fulltoc tre's load ok on the server as when I try to pull a cu weapon from the frog i get no objectCRC error messages on the server, but in the client I get the message "you received a: %TO" and nothing appears.

Just wondering if you'd seen this in your trials, gonna keep playing today, see if I can see what I missed.
 

duffstone

New Member
Sep 13, 2013
179
0
0
#4
Yeah, you have to update your object_crc table in the /misc folder for each item you add. Every object needs a crc in that file.
 

duffstone

New Member
Sep 13, 2013
179
0
0
#6
YW, but don't thank me too quickly. it's a real PITA to keep up with that file and what is and is not in it. CRCEdit doesn't have a sort feature so it's really tough to figure out crc issues. they're easy enough to recognize tho, the core will spit out a can't find crc message, or your client will load the object but you can't see it.

One of these day's I'll write something to make it easier, but I'm too wrapped up in "discovery" mode to actually settle into "development" mode. :)

-Duff
 

kinshi

New Member
Dec 13, 2011
18
0
0
#7
The SOE scheme was really designed around dial up connections, thus why so many smaller files. Consolidating TREs is not a bad idea at all in this age of broadband. (besides you wind up knowing what is in the final TRE much more so than having it spread all over multiple TREs)

As far as the object CRCs, what I have been doing is building off the Pub 14.1 CRC and adding to it one NGE object at a time, and have my own CRC list (as the SOE one is just too big to sift through, and its useful to know exactly what is in there by virtue of adding it in there myself)

I had gone thru the gross process of extracting all the TREs having gotten sick of all the cherry picking, then put my post-Pub 14.1 stuff under version control.
 

duffstone

New Member
Sep 13, 2013
179
0
0
#8
It would be nice to have a new CRC tool that would give better functionality. I thought about doing it in .net, but i don't understand any of the existing CRC code and how they're generated. I've tried, but didn't come close and bitwise operations kick my arse...

I'd like some sort of merging ability so I can maintain my own CRC file, and keep the 14.1 CRC file pure, and then merge them before packing a new .tre. A search / sort feature would be nice too. :)

maybe if the operation is something Uli or Timbab understand they can help me out so I can put all the pieces together in .net.

-Duff
 

Lasko

Moderator
Staff member
Moderator
Feb 13, 2012
249
2
18
#9
You can use tre explorer to search for object strings already in the crc file if that's any help, I use that to check what is actually contained in the file or if something I added was successfully updated. Dose not sort tho.

On a side note, does the addition of stuff like post 14.1 stuff into that file get ignored by the core until it is called for or will it cause conflicts somewhere? Just wondering about using the final version of that file and re adding items that was taken out over the years instead of working forward and adding Cu and NGE crc's. Going to try it anyways, just wondered if anyone had tried it??
 

duffstone

New Member
Sep 13, 2013
179
0
0
#10
if you try to use the very last version of the object_crc file, you'll get a CDT. I have not investigated why yet. but I did try that first.

-Duff
 

Lasko

Moderator
Staff member
Moderator
Feb 13, 2012
249
2
18
#11
Yep, tried it and CDT. No server error messages

Working forward it is then.... CU mobs complete now, adding Musty mobs.

Lets hope it's the Ewok love wings that cause the CDT:)
 

duffstone

New Member
Sep 13, 2013
179
0
0
#12
not sure if you caught the other thread but working on a new CRC tool that will fill some gaps for us. may take a couple weeks, although it's mostly interface and i/o stuff now. I hope. LOL... - Duff
 

Valkyra

Super Cool Person
Staff member
Moderator
Aug 31, 2010
167
5
18
Murica
#13
Not sure if this is relevant, but...

Core3's TRE lib extension cannot currently read .toc files, and TheAnswer initially told me they had no planned support for it.

If you saw a post I made on the SWGEmu forums, I had been working with making a fork per se for a CU-based emulator. I don't know if you can get the Pre-CU client to load every single file past 14.1 which from what I skimmed in this thread seems you are doing?

Anyway, I was using a Publish 24 CU client, the same one used by the now defunt OpenSWG project. However, what I had to do to get the server to read the files without a toc was open each TRE from publish 15-26 (I think was the number, whatever corresponds to Publish 24 CU the one before the NGE), then save all their files in the correct directory structure, and I made 2 .tre's out of them, one is _client and one is _server, like the TRE formats they have after publish 17. These were purely for the server to read, rather than the client.

I got to the point where I could zone in, and I even had some code in to load the datatables for buildout, since as you know on Kashyyyk and Mustafar, they used datatables instead of .ws world snapshot files.

TA added the ability to load parse server heightmaps from TGA, which is what mustafar and kashyyyk use. I at one point I was able to successfully zone onto Mustafar, in my client on Core3, and run around with most of the packets working correctly.

I still have the 2 TRE's on my HDD on my PC, but my motherboard broke December 5th and RMA'ing it with ASUS has been a nightmare. I should have it repaired by Monday of next week after this weekend so I can access them if anyone wants.

I'll probably be working on a CU emulator based off Core3 framework in the future, nothing soon though since it takes time to figure out things and I'd rather have Core3 base implementations all in and fixed to help reach 1.0 sooner.
 

duffstone

New Member
Sep 13, 2013
179
0
0
#14
sounds awesome from a functional point of view, I'm still on the 14.1 "Style" of game play band wagon, with all the NEW assets made available however. But either way the more heads we have working on what "Can" be done, the better equipped we'll be to make it how we want it. :)

-Duff
 
Jun 18, 2013
37
0
0
34
tarkin.org
#17
Didn't want to start a new thread since I'm asking directly about something in this one...

swgemu_live example:
maxSearchPriority is always total number of files to load. searchTree_00_x= always begins with zero and counts up to the newest file.
[SharedFile]
maxSearchPriority=3
searchTree_00_2=mytre.tre
searchTree_00_1=alltre02.tre
searchTree_00_0=alltre01.tre
Can you explain the way it works now:

searchTree_00_14=patch_11_00.tre
searchTree_01_13=data_sku1_05.tre
searchTree_01_13=data_sku1_04.tre
searchTree_01_13=data_sku1_03.tre
searchTree_01_13=data_sku1_02.tre
searchTree_01_13=data_sku1_01.tre
searchTree_01_13=data_sku1_00.tre
searchTree_00_12=patch_10.tre
searchTree_00_11=patch_09.tre
searchTree_00_10=patch_08.tre
searchTree_00_9=patch_07.tre
searchTree_00_8=patch_06.tre
searchTree_00_7=patch_05.tre
searchTree_00_6=patch_04.tre
searchTree_00_5=patch_03.tre
searchTree_00_4=patch_02.tre
searchTree_00_3=patch_01.tre
searchTree_00_2=patch_00.tre
searchTree_00_1=data_other_00.tre
searchTree_00_1=data_static_mesh_01.tre
searchTree_00_1=data_static_mesh_00.tre
searchTree_00_1=data_texture_07.tre
searchTree_00_1=data_texture_06.tre
searchTree_00_1=data_texture_05.tre
searchTree_00_1=data_texture_04.tre
searchTree_00_1=data_texture_03.tre
searchTree_00_1=data_texture_02.tre
searchTree_00_1=data_texture_01.tre
searchTree_00_1=data_texture_00.tre
searchTree_00_1=data_skeletal_mesh_01.tre
searchTree_00_1=data_skeletal_mesh_00.tre
searchTree_00_1=data_animation_00.tre
searchTree_00_1=data_sample_04.tre
searchTree_00_1=data_sample_03.tre
searchTree_00_1=data_sample_02.tre
searchTree_00_1=data_sample_01.tre
searchTree_00_1=data_sample_00.tre
searchTree_00_1=data_music_00.tre
searchTree_00_0=bottom.tre
Not all of them, there, just wanted enough to illustrate. Why are there so many _00_1's, what does that mean that they all end in the same #?

And how about searchTree_01_13? There's no _00_13, only _01_13, what does the 01 first indicate?

Thirdly, regarding adding new NGE things - I've heard that certain things need to load before other things in order for stuff to work properly. Can anyone clarify what that means? Obviously I know I want any crc's and stfs that have been modified to go on top of older tres that they'll need to overwrite. But how about shaders, efts, that kind of thing?
 

hawkbs

New Member
Feb 23, 2013
7
0
0
#19
duffstone said:
if you try to use the very last version of the object_crc file, you'll get a CDT. I have not investigated why yet. but I did try that first.

-Duff
Nifty little thing I just discovered using Sytners IFF Editor 3.
if you open up the sku0_client.toc file then click the Tab that says Extract All History it will extract every single patch from launch to sunset into their own respective folders. This way hopefully we can grab every single object crc file that was ever released. Theres something like 193k files though that are going to be extracted so make sure you have room and create a folder somewhere to put them into. I just created one called Patch History. see attached pic.
N.B mines still extracting at this time so will have to wait and see but nonetheless this is a fantastic feature.
Hope this helps everyone!
 

Attachments

Timbab

Administrator
Staff member
Administrator
Oct 6, 2010
999
5
18
27
Magna Germania
#20
Nope, you can't extract every file like that, there are a bunch you won't see, due to the .toc only being able to show you the most recent files (No historical view), at least, for anything post patch 17.

That's one of the reasons why I've been collecting tocs from different eras to create a more complete picture. Due to requests, I'll probably start creating specific tocs for each tre post 17 that combines all the information from the tocs I have sometime this week.

Context to why you won't get information post 17; Version 5 .tre files had mini tocs inside of them, containing information of what was inside, with patch 17, they removed it and used a centralized .toc, which only ever kept track of the latest version of each used file.