Vertex/Pixel Shader v4.0?

[Borrie BoBaka]

So this might be a strange one to talk about. 

I've been on a quest to get my game looking as fantastic as possible, but I recently noticed there was some discrepancies between what my client had been seeing, and what I had seen a few generic Screenshots of SWGEmu displayed. Most notably, I had started to see screenshots of Protocol Droids, Chrome Rims, and other shiny materials actually displaying a reflection, and had remembered that this was a thing I saw back on live, before the NGE.

So I did some research and it seems reflections of that calibre are related to the Pixel Shader version being used. When my client uses 'Optimal' it has dull, matte reflections, none at all. 2.0 did the same thing. 1.4 gives me these reflections but they're static on the model (you can see the reflection almost plastered on the texture, not a true reflection).

I did some more research and discovered some references to a Vertex/Pixel Shader Version 4.0, but my client does not have this available. I am wondering if this is just a wild Unicorn and 4.0 either doesn't exist or was an NGE only thing, or if my client is significantly missing a graphical upgrade. And if it is, how do I upgrade it?

Thanks for any assistance, I am hoping to get my client as graphically updated as possible, and don't want the base client itself not being all it can be to start with.

 

[Timbab]

Afaik, all the shaders both in preCU and NGE are written in in 2.0 or below (Check vertex_program/ and pixel_program/). But since SWG is dx9, you're stuck with 3.0, 4.0 came with dx10.

I'm the wrong person to answer this, but unless I'm mistaken, SWG should therefor be able to support 3.0.

That said, NGE and PreCU had different shading pipes and using NGE shaders breaks a PreCU client for the post part, or at least shades it wrong. Unsure why this is atm, I never really looked into it but plan to at some point, some can probably be rewritten to work with the PreCU client.

Unfortunately, we don't have any mod that does anything with the HLSL shaders, any mod claiming to in the past was lying and/or touched simpler stuff like .SHT files (Texture 'container', links to textures and gives some 'effect' instructions along with the link to an .EFT, which links to the HLSL shaders).

Wish someone experienced would take the time!

 

[Sytner]

What Timbab said is right regarding SM4.0, but either way it wouldn't be a silver bullet to solve the type of issue you talked about. There's really nothing we could do to harness the new features added between SM2.0 and SM4.0 without the ability to easily modify the game's source code. Generally speaking, to be able to make good use of the new features you need to provide additional information to the shader. The game code itself is responsible for providing this information and that's going to be hard to change. What someone actually needs to do is take a look at the SM1.4 and SM2.0 shaders we have (iirc the 1.4 ones may be in assembly so it's a bit trickier), see what's different and then see if the 2.0 version can be fixed/improved in a reasonable way.

 

[Borrie BoBaka]

Thank you for the replies. When I used to do modding for SWG a long time ago I hadn't fully understood the limitations of the game's engine in such a way that I could tell the difference. I had suspected that 'Optimal' was 3.0 and so I was looking for a 4.0, but I suspect my top version is 2.0, as no 3.0 appears on the list.

The primary reason I started this thread is because I have been wondering why 1.4 looks better than 2.0, and if I was perhaps missing something in the pipeline. For instance, below, I've taken a comparison shot of 2.0 and 1.4.

https://dl.dropboxusercontent.com/u/2165377/Archive/Misc/ShaderVersionDifferences.png

I was wondering why it was showing an Envmap for reflections in 1.4 but not in 2.0, which should be that and more. You could wager the Protocol Droid looks better in 2.0, which I wouldn't immediately disagree, but the lack of the envmap is a bit jarring and makes it look matte and dull. I was happy with the result but I noticed that the reflections do not move or alter when you move your camera, just when the character moves. I do not know if this was how it always was, but it feels a bit off in my opinion, so I was in search for answers.

I suppose the final question I should ask is, is 3.0 compatible with PreCu SWG, and if so, am I missing it, or is it hiding?

https://dl.dropboxusercontent.com/u/2165377/Archive/Misc/SWGclientSetup1.png

That is all that shows up when I go to select the version of the shader. If I am missing 3.0, how would I go about getting it?

 

[Sytner]

It's not supported and there are no existing shaders that would take any advantage of it, but SM3.0 is in theory possible with some modifications. There's really no reason for anyone to bother trying to add it though - it would gain us nothing without other serious changes.

I agree the 1.4 shots look better. Unless we can't get the client to provide the necessary information it shouldn't be too hard to fix up the 2.0 ones to do the same thing (or better).

 

[zelvader]

even though i do want the shader v4.0, we've gonna have to figure it out later but for right now it's not going to happen until one of us will have to make a special tool to do this.

 

[Borrie BoBaka]

Thank you for all the feedback, everyone. I was looking for clarification and I got it. 

When I played SWG for the first time back in '03, I wasn't running with the greatest graphical capabilities at the time, so I never really knew what was causing the jump between dull graphics to the shiny ones. The envmap usage does make all the difference in the world visually when using 1.4 over 2.0, and I was really wondering why the latest version of the shader was omitting something like that. 

It sounds like this topic is knocking on the door of a rather large technological boost to SWG if the time, energy, work and research was there, so I'll let sleeping dogs lie until someone takes the charge to figure that out. But now I understand my client isn't lacking something. 

1.4 over 2.0 is going to work fine for me for now, the only thing I seemingly lose is aging on characters, oddly enough. As though the normal maps don't render appropriately with 1.4. On that note, actually, I might see if 1.4 is rendering normal maps at all. If that's the case, I may have to choose Normal Maps over Envmap metallic shine, which will be a bit of a tough decision.

 

[Uli]

even though i do want the shader v4.0, we've gonna have to figure it out later but for right now it's not going to happen until one of us will have to make a special tool to do this.

I don't think you read the post fully.

The game needs to be running in DirectX 10 or Higher for 4.0 support, I'm curious what specific features are available in the 4.0 shader model do you want to utilize which are not part of 3.0?

It's not exactly easy nor worth the time to be able to turn the game from DX9 to DX10+, requires rewriting the entire render pipeline for the game.

 

[Timbab]

even though i do want the shader v4.0, we've gonna have to figure it out later but for right now it's not going to happen until one of us will have to make a special tool to do this.

That's never going to happen. 

Shader 4.0 is DirectX 10, SWG is DirectX 9.

Thank you for all the feedback, everyone. I was looking for clarification and I got it. 

When I played SWG for the first time back in '03, I wasn't running with the greatest graphical capabilities at the time, so I never really knew what was causing the jump between dull graphics to the shiny ones. The envmap usage does make all the difference in the world visually when using 1.4 over 2.0, and I was really wondering why the latest version of the shader was omitting something like that. 

It sounds like this topic is knocking on the door of a rather large technological boost to SWG if the time, energy, work and research was there, so I'll let sleeping dogs lie until someone takes the charge to figure that out. But now I understand my client isn't lacking something. 

1.4 over 2.0 is going to work fine for me for now, the only thing I seemingly lose is aging on characters, oddly enough. As though the normal maps don't render appropriately with 1.4. On that note, actually, I might see if 1.4 is rendering normal maps at all. If that's the case, I may have to choose Normal Maps over Envmap metallic shine, which will be a bit of a tough decision.

Most likely all that's needed, is modifying the HLSL shdaders, which are found in the .TRE files. They've just written the 1.4 differently than the 2.0 one (This is one shader we're talking about, for this specific example), but most likely can be changed without too much hassle to what you desire. No one to my knowledge has toyed with the HLSL shaders though, but they're not exactly hidden. 

Just look at:

shared_program/
pixel_program/ (These are compiled and in an .IFF container, but you find a copy of the code in the PSRC chunk)
vertex_program/

Edit 1:

To make it more clear how the shaders are linked to a texture (I've also slapped each one onto pastebin):

• shader/pak_protocol_droid_red_l2_aes17.sht
  • effect/a_envmask_specmap.eft
    • Shader 2.0
      • vertex_program/a_envmask_spec_vs20_for_ps20.vsh
      • pixel_program/a_envmask_specmap_ps20.psh
    • Shader 2.0 - No Dot3
      • vertex_program/a_envmask_spec_vs20_nodot3.vsh
      • pixel_program/a_envmask_specmap_ps20_nodot3.psh
    • Shader 1.1
      • vertex_program/a_envmask_specmap_vs11_for_ps11.vsh
      • pixel_program/a_envmask_specmap_ps11.psh

Edit 2:

Actually the only thing that's written in ASM are the shader model <1 I believe shaders, for backwards compatibility at the time. Some might still use this for some things though, unsure atm.

Any vsh/psh marked with vs or ps11/vs or ps20, etc should be in HLSL and not asm, anything with no _vs/_ps should be in asm.

These files for example:
ASM: vertex_program/a_3uv_blend.vsh
SM1.1: vertex_program/a_3uv_blend_vs11.vsh
SM2.0: vertex_program/a_3uv_blend_vs20.vsh


Hopefully that'll make it a bit easier to understand, at least in the sense of why X looks different than Y. It's the shaders themselves and not just an update to a new shader model version.

 

[eqsanctum]

even though i do want the shader v4.0, we've gonna have to figure it out later but for right now it's not going to happen until one of us will have to make a special tool to do this.

That's never going to happen. 

Shader 4.0 is DirectX 10, SWG is DirectX 9.

Thank you for all the feedback, everyone. I was looking for clarification and I got it. 

When I played SWG for the first time back in '03, I wasn't running with the greatest graphical capabilities at the time, so I never really knew what was causing the jump between dull graphics to the shiny ones. The envmap usage does make all the difference in the world visually when using 1.4 over 2.0, and I was really wondering why the latest version of the shader was omitting something like that. 

It sounds like this topic is knocking on the door of a rather large technological boost to SWG if the time, energy, work and research was there, so I'll let sleeping dogs lie until someone takes the charge to figure that out. But now I understand my client isn't lacking something. 

1.4 over 2.0 is going to work fine for me for now, the only thing I seemingly lose is aging on characters, oddly enough. As though the normal maps don't render appropriately with 1.4. On that note, actually, I might see if 1.4 is rendering normal maps at all. If that's the case, I may have to choose Normal Maps over Envmap metallic shine, which will be a bit of a tough decision.

Most likely all that's needed, is modifying the HLSL shdaders, which are found in the .TRE files. They've just written the 1.4 differently than the 2.0 one (This is one shader we're talking about, for this specific example), but most likely can be changed without too much hassle to what you desire. No one to my knowledge has toyed with the HLSL shaders though, but they're not exactly hidden. 

Just look at:

shared_program/
pixel_program/ (These are compiled and in an .IFF container, but you find a copy of the code in the PSRC chunk)
vertex_program/

Edit 1:

To make it more clear how the shaders are linked to a texture (I've also slapped each one onto pastebin):

• shader/pak_protocol_droid_red_l2_aes17.sht
  • effect/a_envmask_specmap.eft
    • Shader 2.0
      • vertex_program/a_envmask_spec_vs20_for_ps20.vsh
      • pixel_program/a_envmask_specmap_ps20.psh
    • Shader 2.0 - No Dot3
      • vertex_program/a_envmask_spec_vs20_nodot3.vsh
      • pixel_program/a_envmask_specmap_ps20_nodot3.psh
    • Shader 1.1
      • vertex_program/a_envmask_specmap_vs11_for_ps11.vsh
      • pixel_program/a_envmask_specmap_ps11.psh

Edit 2:

Actually the only thing that's written in ASM are the shader model <1 I believe shaders, for backwards compatibility at the time. Some might still use this for some things though, unsure atm.

Any vsh/psh marked with vs or ps11/vs or ps20, etc should be in HLSL and not asm, anything with no _vs/_ps should be in asm.

These files for example:
ASM: vertex_program/a_3uv_blend.vsh
SM1.1: vertex_program/a_3uv_blend_vs11.vsh
SM2.0: vertex_program/a_3uv_blend_vs20.vsh


Hopefully that'll make it a bit easier to understand, at least in the sense of why X looks different than Y. It's the shaders themselves and not just an update to a new shader model version.

So as an outsider to this, considering Larce is doing hefty in depth FAQs, should we recommend using 1.4 instead of 2.0 or optimized in client settings

 

[Timbab]

I'd expect optimized to automatically set it to 2.0 internally (I haven't actually checked/looked, but it's easy to confirmed).

Unless it changes stuff on the fly.

Any modern GPU can handle 2.0 with ease though, will need to look into how it's handled internally with optimized then. So safest bet is to do 2.0.

 

[Sytner]

Optimal will result in 2.0 getting picked. If you think 1.4 looks better, and it does seem to for the shots you posted at least, then sure it seems reasonable to suggest that people use it. Or at least let them know about the differences.

even though i do want the shader v4.0, we've gonna have to figure it out later but for right now it's not going to happen until one of us will have to make a special tool to do this.

:huh:

 

[eqsanctum]

Optimal will result in 2.0 getting picked. If you think 1.4 looks better, and it does seem to for the shots you posted at least, then sure it seems reasonable to suggest that people use it. Or at least let them know about the differences.

Sounds good thank you.

 

[eqsanctum]

I'd expect optimized to automatically set it to 2.0 internally (I haven't actually checked/looked, but it's easy to confirmed).

Unless it changes stuff on the fly.

Any modern GPU can handle 2.0 with ease though, will need to look into how it's handled internally with optimized then. So safest bet is to do 2.0.

I'll have to double check on what it changes I'll get back to you.

 

[Sytner]

I'd expect optimized to automatically set it to 2.0 internally (I haven't actually checked/looked, but it's easy to confirmed).

Unless it changes stuff on the fly.

Any modern GPU can handle 2.0 with ease though, will need to look into how it's handled internally with optimized then. So safest bet is to do 2.0.

I'll have to double check on what it changes I'll get back to you.

I checked yesterday and it will definitely pick SM2.0. If you pick a value other than optimal the config file stores the maximum minor and major shader versions permitted. When you set it to optimal the config file is left empty. Then when the client loads the maximum shader version is initialised to int max. The client then determines the shader capabilities from this maximum version. This will result in SM2.0 being picked as it is the highest supported by the client.

I had a look at what it would take to use SM3.0 shaders. For the pixel shaders it is sufficient to just compile them using SM3.0 and put them in the SM2.0 part of the .psh file. For vertex shaders a bit more work is required since they're compiled at runtime. There is a string in the client, "vs_2_0" that needs to be changed to "vs_3_0" to switch this compilation. This requires then either a modified client or a patch that's applied when launching.

 

[Borrie BoBaka]

This may potentially be unrelated but I wanted to make sure that the shader versions aren't having an effect on this:

I've noticed since getting more involved with the Emu, that the game has some strangeness with Z-Fighting when two objects collide but don't have the same co-ordinates. I know Z-Fighting is when two planes try to occupy the same space and they fight over which texture to display constantly. However, the effect of Z-Fighting appears to be happening at any intersection of objects, primarily paintings and flat models, but more than that as well. 

I made a gif of the effect in play, of a rug intersecting a ceiling wall. There shouldn't be any Z-Fighting at all, theoretically, but it is still jittering on the edges. 

https://dl.dropboxusercontent.com/u/2165377/Archive/Misc/ZfightingGif1.gif

Is this related to the shaders in play? Or is it something completely unrelated, like a mesh issue, resolution, or a newer computer trying to run an older game?

 

[Sytner]

This may potentially be unrelated but I wanted to make sure that the shader versions aren't having an effect on this:

I've noticed since getting more involved with the Emu, that the game has some strangeness with Z-Fighting when two objects collide but don't have the same co-ordinates. I know Z-Fighting is when two planes try to occupy the same space and they fight over which texture to display constantly. However, the effect of Z-Fighting appears to be happening at any intersection of objects, primarily paintings and flat models, but more than that as well. 

I made a gif of the effect in play, of a rug intersecting a ceiling wall. There shouldn't be any Z-Fighting at all, theoretically, but it is still jittering on the edges. 

https://dl.dropboxusercontent.com/u/2165377/Archive/Misc/ZfightingGif1.gif

Is this related to the shaders in play? Or is it something completely unrelated, like a mesh issue, resolution, or a newer computer trying to run an older game?

I can't view your image at work but I think this is very unlikely to be a shader issue unless they were doing custom depth buffer writes inside them. e.g. to achieve a logarithmic depth buffer. I've never heard of SWG doing anything like this.

It may be more noticable now than on live since we all have better machines and are likely maxing the draw distance. A higher draw distance will mean less precision in the depth buffer. Less precision means that triangles close but not necessarily intersecting have a greater chance of causing z-fighting.

 

[eqsanctum]

I'd expect optimized to automatically set it to 2.0 internally (I haven't actually checked/looked, but it's easy to confirmed).

Unless it changes stuff on the fly.

Any modern GPU can handle 2.0 with ease though, will need to look into how it's handled internally with optimized then. So safest bet is to do 2.0.

I'll have to double check on what it changes I'll get back to you.

I checked yesterday and it will definitely pick SM2.0. If you pick a value other than optimal the config file stores the maximum minor and major shader versions permitted. When you set it to optimal the config file is left empty. Then when the client loads the maximum shader version is initialised to int max. The client then determines the shader capabilities from this maximum version. This will result in SM2.0 being picked as it is the highest supported by the client.

I had a look at what it would take to use SM3.0 shaders. For the pixel shaders it is sufficient to just compile them using SM3.0 and put them in the SM2.0 part of the .psh file. For vertex shaders a bit more work is required since they're compiled at runtime. There is a string in the client, "vs_2_0" that needs to be changed to "vs_3_0" to switch this compilation. This requires then either a modified client or a patch that's applied when launching.

Hmm, have to sit down with Aso about this one. As far as Larce though I'll have him recommend the optimal setting for the shaders.