|
|
|
This thread is locked; no one can reply to it.
|
| Game Modding Texture problems |
|
Dario ff
Member #10,065
August 2008
 |
I've been trying at this over the past days and the solution evades me. I'm trying to mod Sonic Generations(released on Steam 2 months ago), to import the geometry stages from an old Xbox 360 game, Sonic Unleashed, which uses pretty much the same engine. The evidence is clearly there since I just copied and modified some of the stage files, and the geometry loads. The problem is the texture mapping is all wrong. My question for the 3D techies here is, what could the problem be? Perhaps manually modifying the textures to be flipped or something could solve it? Or would I have to fix the UV mapping? It's all in binary format so it would require heavy hex editting to figure it out, so perhaps I can just do a batch conversion of the textures and fix it. Here's an example of the issue, and the source texture to spot the problem: This is a screencap of the original game(youtube video, bad quality): And here it's ported over to the PC game: I've been focusing in those small colored strips to spot the texture problem. Here's the original texture: As you can see, the original game just renders the "Green" part I marked below, but the PC game seems to render the "Red" section I marked. Is there any texturing differences on the Xbox 360 I should be aware off? EDIT: I can't say how accurate that "red" section I marked is. After all, it's not really different from any other tall rectangle you could mark in the texture. EDIT2: It seems rotating the image and keeping the same size, plus stretching and flipping fixed it. TranslatorHack 2010, a human translation chain in a.cc. |
|
Ron Novy
Member #6,982
March 2006
 |
I'm not aware of any difference in PC and XBox uv coordinates, but if you're modding a game then it's most likely a difference in the way the game is on PC and XBox... Honestly though, it looks like they are simply rotated as you said. Have you tried rotating the textures or is that just too complex? Also, the XBox 360 uses big-endian byte ordering and on the PC you have little-endian byte ordering. Could be an endianness issue... ---- |
|
Dario ff
Member #10,065
August 2008
|
Curiously enough, a 90 degrees CW rotation and horizontal flip solves the problem for JUST a few textures. Luckily, it's the majority, but stuff such as skies for example don't need to be flipped. I think it's just weird handling by their engine or something... Luckily, if I hadn't created this thread I wouldn't have noticed it tho. Ron Novy said: Also, the XBox 360 uses big-endian byte ordering and on the PC you have little-endian byte ordering. Could be an endianness issue...
That reminds me, I've been also trying to port over the Havok binary files from the game. Apparently, they're saved specifically for the XBox 360(I've seen the option on the SDK), and it's not compatible with the PC version... from looking over it, the only difference is the endianness in fact. I suppose I don't really have a way of converting the file without knowing the format, do I? It's closed source. TranslatorHack 2010, a human translation chain in a.cc. |
|
Ron Novy
Member #6,982
March 2006
|
Oh no! 1MB. Having worked on AATranslator I have helped work out a few file formats here and there. I working on a new library for AAT right now but need a distraction for the moment... ---- |
|
Dario ff
Member #10,065
August 2008
|
This is the Xbox 360 hkx binary file from Sonic Unleashed. It handles the level collision data. I've downloaded the Havok SDK PC tools and they can't just open it, claiming it's not for the specified platform. According to their page, I would need a fully licensed Havok SDK for opening it(SAVING to the 360 version is possible, but not opening), but my guess is that it can probably be converted... Since of course Sonic Unleashed was never released on PC, there's no PC version of those binary files to compare to. Generations uses a similar engine ported over to PC tho, so I took the chance to convert and compare the same binary hkx file. The (free) Havok content tool can open them since they were saved for the PC after all. Green Hill Zone PC HKX TranslatorHack 2010, a human translation chain in a.cc. |
|
Ron Novy
Member #6,982
March 2006
|
Well, so far the Green Hill Zone PC HKX resaved to XBOX360 format looks to be in little-endian as well, not sure why. It is different, just no change in endianness... [edit] You also seem to have an older version of the SDK then I do. And some of my SDK programs are not working for some reason... ---- |
|
Dario ff
Member #10,065
August 2008
|
That's because I'm a retard and uploaded the wrong binary file. Here's the real converted 360 file. Double checked this time, I can't open it on the havok tools. As for the older version, that's intentional because that's the version the game on the PC is using. TranslatorHack 2010, a human translation chain in a.cc. |
|
Ron Novy
Member #6,982
March 2006
|
Ahh... So I can confirm then, the byte at 0x11 determines endianness of the file. That's the only thing I'm really sure of here... 1struct hvk_head {
2 char o0x00_57E0E05710C0C010[8]; // Always the same 8 bytes...
3 int32 o0x08_unknown; // ??? endianness switches.
4 int32 o0x0C_unknown; // ??? endianness switches.
5 char o0x10_flag_bytes[4]; // 0x11 identifies endianness...
6 int32 o0x14_unknown[5]; // ??? 5 integers. endianness switches.
7 char o0x28_havoklibname[24]; // Anything after byte 15 seems to have no affect.
8};
9
10/* 3 sections starting after hvk_head... */
11struct hvk_section {
12 char name[16]; // Section name. __classnames__, __types__ or __data__
13 int32 o0x10_000000ff; // ??? Not affected by endianness.
14 int32 offset; // Literal file offset to data for this section. Is affected by endianness.
15 int32 sub_offset[6]; // Offset to sub section. affected by endianness.
16 // Literal file offset to subsection 'x' is 'offset + sub_offset[x]'
17};
18
19/* Class names */
20struct hvk_classname {
21 int32 o0x00; // ?? Affected by endianness.
22 char o0x04_0x09; // ?? Always seems to be '0x09' which may indicate that you need to read a null terminated string.
23 char *o0x05; // Null terminated string.
24};
Trying to figure out where to go from there... Have to do some other stuff right now though... ---- |
|
Arthur Kalliokoski
Second in Command
February 2005
 |
If it's confusing rows with columns in the image, I'd suspect much more than endianness. They all watch too much MSNBC... they get ideas. |
|
Ron Novy
Member #6,982
March 2006
|
Edited the structure hvk_section in my post above. It actually gets a lot easier the more you dig into the format. It's even easier if you know what data is already there. Right now my Havok SDK doesn't seem to be working. It could be because the computer I'm trying to run it on is about 10 years old now... As long as you double check the endianness between the PC and 360 version you should be able to create a nice conversion utility... Just can't work on it anymore myself right now... Need to sleep. Lots to do tomorrow... ---- |
|
Dario ff
Member #10,065
August 2008
|
Wow, thanks for that. The Havok reader won't load up on your PC if you haven't installed it. If it's a folder from an old installation, it won't work. It needs some system dlls, so a proper reinstall should do the trick. AFAIK, any new version will read the old versions without any trouble. These collision files don't really have much visual data to preview on their tool tho, it's just loads and loads of vertexes(someone who was researching them told me it was a set ammount of objects, and depending on that the vertex data would be interpreted). Given that it's the same game engine underneath, switching the endianness at the right spots where it's affect, and a compatible header, should make it work. EDIT: I IS DERP.
Turns out the headers do hold some relevant information which would probably save you a stupid ammount of research... Here's a folder of the headers. Some relevant files: hkPackfileHeader.h, hkPackfileSectionHeader.h. I guess there might be a way after all. 1
2#ifndef HKSERIALIZE_SERIALIZE_BINARY_HKPACKFILEHEADER_XML_H
3#define HKSERIALIZE_SERIALIZE_BINARY_HKPACKFILEHEADER_XML_H
4
5
6/// hkPackfileHeader meta information
7extern const class hkClass hkPackfileHeaderClass;
8
9/// The header of a binary packfile.
10class hkPackfileHeader
11{
12 //+version(1)
13 public:
14
15 HK_DECLARE_NONVIRTUAL_CLASS_ALLOCATOR(HK_MEMORY_CLASS_SERIALIZE, hkPackfileHeader);
16 HK_DECLARE_REFLECTION();
17
18 /// Default constructor
19 hkPackfileHeader()
20 {
21 hkString::memSet(this, -1, sizeof(*this));
22 m_magic[0] = 0x57e0e057; // both endian agnostic (byte symmetric)
23 m_magic[1] = 0x10c0c010;
24 m_contentsVersion[0] = 0;
25 m_flags = 0; // Set to 1 when a packfile is loaded in-place to signify that the packfile is loaded
26 }
27
28 public:
29
30 /// Magic file identifier. See constructor for values.
31 hkInt32 m_magic[2];
32
33 /// This is a user settable tag.
34 hkInt32 m_userTag;
35
36 /// Binary file version. Currently 9
37 hkInt32 m_fileVersion;
38
39 /// The structure layout rules used by this file.
40 hkUint8 m_layoutRules[4];
41
42 /// Number of packfilesections following this header.
43 hkInt32 m_numSections;
44
45 /// Where the content's data structure is (section and offset within that section).
46 hkInt32 m_contentsSectionIndex;
47 hkInt32 m_contentsSectionOffset;
48
49 /// Where the content's class name is (section and offset within that section).
50 hkInt32 m_contentsClassNameSectionIndex;
51 hkInt32 m_contentsClassNameSectionOffset;
52
53 /// Future expansion
54 char m_contentsVersion[16];
55
56 ///
57 hkInt32 m_flags;
58 hkInt32 m_pad[1];
59};
60
61#endif // HKSERIALIZE_SERIALIZE_BINARY_HKPACKFILEHEADER_XML_H
1
2#ifndef HKSERIALIZE_SERIALIZE_BINARY_HKPACKFILESECTIONHEADER_XML_H
3#define HKSERIALIZE_SERIALIZE_BINARY_HKPACKFILESECTIONHEADER_XML_H
4
5#include <Common/Serialize/Resource/hkResource.h>
6
7/// hkPackfileSectionHeader meta information
8extern const class hkClass hkPackfileSectionHeaderClass;
9
10/// Packfiles are composed of several sections.
11/// A section contains several areas
12/// | data | local | global | finish | exports | imports |
13/// data: the user usable data.
14/// local: pointer patches within this section (src,dst).
15/// global: pointer patches to locations within this packfile (src,(section,dst)).
16/// finish: offset and typename of all objects for finish functions (src, typename).
17/// exports: named objects (src,name).
18/// imports: named pointer patches outside this packfile (src,name).
19class hkPackfileSectionHeader
20{
21 public:
22
23 HK_DECLARE_NONVIRTUAL_CLASS_ALLOCATOR(HK_MEMORY_CLASS_SERIALIZE, hkPackfileSectionHeader);
24 HK_DECLARE_REFLECTION();
25
26 /// Default constructor
27 hkPackfileSectionHeader()
28 {
29 hkString::memSet(this,0,sizeof(this));
30 }
31
32 void* getStart( void* absoluteStart )
33 {
34 return hkAddByteOffset(absoluteStart, m_absoluteDataStart);
35 }
36
37 /// Size in bytes of data part
38 int getDataSize() const
39 {
40 return m_localFixupsOffset;
41 }
42 /// Size in bytes of intra section pointer patches
43 int getLocalSize() const
44 {
45 return m_globalFixupsOffset - m_localFixupsOffset;
46 }
47 /// Size in bytes of inter section pointer patches
48 int getGlobalSize() const
49 {
50 return m_virtualFixupsOffset - m_globalFixupsOffset;
51 }
52 /// Size in bytes of finishing table.
53 int getFinishSize() const
54 {
55 return m_exportsOffset - m_virtualFixupsOffset;
56 }
57 /// Size in bytes of exports table.
58 int getExportsSize() const
59 {
60 return m_importsOffset - m_exportsOffset;
61 }
62 /// Size in bytes of imports table.
63 int getImportsSize() const
64 {
65 return m_endOffset - m_importsOffset;
66 }
67
68 /// Extract exports from the given section.
69 void getExports( const void* sectionBegin, hkArray<hkResource::Export>& exports ) const;
70
71 /// Extract imports from the given section.
72 void getImports( const void* sectionBegin, hkArray<hkResource::Import>& imports ) const;
73
74 public:
75
76 ///
77 char m_sectionTag[19];
78
79 ///
80 char m_nullByte;
81
82 /// Absolute file offset of where this sections data begins.
83 hkInt32 m_absoluteDataStart;
84
85 /// Offset of local fixups from absoluteDataStart.
86 hkInt32 m_localFixupsOffset;
87
88 /// Offset of global fixups from absoluteDataStart.
89 hkInt32 m_globalFixupsOffset;
90
91 /// Offset of virtual fixups from absoluteDataStart.
92 hkInt32 m_virtualFixupsOffset;
93
94 /// Offset of exports from absoluteDataStart.
95 hkInt32 m_exportsOffset;
96
97 /// Offset of imports from absoluteDataStart.
98 hkInt32 m_importsOffset;
99
100 /// Offset of the end of section. Also the section size.
101 hkInt32 m_endOffset;
102};
103
104#endif // HKSERIALIZE_SERIALIZE_BINARY_HKPACKFILESECTIONHEADER_XML_H
EDIT #2:
I haven't got much time to crack on the format yet, but I got the header to be recognized by the Havok content tools as compatible at least. It won't say "wrong platform for packfile" now, but it will fail to load. Here's the little application I wrote for converting the havok header and the section headers. Now I need to figure out what all these offsets mean. main.cpp 1#include <iostream>
2
3using namespace std;
4
5
6inline void int32_endian_swap(int& x) {
7 x = (x>>24) |
8 ((x<<8) & 0x00FF0000) |
9 ((x>>8) & 0x0000FF00) |
10 (x<<24);
11}
12
13
14
15class hkPackfileHeader_conv {
16 //+version(1)
17 public:
18 /// Default constructor
19 hkPackfileHeader_conv()
20 {
21 m_magic[0] = 0x57e0e057; // both endian agnostic (byte symmetric)
22 m_magic[1] = 0x10c0c010;
23 m_contentsVersion[0] = 0;
24 m_flags = 0; // Set to 1 when a packfile is loaded in-place to signify that the packfile is loaded
25 }
26
27 public:
28
29 /// Magic file identifier. See constructor for values.
30 int m_magic[2];
31
32 /// This is a user settable tag.
33 int m_userTag;
34
35 /// Binary file version. Currently 9
36 int m_fileVersion;
37
38 /// The structure layout rules used by this file.
39 unsigned char m_layoutRules[4];
40
41 /// Number of packfilesections following this header.
42 int m_numSections;
43
44 /// Where the content's data structure is (section and offset within that section).
45 int m_contentsSectionIndex;
46 int m_contentsSectionOffset;
47
48 /// Where the content's class name is (section and offset within that section).
49 int m_contentsClassNameSectionIndex;
50 int m_contentsClassNameSectionOffset;
51
52 /// Future expansion
53 char m_contentsVersion[16];
54
55 ///
56 int m_flags;
57 int m_pad[1];
58
59 void convertToPC() {
60 m_userTag=0;
61 int32_endian_swap(m_fileVersion);
62 m_layoutRules[0] = 4;
63 m_layoutRules[1] = 1;
64 m_layoutRules[2] = 0;
65 m_layoutRules[3] = 1;
66
67 int32_endian_swap(m_numSections);
68 int32_endian_swap(m_contentsSectionIndex);
69 int32_endian_swap(m_contentsSectionOffset);
70
71 int32_endian_swap(m_contentsClassNameSectionIndex);
72 int32_endian_swap(m_contentsClassNameSectionOffset);
73
74 int32_endian_swap(m_flags);
75 }
76
77 void print() {
78 printf("File number sections: %d\n", m_numSections);
79 printf("Content Section Index: %d\n", m_contentsSectionIndex);
80 printf("Content Section Offset: %d\n", m_contentsSectionOffset);
81
82 printf("Content Class Section Index: %d\n", m_contentsClassNameSectionIndex);
83 printf("Content Class Section Offset: %d\n", m_contentsClassNameSectionOffset);
84 }
85};
86
87
88
89/// Packfiles are composed of several sections.
90/// A section contains several areas
91/// | data | local | global | finish | exports | imports |
92/// data: the user usable data.
93/// local: pointer patches within this section (src,dst).
94/// global: pointer patches to locations within this packfile (src,(section,dst)).
95/// finish: offset and typename of all objects for finish functions (src, typename).
96/// exports: named objects (src,name).
97/// imports: named pointer patches outside this packfile (src,name).
98class hkPackfileSectionHeader_conv {
99 public:
100
101 /// Default constructor
102 hkPackfileSectionHeader_conv()
103 {
104
105 }
106
107 /// Size in bytes of data part
108 int getDataSize() const
109 {
110 return m_localFixupsOffset;
111 }
112 /// Size in bytes of intra section pointer patches
113 int getLocalSize() const
114 {
115 return m_globalFixupsOffset - m_localFixupsOffset;
116 }
117 /// Size in bytes of inter section pointer patches
118 int getGlobalSize() const
119 {
120 return m_virtualFixupsOffset - m_globalFixupsOffset;
121 }
122 /// Size in bytes of finishing table.
123 int getFinishSize() const
124 {
125 return m_exportsOffset - m_virtualFixupsOffset;
126 }
127 /// Size in bytes of exports table.
128 int getExportsSize() const
129 {
130 return m_importsOffset - m_exportsOffset;
131 }
132 /// Size in bytes of imports table.
133 int getImportsSize() const
134 {
135 return m_endOffset - m_importsOffset;
136 }
137
138 public:
139
140 ///
141 char m_sectionTag[19];
142
143 ///
144 char m_nullByte;
145
146 /// Absolute file offset of where this sections data begins.
147 int m_absoluteDataStart;
148
149 /// Offset of local fixups from absoluteDataStart.
150 int m_localFixupsOffset;
151
152 /// Offset of global fixups from absoluteDataStart.
153 int m_globalFixupsOffset;
154
155 /// Offset of virtual fixups from absoluteDataStart.
156 int m_virtualFixupsOffset;
157
158 /// Offset of exports from absoluteDataStart.
159 int m_exportsOffset;
160
161 /// Offset of imports from absoluteDataStart.
162 int m_importsOffset;
163
164 /// Offset of the end of section. Also the section size.
165 int m_endOffset;
166
167 void convertToPC() {
168 int32_endian_swap(m_absoluteDataStart);
169 int32_endian_swap(m_localFixupsOffset);
170 int32_endian_swap(m_globalFixupsOffset);
171 int32_endian_swap(m_virtualFixupsOffset);
172 int32_endian_swap(m_exportsOffset);
173 int32_endian_swap(m_importsOffset);
174 int32_endian_swap(m_endOffset);
175 }
176};
177
178
179int main()
180{
181 char *data=NULL;
182
183 FILE *src=fopen("source.hkx", "rb");
184 if (!src) {
185 printf("Couldn't open %s, file doesn't exist.\n", "source.hkx");
186 return 1;
187 }
188
189 FILE *dest=fopen("out.hkx", "wb");
190 if (!dest) {
191 printf("Couldn't create %s for writing, is the directory write-protected?\n", "out.hkx");
192 return 1;
193 }
194
195 fseek(src, 0L, SEEK_END);
196 int sz = ftell(src);
197 fseek(src, 0L, SEEK_SET);
198
199
200 hkPackfileHeader_conv header;
201
202 fread(&header, sizeof(hkPackfileHeader_conv), 1, src);
203 header.convertToPC();
204
205 fwrite(&header, sizeof(hkPackfileHeader_conv), 1, dest);
206
207 for (int i=0; i<header.m_numSections; i++) {
208 hkPackfileSectionHeader_conv section_header;
209 fread(§ion_header, sizeof(hkPackfileSectionHeader_conv), 1, src);
210 section_header.convertToPC();
211
212 fwrite(§ion_header, sizeof(hkPackfileSectionHeader_conv), 1, dest);
213 }
214
215 header.print();
216
217 sz-=sizeof(hkPackfileSectionHeader_conv)*header.m_numSections + sizeof(hkPackfileHeader_conv);
218
219 data=(char *) malloc(sz);
220 fread(data, sz, 1, src);
221
222 fwrite(data, sz, 1, dest);
223
224 fclose(src);
225 fclose(dest);
226
227 return 0;
228}
I forced the layout rules to be 4101 to be recognized as a PC header. It takes a *source.hkx*(the xbox 360 havok file) and writes to out.hkx. I'm gonna try to read the classnames now, but the order seems different than the PC one. And so do some of the int32s before it. Some were affected by endiannes, but others seem totally different values. TranslatorHack 2010, a human translation chain in a.cc. |
|
Ron Novy
Member #6,982
March 2006
|
Arg. I don't seem to have those header files... well I can't seem to find them. A quick look in that shows header files in "Packfile/Binary" which does look interesting though... Haven't had any time to work on it myself though. Still working on this other library. Yes, those values before the class names are probably some kind of ID and not offsets. The differences would probably be due to different versions of the SDK creating the files... That's my guess at least... I'm guessing that if you can find every int16, int32 and int64 that needs swapping and just swap them then it should be ok as long as you change the endianness flag in the header. No need to really find out what anything does until another problem comes up... Not sure if it was mentioned already, but another thing I noticed was that each section was aligned on a 16 byte boundary. The character 0xff was used to fill in any gaps... Anyway, it looks like you've made some progress at least. Have to go again. ---- |
|
Dario ff
Member #10,065
August 2008
|
I gave up on trying to convert the file. Instead, I decided on researching how to import a brand new collision mesh into the game. And I succeeded. \o/ Blessed be the Havok Content tools:  There's still the problematic texture/lighting problems, but at least it's playable. TranslatorHack 2010, a human translation chain in a.cc. |
|
Ron Novy
Member #6,982
March 2006
|
Ha... That's awesome. I liked that quote at the end of the video. [edit] ---- |
|
Dario ff
Member #10,065
August 2008
|
I think the bit depth color theory is actually right for the Global Illumination textures. After struggling with them for a while, I figured out that they need to be uv-flipped as well. But they are big textures which hold several sub-textures most of the time. So I looked at a little binary file next to them called "atlasinfo". Thanks to your previous binary guessing efforts, I learned from that and REd the whole structure. It contained the UV coords of each sub-texture, so I just swapped the U with V in there and walla!  Now the global illumination AND the textures are correctly mapped. The only thing left to figure out so far is the Shadow casting but, the stage shouldn't look this dark. How could I convert the texture to the right color values if it perhaps was orignally 16-bit, and now must be switched to 32-bit? The final color should look as if it was lit by the same intensity as you can see in the rest of the objects. If it's something doable in Photoshop 7 with automated batch, even better. TranslatorHack 2010, a human translation chain in a.cc. |
|
Ron Novy
Member #6,982
March 2006
|
Nice job. Dario ff said: How could I convert the texture to the right color values if it perhaps was originally 16-bit, and now must be switched to 32-bit? Well, if each texture is it's own individual file then it could be easy. You might change the format to 32-bit and multiply each color value by 8 if they are still too dark. The value for the green component might need to be multiplied by 4 if it was actually 565 16-bit color... If each texture is inside a larger file then it might be more difficult. You'd need to figure out the structures of that file and create a new one with the new textures. It could just be some color values near the UV coordinates that need shifting too, but all this is just theory without knowing more about the game files. [edit] [edit2] ---- |
|
|
I hope allegro 5 has dds support so I could write a batch utility.
(And the file is pretty darned big, nearly 1 MB)
Do you think switching the byte at 0x11 could probably make it load up if you modified the header to be compatible with the PC version? 
Well perhaps their reader will only accept little endian on PC, but I wouldn't have problem writing a conversion utility. I'm just a little dumb at trying to figure out file formats.
