Allegro.cc - Online Community

Allegro.cc Forums » Programming Questions » Data packer via function calls

This thread is locked; no one can reply to it. rss feed Print
 1   2 
Data packer via function calls
Graham Goring
Member #2,367
May 2002

Okay, I've got to the stage of my project where I really need to package up all the bits and bobs it loads into one lump. Thing is that the list of bits and bobs I need to package up is procedurally generated by the program so I don't want to hand-assemble the package. In addition to this I need them stored in a format whereby fmod can load them and not just Allegro (although streamed sounds like mp3s won't be packaged). Any advice on what route I should go down?

Neil Walker
Member #210
April 2000
avatar

Can't you just use the dat/pack file routines for storing then the same again in collaboration with the magic # for reading? e.g. find_datafile_object() or pack_open() with filename.dat#object_name

Or am I missing the question.

Neil.
MAME Cabinet Blog / AXL LIBRARY (a games framework) / AXL Documentation and Tutorial

wii:0356-1384-6687-2022, kart:3308-4806-6002. XBOX:chucklepie

A J
Member #3,025
December 2002
avatar

you should stick all your 'assets' into a sub folder from your main application, and stop this completely daft idea that everything needs to compressed into a single file.. use the file system to orgainze your data, thats what its for.

___________________________
The more you talk, the more AJ is right. - ML

Neil Walker
Member #210
April 2000
avatar

That's what I do, I use an XML file to reference everything, but Graham asked a question about packing files as he probably has a good reason for doing so (e.g. it stops the casual user from ripping graphics/sound).

Neil.
MAME Cabinet Blog / AXL LIBRARY (a games framework) / AXL Documentation and Tutorial

wii:0356-1384-6687-2022, kart:3308-4806-6002. XBOX:chucklepie

Graham Goring
Member #2,367
May 2002

Okay, here are my options:

Either pack everything (to stop people just ripping off the assets - that's the main reason) or leave everything in the structure but obfuscate the assets by scrambling the pixels of the graphics and descrambling at load time. Alas I can't see a way to do that with the sound, so it kinda' rules out that option.

Btw, Neil, I didn't realise that you could could use the packer as a library, I thought you had to manually build the packfiles via a program. Does it preserve directory structure when you pack stuff?

Neil Walker
Member #210
April 2000
avatar

The manual is your friend. The DatFile routines are for reading and the packfiles are the lower level functions used for reading/writing and what (I think) the DatFile routines use:

//the datfile functions and for setting up your own type
http://www.allegro.cc/manual/api/datafile-routines/

//packfile (the pack_ are probably what you want)
http://www.allegro.cc/manual/api/file-and-compression-routines/

I've never actually used them, but I think they are what you want.

If you're stuck, just go to the tools directory and look at grabber.c

Neil.
MAME Cabinet Blog / AXL LIBRARY (a games framework) / AXL Documentation and Tutorial

wii:0356-1384-6687-2022, kart:3308-4806-6002. XBOX:chucklepie

Kitty Cat
Member #2,815
October 2002
avatar

Quote:

you should stick all your 'assets' into a sub folder from your main application, and stop this completely daft idea that everything needs to compressed into a single file.. use the file system to orgainze your data, thats what its for.

There's something to be said for cleanliness. One file looks a lot nicer than an entire directory tree. Also, a bunch of files in a datafile will usually take up less room than a directory tree with multiple files.

I can already hear you saying "but disk size isn't important these days", and ultimately, I'd say that's wrong. I've had to deal with running out of disk space before, and it's not fun. Every program that doesn't worry about size contributes more to the disk filling up, so the less of this that happens, the better. No matter what size drives you have, they will fill up. And I'm not one that can go affording a new bigger drive everytime that happens.

If size wasn't an issue, then you might as well use all plain .bmp and .wav files. It's a lot more efficient to read them in since you don't need to decompress them (wasting RAM and CPU time).

--
"Do not meddle in the affairs of cats, for they are subtle and will pee on your computer." -- Bruce Graham

Graham Goring
Member #2,367
May 2002

Ta' Neil, looking through them and at first it looked a bit bespoke with the object type stuff but if I treat everything as a binary data chunk I should be okay. I think...

EDIT(AGAIN): Oh, it's just been pointed out to me that fmod can load from memory pointers.

Now I just need to understand the grabber completely so I can pass it a filename and it'll return the index of it within the .dat file's structure. Erk.

Peter Hull
Member #1,136
March 2001

How about using the dat utility via a batch file to rebuild your datafile automatically. You can also set a password which will stop the casual hacker from nicking your stuff.

Pete

Tobias Dammers
Member #2,604
August 2002
avatar

Quote:

If size wasn't an issue, then you might as well use all plain .bmp and .wav files. It's a lot more efficient to read them in since you don't need to decompress them (wasting RAM and CPU time).

Actually, with today's cpus, loading and playing an mp3 is faster than loading and playing wav - the bottleneck here isn't cpu clocks, but disk i/o.

---
Me make music: Triofobie
---
"We need Tobias and his awesome trombone, too." - Johan Halmén

HoHo
Member #4,534
April 2004
avatar

Quote:

the bottleneck here isn't cpu clocks, but disk i/o.

Also even if all the data is loaded in ram it would probably be a bit faster to decode/present if it is as small as possible because the data path between CPU and ram is not exactly big compared to the processing power of the latest and greatest CPU's. The more data fits in L1/2/3 the better, especially on P4 CPU machines (high-end a64 usually has about 1/2 to 1/3 memopry latency of P4's, ~35-50ns vs 100-120ns according to everest home edition).
Sure, I doubt anyone has hundreds of MiB's of sounds to crunch through every second so generally this bottleneck doesn't show up in most scenarios.

[edit]

Fixed the program name.

__________
In theory, there is no difference between theory and practice. But, in practice, there is - Jan L.A. van de Snepscheut
MMORPG's...Many Men Online Role Playing Girls - Radagar
"Is Java REALLY slower? Does STL really bloat your exes? Find out with your friendly host, HoHo, and his benchmarking machine!" - Jakub Wasilewski

A J
Member #3,025
December 2002
avatar

Quote:

will usually take up less room than

like the disk space savings your talking about can even be counted ?

Quote:

And I'm not one that can go affording a new bigger drive everytime that happens.

yeah, cause a 1 gig of HD < $1.. is just too much to spend ::)

Quote:

It's a lot more efficient to read them (BMP/WAV) in since you don't need to decompress them (wasting RAM and CPU time).

disk access is much slower than RAM/CPU, and bus bandwidth is considerably limited compared to CPU cycles. The number of CPU cycles wasted whilst the disk churns is in the millions.

___________________________
The more you talk, the more AJ is right. - ML

Kitty Cat
Member #2,815
October 2002
avatar

Quote:

like the disk space savings your talking about can even be counted ?

Depends on the file system, how many (sub)directories, how many files, the size of the files, etc...

Quote:

yeah, cause a 1 gig of HD < $1.. is just too much to spend ::)

When you have a grand total of $0 to spend, yeah. Lucklly that is not currently the case with me, but as I've learned through life, you can never depend on money being there when you want it. I couldn't even afford this computer when I got it. Luckilly a friend was willing to donate it to me because the parts were "old" (same with my current 80GB HD).

--
"Do not meddle in the affairs of cats, for they are subtle and will pee on your computer." -- Bruce Graham

Jonatan Hedborg
Member #4,886
July 2004
avatar

Internal fragmentation should'nt be too much of an issue, but if you have a few thousand (very) small files, it will add up.

Personally i prefer to have a few datafiles split up acording to content, ie music, graphics and so forth. Also, remember that it's faster to load a single file and jump around in it than it is to open lots of tiny files and read all data from it in one go... The problem with big files however is that they tend to get fragmented on fidk (assuming the average user), which can slow down reading a lot.

Graham Goring
Member #2,367
May 2002

Well, the solution I ended up with is somewhat bizarre. To create my data file I output a list of all the files I need to a batch file (as Peter suggested) inside lines of "dat datapack.dat -a filename.ext -k" and with whatever other flags I need to set, and then I call that with system() and then erase it. Somewhat roundabout but as it's really for the release version of the game it's a small price to pay. The main problem is that as I specify flags with each file I have to execute dat.exe a ton of times and obviously it gets incrementally slower as it adds more and more. Tsk, eh?

Anyhoo, it's all peachy now that it's been pointed out to me that fmod is just as happy loading from memory as from a file.

A J
Member #3,025
December 2002
avatar

do you know about exedat ?

___________________________
The more you talk, the more AJ is right. - ML

ImLeftFooted
Member #3,935
October 2003
avatar

Filesize is extremely important for download size.

Graham Goring
Member #2,367
May 2002

I assume that's the puppy which appends the dat file to the executable? I don't think I'll be taking that approach, thanks.

Jonatan Hedborg
Member #4,886
July 2004
avatar

While that is very true, it is also quite irrelevant for this discussion; When you distribute you program you will most certainly use some manner of compression which nullifies the problem.

EDIT: I was refering to Dustins post

Graham Goring
Member #2,367
May 2002

Yes, heaven forfend that I change the course of the discussion that I started. ;)

Anyhoo, now I have another problem. Generally to use a bitmap from the datapack it seems you just cast the pointer to it's data as a BITMAP * however what if the colour depth doesn't match the current one set up for Allegro? As it's not being loaded then surely it bypasses the conversion and will break?

I only ask as everything in my game is appearing as a big white rectangle. :P

DATAFILE *data_pointer = find_datafile_object(datafile_dat, object_name);
bmps[total_bitmaps_loaded].image = (BITMAP *) data_pointer->dat;

Is what I'm using at the moment. Is there another way?

EDIT: Righty, I've got the graphics displaying now I'm in software mode but the RGB values are scrambled to GBR or something. So is there a way of properly loading a graphics from the PACK file instead of just casting a pointer so it undergoes the proper conversion routine?

EDIT: Oh, I should point out that I'm using AllegroGL at the point where the packfile is loaded.

EDIT: Actually, I should make this a bit clearer, here's the order in which is does things.

1) Spawn a little 32 bit window to select the project
2) Set graphic mode back to text mode
3) Load the packfile
4) Set up an opengl window (set color conversion mode to total and preserve transparency) or a software window at either 16 or 32 bit depending on user preference.
5) fixup_datafile
6) Cast pointers to the bitmaps and upload them as textures.

Now, it should be noted that it seems to work okay in software mode, but if at the point that it turns them into textures, it all goes kerplooie. It seems that fixup_datafile isn't working at all as it doesn't convert the graphics sitting in it to the new colour depth in software mode so if the colour depths at point 1 and 4 don't match the graphics are all stretched. Either way, even if the depths match if I try uploading said textures in hardware graphics AllegroGL mode it doesn't work.

Neil Walker
Member #210
April 2000
avatar

I think set_colour_conversion(COLORCONV_NONE) may be your friend? and maybe chuck in a fixup_datafile just for a laugh.

Neil.
MAME Cabinet Blog / AXL LIBRARY (a games framework) / AXL Documentation and Tutorial

wii:0356-1384-6687-2022, kart:3308-4806-6002. XBOX:chucklepie

Graham Goring
Member #2,367
May 2002

I've noticed that you can't us fixup to change from 16 to 32 bit and vice versa so I'm gonna' split my pack up (ouch!) and only load the graphics pack once the screen mode has been set up properly. Fingers crossed!

EDIT: Well, it's fixed the whole 16/32 bit thing, however when I try turning them into textures they still come out all white. Blecht!

Neil Walker
Member #210
April 2000
avatar

Did the set_colour_conversion(COLORCONV_NONE) not work?

Neil.
MAME Cabinet Blog / AXL LIBRARY (a games framework) / AXL Documentation and Tutorial

wii:0356-1384-6687-2022, kart:3308-4806-6002. XBOX:chucklepie

Peter Hull
Member #1,136
March 2001

What code are you using to upload the textures?

Pete

Graham Goring
Member #2,367
May 2002

allegro_gl_make_masked_texture (BITMAP *);

 1   2 


Go to: