Okay, I've got to the stage of my project where I really need to package up all the bits and bobs it loads into one lump. Thing is that the list of bits and bobs I need to package up is procedurally generated by the program so I don't want to hand-assemble the package. In addition to this I need them stored in a format whereby fmod can load them and not just Allegro (although streamed sounds like mp3s won't be packaged). Any advice on what route I should go down?

Can't you just use the dat/pack file routines for storing then the same again in collaboration with the magic # for reading? e.g. find_datafile_object() or pack_open() with filename.dat#object_name

Or am I missing the question.

you should stick all your 'assets' into a sub folder from your main application, and stop this completely daft idea that everything needs to compressed into a single file.. use the file system to orgainze your data, thats what its for.

That's what I do, I use an XML file to reference everything, but Graham asked a question about packing files as he probably has a good reason for doing so (e.g. it stops the casual user from ripping graphics/sound).

Okay, here are my options:

Either pack everything (to stop people just ripping off the assets - that's the main reason) or leave everything in the structure but obfuscate the assets by scrambling the pixels of the graphics and descrambling at load time. Alas I can't see a way to do that with the sound, so it kinda' rules out that option.

Btw, Neil, I didn't realise that you could could use the packer as a library, I thought you had to manually build the packfiles via a program. Does it preserve directory structure when you pack stuff?

The manual is your friend. The DatFile routines are for reading and the packfiles are the lower level functions used for reading/writing and what (I think) the DatFile routines use:

//the datfile functions and for setting up your own type
http://www.allegro.cc/manual/api/datafile-routines/

//packfile (the pack_ are probably what you want)
http://www.allegro.cc/manual/api/file-and-compression-routines/

I've never actually used them, but I think they are what you want.

If you're stuck, just go to the tools directory and look at grabber.c

There's something to be said for cleanliness. One file looks a lot nicer than an entire directory tree. Also, a bunch of files in a datafile will usually take up less room than a directory tree with multiple files.

I can already hear you saying "but disk size isn't important these days", and ultimately, I'd say that's wrong. I've had to deal with running out of disk space before, and it's not fun. Every program that doesn't worry about size contributes more to the disk filling up, so the less of this that happens, the better. No matter what size drives you have, they will fill up. And I'm not one that can go affording a new bigger drive everytime that happens.

If size wasn't an issue, then you might as well use all plain .bmp and .wav files. It's a lot more efficient to read them in since you don't need to decompress them (wasting RAM and CPU time).

Ta' Neil, looking through them and at first it looked a bit bespoke with the object type stuff but if I treat everything as a binary data chunk I should be okay. I think...

EDIT(AGAIN): Oh, it's just been pointed out to me that fmod can load from memory pointers.

Now I just need to understand the grabber completely so I can pass it a filename and it'll return the index of it within the .dat file's structure. Erk.

How about using the dat utility via a batch file to rebuild your datafile automatically. You can also set a password which will stop the casual hacker from nicking your stuff.

Pete

Actually, with today's cpus, loading and playing an mp3 is faster than loading and playing wav - the bottleneck here isn't cpu clocks, but disk i/o.

Also even if all the data is loaded in ram it would probably be a bit faster to decode/present if it is as small as possible because the data path between CPU and ram is not exactly big compared to the processing power of the latest and greatest CPU's. The more data fits in L1/2/3 the better, especially on P4 CPU machines (high-end a64 usually has about 1/2 to 1/3 memopry latency of P4's, ~35-50ns vs 100-120ns according to everest home edition).
Sure, I doubt anyone has hundreds of MiB's of sounds to crunch through every second so generally this bottleneck doesn't show up in most scenarios.

[edit]

Fixed the program name.

like the disk space savings your talking about can even be counted ?

yeah, cause a 1 gig of HD < $1.. is just too much to spend

disk access is much slower than RAM/CPU, and bus bandwidth is considerably limited compared to CPU cycles. The number of CPU cycles wasted whilst the disk churns is in the millions.

Depends on the file system, how many (sub)directories, how many files, the size of the files, etc...

When you have a grand total of $0 to spend, yeah. Lucklly that is not currently the case with me, but as I've learned through life, you can never depend on money being there when you want it. I couldn't even afford this computer when I got it. Luckilly a friend was willing to donate it to me because the parts were "old" (same with my current 80GB HD).

Internal fragmentation should'nt be too much of an issue, but if you have a few thousand (very) small files, it will add up.

Personally i prefer to have a few datafiles split up acording to content, ie music, graphics and so forth. Also, remember that it's faster to load a single file and jump around in it than it is to open lots of tiny files and read all data from it in one go... The problem with big files however is that they tend to get fragmented on fidk (assuming the average user), which can slow down reading a lot.

Well, the solution I ended up with is somewhat bizarre. To create my data file I output a list of all the files I need to a batch file (as Peter suggested) inside lines of "dat datapack.dat -a filename.ext -k" and with whatever other flags I need to set, and then I call that with system() and then erase it. Somewhat roundabout but as it's really for the release version of the game it's a small price to pay. The main problem is that as I specify flags with each file I have to execute dat.exe a ton of times and obviously it gets incrementally slower as it adds more and more. Tsk, eh?

Anyhoo, it's all peachy now that it's been pointed out to me that fmod is just as happy loading from memory as from a file.

do you know about exedat ?

Filesize is extremely important for download size.

I assume that's the puppy which appends the dat file to the executable? I don't think I'll be taking that approach, thanks.

While that is very true, it is also quite irrelevant for this discussion; When you distribute you program you will most certainly use some manner of compression which nullifies the problem.

EDIT: I was refering to Dustins post

Yes, heaven forfend that I change the course of the discussion that I started.

Anyhoo, now I have another problem. Generally to use a bitmap from the datapack it seems you just cast the pointer to it's data as a BITMAP * however what if the colour depth doesn't match the current one set up for Allegro? As it's not being loaded then surely it bypasses the conversion and will break?

I only ask as everything in my game is appearing as a big white rectangle.

DATAFILE *data_pointer = find_datafile_object(datafile_dat, object_name);
bmps[total_bitmaps_loaded].image = (BITMAP *) data_pointer->dat;

Is what I'm using at the moment. Is there another way?

EDIT: Righty, I've got the graphics displaying now I'm in software mode but the RGB values are scrambled to GBR or something. So is there a way of properly loading a graphics from the PACK file instead of just casting a pointer so it undergoes the proper conversion routine?

EDIT: Oh, I should point out that I'm using AllegroGL at the point where the packfile is loaded.

EDIT: Actually, I should make this a bit clearer, here's the order in which is does things.

1) Spawn a little 32 bit window to select the project
2) Set graphic mode back to text mode
3) Load the packfile
4) Set up an opengl window (set color conversion mode to total and preserve transparency) or a software window at either 16 or 32 bit depending on user preference.
5) fixup_datafile
6) Cast pointers to the bitmaps and upload them as textures.

Now, it should be noted that it seems to work okay in software mode, but if at the point that it turns them into textures, it all goes kerplooie. It seems that fixup_datafile isn't working at all as it doesn't convert the graphics sitting in it to the new colour depth in software mode so if the colour depths at point 1 and 4 don't match the graphics are all stretched. Either way, even if the depths match if I try uploading said textures in hardware graphics AllegroGL mode it doesn't work.

I think set_colour_conversion(COLORCONV_NONE) may be your friend? and maybe chuck in a fixup_datafile just for a laugh.

I've noticed that you can't us fixup to change from 16 to 32 bit and vice versa so I'm gonna' split my pack up (ouch!) and only load the graphics pack once the screen mode has been set up properly. Fingers crossed!

EDIT: Well, it's fixed the whole 16/32 bit thing, however when I try turning them into textures they still come out all white. Blecht!

Did the set_colour_conversion(COLORCONV_NONE) not work?

What code are you using to upload the textures?

Pete

allegro_gl_make_masked_texture (BITMAP *);

Feel free to send me a $10 hd then, and let me know where to send the money

Eep! Ignore me, the texture upload thing wasn't working because I'm an absolute bloody moron. So, the usual, then.