ZIP file reader encapsulated in PACKFILE functions
Tobi Vollebregt

A few days ago I downloaded the 4.1.18 WIP. I noticed the packfile functions used vtables since 4.1.18. So immediately I thought finally zip support!

Anyway, I've been writing the attached zip file reader today, and I need some people to test it and I need some feedback, so it would be great if you could just make a little test with it and report back here whether you had any problems.

You need Allegro 4.1.18 and zlib installed.

documentation (also in zipfile.h):

1/* Usage example:
2 * Suppose you have a file archive.zip containing the files
3 * foo and bar, you would read from them as follows:
4 *
5 * PACKFILE *zip, *foo, *bar;
6 *
7 * zip = pack_fopen_zip_dir("archive.zip");
8 *
9 * if (!zip) { show error message }
10 *
11 * foo = pack_fopen_zip(zip, "foo");
12 * bar = pack_fopen_zip(zip, "bar");
13 * pack_fclose(zip);
14 *
15 * if (!foo || !bar) { show error message }
16 *
17 * { read from packfiles foo and bar like
18 * you read from any other packfile }
19 *
20 * pack_fclose(foo);
21 * pack_fclose(bar);
22 */

EDIT: attached new version
EDIT: attached new version

razor

Sweet, I'll try it later, but it sounds really cool.

Peter Hull

I've thought for a while that this was the way to go. Then we won't need the dat utility any more; we can just use (Win)Zip.

Nice one Tobi!

Pete

Avenger

I've seen a few games using zip for compression.. It is really great that we now have an addon to do it:). Thanks a lot!

Elias

Add a complete example, I'm too lazy to try it without one :P Sounds very useful in any case.

Evert

This sounds simply awesome. I'll grab a copy of it and see if I can check it out.

A J

have i missed the point ?
why use compression ?

Richard Phipps

To save space?

Tobi Vollebregt

Thanks!

I'll do some more testing by myself today and then I'll upload it to the depot. I'll also add an example.

And btw, the main reason IMO to use zip for game data is because of easy maintenance, not necessarily compression (you would zip the download anyway). Just make a dir with game data, right click->add to zip and voilĂ , there's your datafile.

Kitty Cat
Quote:

not necessarily compression (you would zip the download anyway)

It's still nice and polite to leave as small of a foot print on disk as reasonably possible.

A J

hard drive space is cheap.
too cheap.. so cheap it is not worth the effort of compressing individual bits.
i no longer see it as useful, rather as a distraction and possible cause of problems. not part of the solution.

Avenger

I would like to use uncompressed zips in my program.. mainly because that would make my game easier to modify.. And it is much better to have one big file instead of thousands of small ones

Dennis
Quote:

it is not worth the effort of compressing individual bits.

an "individual bit" can't be compressed any further:P
Sorry for the OT but this silly joke felt so burning under my nails.;D

razor

It is nice to have all your files in one file. That way you don't flood peoples system with files, I know that I've noticed windows deals with lots of files much slower than a single large file.

Tobi Vollebregt

I've updated the zip attached to my first post. Fixed some unicode handling with trace, added some assertions and an example.c and example.zip

btw, I'm thinking about a way to read the directory entries from the zip file. Now I would like to know, would you think it's useful to know more about a file than name+size (like crc, attributes, compressed size, comment, etc.)? And what API would you prefer?

Like this? Or something else?

int zip_get_num_dir_entries(PACKFILE *f);
const char *zip_get_name(PACKFILE *f, int i);
long zip_get_size(PACKFILE *f, int i);

Elias

It could be interesting to use an API similiar to allegro's directory API:

al_find_first, al_find_next, al_find_close, for_each_file_ex

But then, I'd actually prefer to get the number of entries and then use an index, myself :)

Hm, and this makes me wonder.. haven't some people worked on designs for a simple virtual file system? Seems like a good time to look at such designs now - they already must include an API to list files and get various information about them.

Indeterminatus

That's some nice work you did there! Great job! About the additional information (crc, attributes etc.) I'd say it wouldnt hurt to have them ready ;).

Kitty Cat
Quote:

hard drive space is cheap.
too cheap.. so cheap it is not worth the effort of compressing individual bits.
i no longer see it as useful, rather as a distraction and possible cause of problems. not part of the solution.

I whole heartedly disagree. Hard drive space may be cheaper, but not everyone is able to upgrade as needed. The more games that go by uncompressed, the more hard drive space will be needed, thus forcing the user to upgrade to bigger drives. Cheap or not, it's still $$$ not everyone has or is able to spend. "It's cheap" is a poor excuse because "it's cheap" is still too expensive for some. Do you think I'd be stuck with a Geforce 2 MX video card if I could get a Geforce 4 Ti because "it's cheap"?

Durring development is a different matter of course, but if HD space was such a non-issue, why use PNG, MP3/Ogg, and MPG/AVI/Ogg at all when TGA, WAV, and uncompressed BMPs have been around forever (and are directly supported by Allegro) and serve the same purpose? Now, if, and that's a big if, all your game data is precompressed PNG and Ogg (and even FLAC), then I can see compressed zip files as being unneeded (and potentially hindering), but your average game (especially average Allegro game) has plenty of uncompressed data.

Tobi Vollebregt

The API is now like this:

int zip_get_num_dir_entries(PACKFILE *f);
char *zip_get_name(PACKFILE *f, int i, char *dest);
long zip_get_size(PACKFILE *f, int i);

I'm gonna send a mail to matthew to add it to the resource dir. I think the latest version (only on my HD at this moment) is fully unicode-aware now (even when debugging) and all others things seems to be fine. If you want to know the crc or other file props I'll just let you hack into my sources :P, I don't really feel the need to add it.

Steve Terry
Quote:

an "individual bit" can't be compressed any further

Lzip will prove you wrong!

Richard Phipps

With Quantum Mechanics it could.. :)

Jose Cuervo

Has anyone thought of adding tar.gz support? Because I generally have smaller files with it, like 1/5 smaller.

Thomas Fjellstrom

tar.gz would be more difficult, as the directory format (tar) is compressed globably, so it would have to be completely uncompressed and loaded into memory before anything can be read.

Daniel Schlyder

And if you want better compression, you should use LZMA.

Thomas Fjellstrom

Or if you want a truely fast decompressor, try LZO. On a p1 133 it does about 16MB/s, LZMA does about 8MB/s on a p3 1ghtz...

Gideon Weems
Thomas said:

Or if you want a truely fast decompressor, try LZO

Heh. You know, I've read about that algorithm before, and I only remembered after seeing the pic on the main page. "Space Grade Technology".

But that shouldn't overshadow Zip support; it's awesome. I'm grateful for the people who made it possible. Thanks Tobi, vtable authors, et. al. I'll do some testing.

BAF

We should add this right into allegro, after it is more complete. Then that would make allegro dependent on zlib, but if the license allows it, we could include the needed zlib code right in allegro

Elias
Quote:

We should add this right into allegro, after it is more complete. Then that would make allegro dependent on zlib, but if the license allows it, we could include the needed zlib code right in allegro

Nah, the better route would be to include this into an extra distribution, which contains allegro + some useful addons, all packed up as a single download and easy to install.

CGamesPlay

As far as that goes, what about the licensing of Allegro? zlib isn't restrictive :) It's what I release under.

Matthew Leverton

zlib is more restrictive than Allegro, but only in stuff that doesn't matter. (Ie, don't abuse our generosity vs please don't abuse our generosity.)

Allegro could take zlib and bundle it, but then Allegro's license would have be to be modified to be "Giftware, with some parts under the zlib license".

BAF

I think that may be a good idea myself.

Evert

As Elias said, I think this should be a supported or approved or bundled or gift wrapped or whatever addon, not part of the vanilla Allegro distribution.
Depending on zlib shouldn't be too much of a problem for the bigger package. We'd want PNG support in there anyway.

Elias

Yes.. maybe with 4.2.0, someone will create such a thing. A distribution which has zlib, libpng, loadpng, AllegroGL, glyphkeeper, jpgalleg, whatever else, all bundled together. The make process then also would build the examples and documentation of all the addons. But there still would be no need to move zlib into allegro's CVS repository.. everything would stay nicely modular (except the allegro core itself, which still is quite big :P).

Richard Phipps

Maybe there should be a stripped down Allegro core then since this will get pretty big in terms of file size.

Thomas Fjellstrom

There will always be a vanila allegro distro.

Thread #467091. Printed from Allegro.cc