Allegro.cc - Online Community

Allegro.cc Forums » Game Design & Concepts » Tilemap Storage and Implementation

This thread is locked; no one can reply to it. rss feed Print
Tilemap Storage and Implementation
Anomie
Member #9,403
January 2008
avatar

I always used to store my tilemaps as bitmaps, reading pixels for tiles...but I've just finished working on a way to store tilemaps as binary files! This means that my 1000px * 1000px bitmap (around 2.86 MB) is replaced by a binary file storing 1,000,000 tiles (116 KB! (and drops to 18 KB when compressed! (that's ~84.8% compression! (and the compression only gets better as the file grows!)))). This wouldn't mean much to me if I couldn't use a map that size in a game, and so I also redid the way I render a tilemap, giving me a huge performance boost, totally independent of the map size (which had been my limiter before).

All these crazy new improvements got me excited! And so I thought I'd ask if anyone knows of a way to store a bigger tilemap in a smaller space, or render a 640*480 screen full of 32*32 tiles faster than my method does.

Anyone?

______________
I am the Lorax. I speak for the trees. I speak for the trees for the trees have no tongues.

Indeterminatus
Member #737
November 2000
avatar

Well, you didn't really tell us how you do it, so ... I don't know. :P

_______________________________
Indeterminatus. [Atomic Butcher]
si tacuisses, philosophus mansisses

Onewing
Member #6,152
August 2005
avatar

Quote:

faster than my method does.

You didn't really specify your method. ;)

Anyway, using a binary file is pretty standard for simple tilemap storage, so seeing all these exclamations is kind of funny ("look how much better my art is now that I color inside the lines!").

The best way to render a tilemap is to just worry about the tiles that are seen. If you spending any time on a tile that's outside what you seen on the screen, then you are wasting time.

As for logic, no one really cares if a monster is has collision with a wall 3-hour's walking distance away from where you currently are. That's a simulation, not a game. However, you don't want to just stop logic with anything that's not on the screen, otherwise as soon as something walks off the screen, it'd get frozen and one step towards it and there's where it be. For logic, I'd still move things that are at least an entire screen away, but this is an arbitrary value, it can adjusted to whatever you feel is necessary.

------------
Solo-Games.org | My Tech Blog: The Digital Helm

Anomie
Member #9,403
January 2008
avatar

Quote:

Anyway, using a binary file is pretty standard for simple tilemap storage, so seeing all these exclamations is kind of funny ("look how much better my art is now that I color inside the lines!").

Yeah...but I'm the kind of guy who gets real giddy when some new information clicks. I've been told I was grinning and wringing my hands when I first read about classes and inheritance and polymorphism. (also, I'm a sucker for tiny file sizes, having had dial-up until a few years ago)

By the way, with the storage stuff, I'm getting super-crazy-amazing compression when the binary file gets big. A 5000 tile by 5000 tile map is ~3MB, and is compressed down to 90K when I stick it in a .rar...is that normal with raw binary files?

The old way that I was rendering the tilemap was to check each tile to see if it was within view. I'd done a lot of maps that way, and never noticed an issue. (even though it may seem totally ridiculous now)

But my new way gets ~460fps at 640 by 480 on my old clunker-box!

(also, a 5000 tile by 5000 tile map takes up 512MB of RAM on it's own. Any suggestions on having tiles take up less space?)

[edit] Sorry... 'my new way' = Finding the tile that (cam_x, cam_y) is over (this would be the top left corner of the 'viewport' (or whatever). Then just go through and draw the next n tiles in the vector (where n = SCREEN_W/TILESIZE)... When that row's done, drop down and draw the next one the same way, until the distance down from the first tile = SCREEN_H/TILESIZE.

[edit] Here's the function:

void draw_map(BITMAP *bmp)
{
  short _x = cam_x/TILESIZE;
  short _y = cam_y/TILESIZE;

  if(_y <= 0) _y = 1;
  if(_x <= 0) _x = 1;

  for(short ry = 0 ; ry < SCREEN_H/TILESIZE + TILESIZE ; ry++)
    for(short rx = 0 ; rx < SCREEN_W/TILESIZE + TILESIZE ; rx++)
      vTile[(MAPWIDTH * (_y+ry)) - (MAPWIDTH - (_x+rx)) - 1]->draw(bmp);

}

That '(MAPWIDTH * (_y+ry)) - (MAPWIDTH - (_x+rx))' is the juicy little jewel that I sat and worked out myself...that's the bit that makes me giddy. (that, and finally realizing that all I needed to save for any of my first-layer-tiles was a single char for 'type')

______________
I am the Lorax. I speak for the trees. I speak for the trees for the trees have no tongues.

james_lohr
Member #1,947
February 2002

Quote:

5000 tile by 5000

5000x5000 at 2 bytes per tile is 50mb, and with ordinary sized tiles that is like 20k+ screens which is a bit ridiculous. Also, you can quite easily use RLE for tiles if you have a lot of repetition, or macro tile sets or something.

If you really really need massive and fully customizable maps, then you can calculate the base tiles using a clever function and then only store the modifications (difference) to this base using quad tree data structure.

[edit] To elaborate on this: your function could define a generic landscape. You then modify this landscape to exactly what you want and store just these modifications. The actual value for a tile is given by tile_value = generic_function(x,y) + modification(x,y), where the modification could be implemented as any number of data structures (hash for example).

There are loads of tricks used in image processing and compression that extend to this.

Thomas Harte
Member #33
April 2000
avatar

Quote:

Sorry... 'my new way' = Finding the tile that (cam_x, cam_y) is over (this would be the top left corner of the 'viewport' (or whatever). Then just go through and draw the next n tiles in the vector (where n = SCREEN_W/TILESIZE)... When that row's done, drop down and draw the next one the same way, until the distance down from the first tile = SCREEN_H/TILESIZE.

I think that's the 'normal' way of doing it. A lot of the old tilemap hardware (i.e. the NES, Master System, etc) has a video buffer to which you can only store just enough tiles to cover the screen. Then you have to shuffle contents in from your real tilemap as the game progresses to show other bits of the level.

Anomie
Member #9,403
January 2008
avatar

Quote:

5000x5000 at 2 bytes per tile is 50mb

Whuh? Aren't chars a single byte? And...why is that ~23MB fitting into ~3MB, which then will compress into 90KB? (not that I'm complaining, mind you...)

And...5000 tiles by 5000 tiles is only...250 screen widths! So ha. I don't ever plan to use a map that size, but whenever I'm working on something that will work with other 'modules', I like to make sure that it runs smoothly under several times the load it will actually carry, to (attempt to) make sure that it will leave enough resources to run whatever it's going to be combined with. (collision, random AI, whatever)

[edit]

Quote:

I think that's the 'normal' way of doing it.

Yeah, sorry if I was making it seem like 'Hey, world, check out this new amazing way of rendering tiles!'. I just meant to say that thinking through all this stuff on my own really got me excited about it, and I'd love to see if someone can show me another step to take.

______________
I am the Lorax. I speak for the trees. I speak for the trees for the trees have no tongues.

Thomas Harte
Member #33
April 2000
avatar

Quote:

Yeah, sorry if I was making it seem like 'Hey, world, check out this new amazing way of rendering tiles!'. I just meant to say that thinking through all this stuff on my own really got me excited about it, and I'd love to see if someone can show me another step to take.

Oh, I didn't mean it like that! I meant that you've independently figured out the fastest method. Which is obviously a very good thing.

james_lohr
Member #1,947
February 2002

Quote:

Whuh? Aren't chars a single byte? And...why is that ~23MB fitting into ~3MB, which then will compress into 90KB?

Because of redundancy in your data. If you have 5000x5000 tiles that are all 0, then that'll compress down to nothing. Try losslessly compressing 5000x5000 of equally distributed noise with an even frequency histogram and you'll get bugger all compression.

Quote:

250 screen widths! So ha.

250x100 = 20k + screens. So ha. :P

Anomie
Member #9,403
January 2008
avatar

Quote:

250x100 = 20k + screens.

Aw maaan...

So I guess what I need to work on at the moment is the size of my map files, and the amount of RAM it takes to store a tile... I'm real happy with storing a single char for each first-layer tile, but that only works because there has to be a tile in every space. In higher-level tiles, I'll have to store the position, too...

For some reason, I was under the impression that a bool was a single bit... My idea for storing position, then, was to have a maximum map size of 500x500 and use two chars for position, rather than two shorts. Then I'd have a bool right after each char, and if it's set to true, add 255 to the char to get the real position...that'd have me storing...3 bytes + 2 bits per tile, rather than 5 bytes...which, I guess, wouldn't help too much, even if I could store a single bit...plus, doesn't the OS round bits up into bytes for storage?

But the RAM usage bothers me more than the file size. ~500MB isn't cool, man...and I'm going to end up having more information in the tile class, too... Is the RAM used equal to the size of all the variables belonging to the object? Or is there more?

______________
I am the Lorax. I speak for the trees. I speak for the trees for the trees have no tongues.

james_lohr
Member #1,947
February 2002

You're not making much sense. Even on a game with massive landscapes and interactive and varied tiles, I can't think of any reason why you'd need more than a few 10s of MB for tile data (not including textures).

Anomie
Member #9,403
January 2008
avatar

Quote:

You're not making much sense.

I'm going to assume you're talking about my effort to squeeze an extra 14 bits of storage space out of my tiles... Yeah...a little crazy... But like I said, I have a thing for tiny files. That's why I've generally avoided tilemap stuff - they usually require a lot of media to be included with a program. I'd feel terrible releasing anything more hefty than Chrono Trigger (excluding audio...I'd end up releasing a 4MB version with MIDIs, and a 498MB version, with totally uncompressed wavs...I'm a total nutjob).

______________
I am the Lorax. I speak for the trees. I speak for the trees for the trees have no tongues.

Paul Pridham
Member #250
April 2000
avatar

Also keep in mind that if you have a screen full of empty tiles, you might as well not store that screen to disk. Huge maps with lots of jumping/flying/falling etc. space are nicely optimized by storing maps in a per-screen format.

Anomie
Member #9,403
January 2008
avatar

Yeah, it seems like there was a lot of empty space in the Rockman games... But I'm planning this for use with a console-style RPG.

______________
I am the Lorax. I speak for the trees. I speak for the trees for the trees have no tongues.

Ben Delacob
Member #6,141
August 2005
avatar

If you haven't made a game before, consider writing a small arcade-sized game to work up your skill. The experience will help you a lot on a larger project.

Use UPX on your final release if you want to compress your program.

__________________________________
Allegro html mockup code thread -website-
"two to the fighting eighth power"

Anomie
Member #9,403
January 2008
avatar

Yeah, I just recently got UPX, and it's awesome. Killer for someone like me.

I've done a couple little games, and lots and lots of small programs that do a single interesting or useful thing... Kinda writing example code for myself as I learn...

I think the only wall I've got in front of me now is scripting. Totally uncharted territory for me.

On a lighter note: I'm not planning on making a whole game (of this size) by myself... My plan is to develop the 'engine' as much as I can without media (this 'engine' includes external tools, like a map editor, or an NPC editor), and then try and get someone (a couple people?) willing to do sprites/tiles, before I try and make a game - which I see as totally separate from what I'm doing now. In my case, making the game should just involve scripting, a bit of imagination and maybe tweaking the technology, if I do a crap job the first time 'round.

[P.S.]Ha! I thought of a better way to store higher-layer tiles! Since there's no reason to store them in their order on the grid, I can store them by type. For the higher layers I can store tiles like this: type1:xyxyxyxyxyxyxyxyxyxyxyxyxyxyxy, type4:xyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxy, etc. Awesome.

______________
I am the Lorax. I speak for the trees. I speak for the trees for the trees have no tongues.

Albin Engström
Member #8,110
December 2006
avatar

Anomie said:

In higher-level tiles, I'll have to store the position, too...

Why do you have to do that? You don't have to store the position of tiles.

for example:
You can just store the dimensions of the map, x y and z.
And then read the map as it would have been saved with those dimensions.

The only reason you would not want to store the tiles without positions would be if you where certain a certain amount of tiles could be "defaulted", for example, if every tile uses 5 bytes, it would be wise to store the position (4 bytes) only if 80% of the tiles isn't used, which isn't very likely. If you really need to default certain tiles you could store a byte which represents how many tiles is to be defaulted after it, if the number actually exceeds 255 you could simply store that tile as what a default tile represents.

Audric
Member #907
January 2001

I think he referred to layers where there's a lot of blank space. In this case, yes it can take less space. However, good luck checking for collisions. (and editing levels)

Thomas Harte
Member #33
April 2000
avatar

The only time I every wrote anything that used a tilemap (other than emulators of tile based systems), I made the moving platforms out of their own mini-tilemaps. Collision detection was easy — just use the relative offsets of the two to figure out where the player is in the platforms' tilemap and do the usual stuff. I would guess that breaking apart a sparse tilemap so that it is actually a series of unconnected tiles or tilemaps would be much the same.

Anomie
Member #9,403
January 2008
avatar

Quote:

Why do you have to do that? You don't have to store the position of tiles.

Quote:

layers where there's a lot of blank space.

Yeah... That does bring up a good point though, Audric...every time I've needed to do something based on the position of the tile, I've taken advantage of the fact that my map is filled left-to-right, and top-to-bottom - and the fact that every slot is filled. I definitely don't think it's a good idea to fill the tile vector up with NULL tiles, either... I've been thinking about writing a function like tile_at(x, y, layer) that would just return the number (to be used with my vector) of the tile at that position on the specified layer. It'd be easy for first layer, but again, slow for any higher layers, unless I fill the empty spaces with filler tiles...

Quote:

I would guess that breaking apart a sparse tilemap so that it is actually a series of unconnected tiles or tilemaps would be much the same.

I'm not sure it's going to be that sparse...like...the second layer would be for things like furniture, walls, hills, tree trunks... In a town or village, I could see the higher layers being 30-40% filled.

______________
I am the Lorax. I speak for the trees. I speak for the trees for the trees have no tongues.

SiegeLord
Member #7,827
October 2006
avatar

In that case, just remove the concept of map size layers altogether and have each grid square have a vector of tiles that it has. I.e. grid square at (5,5) might just have a grass tile. The grid square at (6,6) might have a grass tile, then a tree tile etc.

1SGridSq
2{
3 vector<STile*> tiles;
4};
5...
6SGridSq theMap[mapWidth * mapHeight];
7...
8GetTileAt(int x, int y, int z)
9{
10 /*
11 Naturally, check the bounds first
12 */
13 SGridSq* square = theMap[x + y * mapWidth];
14 if(z >= 0 && z < square->tiles.size())
15 {
16 return square->tiles[z];
17 }
18 else
19 return 0;
20}

EDIT: You still have to hold the NULL tiles in each SGridSq if the tiles are not present at every layer at that position, but this should be much less severe than before.

"For in much wisdom is much grief: and he that increases knowledge increases sorrow."-Ecclesiastes 1:18
[SiegeLord's Abode][Codes]:[DAllegro5]:[RustAllegro]

Anomie
Member #9,403
January 2008
avatar

Doing things that way sounds like it'd work...but then I'd need some separator between tiles, right? That's another char for every tile. That kinda worries me... I guess it'd still be smaller than making whole new layers that're 60% NULL tiles in my vector. Would there be any problems (extra RAM usage?) with having all those different vectors? As many as 1,000,000 per map?

______________
I am the Lorax. I speak for the trees. I speak for the trees for the trees have no tongues.

SiegeLord
Member #7,827
October 2006
avatar

On my system, the vector<>'s size is 12 bytes. Assuming filled layers, a 1000x1000 map, and a byte to store the tile index that is 13 Mb to describe the entire map in RAM. If you have 10 fully filled layers, that will be 22Mb. When stored on disk, this shall be even less, since some sort of compression algorithm will reduce that even further... I do not see your concern.

EDIT: To clarify, the overhead for this size of the map is 12 Mb. However, if we assume that all layers above the 1st layer are only half filled, we get that using this vector based approach takes roughly 17.5 Mb. If we were to use map size layers, this would take 10 Mb, which is actually better... so I really don't see your concern anymore... just store your map as a bunch of layers with the tile 0 being the empty tile. I have 3Gb of RAM here... 10 Mb here or there is insignificant. You are making a tempest in a teacup about this, go make the rest of your game.

"For in much wisdom is much grief: and he that increases knowledge increases sorrow."-Ecclesiastes 1:18
[SiegeLord's Abode][Codes]:[DAllegro5]:[RustAllegro]

Tobias Dammers
Member #2,604
August 2002
avatar

Quote:

But my new way gets ~460fps at 640 by 480 on my old clunker-box!

No you don't. No current hardware can render 460 frames per second. In case it's really an old clunker box, the screen's vertical retrace rate is probably set to something in the <100Hz range, so that's the maximum number of frames your game will ever produce, no matter what you do.

About the file compressing so well: That doesn't surprise me too much, considering you probably used a sample map with mostly empty tiles (or at least, large regions with the same value). Since RAR uses some kind of dictionary algorithm, this kind of obviously redundant data compresses extremely well. Just test it on bitmaps: Make one .bmp file that is completely white (or any other color you fancy, say, pink), and another one that is filled with random color noise. Compare the compressed results and you'll see what I mean.

About the general format of your tilemaps: If you REALLY need such large maps, then you should consider using a more complex format (e.g. tilemaps where there are things to happen, leaving empty space between them; or work with "rooms" and portals connecting them), but for anything normal, just use a flat tilemap.

In the map file, a char per tile is enough if your map doesn't contain more than 255 different tiles; if the game features more possible tiles, but you don't use them all, you can use a tile list. This is basically what I'm using, since my actual tiles are polymorphic, and some feature procedural graphics; the file starts with a tile type list, after that comes the tile map which contains indices into the tile type list rather than actual tile values. This approach is also more flexible if you add more tiles later, or change your graphics, since sprites can be referenced by name without wasting much space (you only state the name once, no matter how often you use the tile).

---
Me make music: Triofobie
---
"We need Tobias and his awesome trombone, too." - Johan Halmén

Go to: