I'm sitting here doing some more work on isometrics, which is still relatively new to me. I've done 3D and lots of 2D, just never Isometrics. As long as I can recall I have always read you need a mouse map when it comes to selecting a tile with your mouse so you know which tile you are clicking on. Well, I found some nice code on selecting isometric tiles without using a mouse map so after some of my own adjustments to it, I thought I would share what I have here.

   // get the map x & y positions
   // Note: with these calculations you don't need a colour mouse map!
   tile_mx = (int)((((float)mouse_y + (float)scroll_y) / (float)TILE_Y) + (((float)mouse_x +
             (float)scroll_x) - ((float)MAPSIZE * (float)TILE_X/2)) / (float)TILE_X);
   tile_my = (int)((((float)mouse_y + (float)scroll_y) / (float)TILE_Y) - (((float)mouse_x +
             (float)scroll_x) - ((float)MAPSIZE * (float)TILE_X/2)) / (float)TILE_X);

I have a lot of (float)'s in there because it is very important that the calculation's floating point precision is retained before it is finally converted to an int.

Here's an explanation of the variables:
mouse_x & mouse_y: allegro variables containing the mouse position on screen.
scroll_x & scroll_y: how much the map has been scrolled, this is for maps larger than the screen can view.
TILE_X & TILE_Y: The size of the tiles (not nessecarily the size of the actual bitmaps). This is the size of teh diamond part of the tile, my bitmaps are 64x64, but the diamond part is 64 x 32, so this contains 64 & 32 respectively. (actually they're 64x31, but are drawn every 32 in the y direction)
MAPSIZE: The size of the map, if the map is 10x10, this will contain 10 for example.

And a screenshot of my editor as it is so far (the hilltops are brown because I plan to add in more layers later on, not added yet):
{"name":"604023","src":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/6\/e\/6ed19fcd9bca812abdb45620adf8effc.png","w":800,"h":600,"tn":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/6\/e\/6ed19fcd9bca812abdb45620adf8effc"}

And for completeness sake, here's code for plotting your tiles on screen:

screen_x = (iso_x - iso_y + MAPSIZE) * (TILE_X / 2);
screen_y = (iso_x + iso_y) * (TILE_Y / 2);

iso_x & iso_y being the isometric map co-ordinates, top most position being 0,0 for example.

if you have a large scrolling map, I would add scroll_x and scroll_y to screen_x and screen_y respectively.

Interesting. Does the selection works fine on all the layers ? (I mean also on tile that are higher than the basic floor)

I haven't added layers yet, but I can't see why not, you just need to adjust for the vertical offset, should be simple enough.

WHat I will probably do is just check for where on the grid you have clicked (on the base layer), then check the height of that layer and simply select the top most tile.

For my layering I plan to have each tile have a height, rather that separate maps for each layer, this will make it more flexible with less wasted space. Something like:

map[y][x].height

Only the top most tile will matter the way I am doing it. If you need to have separate tiles on the same location at differing heights I would just add a linked list with all the tiles at that location.

Edit:

Of course, for editing layers, I would probably have an option to only show one specific layer, or hide all layers above a certain point, especially if you have layers sandwiched between others.

Edit2:

An additional note about my editor. I started out using PNGs with Allegro 4.4 and man alive were they slow to draw! At least with transparency. I ended up switching back to magic pink BMPs. The frame rate with the screen full of map was 9FPS with PNGs, as you can see in t his screenshot it's well over 200... I guess I'll save using PNGs for allegro 5.

I'm not really sure how to explain it, but your solution won't work when you add variable heights to your tiles because you can't know in advance what the height of the tile will be.

For example, a tile in (x,y,z) = (0,0,0) and a tile in (x,y,z)=(1,1,2) should be drawn in exactly the same space on the screen, the same for (2,2,4) and (3,3,6). (it might be different numbers depending on how exactly your solution works, but nevermind that.)
When you point on that 2d location on the screen, you have no way of knowing if the tile in (0,0) is drawn there, or if it is a tile in (1,1) that is 2 high, or a tile that is in (2,2) and 4 high etc.

EDIT: Yeah, alpha blending in A4 is incredibly slow. Just switch to A5

Sure it will work, it is working for me now. I select that tile, get the height of it, adjust the selector image I draw to indicate which tile I am pointing at and voila. This is a selection issue, the tiles all still have the same base location, when you select them, you're selecting that grid location no matter what the height is. getting the height of the tile and adjusting the actual drawing of the selector on it is trivial.

I'll just have to add layers in (which I am planning on anyhow) and post the program when done.

Edit: At this time my map is scroll-able as well, in the image above it is a 50x50 map, fairly large with 64x32 tiles. A layer above is no different than scrolling up whatever the offset is. And currently I scroll all over and have no problems selecting a tile, I just add in the offset to the calculation for the scrolling. It's no different for layers, just add in a vertical offset for the layer's height and select away. How else would you select these tiles with a colour map? You still need to account for a vertical offset before you can select a grid location and then use the colour tile to see how much you need to adjust for, there is no difference, you need to offset for the layer then use the math to get the co-ordinates, you're just replacing a colour image with calculations.

Edit2: I think I see what you are talking about though. Selecting a high layer's tile just by pointing at the map could be difficult, but it is the same no matter what method you use, colour map or the above calculations. If you use a colour map, you'll still end up pointing at a grid location of say a tile higher than what you wish, how do you determine the layer you're pointing at seems to me to be the same problem for both methods. All I can think of is checking the layers for the surrounding tiles, but as I said, this is a problem to consider no matter which method you use.

Lets put it like this, I have a tile in (0,0,0), it is drawn on the screen in coordinate (300,0).
I also have a tile in (1,1,0), it is drawn on the screen in coordinate (300,32).
I also have a tile in (2,2,0), it is drawn on the screen in coordinate (300,64).
I also have a tile in (3,3,6), it is drawn on the screen in coordinate (300,0).
I also have a tile in (4,4,0), it is drawn on the screen in coordinate (300,128).
I also have a tile in (5,5,6), it is drawn on the screen in coordinate (300,128).
I also have a tile in (6,6,6), it is drawn on the screen in coordinate (300,192).
I also have a tile in (7,7,6), it is drawn on the screen in coordinate (300,256).
I also have a tile in (8,8,6), it is drawn on the screen in coordinate (300,288).
I also have a tile in (9,9,6), it is drawn on the screen in coordinate (300,320).

What happens when I click (300,1), do I get the tile in (0,0,0) or the tile in (3,3,6)?
What happens when I click (300,96)? Which tile do I get?
What happens when I click (300,320)? Which tile do I get? Keep in mind that according to what you just said you would get the tile in (8,8,6).

I see that problem, but I don't see how using a coloured tile makes any difference, the problem is the same, and I haven't tackled it yet. In each case you need to determine if what you're pointing at is a higher level or not. Of course, you could simply edit by layer which is what I had in mind as well. IN either case, you have the same problem as with colour tiles or the calculations.

I am new to isometrics so I haven't dealt with layers much yet, but this is an interesting discussion and I thank you for bring it up, I'll look into this and see what I can find. I was thinking about checking surrounding tiles to see what their heights were though, I guess you would only have to check the tiles immediately south of where you are pointing at. I wonder if you kept a separate index of tiles that are drawn on a certain point? hmmm.....

Note: on my map a tile would never go from being at a height of 0 on one tile to being at a height of 6 on the next one down. Tiles would only be one layer difference at most the way I am doing things, but still, it's an interesting problem no matter how you do selecting.

Here's how it works:
Lets assume I have a tile (X,Y,Z), and lets also assume the because of its height it has been drawn to screen coordinates (A,B). If it had a different height it might have been drawn to (A,B-32) or (A,B+32), so I can't really tell that this is the tile I want. (so far I think you agree with me.)
Now, what I do is assign each tile a unique color using a simple formula, for example lets say our tile (X,Y,Z) received the color (r,g,b)=(X,Y,0).
Now, as I draw the tiles to the screen, I also draw their color counterparts to a side bitmap. Since I'm using the same drawing process, if I drew our tile into (A,B) on the screen, it should also be on (A,B) on that bitmap.
This means that if the user had clicked (A,B), I can check what color (A,B) is on the bitmap and I will see that it is colored (X,Y,0), from that I can deduce that it belongs to the tile in (X,Y,Z), so I know exactly which tile I chose and I am now happy
Using this method also helps solve the case where two tiles are drawn to the same coordinates (one is hiding the other), because your drawing algorithm should already take care of that and make sure that the closer of the two tiles is the one that is visible (and is therefore that one that will be drawn both to the screen and to the bitmap), and now you are twice as happy

I'm curious how games like Transport Tycoon and Simcity 2000 solved this? You can have some pretty high hills on both these games and even larger maps. I am thinking that a separate bitmap could end up eating a lot of memory for these really huge maps. I know Transport Tycoon (renamed "Locomotion") has maps that are about 400 square, Simcity 2000 is around 256 square I think, something like that, in either case, some pretty big maps.

I know in my game, which will use the same style maps as these two games the heights between tiles won't be greater than 1, so if you select a tile, you should only need to check the tile you think you are on, then check the tile to the north and south of it to see if there is a height difference. But I can't foresee much of a problem now that I think of it. We'll see. I'm reallu anxious to add in layers now. The way it works in my editor is the same as Simcity2000, where you can click to add in a hill, if you click again, it raises it up higher (adjusting the surrounding tiles accordingly)..... I dunno, I'm a trial and error kind of guy, I'll see when I actually program it.

You have an interesting solution though, I just wonder how effective it will be for really large maps...

You have two possible approaches:
1) Store one giant bitmap - since the map rarely changes, and if it does then it only changes in a small area, you'll rarely have to redraw it. Sure, it can be quite big, but at 64*32 you have can have 256*256 tiles in 128 megabytes, which isn't much for modern games.
2) Use a bitmap as big as the screen and draw to it every time you draw to the screen. This means that drawing takes twice as long, but the extra memory usage is negligible. (And for drawing isometric tiles, drawing time should not be considerable enough that you care about doubling it...)

Perhaps a scaled down version of the larger tiles? If the tiles are 64x32 as mine are, they could be scaled down to 16x8 or so, even half size would work, then just apply the same algorithm for selecting to the smaller tiles which you would use to colourize for the height selection....

Edit:

Seeing as how I am still new to Isometrics and I am not finding ANY tutorials online about selecting tiles at really high heights, I'll just stick to 1 layer above the ground like I have now. I think I may create a seperate project just to experiment with this problem and see what I can come up with. Your ideas about colour maps is a good one, but I am not convinced that's the best idea. I am determined to find out how games like SimCity 2000 did it. When I tested that game it runs extremely fast, scrolling etc... but that may be just due to it using 8bit graphics too. I should check on how much RAM it uses and see if perhaps they use something like what you suggested...

Edit2:

I found an interesting document on the SimCity2000 file format. It doesn't answer my question about how they select tiles, but it is interesting how data is organized and the info contained in a SC2000 level.

SIMCITY 2000 FILE FORMAT (MS-DOS VERSION)

I'm thinking of an isometric view too for the world map.. However, I'm thinking of fake-isometrics instead of the one you are discussing here. By fake I mean using rectangular coordinates as a normal 2d game would use but tiles are 16*8. I guess diablo 2 uses that method. 1 rectangle contains 4 triangles.

@Neil Roy: for some reason or another, I can't seem to be able to click the link...

@J-Gamer: Sorry, typo, I corrected it.

@Hyena: It seems to me that this could have more problems than a normal isometric view. In a normal isometric map you're basically just rotating a square map 45 degrees, but figuring out the grid location is just about the same.

One note about Diablo levels, they don't have the problems with height because Diablo levels are flat, they don't have any height at all, you won't see a hill in Diablo, go ahead and look. A flat map is uber easy to plot and get mouse locations on, I am doing that already, even with small hills, without trouble.

I have considered not even bothering to worry about having hills in my game as most of the really successful games I have noticed have been flat isometric games, they get away with this by having a lot more detail in their game so you're so busy admiring how nice it all looks you don't even notice how flat everything is. There are some Diablo levels that appear to have height (like in a few with stairs) but this is just an illusion, they're still flat.

I may just stick to a flat map with small hills like I posted, there isn't really any big need for large hills anyhow when I start thinking about it. It's certainly not something I wish to stress over.

Edit:

If anyone's curious, here's what I have created so far: isomed.zip
(Windows executable, compiled for use on i586 or better CPUs, I also used ImageCfg on this so it will be forced to only use single CPU on multi-cpu platforms.)

{"name":"604028","src":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/4\/9\/493bc326fe32f4666ba1e467e44e9506.png","w":800,"h":600,"tn":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/4\/9\/493bc326fe32f4666ba1e467e44e9506"}

it's fairly basic at the moment, you scroll the map around with your cursor keys, left clicking and dragging around the terrain defaults to "drawing hills", right click flattens them again. Road drawing mode is similar, just left and and drag your mouse around the terrain to draw roads, it will automatically connect up the roads similar to the way Simcity 2000 does it. Right clicking will erase roads and automatically "disconnect" adjacent roads. I haven't added in code to put roads on top of the hills yet, you can put one on the side of a hill. I'll get to that next. The map is currently 50x50, although it can be any size, this is just one I made up for now. I want to make all this configurable from a menu in the future.

I hope you've looked that page too: http://www.yarrcade.com/2010/09/19/terrain-modificatio-in-grid-based-games-part-1-isometric-landscape/

Yesterday I checked it out.. It is cool but if you start seeing the flaws of this system, it doesn't seem to fit so well to your game. Even when bitmap texture is blitted on the isometric tiles of different angles.

However, I downloaded your zip because I am building a random world generator based on random noise -> heightmap conversion and I was looking for a system to display my results. Age of empires and red alert 2 used your system but still you don't see many hills in these games for some reason. In the other hand, games like Settlers use a lot of hills and they are totally cool looking there:
{"name":"screenshot660-1.png","src":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/1\/c\/1c9d28a8607c1770e6dac833dc5e0a2a.png","w":720,"h":480,"tn":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/1\/c\/1c9d28a8607c1770e6dac833dc5e0a2a"}

So I personally am very tempted to try out the settlers' hexagonal landscape just because it looks a lot more realistic than all the isometric ones. I guess it's the case of choosing a right system for a right kind of a game

EDIT:
Looked your isomed and have to admit it's really impressive. I guess if adding some gradients when the height changes you could squeeze out something beautiful from this system.

EDIT2:
Look at the attachment. In the bottom left corner you'll see what I meant by gradients. You will also notice that the higher ground is brighter than the lower one thus you probably cannot make infinitely high grounds as the lightness would become pure white.

{"name":"604044","src":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/e\/e\/eeaed3eea3ba6f8a4d4150b4a42b23c2.jpg","w":806,"h":625,"tn":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/e\/e\/eeaed3eea3ba6f8a4d4150b4a42b23c2"}

That interesting because I was playing Locomotion (newer version of Transport Tycoon for Windows) and I noticed they had an option to toggle gradients on/off to "smooth" out the edges and that got me to thinking about trying that myself. I also want to redo my graphics so there aren't grid lines and make them something you can toggle on/off. I'll also probably redo my roads so they're done more the way Locomotion does theirs. I', also thinking about making the tiles so that they are higher when you raise them, but I'm not certain about that yet.

I'm thinking about just allowing the hills to be only a couple layers high, they don't really need to be too high anyhow. I had noticed that the tops of hills being the same shade as the land really took away from the look of them being higher, I may just program it to have lighter colours for height, not sure how I will implement this yet, but it does look nicer, now that you have done something up, I really like the looks of that. Add in some trees, brush, real looking grass etc... and it could get quite nice looking I think.

I originally had the editor so you clicked on a tile, then places it, but when trying to make large maps this got VERY tedious so I looking into ways to automate the process so you could just draw them. What I do is have the tiles organized so the corners of the tiles are like a binary...

 1
8/\2
 \/
 4

The main base tile is tile #0, zero meaning no corners are raised. If the top corner is raised, it is tile #1, the right corner, it is #2, if the top and right are raise, you guessed it, #3 (1 + 2). Etc... this way, when I raise a tile up by left clicking it, obviously all 4 corners are raise, all I need to do is take the value of the surrounding tiles and logically OR them with the corner that gets raised, so if I click a tile, the tile above it (Y-1, X-1) has the southern most corner raised up to match, so I take the value of that tile, OR it with 4, and the result is the new tile# that goes there, this makes it really simple to raise all the surrounding tiles up and make a hill. The code to do that is very simple:

I check to make sure the tile the mouse is over is greater than zero (I set tile_mx and tile_my, the tile the mouse is pointing at to -1 if it is pointing off the border of the map).

Tile #15 is all four corners raised as you may have guessed. I want to change that part of this so it instead sets the "height" of that map tile.

The rest of this simply checks to make sure it stays within bounds (less than MAPSIZE, greater than zero) and then ORs it with the appropriate number for the corners effected. Road tile pieces are all greater than 19, which is why it returns if the tile # is >19.

And of coure, lowering the terrain is similar, you lower the main tile (set it to zero) then logically AND all the surrounding tiles so the connected corners are all lowered, which leads to the proper tile #s being drawn.

I do something similar for the roads as well, except with them I call a recursive function which adjusts surrounding roads so they automatically connect up. The road pieces are organized in a "binary" fashion like the terrain.

I may redo the tiles so that they are all WATER at the base location, and you raise up LAND when you click, so you see islands when you click, otherwise it's all water when you start.

That could probably be improved, but it's just my first draft of that function, I don't like all those "if"s in there, but...

It just looks like normals to the light source to me. Height doesn't matter, just the angle.

Oh I seen what you meant. My scenes are lit by the game though, could be interesting to try though.

Edit: While looking over some other websites a possible solution came to me with regard to selecting tiles that are several layers high. Once you select a tile, you could then compare where it is drawn on screen with where the mouse is pointing, then maybe do some maths on the distance and esimate how many tiles "south" of where the program selected you are actually pointing, perhaps "walking" one tile at a time until you get the correct one. Or in the case of valleys, walk north. I guess you would test one tile to the south, one to the north, whichever brings you closer you would continue to "walk" in that direction until you get the correct grid location... I'll have to play around with this idea, I was going to not bother with higher terrain, but it bugs me too much to just drop it like that.

Edit: Upon further reflection, I have come up with a better solution that I think will work (I don't think the above solution will). I based it off of the ray casting concept where you draw a line from the viewer into the scene and see what it intersects with. Well, with isometrics the best way I could come up with that is similar would be to start a fixed number of tiles below the mouse position, then test that location to see if the tile position matches. If not, then move one tile north and test it again. Keep doing this while the top of the tile being tested is below the mouse_y position. Once the top of the tile is above the mouse_y than you have the right tile. If you start at the bottom of the map and work your way up, you will always get the right tile. The number of tiles you start below the mouse position will depend on the maximum height of the hills on your map. You could of course start at the very bottom of the map, but this would probably just waste time. If you have really large maps like I have, you could also just start at the bottom of the visible map.