I would first do the other direction, go from tile to pixel coordinates. Without loss of generality, I chose the direction sloping towards the upper right edge and made it be as high as the tile:

0.............TILE_WIDTH
.      /`-_
.     /    `-_
.    /       /|
.   /   _   / |
.  / _-´ `-/  |
. /-´     / `-| 
.  `-_   / _-´
.     `-/-´
TILE_HEIGHT * 2

So clearly:

screen.x = map.x * TILE_WIDTH / 2 - map.y * TILE_WIDTH / 2
screen.y = map.x * TILE_HEIGHT / 2 + map.y * TILE_HEIGHT / 2 - TILE_HEIGHT * (1 - map.y)

Now just solve for map.x and map.y:

1) screen.x * 2 / TILE_WIDTH = map.x - map.y
2) screen.y * 2 / TILE_HEIGHT = map.x + map.y - (2 - 2 * map.y) = map.x + 3 * map.y - 2
1-2) screen.x * 2 / TILE_WIDTH - screen.y * 2 / TILE_HEIGHT = - 4 * map.y + 2
Therefore:
map.y = screen.y / TILE_HEIGHT / 2 - screen.x / TILE_WIDTH / 2 + 0.5
map.x = screen.x * 1.5 / TILE_WIDTH + screen.y / TILE_HEIGHT / 2 + 0.5
(I probably did that wrong, it's late...)

The same works for the other 3 directions and for arbitrary slopes.

The 4 additional "diagonal" directions depend on how exactly you have them, but probably you split your tile into two triangles where one triangle is just flat and the other slopes towards one corner.

Depends on the game. Most often for something like selecting units with the mouse you don't really want the tile but the unit itself. E.g. if the head sticks out on top of the tile above and you click that head - it still should select the unit and not the tile.

I think you're making it more complicated than it needs to be. If you do it a certain way, all you need to do is apply a scale to each dimension and apply some inequalities. This figure shows the map coordinates in tile units.

You can see that the tile units are just a grid of x,y values, with x increasing to the right and y increasing downwards with 0,0 in the upper left corner. This is the same coordinate system Allegro uses.

{"name":"610536","src":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/b\/b\/bb3158e45a4dabe61c646d94a35ba7f9.png","w":600,"h":545,"tn":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/b\/b\/bb3158e45a4dabe61c646d94a35ba7f9"}

To convert from tile units to map units all you need to do is apply the scale, which is the width and height of the tiles over 2.

int mapx = (tilex * TILE_WIDTH)/2;
int mapy = (tiley * TILE_HEIGHT)/2;

To convert from map coordinates into whole tile coordinates, you can apply a series of transformations and inequalities. Basically, you test whether or not the point falls on a certain side of a specific line in tile space.

{"name":"610537","src":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/9\/8\/9818c2559c390dc7dc591b7909196523.png","w":1024,"h":689,"tn":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/9\/8\/9818c2559c390dc7dc591b7909196523"}

In the figure above, there are 4 lines to test against, numbered one through four, with the corresponding condition indicated by a or b depending on which side of the line it falls. To find the equations of the lines, use the point slope intercept formula :

y = ((y2 - y1)(x2 - x1))*x + b;

This gives us 4 lines

/// Line 1 [Y = -X + 1]
/// Line 2 [Y = X - 1]
/// Line 3 [Y = X + 1]
/// Line 4 [Y = X + 3]

First, find the grid sector where the point falls. I'll call this tile quadrant x and y. Simply apply the inverse scale from above to go from map coordinates to tile space.

float TileQuadrantX = ((float)mapx)/(TILE_WIDTH/2.0f));
float TileQuadrantY = ((float)mapy)/(TILE_HEIGHT/2.0f));

Next we want the base tile x and y to the nearest multiple of 2. Integer division works perfect for this. The reason is that we will be testing against a 2x2 space in the next step.

int BaseTileX = ((int)TileQuadrantX)/2)*2;
int BaseTileY = ((int)TileQuadrantY)/2)*2;

To test against our line pattern we need the tile quadrant position modulus 2.0 to find the position relative to our 2x2 space.

float QuadrantX = fmod(TileQuadrantX , 2.0f);
float QuadrantY = fmod(TileQuadrantY , 2.0f);

Next we test against the lines that divide our 2x2 space. This will give us the correct tile offset for our base tile position.

Now we have our tile coordinates from our map coordinates. I suggest making two functions to convert from map to tile and tile to map coordinates.

However, despite all this, I may have misunderstood the question. If you're asking for fractional tile coordinates I'm not sure I know how to do that. I think it's just a series of transformations though.

EDIT
I think I see what you're asking. To answer your questions, to get the 8 slopes, just use map coordinates for each tile. So to get from one tile to another, make a line from the two points in map coordinates. It will still travel the same direction.

Just a question, why do you need fractional tile coordinates? What does that get you? And if you solved it, you should provide the solutions so people asking the same question in the future get to see the answer and don't have to ask it again.

And my question is, what do you use that for? Why not just use map coordinates? Do you really need fractional tile coordinates?

You're totally missing what I'm getting at. What I'm saying is why can't you simply track your game object's coordinates in map coordinates (pixels?). You can convert map coords to whole number tile coords any time you need to. And you can more easily track offsets in pixels than you can in floating point tile coordinates (which you have to convert back to map coordinates (pixels) to draw them anyway).

This is the first time you mentioned 3D. I don't think you would need fractional tile coordinates if it was just a standard 2D isometric map though. I won't bug you any more about it.

Please do post the solution though, for future reference for other's sake.

I only used a 2 color mouse map: do determine if I'm over the tile or not.
since I use a heightmap it's possible to have tiles behind each other.
So I can't predict all the intersection types.

I'll post the solution when i get it working. It's still broken since I only get to work on it in my spare time, and my math skills aren't that great.

It's isometric 2.5D with heightmap support.
I did say that I'm also using a mousemap. I'm just not familiar with your multi-color technique.
My mouse-map only has 2 colors, for on or off.