Your tables should have as immediate children just tbody and thead elements, with the rows within*. So, amend the HTML to be:
<table border="1" width="100%" id="test">
<tbody>
<tr>
<td>
<table border="1" width="100%">
<tbody>
<tr>
<td>table 2</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr><td>table 1</td></tr>
<tr><td>table 1</td></tr>
<tr><td>table 1</td></tr>
</tbody>
</table>
Then amend your selector slightly to this:
#test > tbody > tr:last-child { background:#ff0000; }