Yes only to the attacker and I might add should you ever PvP yourself the points go into the tower that has been
* unlocked by you (so for example if you had Colonial era troops but hadn't won the tower on the map the points wouldn't count at all)
* go into the tower of the highest era troop you put into the army. So if you had 6 Iron Age and 1 Early Middle Age troops the points would only go into the EMA tower.
As a defender with the current way PvP is set up you don't get any points. Just make sure you're only putting in unattached units into your defending army and that you put one in or they'll only have to win against 2 spearfighters.