on small maps just target their miners.
You're assuming the opponent wants to play a game of "tag" with your military. What if he harasses your economy in turn?
Get a castle archer? The villain can afford a castle archer too, of course.
Except your sword+archer combo will get shredded, whilst his spearton could afford to take a nap while getting shot at.
The Spearton owner thus not only gains a military advantage (the reason you won't confront him is because he's stronger, yes?), but also an economic advantage, because you have to retreat first, lest your troops die for nothing.