Swords and archers are very cheap, quick low tier units, less so spears. You should not need many resources to max out your population.
If he's going full size giants and allbows, he needs a lot more resources to get max population.
There is the key to your basic strategy - max out and attack as soon as possible.
If you hang around at max population without attacking, his army is growing in strength and yours is not (being maxed allready).
However, you should not be defensive to max out your population size. If you are, he's gonna spam miners then spam high tier units, then ROFLSTOMP you.
You shoudl be attacking to secure the tower from the soonest moment you have units, and from there you should be attacking to supress his miners, slowing down his building while you quickly mass your cheap units.
This is a more advanced version of your strategy.
Walkthrough;
start sword and archer.
secure the tower.
harass his miners - initially he has no castle archer, so charge in with the sword(s) while shoting from behind with archer(s) untill he gets a castle archer, then particularly harass those on outer gold which your archer can kill from outside castle archer range. Use spears and swords to protect archer if he attacks - making sure to fight outside castle archer range.
build an economy while building your cheap units - just enough units to keep the tower in the early stages so you can pump up the economy and attack his miners.
Get tower spawn 1
When you have enough economy, build your units as quickly as you can.
do hit and runs to supress his economy as you are building more units. When his miners garison, hit his statue a bit on each spawn - retreat as ghost dies (assuming he has castle archer) Heal any units with garison or meric.
finally, focus fire on the statue and type 'GG'
Sorry, its not defensive. But it isnt a bad plan based on what you want to do - use swords, spears and archers.