Its better in terms of 'investment to return' to buy extra miners rather than passive income. And in those terms alone, 'no passive income is not worth it'.
But thats an incomplete picture and therefore misleading. (See the exampe of logic based on an incomplete picture below if thats not obvious).
The other factors are that
1. you dont have to put any population cost towards the return of both gold and mana, where as miners cost population and so reduce the military you can build (i.e. you get a bigger military if you invest in passive income 3 than if you invest the same costs in miners - which is huge once you reach population cap.)
2. you cant have the income killed or stopped if the enemy attacks your base. If he kills your miner, or forces you to garison, your miner is not as good an investment as you thought.
3. your income continues when the gold reserves have been used up. If your in a longer game, often the first line of gold is finished, which reduces the number of miners that can work gold - if both players are in the same state, having passive income may be a huge benifit as you have a better economy than him, and you have the population slots free too.
So the complete answer has to be there are some good reasons to buy passive income, and in some situations its a good idea - and if you do so, its better to buy all three than just 1 or 2 levels as the return is better. So; dont screw your early military or economy to rush for it, hiner hustle is a better early buy. But dont ignore it mid game if you have income to spare.
Cars are dangerous, many people die, so we need to drive as safely as possible. (true)
The more time we spend in cars, the more chance of an accident. (true)
The faster you drive, the less time you spend in the car. (true)
Therefore logic shows that the faster I drive the less chance theer is that I will have an accident. (true logically, but untrue in fact)
The conclusion is unsound as the logic is based on incomplete evidence - the fact that driving fast over a certain period of time increases the chance of a crash during that period of time has not been allowed for - showing that sound logic on incomplete facts produces unsound conclusions.
The roundabout is the most dangerous part of the trip
I should spend the least time possible in the most dangerous part of the trip
Therefore I should attain maximum speed on the roundabout