The real sh*t fight is when you get to Immortal difficuility. You shouldn't automate your workers on ANY difficuilty. Rapid expansion is really a situational strategy. You should reason weather you should expand outwords or inwards. One massive city can be just as effective as a commune.
AI rely on numbers, not brains. For every 1 unit you have, at emprorer they have 3. At deity it's pretty much impossible unless you've got the envirement to your advantage(The only time I've pulled off a deity conquest was when I waited for Montezmu to assume control of the world, then sent a small army to capture his capital).
At the same time, your war-time tactics should shift from mainstream battles to ambushes and Total war tactics. Attack EVERYONE. Even if it's a worker you don't need, you can destroy and disband him and still get 30 gold for deleting the unit. You should NEVER resort to guirella, unless your in the rare situation where the Iraquious national ability is actualy useful. You can only really fight guierlla defensively, and by the time you've defeated one wave as a guirilla nation, the enemy have already prepared 10 more.
Don't try and capture cities unless they've already been weakened. Expanding outwards too quickly will force the AI into direct combat, and they will completely disregard diplomocy, friend or foe. At the same time, the AI at immortal difficuilty will have Rifles before you have swords.