Why? just check history. And I don't mean just civ's. Games with bad AI never got their AI fixed substantially. MoO series, MoM, SMAC and TW series to name a few, all had bad AI. Fixes and unofficial patches/mods always came out and dealt with the most urgent matters, yet the AI always remains on the same general stupidity level. It's really hard to write a decent AI to games, and it's even harder fixing it and raising its level from "bad" to even "barely good".
It's history folks, and it's hard to argue with that.
And it also makes me quite sad. I love Civ and I kinda like this game, despite all the issues.