I'll use Civ4 as a clear example. One avid Civ4 player has spent his own time making changes to the AI following Firaxis's decision to release tools that made this possible. These are really good changes and I'm sure could have been done by Firaxis (they actually took several of them for their most recent patch, so it's not a quality issue). But they don't care enough to spend the resource - improved AI matters to too few people at the end of the day, and spanky new 'cool' features and nice graphics matter to more given the current market composition.
It seems to me it's more a function of their own skill. Blake is a really, really good Civ 4 player, and we've had a year to figure out many of the best strategies. It seems to me that the AI's priorities were just wrong (and Firaxis has focused mostly on bugs/balance in patches). That's different from, say, introducing entire new algorithms to the AI.
And I think the reason AI progress has been slow is because it's hard.
.

at Asher(and the bunch of guys who really didnt understand what i was trying to draw attention to - probably my fault for not being arsed to write a huge document in detail with clear examples etc etc. Luckily many other folks have got it and made some very good and pertinant posts on some of the aspects)
Comment