Really?
What I read about the way people live in the Imperium (Hive worlds, factory worlds, killing 1000 psykers for the throne daily, such stuff) doesn't seem any less grimdark to me than the combat.
Aggri-worlds are supposed to be relativly comfortable, worst case you're a serf breaking your back ploughing fields but at least you have breathable air & get to eat decently (no point in transporting corpse-starch to an aggri-world). And occasionally you'l even get to sit in the sun for a minute or two. Plus, less crowded since they need the space to grow crops. Anything higher than the lowest serf should have it fairly comfortable on an aggriworld.
On hive worlds it depends a lot on were you live. The underhive is horrid. But the higher you go the better it gets. A good representation of it I think is Taris in SW KoToR: the undercity is riddled with crime, monsters, mutants & desease. Virtually noone has any sort of (legal) income. The lower city is riddled with gangs and derelict areas, but you can already get some luxuries and there are legal jobs here and desease is no longer rampant. It's more akin to modern day ghetto's, it's not great and you're probably going to end up shot in a random turfwar but it isn't exactly hell either. And the upper city is clean and for the most part comfortable, no worse than modern day society (apart from maybe the fact that noone has a yard..). With the proper upper class like the gouvernour living in luxury.
Factory worlds are fairly awefull as they're polluted to the extreme. So where hive-worlds
eventually start to get comfortable factory worlds really only become comfortable inside the areas for the upper class. Anything below the nobility is kinda screwed.
Also bear in mind the imperium is humongous, a hive city houses anywhere from 10 billion to a 100 billion people and planets can easily have multiple hive cities. Even if 99% of the imperium lived comfortably in the obscene wealth of nobility there'd still be untold billions of people living in abject squalor or in active warzones. So there's plenty of people stuck in one grimdark horror or another to write stories about.
Anyways, the imperium (as a whole) are ultimatly the "good" guys (alongside Tau, and the less stuck up Eldar from there respective points of view). Attempting to protect the innocent, and would if they got the chance build a utopia, doing away with those grimdark horrors and the institutions they themselves created to fight those horrors. There'd obviously be some issues with corrupt governours and powerhungry inquisitors if the alien horrors would dissappear overnight. But ultimatly their goals do boil down to "making the universe a safe and comfortable place for humanity". Hell given enough time the imperium would even be able to overcome it's xenophobia, though that'd probably take the work of several primarchs or the emporer himself to deal with the millenia of indoctrination...
I think we all agree that even the "nicest" civilization in the 40k universe (probably the Tau, who are basically Chinese communism turned up to eleven. Even the humans under their reign seem to live better than under the Imperium) would pass as the evil guys in basically every other universe.
About the being evil being necessary... I am not sure I agree.
For example I don't see it as a necessary evil to basically kill everybody who invents something. That's just dumb.
Meh, the killing of inventors has 3 reasons as far as I can tell.
1) It's nice and grimdark, so obviously the writers are adding that no matter how stupid it is.
2) The adeptus mechanicus barely understands anything of what they're doing and how their tools work. So tinkering with it tends to result in disaster. Both cuz you end up accidently breaking stuff they don't know how to fix, and cuz you might accidently blow up half a planet.
3) Technology in general seems to have the nasty tendency of doing horrible things in 40K (see men of iron)
So based on 2& 3 they'd rather be safe than sorry and not allow you to do anything unless they reaaaaaaaally trust you.
It's still kinda dumb though, and mostly due to the whole "it's grimdark so we'l act like idiots even if it doesn't make any sense"
