By mods do you mean mods you use in games?
It can come down to anything, might be processing power e.g. a lot of people on-screen at once (CPU & GPU). High fidelity textures and shaders (GPU).
But it all comes down to how the game runs, but if you want to highly mod a game such as Skyrim, I'd suggest a good CPU and GPU with enough RAM (4GB at the least).