ideally you would use an unamplified "line-out" or "line-level" connection on your computer for the lowest possible %THD (distortion).
if you do not have that as an option, you could get away with using your headphone jack.
3.5mm to rca splitter. they are more common than dirt.
if you're looking to improve the quality of audio over onboard, just an amp alone will not do this (you would need a dac (either external or soundcard) to bypass onboard. if you just wanted to push more power to headphones to drive them better, just hooking them up as above is fine.
pretty much all of your headphone amps are rca input. some have 3.5mm aux inputs though. its also possible to find some combination dac+amp units (such as the fiio e10k) which...