Connecting new TV to very old HI-Fi?

We've had a rather snazzy Samsung 50" 3D tv for a while now, but we had been using the tv's speakers. Sitting right next to it, is a hi-Fi system that was bought a very long time ago, but me, not being content with the tv speakers decided this morning to hook it up to the system and see how things went. I turned the hi-Fi round to find a red and white connector with the words audio (video) beneath them, and thought-great, this might connect to my tv! Venturing around the back of the TV I discovered it had another red and white connector. After rummaging around in a draw for several minutes I found a scart cable sort of thing-red, yellow and white on both ends. I decided to hook it up from the tv to the hi-Fi using just the red and white connector, as from my understanding they do the same thing as a lead with just red and white. I switched onto external speakers on my tv and "video"on my hi-Fi expecting bear grylls to come shouting though the speakers about survival. Silence from the speakers, but going around the back of the tv and unplugging the cables I found that if I pressed the lead on the outside of the port I got the buzzing noise from the speakers.
I've tried all the options on my tv and hi-Fi, but cannot figure out for the life of me why it won't work. I see no reason why not? Not being too up on sound I may have missed something obvious to another. What's wrong?