Most TVs will not go to an input without a video signal so you can't just connect audio to an input.
The work around is to connect the composite video output of one of your turned on sources (cable box, DVD, etc) to the same input that your phone audio out is connected. That sources HDMI output will still be active for normal use.
If you don't have MHL on your phone or use an apple AV adapter if you have an iPhone then you might be able to use a chromestick.