Depends on do you mind carrying a phablet around everywhere or not.
Do you like using or plan to use a stylus a lot?
The easiest way for me to assist you would be for you to give me some insight on your use of a mobile phone i.e:
Do you mostly just listen to music and browse the internet, using social media and social media apps and making calls on and off
or do you mostly game
Both have memory card slots so high storage is not an issue as you can always purchase new memory cards and interchange between them
Important things to note:
One thing to keep in mind is they both have the same processor, yet the Note will be faster
The note will be much heavier, but if you don't mind the size you get use to the weight in about a week or 2.
The S6 will have slightly better cameras but you will not notice it.
The Note 4 will have a much better battery life, they always do.
The Note 4 will be slightly cheaper compared to the S6 so that might appeal to you if you want to save a few bucks.
All in all they are comparatively similar with only a couple of changes.
If I were you I would save myself a few bucks and go Note 4 if I was buying cash, if however you were on contract I would go S6 simply because it's lighter ( extra weight means discomfort even if it's just inside your pocket ). The battery life is better, which for me is a plus.
Although in the end it comes down to what your needs are.