Hello everyone,
 
I have a problem with setting up ipsec/l2tp vpn server on Windows XP.
 
I am able to connect to it when pptp protocol is used but no luck with l2tp/ipsec. Below my vpn infrastructure specification:
 
1. Certificates are used to secure a connection. They were generated on Windows 2003 Server machine with Certificate Authority in Stand-alone mode. This machine only generates certificates and is not used in vpn client-server vpn infrastructure.
2. Both certificates (CA cert and client cert) are installed on both xp machine which should perform as vpn server and xp client machine  
 
No matter what kind of protocol i use on a client side (chapv2, eap) i get a 789 error  
"The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer"
 
Microsoft's Tech net says that in that case i should use preshared keys instead certificates but i can't do that for imposed security reasons.
 
Have anyone has a clue what could be done.  
 
 

Well depending on the size of your network hamachi is free for up to 200 people on a VPN.  I've found that it just works (it's UDP punching technique tunnels through anything and it's very secure).  A more professional solution might be using openVPN on Linux but that is a lot more work.  If Microsoft refuses to give you a simple solution to fix what you have I'm sorry but I can only suggest alternatives.

Ah crap. I just covered that exact error in my MCSE class about a month ago.
 
It has to do with your certificates - that I can assure you. When I get home I'll take a look at it and post it back up here. I have to study anyhow.

I believe I've tried everything to set up vpn server on winxppro but with no luck. I've just installed WIN2003Server with iss,rras and certificate authority and everything works just fine. Anyway, brw02005 and Riser thanks for help.