The past few days I have been working at a client that is ultra-secure and have been tasked with installing BizTalk with insane security. I have worked in environments in the past that religiously use the DISA gold disks (http://iase.disa.mil/stigs/SRR/index.html) had to follow the NSA security review documents. Another similar suite of security checkers is the Microsoft Baseline Security (MBSA) tool. The MBSA tool basically enables an IT administrator to lock down a system so tight that Microsoft server products are unable to run. In security alerts released by Microsoft, MBSA provides a way for administrators to harden a system through group policy settings, IPSec policy, among many other configuration settings. The security alert articles (like http://www.microsoft.com/technet/security/bulletin/MS05-051.mspx) even mention that modification of settings in this way will break certain Microsoft server products like BizTalk, SQL Server, MSMQ, etc. So I realized it was really important for people to know that if they are in an ultra-secure environment and are being asked to setup a Microsoft server product, you should defintely start with a baseline that was not already infinitely locked down and then slowly harden it while testing. Otherwise you will wind up going down a million rabbit holes trying to get connectivity working and may never make much progress.
In other words, make sure you have a test and development strategy that corresponds to a security baselining strategy. If you do not have more than one quality environment (your production version is your test version), then you will probably get stuck debugging security issues. Otherwise, be sure that baselines are created or established after doing functional testing rather than before. Some people will argue that its better to do it right the first time. I generally agree with this, but if you are in an ultra-secure environment where it is hard to know if a product will work at all due to the security, you will be better off knowing it will work first and then hardening the application over time.
Here is a list of some of the things I have had to do just to get BizTalk to install and partially configure:
Install network COM+, network DTC, Configure group policy to enable COM+, DTC to run on multiple servers, Configure DTC settings so that Remote Clients and Network DTC access exists, configure COM+ NTFS permissions at %windir%registration (see http://support.microsoft.com/kb/909444 for an example of ultra security – configuring the security beyond even Microsoft’s recommendations).