Updated on 7/25/2009.

Visual Studio setup project produced MSI packages with problems for a while now. It looks like with introduction of Visual Studio 2008, setup project added a couple of new twists, compared to VS'05:
1. Order in which custom actions are called has changed.
2. When installation is an upgrade, old binary files (EXE and DLL) will only be replaced by new ones if new binaries have higher file version.

Here's what used to happen. MSIs produced by VS 2005 had the intuitive order:
- Uninstall step from the old installer version.
- Install and Commit steps from the new installer version.

In other words, installing an MSI created by VS 2005 was a very rough equivalent of uninstalling the old version, followed by installing the new one. VS 2008 order has become a bit more complex, arguably smarter, but also less intuitive.

MSIs produced by VS 2008 seem to have this new order:
- Install step from new installer version.
- Commit step from the new installer version.
Note that Uninstall step of old version installation is not called at all during an upgrade-installation of an MSI generated by VS'08 (with RemoveOldVersion set to True). Funny, but even though Uninstall steps does not seem to be invoked, the custom action assembly of previous version is still getting loaded, which may lead to installation crash if old version uses .NET Framework 1.x. Custom action order change is the biggest departure from VS'05 MSIs. Also, only files that have been changed in the new version of the installed package, will be replaced in the upgrade mode, while unchanged files will remain.

Also, upgrade flow of the MSIs generated by Visual Studio 2008 also replaces binary files only if their FileVersion property has changed. Since this was not a requirement in Visual Studio 2005, you may want to go through your AssemblyInfo.cs files and ensure that they either have the wildcard in their version name ([assembly: AssemblyVersion("1.0.*")]), or you manually increment version before releasing a new build. If AssemblyVersion and FileVersion are in sync, you can remove FileVersion attribute from the AssemblyInfo.cs.

You are most likely going to experience the effect of these changes after porting your Visual Studio 2005 windows service installer project to Visual Studio 2008, and getting the "Error 1001. The specified service already exists." error. Explaining this all would make this rather a long story, but the gist of it is this: ServiceInstaller's Install() method attempts to register the service even if service is already registered, which is the case in the upgrade service installation (remember, Uninstall step, which unregisters a service is not called anymore). ServiceInstaller's Install() throws the above-mentioned exception if service is already registered.

To successfully upgrade a Windows Service, it needs to be stopped before files can be replaced. If service was not stopped and files are replaced - target system is likely to be required to reboot at the end of the installation process. Stopping service during upgrade installation from Install custom action will be too late - at this point files are already replaced and reboot is imminent. You see what happens here: it appears that upgrade installation of a windows service created in Visual Studio 2008 will *always* lead to rebooting the target machine. Given the fact that windows services are very often created for server applications deployed on high-availability machines, it's seems that windows service installation done by the book in Visual Studio 2008 is all but useless.

Here are two solutions.

Solution #1 (for Windows Service Installers): Make your MSI act the old (VS'05) way

Keep your old custom steps and do this. It was a pain to copy the script to clipboard from IE. I had to do View | Source to copy & paste the script. Also, if you save the MSI_SetActionSequence.js file in the solution folder, your post-build event command will be exactly this:
cscript.exe "$(ProjectDir)..\MSI_SetActionSequence.js" "$(BuiltOuputPath)" InstallExecuteSequence RemoveExistingProducts 1525
(Path to MSI_SetActionSequence.js may vary.)

Solution #2 - Update your VS'05 custom actions code to comply with new (VS'08) way

When registering a service, two things need to be done differently compared to how you did it in Visual Studio 2005 setup project:
   1. Invoke Install step only for clean (non-upgrade) installation.
   2. Commit step needs to restart the service in the case of upgrade installation.

Here's a bit more details and a few snippets that should help.

1. First, in your setup project, select Install custom action of the service installer, and set its Condition value to NOT PREVIOUSVERSIONSINSTALLED. This will eliminate calling ServiceInstaller's Install() custom action for upgrade installation.

2. Select Commit action and set its CustomActionData value to /OldProductCode="[PREVIOUSVERSIONSINSTALLED]". This will pass the ProductCode of the old version to the Commit custom action - if it's an upgrade installation, and blank string if it's a new installation. You can use it in the Commit() code to determine whether it's an upgrade installation and restart the service:

private bool IsUpgrade
{
    get
    {
       return ! string .IsNullOrEmpty( this .Context.Parameters[ "OldProductCode" ]);
   }
}

Making VS'05-generated Installers Act Like VS'08-made

Despite breaking windows services installers, changes to the installation process introduced by Visual Studio 2008 do benefit other types of installations, because being able to tell whether it's an upgrade installation and make the installer act accordingly is quite valuable. Developers of Visual Studio 2005 can have the same functionality if they modify their final MSI by running this PostBuildEvent command in your Setup project:
cscript.exe "$(ProjectDir)..\MSI_SetActionSequence.js" "$(BuiltOuputPath)" InstallExecuteSequence RemoveExistingProducts 6650
(Path to MSI_SetActionSequence.js may vary.) 

If you go this route, then you likely will need to use a pattern shown from the "Solution #2" shown above:
- Install custom step should be called on the NOT PREVIOUSVERSIONSINSTALLED condition.
- And to tell whether your code runs in the upgrade mode, Commit custom steps should have /OldProductCode="[PREVIOUSVERSIONSINSTALLED]" parameter passed to it so Commit() implementation could use this.IsUpgrade property as shown above.

All! If you use Visual Studio 2003 or 2005 to create MSI-based setup packages, here's a good one for you: if your installation uses Uninstall and Install/Commit custom actions implemented as an installer class - you are in trouble. In the process of upgrading your product MSIEXEC.exe first loads an assembly with Uninstall custom action implementation - to complete previous version uninstallation. After that it tries to load installer class of the new version to do Install and/or Commit custom actions of the new version. At this point things can get really bad. If your custom action assembly is not signed/strongly named (and in my experience sometimes even if it is signed) MSIEXEC.EXE will fail to load custom action assembly from the new version and will run Install/Commit custom steps from the old one. This means that if you added new code to your Install/Commit steps it simply won't be executed during upgrade. Even worse: Install/Commit custom actions of the old version will run instead of the new one!

This happens due to completely bizarre, to put it mildly, logic of .NET Assembly.LoadFrom() method. .NET Framework has a rule that after assembly is loaded it can't be unloaded unless it was loaded into a separate AppDomain: appdomains can be unloaded and assemblies can't. Two assemblies may end up looking the same to LoadFrom() if they have the same name even if they are located in different folders or have different versions. So what happens here is this: after MSIEXEC.exe loaded assembly named 'X' to do Uninstall custom step, the subsequent attempt to load assembly named also 'X' from another folder to do Install/Commit step does not happen. But get this: one would expect that if you asked LoadFrom() to load assembly 'X' from folder 'Y' it should either load it or tell you it can't. Instead due to some truly twisted logic, LoadFrom() won't fail if it can't load new 'X' assembly - it will simply return the reference to the one that is already loaded. So much for solving DLL hell problem!

Microsoft knows about the problem since 2004
http://support.microsoft.com/kb/555184/

It didn't, however, fix it yet:
http://support.microsoft.com/kb/906766

They recommend giving unique names to custom action assemblies for each new release. Alternatively they say signing an assembly will make problem go away. I tried signing and in my small test project it made problem go away, but not in the "real" one. I am stuck with having to rename custom action installer assemblies for each release. All Microsoft needed to do is this: force installer to create new appdomain and load old version's Uninstall custom steps assembly there and let it run. After it's done, unload the appdomain and create the new one where you load new version's custom action assembly with Install step implementation. That would make it unnecessary to give assemblies unique names - strong or physical. My understanding is that Visual Studio adds a small shim DLL to the MSI package that loads .NET installer classes from the custom action assemblies. This means they don't even need to wait for another MSI API release to fix it - every new Visual Studio or a even a Service Pack for Visual Studio could have fixed the issue that is still with us more than three years later.

In Windows XP one could just double-click an .MSI (Windows Installer) file to start package installation: MSIEXEC.exe is associated with the .MSI extension and if user had administrator rights installation would go forward. Clicking .MSI file was functionally identical to running Setup.exe bootstrapper, provided Setup.exe didn't have additional functions other than starting the installation.

In Windows Vista things are different. When Vista's User Account Control (UAC) is turned on, launching Setup.exe is not quite the same as running MSIEXEC.EXE /i mypackage.msi. The difference is that when Setup.exe is started, Vista runs it in "elevated" mode, which gives the process more privileges. MSIEXEC.EXE does not seem to run in elevated mode and therefore behavior of the installation may be different.

The issue seems to be manifesting itself most often when an MSI setup package made using Visual Studio executes custom action steps implemented as an Installer class. I am not sure what exactly happens but I noticed that MSI error 2689, which is a common result of failed custom action, will go away if installation initiated using Setup.exe instead of just clicking on .MSI file.

Bottom line: On Vista always start installations by launching Setup.exe instead of double-clicking .MSI file.

Another possibility to consider: if you were not a victim of computer virus attack in the last five years (Windows XP lifetime), then you are may want to simply turn Vista UAC off.