Building VDPROJ (with workaround) doesn't seem to work with VS 2017 / TFS 2017

[joeskeen]

I cloned this repo and created a copy for VS2017 (that pointed to the correct paths for VS 2017):

SET Command_Prompt_Path="C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\Tools\VsDevCmd.bat"

I also adjusted the workaround and included it:

REG ADD HKCU\SOFTWARE\Microsoft\VisualStudio\15.0_Config\MSBuild /t REG_DWORD /v EnableOutOfProcBuild /d 0 /f

I see in the build log that it is using the correct paths and making the registry call, but I still get the error:

ERROR: An error occurred while validating.  HRESULT = '8000000A'

Any ideas why?

My best guess is that MS changed the registry key they check. When I explored in RegEdit (before trying this), I noticed that the key Computer\HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\15.0_Config didn't exist at all (where the previous versions had lots of stuff in them). I couldn't see any obvious alternative key to use, and even using the keys for the previous VS versions didn't help.

UPDATE: I tried as a workaround using the VS2015 version of the batch script. I am still getting the same error using VS2015 as well. :(

[it3xl]

Sorry, I can't interrupt my other current activities and take a deep into digging here. This will harm to other people, for now.
All I did for this project I used internet search and a bit of my experience.
I hope you won't forget me if you will find a decision, because I am also planning to migrate to VS 2017, sooner or later.

[DyonBeumer]

For people still looking; it seems that Microsoft has recognized this problem and added a solution:
There is now a program to register the required reg key to be found in
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\CommonExtensions\Microsoft\VSI\DisableOutOfProcBuild\DisableOutOfProcBuild.exe

Hope this helps and good luck.

[yanxiaodi]

Hi guys, I have the same problems. I know Microsoft has provided a tool to fix this issue. But how to use it in the build process in VSTS? I placed this file:
\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\VSI\DisableOutOfProcBuild\DisableOutOfProcBuild.exe in my solution folder and added a task to run it but the build log showed:

C:\Build\_work\17\s\DisableOutOfProcBuild\DisableOutOfProcBuild.exe
Failed to find Visual Studio instance for path 'C:\Build\_work\17\s'.
Failed to disable out of process build for Visual Studio instance with path 'C:\Build\_work\17\s'.
To do this manually you need to set the DWORD value 'EnableOutOfProcBuild' to 0 on registry key with path 'HKEY_CURRENT_USER\SOFTWARE\Microsoft\VisualStudio\15.0_<Visual_Studio_Instance_ID>_Config\MSBuild'.
Where <Visual_Studio_Instance_ID> is replaced by your unique VS instance ID, which will be something like '206b3be1'.
Process completed with exit code 2.

How to run it in VSTS build process?

[it3xl]

Hmm, I'm researching how to help you. It looks simple.
Just started to migrate everything to VS 2017.

[it3xl]

I'll try to decomplile DisableOutOfProcBuild.exe
but it locks like it does just
https://stackoverflow.com/questions/8648428/an-error-occurred-while-validating-hresult-8000000a/41788791#41788791
or my
https://github.com/it3xl/MSBuild-DevEnv-Build-Server-Workarounds/blob/master/wokraround/DevEnv-Vdproj-HRESULT-8000000A-EnableOutOfProcBuild.bat
Possible the reg key locations was changed for VS 2017.
I'll be back.

[it3xl]

And one note.

Microsoft Visual Studio 2017 Installer Projects doesn't compile at all, if any project from SSDT (SQL Server Data Tools) is used in your solution.
I.e. Integration Services - SSIS, Reporting - SSRS, etc.
And nobody fixes this for a long period.

VS Installer Project has really poor reputation. It is like a cow without an owner.
It constantly has awful troubles with any new release of VS.

I replaced the 2017 Installer Project completely by WiX Toolset. It was tough but I did.

[it3xl]

@yanxiaodi , it was your wrong to copy DisableOutOfProcBuild.exe file to your solution.

1. You cannot call "DisableOutOfProcBuild.exe" outside of its installation folder.
2. You cannot call "DisableOutOfProcBuild.exe" if the CMD current directory (%CD%) points to another location.

i.e. you'll do

Firs, in CMD, you must to set the current directory "CD" to the location of DisableOutOfProcBuild.exe.
Otherwise "DisableOutOfProcBuild.exe" doesn't work.

CD "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\VSI\DisableOutOfProcBuild"

Second, you do
CALL DisableOutOfProcBuild.exe
That's it.

@joeskeen
You can call DisableOutOfProcBuild.exe on your build-machine as I have offered above.
You can't call .vdproj without the DevEnv, so you must have an edition of VS 2017 installed on your build-machine.

Or, I'll try to create a version of my "DevEnv-Vdproj-HRESULT-8000000A-EnableOutOfProcBuild.bat for VS 2017.bat"
They use a simple string formatting for registry key inside of DisableOutOfProcBuild.exe
string keyName = string.Format("HKEY_CURRENT_USER\SOFTWARE\Microsoft\VisualStudio\{0}_{1}_Config\MSBuild", (object) "15.0", (object) instanceId);
But possible it is a bad idea. Silly DisableOutOfProcBuild.exe looks better.

By the way I'll fix a typo in my "wokraround" folder.

[it3xl]

So, here solution is my (copy-past and go).
VS2017 Community
VS2017 Professional
VS2017 Enterprise

I'll close the issue after a while if there's no more questions.

[SoleilLapierre]

Thanks for this thread; it was helpful. I ran into the same error again when trying to build a VS2017 project under GitLab CI using a Win10 virtual machine, despite all of the above.

I found the solution in the last paragraph here: https://telaeris.com/kb/jenkins-build-error-vdproj/

The gist of it is that VS2017 uses an additional per-user registry overlay, and you may need to set the EnableOutOfProcBuild key there too.

The steps are:

1. In RegEdit, select HKEY_USER then go File->Load Hive.
2. Navigate to C:\Users\USERNAME\AppData\Local\Microsoft\VisualStudio\15.0_HASH, where USERNAME is the username of the account doing the builds and HASH is the VS2017 instance ID. From there, open privateregistry.bin. You will be asked to enter a name for it - this is just a temporary name for the mount point of the private registry in the regedit tree. For the example below I use VS2017PR.
3. Check that the EnableOutOfProcBuild keys you already added under HKEY_CURRENT_USER are still there - in one case for me, opening the private registry file masked them and in another it didn't. If they are gone, add them again.
4. Also browse to HKEY_USERS/VS2017PR/Software/Microsoft/VisualStudio/15.0_HASH_Config/MSBuild and add the EnableOutOfProcBuild=0 key there too.
5. Select KEY_USERS/VS2017PR and go File->Unload Hive.

Hope this helps anyone still beating their heads against this problem.

[dorChuck]

Thanks for the answer @it3xl and thanks to @joeskeen for the question. If anyone is having the same issue in an azure devOps yaml build here is an example:

- task: BatchScript@1
  inputs:
    filename: 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\VSI\DisableOutOfProcBuild\DisableOutOfProcBuild.exe'
    workingFolder:  'C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\VSI\DisableOutOfProcBuild\'
- task: DevEnvBuild@2
  inputs:
    project: 'projectfolder\visualstudioSetupProject.vdproj'
    buildCfg: '$(buildConfiguration)'
    buildPlatform: '$(buildPlatform)'
    vsVersion: '15.0'
    clean: true

- task: BatchScript@1
  inputs:
    filename: 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\VSI\DisableOutOfProcBuild\DisableOutOfProcBuild.exe'
    workingFolder:  'C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\VSI\DisableOutOfProcBuild\'
    arguments: 'undo'

[Sathish6]

I'm getting the below error.

##[debug]Entering Invoke-VstsTool.
##[debug] Arguments: '/D /E:ON /V:OFF /S /C "CALL "E:\agent\Build_2_work_temp\365e3b0a-63cf-4ee4-9ce2-08d24b0a0d70.cmd""'
##[debug] FileName: 'C:\Windows\system32\cmd.exe'
##[debug] WorkingDirectory: 'E:\agent\Build_2_work\689\s'
"C:\Windows\system32\cmd.exe" /D /E:ON /V:OFF /S /C "CALL "E:\agent\Build_2_work_temp\365e3b0a-63cf-4ee4-9ce2-08d24b0a0d70.cmd""

Start 365e3b0a-63cf-4ee4-9ce2-08d24b0a0d70.cmd

Seting the current path to the DisableOutOfProcBuild.exe installation folder.

'DisableOutOfProcBuild.exe' is not recognized as an internal or external command,
operable program or batch file.

Revert the previous current directory.
CD is set to E:\agent\Build_2_work\689\s
End 365e3b0a-63cf-4ee4-9ce2-08d24b0a0d70.cmd

##[debug]Exit code: 0
##[debug]Leaving Invoke-VstsTool.
##[debug]Leaving E:\agent\Build_2_work_tasks\CmdLine_d9bafed4-0b18-4f58-968d-86655b4d2ce9\2.163.0\cmdline.ps1

[it3xl]

Did you run my files?

[Sathish6]

Yes, I integrated this batch file as task in azure devops. While building this task i got this error.

[it3xl]

@Sathish6 , I only started to adapt this project for VS 2019. Are you using VS 2019?
If yes then modify 2017 to 2019 in any used "DevEnv-Vdproj-VS2017_bla_bla.bat" file.
CD "%ProgramFiles(x86)%\Microsoft Visual Studio\2017\...

If not then let me know and I'll investigate this.