|T O P I C R E V I E W
||Posted - Nov 03 2020 : 8:01:34 PM
I have installed the latest available general release build 2393 of VA
(Release date: 2020.10.28, Version 10.9.2393) over Visual Studio Enterprise 2019 Version 16.7.7. VA keep crashing VS at random times soon after I load a C++ solution. At the time of the crash VA displays dialog that tells me to grab a crash dump and contact Wholetomato. Then VS terminates (after you click thr OK button in that dialog)
The only VA callstack I could grab at the time of crash is this:
[0x0] va_x!PeekMessageWHook + 0x1d85ab
[0x1] va_x!DllUnregisterServer + 0x23f0b
[0x2] va_x!PeekMessageWHook + 0x1d9497
[0x3] va_x!PeekMessageWHook + 0x1ed4f1
[0x4] va_x!PeekMessageWHook + 0x1e949d
[0x5] va_x!PeekMessageWHook + 0x1eeb4b
[0x6] va_x!PeekMessageWHook + 0x1eeecc
[0x7] KERNEL32!BaseThreadInitThunk + 0x19
[0x8] ntdll!__RtlUserThreadStart + 0x2f
[0x9] ntdll!_RtlUserThreadStart + 0x1b
VA_X.dll file version 10.9.2393.0 built 2020.10.28
DevEnv.exe version 16.7.30621.155 Enterprise
msenv.dll version 16.0.30611.6
Comctl32.dll version 6.10.19041.488
Windows 10 10.0 2009 Build 19042.572
16 processors (x86-64, WOW64)
Language info: 1252, 0x409
Microsoft Visual Studio Enterprise 2019
Microsoft .NET Framework
|24 L A T E S T R E P L I E S (Newest First)
||Posted - Feb 11 2021 : 9:39:36 PM
Does anyone experiencing this issue have controlled folder access enabled? If so, does disabling it help?
||Posted - Feb 10 2021 : 11:15:15 AM
I have the logs, many thanks for these. Reading the first process monitor log, a couple of things stand out. Firstly, there are a lot of "path not found" results, over 32,000 in fact. Some of these are just files that VA is checking for, in case they exist, but they don't, but quite a lot of these read like they are the source code in your solution, but they are not being found.
If you load the "pm_crash2399" log file and add the filters:
Process Name is devenv.exe then Include
Result is PATH NOT FOUND then Include
Path begins with E:\source\ then Include
you will see what I mean. Does this make sense?
The other thing that is interesting is that just before the crash, the IDE, and VA, does not seem to be parsing any code from your solution at all. This makes me wonder if somehow the problem is being triggered by something your solution or one of your projects.
Would it be possible to get a copy of just your SLN file and your project files, without any of the code files? I appreciate that code normally cannot be shared, but maybe it will be OK to share just these files, purely for testing purposes. I can try opening them here, which will work even without the code, and see if I can get any sign of any problems.
For C++ and C# the project files are *.VCXPROJ and *.CSPROJ, but I don't know what file extensions you might need to search for for a SQL project. All of the project files should be listed clearly in the SLN file though, if you open this in a text editor.
||Posted - Feb 10 2021 : 06:14:41 AM
I have created the Process Monitor Log and uploaded it on your ftp. Called pm_crash2399.zip & pm2nd_crash2399.zip. The log was activated before the IDE was started until the IDE was crash and disappeared without any user feedback. The rebuild of the symbol DB was triggered. At the next IDE start, the log was captured.
||Posted - Feb 09 2021 : 1:05:42 PM
AlfredS, thank you for testing this, and I apologise for all of the problems you are having here.
Unfortunately the log files aren't really telling us much. Normally I would expect more information in the logs, and for them to be of more use.
Can you please try the following test. First turn Off multithreaded parsing:
VA Options -> Performance -> Enable multithreaded parsing = Off
just to try and keep things more orderly. Now, before loading one of your solutions, can you please try downloading and running Process Monitor, from here:
When you first load the program it will start logging lots of activity, press Ctrl-E to stop the capture. Now use Ctrl-L to open the filter dialog. Add the following filter:
Process Name is devenv.exe then Include
Press OK to accept this new filter.
Now press Ctrl-X to clear the current logs, and then Ctrl-E to turn capture back on. Now load one of your solutions, and when the IDE crashes go back to Process Monitor, press Ctrl-E again to stop the capture, send me the log?
I am using this to find out the last code file that the IDE, and thus VA, opened and read, on the theory that the crash is being triggered by a specific file, or group of files. If so, and this is what is happening, we should see the same file being the last file opened on different crashes.
If so, if you can temporarily rename this file on the hard drive, so it won't be opened / parsed by VA on solution load, this would tell us if the file is the problem.
||Posted - Feb 09 2021 : 12:59:24 PM
pbrown, apologies for the problems. You can use the button:
VA Options -> Performance -> Rebuild symbol databases
to tell VA to rebuild its symbol database, which should fix any problems if they are caused by corruption inside the symbol database. I am not sure why you are seeing what you are seeing though, but it does sound slightly similar, I agree.
||Posted - Feb 09 2021 : 12:33:23 PM
For what it's worth, I had been seeing crashes with build 2393 that are similar to what is described in this thread:
I haven't had the bandwidth to take a crash dump and am not in a position where I can share any of the source or project files involved in the project files. I have since upgraded to 2399 and have still seen some crashes.
Despite the crashes, I have not downgraded because they aren't affecting me too badly. Based on what I've seen with a number of different project files (I have a separate copy in N different source trees), it looks like what's going on is:
- When I get the crash, I am seeing a dialog like the one described by andrewf. It seems to pop up after VAX has been indexing the contents of my large projects.
- If I restart Visual Studio on the same project, I don't seem to get any crashes or unexpected behavior.
- When I created a brand new Git worktree holding my source code, I did not notice any crashes when opening the project files for the first time.
Based on this data, my best guess is that the crash may be occuring when VAX is "upgrading" the internal data it maintains for a project from an older version (2389) to a newer one (2393 or 2399). If that guess is correct, it seems that the crash did not actually break the upgrade itself, since VAX only seems to crash once for any given project file.
||Posted - Feb 09 2021 : 09:18:27 AM
Yes. E is a local "normal" SSD.
I have done your tests with the mongoDb source folder and your provided sln file. I repeated the tests 5 times and never happens a crash.
I open one of our sln and the crash happened on the first try when VA has begun to parse some files. I activated the Setting Performance/Enable logging. The files I uploaded to your ftp.
||Posted - Feb 09 2021 : 08:18:52 AM
Nothing really seems odd here. I assume that E drive is a normal, local SSD?
As a, hopefully, fairly quick and simple test to see what is triggering this problem, can you please try downloading the zip of the code for the MongoDB project from GitHub:
and unzipping this on your hard drive. I have just done this, and it is a 72meg zip file.
There is no SLN file in the root folder when you unzip this, but if you just ask Visual Studio 2019 to open the folder, VA then reports 25,250 files, which makes this a reasonable sized test case.
I have also replied to your email in case 144158 with a simple SLN and matching single project file that lists all of the C and C++ code in the directory tree, which when opened, VA lists 10,678 files.
I have opened this both ways here, and there are no crashing problems for me. Can you please trigger a VA symbol database rebuild via:
VA Options -> Performance -> Rebuild symbol databases
and then try the same tests on your system?
If you still get the crash this will suggest there is something in your stable include directory's that is part of the problem, which narrows things down quite a bit. If this loads correctly then this suggests there is something strange in your solutions that is upsetting VA, so at least we know where to focus.
||Posted - Feb 09 2021 : 02:23:22 AM
They share some header and cpp files.
They have similar project settings.
All solutions have many (21 to 261) projects inside.
They have different types of projects. c++, c#, cmake, sql
If all that matters? I don't know.
License: Standard (....-.....-58X9FL-3BQD) Support ends 2022.08.28
VA_X.dll file version 10.9.2399.0 built 2021.01.26
DevEnv.exe version 16.6.30320.27 Enterprise
msenv.dll version 16.0.30309.148
Comctl32.dll version 6.10.18362.1256
Windows 10 10.0 1909 Build 18363.1256 (remote)
28 processors (x86-64, WOW64)
Language info: 1252, 0xc07
Platform: Project defined
C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\Include\um;
C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\cppwinrt;
C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\winrt;
C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\shared;
C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\um;
C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\ucrt;
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\VS\include;
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.26.28801\atlmfc\include;
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.26.28801\include;
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\VS\UnitTest\include;
C:\Program Files\Microsoft SQL Server\Client SDK\OLEDB\182\SDK\Include;
C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\Include\um;
Stable Source Directories:
C:\Program Files (x86)\Windows Kits\10\Source\10.0.19041.0\ucrt;
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\VS\src;
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.26.28801\crt\src;
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.26.28801\atlmfc\src\atl;
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.26.28801\atlmfc\src\mfcm;
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.26.28801\atlmfc\src\mfc;
||Posted - Feb 08 2021 : 12:59:18 PM
Is there anything obvious that the different solutions that cause this problem have in common?
Do the different solutions that cause the crash share common projects or code? I am wondering if something in your solution is really causing problems for our parser, and that this is somehow the trigger. The solution size, number of projects and memory usage are all quite reasonable, and on their own should not be of any concern.
Can you please go to:
VA Options -> System Info -> Copy Info
and paste the details (from the clipboard) into your reply. This will give us the basic information about your setup. I am interested in your stable include directories, in case there is anything odd or unusual there.
||Posted - Feb 08 2021 : 10:01:48 AM
Yes, I have full rights to read all files. But as I mentioned above once I had the crash also in admin.
With "Show only files in solution" active:16128 inactive:25235. Not all projects are loaded. Which are loaded (91 of 236) is configured by a solution filter. The projects are native C++, c#, and one is SQL.
The crash happen also when I loaded the whole solution and it happens also with other solutions.
All sources are on a local SSD and all drives have enough free space.
||Posted - Feb 08 2021 : 09:33:33 AM
I have asked our developers to have a look at the dump file, thank you for that.
Do you have full permission for VA to read all of the source code in your solution when you are not running as admin?
If the location of VA's symbol database does not matter, and apparently it doesn't, then I am wondering if there is somehow a problem reading the source code to update the database.
How many files do you have in your solution?
If you open VA's Open File in Solution dialog (Alt-Shift-O) the title bar contains two numbers. The first number is the number of files currently listed, which changes as you filter the list. The second number is the total number of files in the list, which is normally the number of files in your solution. What is this second number?
I am wondering about suggesting an open source solution of vaguely similar size you could download and open, with the IDE running without admin rights, to try and make sure that VA is able to actually build its symbol database normally for a separate solution, in case there is something strange about your main solution, or the location of the main solution, perhaps mapped network drive files, etc.
||Posted - Feb 08 2021 : 09:07:26 AM
I have changed the setting but unfortunately the same problem.
In the configured directory is a file "Startup.log". The content of that file is:
InitInst::159 2/8/2021 14:55:14 0x9a3c
CRT Exception pure call:0 2/8/2021 14:55:51 0x9960
CRT Exception pure call:0 2/8/2021 14:55:51 0x4fc8
And 2 subdirectories. In vs16_1 is a file error.log with content: CRT Exception pure call:0 2/8/2021 14:55:51 0x9960
||Posted - Feb 08 2021 : 07:51:14 AM
Apologies for the delay, we don't monitor the FTP server for uploads, since it is rarely used. I have the files now.
Since this problem seems to be related to VA's symbol database, can you please try using the setting:
VA Options -> Performance -> Use alternative directory for symbol database (requires restart)
and setting a location that you will have full read / write access to when running the IDE without admin rights, and see if this makes any difference?
||Posted - Feb 08 2021 : 02:25:28 AM
Yes. The upload was done on 30. Nov. 2020 and it's still there under the folder .../Case 144158.
I have also uploaded 2 crashdumps of 2399 and screenshots from the process explorer to see the memory usage, thread activities, and stack from the most active thread.
In the time when I watched the memory usage a crash happens. This is the dump from today. ~3% means full load for a thread (14Core+HT).
The dump from today I created when the message box about the crash appears.
The virtual memory use of the devenv.exe is ~2.2 GB. Fare away from the critical size.
I hope this will help. If something is unclear don't hesitate to contact me.
||Posted - Feb 05 2021 : 09:42:40 AM
Apologies for this.
Did you ever upload the dump file you have? Looking at case=144158 and the email I sent to you about this, no reply ever showed up in our system.
Would it be possible to get a dump file from VA 2399 crashing please, using the latest version always makes debugging problems slightly easier.
If you were to run Windows Task Manager in detailed mode, how much memory do you see "devenv.exe", the IDE process, using before it crashes? It is possible that running out of memory is a factor here, if so the memory usage should be around 3GB, the limit for a 32bit process.
||Posted - Feb 05 2021 : 01:57:22 AM
What's going on with this defect? In 2399 the problem still exists. Also when I start the IDE with admin rights sometimes the crash happens. Also other colleges have this problem. All we are obligated to downdate to 2382.
If you are interested on a new dump for 2399 please tell me where I should upload it.
||Posted - Nov 30 2020 : 07:27:01 AM
Thank you for this, I have emailed you details of how to upload the file:
||Posted - Nov 30 2020 : 02:20:08 AM
I have done some future investigations and was able to capture a dump file. With an attached 2nd studio I hold on the 1st instance when the dialog was shown.
MessageBox: "A fatal problem has been brought ..."
In that situation, I have created the dump. The dump is ~370MB compressed. If you are interested on it, please tell me where I should store it.
When the dialog was shown, the studio uses ~21% CPU and after some seconds (~10) it was disappeared. I have also tried with the deactivated setting "Enable multithreaded parsing" but the same effect.
||Posted - Nov 27 2020 : 07:56:00 AM
While we look into any mini dumps you can capture, which anti-virus are you using?
Have you tried excluding the VA's symbol database directories from being scanned? I am just wondering if somehow anti-virus could be interfering, somehow. VA should be installed under:
where the "16.0_xxxx" directory has a hash as the last part of the directory name, and VA will be installed into a hashed sub-directory under Extensions. Looking for the "va_x.dll" will help you to locate the correct directory. The symbol database is then stored in the "Data" sub-directory.
||Posted - Nov 27 2020 : 02:32:30 AM
While I do not know of a workaround at this time, a crashdump would help us diagnose the issue.
Here are a few different ways you can capture a dump. Contact email@example.com for SFTP details to upload the dump if you capture one. We would be very grateful.
||Posted - Nov 27 2020 : 02:30:03 AM
I have a similar problem when the studio is not running as administrator.
After the studio has started and the rebuild of the database begun the studio closes without any message. If the DB was successfully built as admin I'm able to start the studio without admin right until I do some changes on my code. With a down date to 2382 the problem is gone.
If someone knows a workaround please let me know.
||Posted - Nov 04 2020 : 12:32:54 AM
Could you please take a screenshot of the dialog VA displays after crashing?
How frequently does the crash happen? Are you able to reproduce it somewhat reliably?
Are you working on a publicly available codebase we could try, like Unreal Engine?
Are you able to capture a crash dump using the "Windows Error Reporting" method below?
It might be better to contact firstname.lastname@example.org in case we need to exchange sensitive information.
||Posted - Nov 03 2020 : 8:26:43 PM
Looks like VA build 2393 might be an unstable build. I've downgraded to VA build 2387 (2020.08.25 General release) and the crashing issue has disappeared