| T O P I C R E V I E W |
| Maksym.Bozhko |
Posted - Aug 14 2020 : 04:10:50 AM
Hello, I have quite big project, around ~6.5 MLOC.
I have multi-threaded parsing enabled, thread count set to match cores*2
I have symbol database location set to folder which is in anti-virus exceptions.
I have all the check-boxes that cause re-parsing unchecked.
(parse all files on start, look for externally modified files unchecked)
However after generating new Visual Studio project for my code and opening it,
I cannot use visual studio while it parses solution, and it often crashes and only uses 1 thread to do so.
Any suggestions?
Pasting info below:
License: Non-renewable Personal ([email protected] / MD7L-GUAD3B-KW79NK-CBCD) Support ends 2021.07.13
VA_X.dll file version 10.9.2382.0 built 2020.06.25
DevEnv.exe version 16.7.30406.217 Enterprise
msenv.dll version 16.0.30329.88
Comctl32.dll version 6.10.19041.1
Windows 10 10.0 2004 Build 19041.421 (remote)
12 processors (x86-64, WOW64)
Language info: 1252, 0x409
Platform: Project defined
Stable Includes:
C:\Program Files (x86)\Windows Kits\NETFXSDK\4.7.2\Include\um;
C:\Program Files (x86)\Windows Kits\10\Include\10.0.18362.0\cppwinrt;
C:\Program Files (x86)\Windows Kits\10\Include\10.0.18362.0\winrt;
C:\Program Files (x86)\Windows Kits\10\Include\10.0.18362.0\shared;
C:\Program Files (x86)\Windows Kits\10\Include\10.0.18362.0\um;
C:\Program Files (x86)\Windows Kits\10\Include\10.0.18362.0\ucrt;
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\VS\include;
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\atlmfc\include;
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\include;
Other Includes:
Stable Source Directories:
C:\Program Files (x86)\Windows Kits\10\Source\10.0.18362.0\ucrt;
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\VS\src;
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\crt\src;
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\atlmfc\src\atl;
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\atlmfc\src\mfcm;
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\atlmfc\src\mfc;
|
| 30 L A T E S T R E P L I E S (Newest First) |
| feline |
Posted - Sep 25 2020 : 11:04:20 AM
Thank you for the update, and apologies for the problems this bug has caused. This is down as a high priority bug, but I don't currently have an estimate for when we will be able to fix it. |
| Maksym.Bozhko |
Posted - Sep 25 2020 : 06:06:45 AM
I managed to open my solution with reduced count of configurations after restarting PC. But I still hope it will be fixed in future updates |
| Maksym.Bozhko |
Posted - Sep 23 2020 : 09:59:28 AM
Thank you, I will try restarting my PC, this might help, because I rarely do that, and I had to kill Visual Studio process that hang very often, I will try and see if it helps. |
| feline |
Posted - Sep 22 2020 : 2:39:11 PM
I am sorry to hear that you are having so many problems with this, and I agree, the problems are out weighing VA's help.
In my testing, I manually removed all of the configurations from the solution file without having any effect on the hanging IDE problem. When trying to find a specific trigger in the project file I discovered that a tiny project file would produce the hang after killing off a hung Visual Studio, but after a reboot exactly the same smaller project file did not produce any problems at all.
As far as I can tell killing off the hung IDE is causing some form of resource leak that is fixed by rebooting Windows. I was not able to locate any lingering background processes that caused the problem, but its possible I missed them.
In case it helps, if you are interested, I have a simple script that will generate a dummy solution and project that simply includes all of the code files inside a directory tree. The result is similar to the IDE's open folder feature, but leaves you with a solution you can configure if you want. |
| Maksym.Bozhko |
Posted - Sep 22 2020 : 08:35:57 AM
Well I have to go through long list of configurations and delete them manually, while making sure I don't accidentally delete the one I need, or I need to create a complex regex to delete all configurations except the ones I need. I cannot modify script that generates our visual studio project, because it creates configurations based on build system. Even when I reduced number of configurations to 10% of original count I still have Visual studio hanging. At this point going through all this pain to just open my solution outweighs the benefit of using Visual Assist over default intellisense for me. |
| feline |
Posted - Sep 22 2020 : 07:21:18 AM
No updates on this problem yet, but it is a fairly serious bug, and we would like to get it fixed as soon as possible, but I don't currently have an estimate for this.
Is reducing the number of configurations an option for you? I only ask since in my testing reducing the number of configurations helped to ease / fix the problem. The results were somewhat random, so I am not exactly sure how much you would need to reduce them to help, but if this is a simple option for you it might be worth a try.
The other obvious option is to bypass the SLN file and open the directory instead. Is this a reasonable work around for you?
Would creating a simplified version of the normal solution and project that VA handles without any problems be an option? This could be quite independent of your main, regenerated solution and project files. The obvious problem here is if the file list in the solution changes, this project won't reflect these changes until it is updated. |
| Maksym.Bozhko |
Posted - Sep 22 2020 : 06:10:04 AM
Unfortunately removing configurations is not an option for me. Our project is generated by script, so every time it is updated somehow, I have to regenerate it. I could modify script to reduce number of configurations to subset I actually need currently, but that subset is still fairly large. Can I expect any improvements to this in future updates? |
| Maksym.Bozhko |
Posted - Sep 17 2020 : 06:37:59 AM
Thank you, I will try that. |
| feline |
Posted - Sep 15 2020 : 12:26:38 PM
The problem is being triggered by the very large VCXPROJ file, but once you start editing the file to remove blocks of the different project configurations the problem becomes a bit random in if / when it appears. I have put in a bug report for this:
case=142875
If you want to keep on using the solution, and don't need all of the different configurations, then deleting the bulk of them from the project and solution files, which can be done in a text editor, looks like a reasonable work around. |
| feline |
Posted - Sep 11 2020 : 10:24:55 AM
If you don't mind sending these files along then please do. I am tempted to say they won't matter, but I honestly don't know why the solution or project files would be causing this problem for you. |
| Maksym.Bozhko |
Posted - Sep 11 2020 : 09:00:27 AM
I also have .vcxproj.filters and .vcxproj.user next to .sln. Do you need these as well, or just .sln and .vcxproj? |
| feline |
Posted - Sep 10 2020 : 12:13:29 PM
Source control bindings probably depend on the system you are using, but if you don't know about them, it suggests they don't exist.
Would it be possible to get a copy of just the SLN and VCXPROJ files, without any source code, from your problem solution? I can then try opening these files here, along with a set of dummy source code files, and see if I can reproduce the problem here, and get a sense of what might be going wrong. |
| Maksym.Bozhko |
Posted - Sep 10 2020 : 09:39:42 AM
I have only one "Project(" line when I opened .sln with text editor and that project has extension ".vcxproj". Where do I check source control bindings? I wasn't able to find anything related. |
| feline |
Posted - Sep 09 2020 : 09:57:10 AM
Thank you for the update. The occasional freeze while "reading solution projects" is probably normal, especially given the sheer size of your solution and the fact that the CPU is running at 100% usage.
At least this tells us that there isn't anything inside your code its self that is causing problems for VA.
Do you know if your solution has source control bindings set? I am trying to think of anything that your solution could be doing that might be a factor.
If not, can you please open your SLN file in a text editor, and look for the "Project(" lines. Do any of these lines list a project file extension other than ".vcxproj" ? I am wondering if there is a project referenced by the solution that VA does not recognise, and that somehow this is triggering this problem. |
| Maksym.Bozhko |
Posted - Sep 09 2020 : 05:38:17 AM
Feline, I opened folder with my sources in Visual studio as you suggested, and it worked: Visual assist used up to 100% CPU and Visual Studio was responding, I sent you 54 MB log file I collected. Visual studio did hang for 2-3 seconds occasionally with
message "Reading solution projects" popping up, but then it would unfreeze and go back to normal. It does prove that solution is the main reason of why Visual Assist fails. Also size of the log gives you a perspective on size of the project. It is a log from my second launch of Visual Studio, on first launch I realized built-in Intellisense database was still turned on, so I disabled it and restarted Visual Studio with "Rebuild VA database". I am not sure if it actually started over, but the most important is it didn't hang and used all available CPU cores. |
| feline |
Posted - Sep 07 2020 : 10:03:33 AM
If you have a mixture of different line endings inside the same code file this can cause the occasional odd problem, but it should not cause anything like this. Files with consistent line endings should be fine, regardless of the file ending being used.
It is possible if you have strange syntax that is not supported by Visual Studio, but is supported by your build tools that this would cause some problems. I would expect this to cause problems only in specific files though, not across the entire solution.
At least we now know that this problem is solution specific, and it is not something about your system, which narrows things down a bit.
VA logs should help to give us some clues, but the logs you got last time were basically empty, as if VA had never actually done anything.
I have run some tests here on VA logging, to give you some reference points.
Before starting I deleted the "va.log" file in my %TEMP% directory, to make sure I was starting from a known state.
My first test is just opening my standard test solution, which is a mixture of C, C++ and C# code. I loaded VS2019, and on the start screen selected "Continue without code", so I could get into the main IDE window, and then the VA options dialog, before anything else was done. I then enabled VA logging and then opened the solution. After VA had finished parsing I closed the IDE, and checked the generated "va.log" file.
This generated a 128kb log file, with 1,549 lines.
For my second test I loaded VS2019, and told VA to rebuild its symbol database. I then deleted the va.log file, and followed the same steps as above to make sure VA logging was enabled before loading the solution. This time the generated log file was 1,138kb in size, and has 20,800 lines.
If you try the same method, do you get a more complete va.log file?
How is your solution laid out in the file system? Does it all exist under a single root directory, or is it split into several different directory trees?
If you have a bit of time, can you please try loading VS2019, and instead of loading your solution normally, instead use the "Open a local folder" option, and then open either your full solution tree, or a part of your solution tree.
This will cause the IDE to load all of the source code in the directory, and all of its subdirectories, that you have selected, and VA will then parse this code, treating it as a solution.
If the problem is something in your SLN or VCXPROJ files then this will allow you to load all of your code without any problems. If instead the problem is being triggered by something inside the code files then you should be able to load at least some subdirectory trees of code without any problems.
I don't like this approach so much, since it puts a lot of the hunting back into your hands, but it should produce some helpful results fairly quickly, if you have time to try.
One final thought, you could try excluding certain file extensions from being parsed, via:
VA Options -> Projects and Files -> Extensions to ignore:
https://support.wholetomato.com/default.asp?W372
and see if this fixes the problem. Again, it is a way to possibly narrow down the files we need to consider and study. |
| Maksym.Bozhko |
Posted - Sep 07 2020 : 06:28:29 AM
So I did test qt source folder while using test profile Visual Studio you suggested earlier, It used all CPU, with plenty of activity in VaDbMtx.exe and worked fine in general. For my work project it hang as usual, I had some activity in VaDbMtx.exe when I launched visual studio(also test profile), but 0 activity after I opened my solution. It seems something about my solution confuses Visual Assist, this is a huge project with many quirks, I am not sure which one exactly is causing this. We have irregular file endings, we do not use visual studio IDE to build, we build from command line using a custom build system(mix of gn and makefiles).We have quite many macros to switch between platforms. We have many different file extensions under project, but most of sources are C++ and then some in C. Could you please list the most likely issues my project might have to cause such behavior? I can't really guess because I don't know how VA works inside. |
| feline |
Posted - Sep 04 2020 : 12:36:55 PM
Thank you for trying this. I had hoped it would produce a different result, but obviously not.
This might be a silly question, but how did you trigger the "analyze wait chain" command?
Doing a quick search tells me that this is offered in the context menu for the process in Windows Task Manager. But I never get this option, even when I have VS2019 parsing a very large C++ solution, with the CPU running at basically 100% usage.
So I am wondering if you triggered this by some other method, or if the command is not being offered to me because the IDE is not considered to be hung / frozen.
I have worked out a test we can both run, to compare results, if you have a few minutes to spare to try this. First up I downloaded the source code as a zip file from this github page:
https://codeload.github.com/qt/qt/zip/4.8
which gave me a 200meg zip file, holding nearly 10,000 cpp files.
Next I loaded Process Monitor, and set the filter to default, and then added the filter rule:
Process Name is VaDbMtx.exe then Include
and made sure it was actively monitoring my system. I then loaded VS2019 and then opened the folder I unpacked the zip file into, "C:\src\qt-4.8\" in this example.
I let the IDE and VA parse the code for a few minutes, and then closed the IDE. I am only using this folder as a common test solution that has enough code to require parsing for a few minutes, so we can compare behaviour on your system and my system. The CPU was running at 100% usage, across several cores, for basically all of this time, as expected.
5 minutes of parsing and monitoring returned just 229 entries in Process Monitor. About half of them are registry activity and the rest are file system activity. How does this compare to what you are seeing? |
| Maksym.Bozhko |
Posted - Sep 04 2020 : 09:09:09 AM
Feline, I tried as you suggested and I see absolutely no difference with test profile. Visual studio still hangs, still only 10% total CPU usage, I don't see any difference compared to my usual Visual Studio profile |
| feline |
Posted - Sep 01 2020 : 10:05:29 AM
The VA logs are virtually empty, it is as if VA has not really done anything at all. But if so, why is your single CPU core so very busy...
Assuming the Set Affinity setting doesn't help, can you please make sure all instances of the IDE are closed, and then run this command:
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\devenv.exe" /RootSuffix VATest
this will create and load a new, default profile for Visual Studio, so you will be asked which basic settings you want to use, and you should only have the standard, default IDE extensions installed. Visual Assist will not be installed into this profile, so you will need to go into the dialog:
IDE Extensions menu -> Manage Extensions
and install Visual Assist from the online extension store, to install VA for this test profile.
To load your normal, default profile just load the IDE normally. To return to this test profile again, pass the /RootSuffix command line switch when loading the IDE. You can run both profiles at the same time, next to each other, but this tends to get confusing, so is normally best avoided. If you export your IDE settings from your main profile you can them import them into the test profile.
Do you see the same problem when loading your solution into this test profile? I am trying to find out if the problem is somehow related to something in your IDE profile. I have seen profile specific problems occasionally, so it's possible. |
| feline |
Posted - Sep 01 2020 : 09:44:51 AM
I have the log files, thank you, I am looking at these now. The CPU usage does suggest that somehow parsing is limited to one thread / CPU, and this is why everything is running so slowly.
One thought on this while I look at the log files. If you run Process Explorer, locate "devenv.exe" in the list of running processes, right click on it, and select "Set Affinity..." you will get a dialog showing check boxes, one for each CPU core on your system, so 12 active. How many of these are checked?
This setting controls how many CPU cores the process is allowed to access. If, for some reason, the IDE has been set to only allow access to a single core then this would explain the CPU pattern you are seeing. |
| Maksym.Bozhko |
Posted - Sep 01 2020 : 07:17:02 AM
Hello, I did everything as you suggested, sent logs and CPU activity screenshot in email with this TOPIC_ID in subject. I launched VA, which caused Visual Studio to be "not responding", I let it hang while recording logs, looking at Resourse�monitor and process monitor. I didn't see any other process doing anything in my project directory except devenv.exe. Git.exe did something on sources folder once or twice, but not on VA database folder. The CPU activity pattern is one core is near 100% almost always while other cores have very low activity. |
| feline |
Posted - Sep 01 2020 : 04:43:31 AM
You would need to check the CPU load for each core to see if one core is running close to 100%. Given that code parsing should use multiple threads its not easy to see how this would happen, but it's possible.
To easily check this run "Resource Monitor", which is a built in Windows 10 tool. Select the CPU tab, and on the right hand side of the display you will see each CPU core listed individually, so you can see if one (or more) cores are running at nearly 100%. |
| Maksym.Bozhko |
Posted - Aug 31 2020 : 11:47:04 AM
Thank you, I will try that tomorrow. About being stuck in a loop, if 1 thread is stuck in a loop, my understanding is I will see 1/12 % CPU usage or 1/6 for 6-core system, which I do see(about 10-15%). The only time when I see 100% CPU is when I compile for example and I see many instances of CL.exe at the same time, which adds up to 100%. That's why I don't think 1 thread being stuck in a loop will be shown as 100% CPU. It is 100% of one core, but not 100% of 6 core on a processor with hyperthreading. |
| feline |
Posted - Aug 31 2020 : 10:15:42 AM
I am starting to wonder if the real problem is something inside the code is confusing our parser, basically leaving it stuck in a loop, but the problem with this theory is that this should be using lots of CPU time, rather than basically none.
The only way I can see this theory making sense is if one core is running at nearly 100%, and all of the other parsing VA can do is stuck, waiting for this thread to finish its loop. Are you seeing a pattern like this?
The only other idea that I have is that another process, for some reason, is blocking access to VA's symbol database directory. You have already checked anti-virus for this, but have you tried running Process Monitor, setting it to monitor the directory holding the VA symbol database, and checked to see if another process is also active in this directory? It might be worth a try, since it should be a fairly fast and simple test.
Let's try a different approach. Can you please change your VA settings to:
VA Options -> Projects and Files -> Upon opening a file, parse all other unparsed files in the same directory:
if Solution is not empty = ON
VA Options -> Performance -> Parse all files when opening a project = OFF
the idea here is that VA will parse files one directory at a time, as you open files in the editor. Does this help at all?
Can you please enable VA logging while you are seeing this problem, and then send me the log files. This page explains enabling logging, and where to look for the generated log files:
https://support.wholetomato.com/default.asp?W305
hopefully this will offer some clues as to what is happening here, and what VA is doing. Please send me the files via email:
[email protected]
including this thread ID or URL in the description, so we can match it up. |
| Maksym.Bozhko |
Posted - Aug 31 2020 : 04:44:01 AM
I had issue like this before, but when I left Visual Studio hanging like that, it would usually finish building database in the end. But now I am not sure if it is even doing anything, the memory that process is using keeps going up gradually little by little, but I don't see any changes written to database folder, maybe it keeps everything in memory and then writes everything to disk when done, I could try to reboot at the end of work day, and try to rebuild database and let it hang for entire night. |
| Maksym.Bozhko |
Posted - Aug 31 2020 : 04:25:39 AM
Feline, I tried both default database location and custom location and folder which is in anti-virus exceptions. I also tried rebuilding database, that didn't help. I also checked that my user has write permissions for database folder as it is mentioned in some of VA manuals. I tried reboot because I suspected another hidden instance of Visual Studio or mutex not released after crash somehow, that also didn't help. Keep in mind I am doing this on corporate workstation with quite restrictive policies, but I didn't see suspicious anti-virus activity or UAC popups to request any permissions. I also tried launching Visual Studio as administrator in case worker threads don't have enough rights or something like that. |
| feline |
Posted - Aug 28 2020 : 07:13:33 AM
I don't think I have ever seen VA get stuck waiting for its database like this before.
First up, could you reboot your system? I know you have checked to make sure you only have one instance of devenv.exe running, I am just wondering if somehow there could be another, related process running in the background that is causing problems.
Next, if this doesn't make any difference, can you please tell VA to rebuild its symbol database, via the button:
VA Options -> Performance -> Rebuild symbol databases
which will require an IDE restart. If this problem is caused by a corrupted VA symbol database then the rebuild should fix the problem.
You have already mentioned that anti-virus is set to ignore the VA symbol database folder, so that should not be a problem.
Have you changed the default location of the VA symbol database, via the setting:
VA Options -> Performance -> Use alternative directory for symbol database (requires restart)
just searching for any possible clues here. |
| Maksym.Bozhko |
Posted - Aug 28 2020 : 04:51:18 AM
Feline, I did what you suggested. I do see ~90 threads for devenv.exe in resource monitor, however devenv.exe still only uses about 8% of CPU and no other process is using any significant amount of CPU and total CPU usage is around 10%. I do see occasional disk usage spikes. I did "analyze wait chain" on devenv.exe, and it says it is waiting on VaDbMtx.exe. I have verified that I only have one instance of visual studio running. Does this mean all the parsing threads fight for one mutex so only one of them is running? |
| feline |
Posted - Aug 27 2020 : 07:00:13 AM
In process explorer you should simply see the devenv.exe process using nearly all of your available CPU, assuming something else is not acting as a bottleneck.
To try and get a better picture of what is happening, I have done the following test. Using Windows 10 and Visual Studio 2019. To avoid the IDE parser confusing the picture, I have set:
IDE tools menu -> Options -> Text Editor -> C/C++ -> Advanced -> Disable Database = True
and for VA I have set:
VA Options -> Performance -> Enable multithreaded parsing = ON
VA Options -> Performance -> Limit the maximum number of concurrent threads to X (requires restart) = OFF
VA Options -> Performance -> Thread priority = Normal
VA Options -> Performance -> Parse all files when opening a project = ON
to encourage VA to use as much CPU time as it can.
In Windows 10 I have loaded "Resource Monitor", and gone to the CPU tab, and set the right hand panel to show both the total CPU usage, and the CPU usage per CPU. Testing this on a machine with 3 CPU's.
Now I have loaded a large C++ solution, to make sure VA has to do plenty of parsing, giving me time to monitor what is happening.
In Resource Monitor "devenv.exe" is reporting 93 threads and between 88% and 95% CPU usage, with the system running at a total of 100% CPU usage.
Going to the Overview tab, the Disk usage keeps on spiking, as you would expect, but is generally fairly low. The network is basically idle, and memory is sitting at 87% Used Physical Memory.
Given so many threads are running under devenv.exe, picking out the ones we want and checking them will take a bit of care, but the Overview tab easily lets us see if something other than the CPU is acting as a bottleneck.
Can you please try setting the same settings, and loading Resource Monitor on your system while the IDE is parsing, to see what results you get?
If something other than the CPU is running at 100% this would help explain the problem you are seeing. |
|
|
