| Author |
 Topic  |
|
|
Nox Metus
Senior Member
43 Posts |
 Posted - Jan 08 2018 : 12:53:07 PM
|
| After many years of usage I decided to depart from Visual Assist. It works only with relatively small projects. With my main project VAX leads to constant crashes of Visual Studio due to out of memory exceptions. I know it's Microsoft's fault: the limit is just 3GB, because it is still a 32 bit executable, with VAX Visual Studio quickly exhausts all available memory and eventually crashes, even if it doesn't, it is sluggish and works only partially. Nevertheless, doesn't matter whose fault it is, I had to stop using it. For the record, this doesn't help: https://docs.wholetomato.com/default.asp?W730 |
Edited by - Nox Metus on Jan 08 2018 12:55:13 PM |
|
|
accord
Whole Tomato Software
United Kingdom
3287 Posts |
Posted - Jan 09 2018 : 09:17:05 AM
|
We're constantly working on improving memory usage by different methods and planning to move more things out of process in the future. How large is your project? Visual Assist is able to work with larger projects, e.g. projects based on Unreal Engine.
Have you tried contacting support before to see if we can help? I'm wondering if your project is too big or if there is something you can do. Something that's not included in https://docs.wholetomato.com/default.asp?W730. |
Edited by - accord on Jan 09 2018 09:20:11 AM |
 |
|
|
Nox Metus
Senior Member
43 Posts |
Posted - Jan 24 2018 : 1:34:30 PM
|
How to measure the size of a project? The amount of lines doesn't matter, right? It's what in these lines that matters. I have 924 cpp files, not sure about all headers that got included, maybe around 10 thousands. Do you have some debug capabilities to spit out some metrics? I will make an experiment for you.
I don't want to compromise the capabilities in order to diminish the memory footprint. I moved my project to 64-bits Eclipse. Eclipse is a one huge glitch that never fully works and requires a lot of effort to setup a project, but it works, albeit slow. A typical memory footprint for Eclipse with my project loaded and indexed is 5.5 GB.
Moving things out of process is a good idea. I would like to know when you do it. |
Edited by - Nox Metus on Jan 24 2018 1:43:41 PM |
|
|
|
feline
Whole Tomato Software
United Kingdom
19014 Posts |
Posted - Jan 24 2018 : 2:26:20 PM
|
Apologies for the many problems you have had here.
Normally the number of files in a solution gives a sense of the size of the project, and 924 cpp files really is not that large compared to some projects people use without these problems. So clearly this is not a useful measure here.
One interesting measure would be the number of symbols found, shown by VA's Find Symbol dialog.
For comparison, a new default, so no game at all yet, Unreal Engine game project is reporting 22,820 files in total, 7,880 cpp files, and 853,775 symbols in total. This test solution runs happily for me, and is a lot smaller than any real game will be.
Do you know if you ever disabled the IDE intellisense? This is step 1, but I wonder how you did this, and which version of the IDE you were working with. |
zen is the art of being at one with the two'ness |
|
|
|
Nox Metus
Senior Member
43 Posts |
Posted - Jan 24 2018 : 5:32:45 PM
|
Without IntelliSense class view is not available and some other features I don't remember now. I don't want to compromise capabilities to reduce the memory footprint. In Eclipse I have everything.
Regarding the size of the project. When the VS IDE starts, it reaches the memory limit in a few seconds, the symbols count at that moment is around 32000. Then the count goes up a little bit. Then it resets to 0, the memory stays occupied, VS becomes sluggish, then it crashes. Without Visual Assist it works, but I don't like the limited capabilities of the bare IDE. With ReSharper it works a tiny bit better, but I don't like ReSharper. |
|
|
|
feline
Whole Tomato Software
United Kingdom
19014 Posts |
Posted - Jan 25 2018 : 12:19:06 PM
|
A few seconds?! This is extremely fast, and when combined with the relatively small number of files and symbols makes me wonder if you could have a corrupt IDE symbol database file. This would sort of explain what you are seeing, but this is not normally how it manifests.
This is easy to test, since if you close the IDE and then delete the file, it will be re-created next time you load the IDE and your solution.
For VS2010 and VS2013 this is the .SDF file in the root directory of your solution.
For VS2015, under the root directory of the solution, look for the ".vs/<SOLUTION-NAME>/v14/ directory, which contains a .SUO file.
For VS2017, under the root directory of the solution, look for the ".vs/<SOLUTION-NAME>/v15/ directory, which contains several different files.
If this makes no difference, could you please try pressing the button:
VA Options -> Performance -> Rebuild symbol databases
and reload the IDE. The rebuilding will take a few minutes, but should only take a few minutes. |
zen is the art of being at one with the two'ness |
|
|
|
Nox Metus
Senior Member
43 Posts |
Posted - Jan 25 2018 : 2:17:05 PM
|
| Sorry, I have already tried all these dances with a tambourine (a kind of magical dances a shaman does in a hope to cure a disease without medicine and knowledge). I was very reluctant to give up, believe me. |
|
|
|
feline
Whole Tomato Software
United Kingdom
19014 Posts |
Posted - Jan 25 2018 : 3:54:44 PM
|
That's fair, I had to ask though, since not everyone tries these sort of checks before reporting a bug.
Are you given the option to save out a mini dump or full dump file when this out of memory crash happens? I am just wondering if we can try to figure out what is causing VA to use such a massive amount of memory, on a solution that on paper does not sound that large. Clearly the file and symbol counts are just not good metrics here.
If you were able to get a dump file, could you send it to us, via the form:
http://www.wholetomato.com/support/contact.asp
including this thread ID or URL in the description, so we can match it up. |
zen is the art of being at one with the two'ness |
|
|
|
Nox Metus
Senior Member
43 Posts |
Posted - Jan 29 2018 : 6:17:32 PM
|
I would be happy to make a memory dump, however it will contain the text from the project, right? The code doesn't belong to me, but to my employer.
Do you have a debug version of your DLLs that logs some diagnostic? I will run it and send back the log, if it doesn't have our proprietary information. |
|
|
|
feline
Whole Tomato Software
United Kingdom
19014 Posts |
Posted - Jan 30 2018 : 08:58:11 AM
|
There are a couple of useful options for logging, but any logging is likely to contain some symbol names from your code. VA log files are a good place to start, but will include symbol names when parsing is being logged.
For these, if you can load the IDE, but not load your solution yet. Then turn on VA logging:
VA Options -> Performance -> Enable logging
then load your solution. Hopefully something useful will be written out before the crash. This page explains locating the log files that are generated:
https://support.wholetomato.com/default.asp?W305
Another option is VMMap from Sysinternals:
https://docs.microsoft.com/en-us/sysinternals/downloads/vmmap
this is specifically designed to record the memory usage information. Can you please run VMMap. When it loads cancel the "Select or Launch Process" dialog, and then set:
VVMap -> Options menu -> Trace Snapshot Interval -> 1 second
now use Ctrl-P to return to the "Select or Launch Process" dialog, and go to the "Launch and trace a new process" tab. Set the path to the IDE in the "Application" field, but leave the arguments blank. Accept the dialog, which will now load the IDE. Once the IDE has loaded, please load your solution.
As I understand it, each snapshot of memory usage is captured by VVMap as it goes along, so it won't cause any problems with the IDE crashes, we just won't get any more snapshots. Can you then please save out the memory usage information as a ".mmp" file, which I can then look at here?
Please submit the files via the form:
http://www.wholetomato.com/support/contact.asp
including this thread ID or URL in the description, so we can match it up.
You will be able to review the file before sending it to me, but it should not tell us much of anything about your actual source code, but it should give us a good idea of where all of the memory is going. |
zen is the art of being at one with the two'ness |
|
|
|
Nox Metus
Senior Member
43 Posts |
Posted - Jan 30 2018 : 3:52:19 PM
|
Of course I ran VMMap :). A static snapshot didn't reveal anything interesting, these 3GB are just marked as �private data�. When I tried tracing, VMMap crashed. :)
Another option can be: you send me the debug symbols, we connect via Skype and you guide me through debugging VAX and getting the information you want. I can sign an NDA for whatever I will see in the VAX, of course. |
Edited by - Nox Metus on Jan 30 2018 3:53:06 PM |
|
|
|
Nox Metus
Senior Member
43 Posts |
Posted - Jan 30 2018 : 4:16:32 PM
|
| I am checking va.log. If I add up all FormatFile::ParseFilecode.length() = N, this will give an estimation of memory VAX using for symbols DB? |
|
|
|
Nox Metus
Senior Member
43 Posts |
Posted - Jan 30 2018 : 4:18:47 PM
|
Closer to the end of the log file I of course see
Visual Assist memory allocation failed. Features may not work correctly.
(47 t)
ERROR: HashTbl::AddReserve CMemoryException (95000) |
|
|
|
sean
Whole Tomato Software
USA
2817 Posts |
Posted - Jan 30 2018 : 5:02:33 PM
|
The W730 wiki page is about all that we have at this time.
We are actively working on moving Code Inspection out of process. If you already have it disabled, that won't be a direct benefit to you, but the exercise will help guide us on moving other systems out of process. That will eventually benefit scenarios like yours. |
|
|
|
feline
Whole Tomato Software
United Kingdom
19014 Posts |
Posted - Jan 31 2018 : 07:34:16 AM
|
OK, which types of tests have you already run? What is left to suggest?  I am impressed with how thorough you have been, I just need to stop suggesting things you have already tried.
Let's try a different approach. Which programming language, or programming languages does your solution use? I know C++, but is it pure C++?
How many projects does it contain?
If it contains more than one project, could you try just loading one project at a time, independently of the main .SLN file, and see if one project has a massive memory load? I don't want to just assume that the memory load is evenly distributed across all of your code.
It's also possible the memory load is somehow tied to your library's, so stable include directories, and not the project its self.
Have you checked the memory load of the IDE it's self, with VA disabled? I assume so, but I am not seeing a figure here. I am just wondering if the IDE on it's own is using a "lot" of memory. If the IDE was hitting 2 gig of memory with VA fully disabled that would be interesting, but not automatically helpful. |
zen is the art of being at one with the two'ness |
|
|
|
Nox Metus
Senior Member
43 Posts |
Posted - Jan 31 2018 : 3:13:54 PM
|
Thank you. :)
I have just one project in the solution. It's a C++ makefile project. What do you mean by libraries? External header files? Yes, there are quite a few. But the symbol count was low at the moment of crash, right?
You were right in telling me what to do :). I have very interesting results. If I just disable VAX, the studio still behaves the same way: quickly reaches 3GB mark, then becomes sluggish, then crashes. But! When I completely uninstalled VAX, the memory footprint is just 520MB. I am curious what VAX is using memory for when it is disabled? Syntax highlight? |
Edited by - Nox Metus on Jan 31 2018 3:21:01 PM |
|
|
|
Nox Metus
Senior Member
43 Posts |
Posted - Feb 01 2018 : 4:12:55 PM
|
| You can disregard my last result. I think I mess something up. It doesn't matter if VAX is just disabled or uninstalled. Without VAX the memory consumption by the IDE is around 500MB. |
|
|
|
feline
Whole Tomato Software
United Kingdom
19014 Posts |
Posted - Feb 01 2018 : 4:46:45 PM
|
I was going to ask how you disabled VA. If VA was enabled at some point with the solution loaded, then VA will still have parsed your code, and loaded information from it's symbol database into memory. So there is a difference between "enabled at first, but then told to stop interacting with the IDE" and "never loaded with the solution loaded."
I was getting all curious there, with 3gig memory usage with no VA, but it's not to be.
Only a single project, and a makefile project. Are you using the IDE / Microsoft compiler for this makefile project? Or an external compiler, and just using the IDE as an editor? It shouldn't matter at all, but if you are using an external compiler I am wondering if you are using another extension to help integrate with it? Still looking for something "different" that would give me a clue as to the problem here.
For the libraries, I am mainly thinking external header files. It depends on what external dependencies your project draws on, and if VA might be trying to parse any of these. For example I remember coming across .TLH files, which were being generated from dll's, if memory serves me correctly. VA is parsing the .TLH files, for the interface information they provide.
If you were referencing several big libraries of external header files, e.g. Boost, this could add a lot of information for VA to consider, without showing up in the size of the project its self.
Do you do a lot of template programming? Perhaps very complex or recursive template code?
Do you have a "va_stdafx.h" file in the root directory of your solution? It's a VA helper file, so it might somehow be a factor.
One last though for now, is your code all stored under a single directory? It would make sense for this to be the case. But if so, is all of the code under this directory your makefile project code, or does the directory tree contain code from several different projects? I am wondering about a few simple tests where dummy solutions are used on just one sub-directory tree of your project, to try and pin down if the memory problems are related to a particular part of your code base.
VS2017 has the menu option:
IDE File menu -> Open -> Folder...
or alternatively I have a script file I can send you that makes test (they won't compile) solutions from the code files under a particular directory tree. |
zen is the art of being at one with the two'ness |
|
|
|
Nox Metus
Senior Member
43 Posts |
Posted - Feb 01 2018 : 10:14:36 PM
|
quote:
Originally posted by feline
if you are using another extension to help integrate with it?
No.
quote:
Originally posted by feline
If you were referencing several big libraries of external header files, e.g. Boost, this could add a lot of information for VA to consider, without showing up in the size of the project its self.
I would have agreed, but the symbol count is low when the IDE crashes.
quote:
Originally posted by feline
Do you do a lot of template programming? Perhaps very complex or recursive template code?
Contemporary C++ is impossible without templates. �Very� is poorly defined. The bulk of the project has very moderate amount of templates.
quote:
Originally posted by feline
Do you have a "va_stdafx.h" file in the root directory of your solution? It's a VA helper file, so it might somehow be a factor.
No.
quote:
Originally posted by feline
One last though for now, is your code all stored under a single directory?
I am not sure what you mean. There is always a directory in the hierarchy of directories, which will contain everything.quote:
Originally posted by feline
But if so, is all of the code under this directory your makefile project code, or does the directory tree contain code from several different projects?
How it matters? .vcxproj has only a specific set of files.
quote:
Originally posted by feline
I am wondering about a few simple tests where dummy solutions are used on just one sub-directory tree of your project, to try and pin down if the memory problems are related to a particular part of your code base.
VS2017 has the menu option:
IDE File menu -> Open -> Folder...
or alternatively I have a script file I can send you that makes test (they won't compile) solutions from the code files under a particular directory tree.
This is what I call dances with tambourin. I have a similar script. Still sounds like it will require a lot of my time, especially considering it's... dancing... i.e. it's an assumption and we don't understand what we are doing.
I tried to run the IDE with VAX disabled, let the IntelliSense parse the files and then enabled VAX. The memory consumption didn't sky rocketed. However the IDE crashed after some time anyway.
I am tired of not targeted random trial-and-error experiments. Dancing with tambourin is exhausting. Unless you provide me with tools for targeted investigation, I am done here. |
Edited by - Nox Metus on Feb 01 2018 10:17:07 PM |
|
|
|
feline
Whole Tomato Software
United Kingdom
19014 Posts |
Posted - Feb 02 2018 : 08:10:52 AM
|
Would you be willing and able to send us the log files VA generates when loading your solution? This will contain some symbol information from your solution, but won't contain blocks of code. This might provide some information about what is going wrong here.
This page explains turning on VA logging, and locating the generated log files:
https://support.wholetomato.com/default.asp?W305
As for smaller projects, I agree this is just a set of blind tests, and I appreciate it feels like a total waste of time. Would you be able to test a version of your makefile project without any settings for include directory paths? I am just wondering if removing all of the library header files will have much of an effect here? The number of files and symbols in your project just does not explain the memory usage, and C++ template programming is normal, and something a lot of our users do. I am simply wondering if you have somehow managed an extremely complex level of template programming. |
zen is the art of being at one with the two'ness |
|
|
|
Nox Metus
Senior Member
43 Posts |
Posted - Feb 02 2018 : 12:56:20 PM
|
quote:
Originally posted by feline
Would you be willing and able to send us the log files VA generates when loading your solution? This will contain some symbol information from your solution, but won't contain blocks of code. This might provide some information about what is going wrong here.
Sorry, cannot do it. Even symbols contain valuable information about future products. This software influences hundreds of million of people. This is absolutely out of question.
quote:
Originally posted by feline
As for smaller projects, I agree this is just a set of blind tests, and I appreciate it feels like a total waste of time. Would you be able to test a version of your makefile project without any settings for include directory paths? I am just wondering if removing all of the library header files will have much of an effect here? The number of files and symbols in your project just does not explain the memory usage, and C++ template programming is normal, and something a lot of our users do. I am simply wondering if you have somehow managed an extremely complex level of template programming.
I don't see a point in this experiment. If I don't specify include paths, nothing will be be parsed, while we are trying to investigate a case where parsing works. Give me an instrumented version of VAX that collects the statistics you need: the number of symbols, size of templates instantiations, memory usage, number of files, size of files, etc... I will collect the statistics and send it back to you. This will be much less waste of time on both sides. |
|
|
|
Nox Metus
Senior Member
43 Posts |
Posted - Feb 02 2018 : 2:56:50 PM
|
The biggest file after a crash in your symbol database directory is VA_N.tmp. It mostly contains PP macros from my project and some typedefs. All of them are repeated several times. The count of unique symbols in this file is ~1e6. I have a lot of very long macro names (up to 100 symbols).
So I bet it's your preprocessor db that uses the memory. I am still waiting for an instrumented version of VA_X. Let's be done with blind testing. |
|
|
|
Nox Metus
Senior Member
43 Posts |
Posted - Feb 06 2018 : 02:34:42 AM
|
| Hey guys. Are we done here? |
|
|
|
ChrisG
Whole Tomato Software
USA
299 Posts |
Posted - Feb 06 2018 : 02:57:38 AM
|
Hi Nox,
Apologies for the delay, besides sending in logs and a crash dump there isn't much we can do about running out of memory at the moment if you have tried everything on the W730 wiki page.
You say you would be willing to send a dump if it did not contain any proprietary information?
A minidump contains minimal memory, but will allow us to see a callstack. Would a minidump be acceptable? If so, please capture one and send it to us using information from the following page. You could also examine the dump yourself after you create it to validate that it does not contain excessive information.
https://docs.wholetomato.com/default.asp?W303
Even with the dump, we may simply have to wait until we can pull VA out of process for these cases. |
|
|
|
feline
Whole Tomato Software
United Kingdom
19014 Posts |
Posted - Feb 06 2018 : 08:20:37 AM
|
Can you give me some guidance on how you are using these PP macros, and what they are doing? VA doesn't have a pre-processor in the traditional sense, since the pre-processor gets to re-write the code, to expand all of the macros, and that's not something we get to do.
Still, having the macro parsing take up all of the memory is a useful clue, so I have been trying to trigger this sort of problem here, but so far without success.
I have defined a macro that creates a struct with 500 members, and declares an instance of the struct. Then I call this macro 500 times, so generating 500 * 500 struct member variables. But this is not having any obvious effect on the IDE's memory usage. So I assume you are doing something else. So if you could point me in the right direction I will try and get heavy memory usage here. If this is caused by macros being expanded into a large number of symbols in VA's symbol database, or a large number of repeats of a few symbols, we should be able to fix this with some entries in a "va_stdafx.h" file, which is used to help VA with macros and anything else that can confuse our parser. |
zen is the art of being at one with the two'ness |
|
|
|
Nox Metus
Senior Member
43 Posts |
Posted - Feb 07 2018 : 6:37:47 PM
|
quote:
Originally posted by ChrisG
if you have tried everything on the W730 wiki page
I have mentioned earlier that I am not interested in resolving the issue by compromising the capabilities.
quote:
Originally posted by feline
Can you give me some guidance on how you are using these PP macros, and what they are doing?
I sent you a PM. |
Edited by - Nox Metus on Feb 07 2018 11:23:14 PM |
|
|
|
feline
Whole Tomato Software
United Kingdom
19014 Posts |
Posted - Feb 08 2018 : 2:57:48 PM
|
| Thank you for the PM, I see now that I have been looking in the wrong places to try and help you with this. I have replied to your PM via email. |
zen is the art of being at one with the two'ness |
|
|
|
foo64
New Member
8 Posts |
Posted - Dec 03 2019 : 6:05:25 PM
|
Are there any updates about moving VAX out of process?
I'm working on a large solution and VAX causes Visual Studio to crash every hour or two, so I regrettably had to stop using it. This happens in Visual Studio 2017 and 2019 for the same solution.
I'm using VAX 2316. |
|
|
|
feline
Whole Tomato Software
United Kingdom
19014 Posts |
Posted - Dec 04 2019 : 06:31:28 AM
|
We are still working on this. But still, you should not be having this problem.
To try and get a sense of how large your solution is, how many files do you have in your solution?
If you open VA's Open File in Solution dialog (Alt-Shift-O) the title bar contains two numbers. The first number is the number of files currently listed, which changes as you filter the list. The second number is the total number of files in the list, which is normally the number of files in your solution. What is this second number?
Assuming that these crashes are related to running out of memory, can you please try the steps listed here, and see if this has any effect on your system stability?
https://support.wholetomato.com/default.asp?W730 |
zen is the art of being at one with the two'ness |
|
|
|
sean
Whole Tomato Software
USA
2817 Posts |
Posted - Dec 04 2019 : 1:40:30 PM
|
| Moving various parts of VA out of process is a work in progress, however the main focus on that work is reducing memory consumption in VS. Getting a crash dump of the crash (https://docs.wholetomato.com/?W303) would help us either confirm the problem is due to memory exhaustion, or point us to some other possible cause that might not be affected by the ongoing work. |
|
|
| |
Topic |
|
I removed Visual Assist due to crashes
