| Author |
 Topic  |
|
MikeGameDev
Junior Member
19 Posts |
 Posted - Jul 03 2018 : 11:05:10 AM
|
Hi,
I've got some very large (>300k lines) auto generated header files in my solution. VA attempts to parse these files which results in very long initialization times. Is there any way I can specify to VA to exclude these files form parsing?
Thanks |
|
|
feline
Whole Tomato Software
United Kingdom
18746 Posts |
Posted - Jul 04 2018 : 07:10:48 AM
|
This is something we are looking into. How are these files "defined"? Are they all in certain sub-directories? A certain project in your solution?
Is excluding them from your solution an option?
Do you often include these files? I am wondering if the option:
VA Options -> Performance -> Parse all files when opening a project
would help here. Turning this Off would stop the files being parsed automatically when you open your solution, but it would not stop the files from being parsed when you open them, or open a cpp file that directly includes them. But this would at least get them parsed "slowly" rather than all at once. |
zen is the art of being at one with the two'ness |
 |
|
|
MikeGameDev
Junior Member
19 Posts |
Posted - Jul 05 2018 : 3:34:43 PM
|
The directory structure is kind of a mess, I don't have any control over it. It's something like this:
FolderA
- MassiveAutogeneratedHeader.h // don't care about this
- ImportantFile.h // need to include this quite often, this file includes "MassiveAutogeneratedHeader.h"
There are several folders in our code base that follows that pattern. The files I want to exclude aren't actually in the solution.
Is there a way I can get VA to list all of the files it has parsed? Just to debug.
The parse all files option is a good tip, I'll try it out. Thanks |
|
|
|
MikeGameDev
Junior Member
19 Posts |
Posted - Jul 05 2018 : 3:58:31 PM
|
| Also, it's worth mentioning that I only us VS + VA for code navigation and coding, building is done by an external tool. If you've got any other performance tips for possible features I could disable, that you'd be greatly appreciated. |
Edited by - MikeGameDev on Jul 05 2018 4:02:17 PM |
|
|
|
feline
Whole Tomato Software
United Kingdom
18746 Posts |
Posted - Jul 06 2018 : 07:05:37 AM
|
Since these files are scattered around, and they need to be considered on a file by file basis, then the normal suggestions are not going to help much here.
What options are there for editing files in your solution? Are you able to edit the calling files? Here you would need to "hide" the call to "MassiveAutogeneratedHeader.h" inside "ImportantFile.h", and anywhere else that VA is going to find it. A search across all files in the solution for the #include line would give you the lines to update. The code that hides the include is:
#ifndef VA_IGNORE_THIS_FILE
#define VA_INCLUDE_HIDDEN "MassiveAutogeneratedHeader.h"
#else
#define VA_INCLUDE_HIDDEN ""
#endif
#include VA_INCLUDE_HIDDEN
So long as the file is not part of the solution its self, and you never open the file in the editor, VA should never find the file, so it will never parse it. |
zen is the art of being at one with the two'ness |
Edited by - feline on Jul 06 2018 07:06:35 AM |
|
|
|
feline
Whole Tomato Software
United Kingdom
18746 Posts |
Posted - Jul 06 2018 : 12:37:11 PM
|
I think I have found another approach, if you are able to modify the generated files. You need to make sure the files start with the code:
#if 1 == 0
_asm {
#endif
which gets VA to believe the entire file is assembler code, that we do not parse, so the file should not be parsed any further. Obviously this would need to be inserted at the top of the file every time it is re-generated, but if this works as expected you won't have to update all of the #include statements. |
zen is the art of being at one with the two'ness |
|
|
|
MikeGameDev
Junior Member
19 Posts |
Posted - Jul 06 2018 : 2:28:34 PM
|
| Hmm. The first option seems to work, I can do a solution wide find and replace to quickly make the change. Is there a way to verify if VA has parsed a file? |
|
|
|
MikeGameDev
Junior Member
19 Posts |
Posted - Jul 06 2018 : 3:05:37 PM
|
| Earlier you asked "Are they all in certain sub-directories?", are we able to exclude entire directories? |
|
|
|
feline
Whole Tomato Software
United Kingdom
18746 Posts |
Posted - Jul 06 2018 : 3:32:23 PM
|
No, there is no list of files that VA has parsed. If you know what symbols the file contains / defines you can try searching for these symbols in VA's Find Symbol dialog, bug this is a very slow and manual method.
Excluding directories is not something we can do yet, but it is something we are considering adding.
Depending on the directory structure you may be able to tell VA that the files in that directory are stable include files, so VA will only parse them occasionally, mainly just when re-building the symbol database. Or perhaps try and fool VA into thinking that the directories contain the Unreal Game engine, which is a stable include library that is part of the solution, so gets some special handling. But this assumes that there are directories where you are happy for VA to treat all of the code files as stable headers. |
zen is the art of being at one with the two'ness |
|
|
|
MikeGameDev
Junior Member
19 Posts |
Posted - Jul 09 2018 : 12:58:18 PM
|
Does VA parse code between inactive pre-processor conditions, for example:
#if 0
// code
#endif
Will the "code" between that block be parsed? |
|
|
|
MikeGameDev
Junior Member
19 Posts |
Posted - Jul 09 2018 : 1:16:26 PM
|
I should elaborate on that last question, each of the files I want to exclude has the following design:
#ifndef FILENAME_H
#define FILENAME_H
// lots of auto generated code
#endif
So if VA doesn't parse inactive blocks, I can just go into the the visual studio project settings and define the "FILENAME_H" symbols so the block will always be inactive. |
|
|
|
accord
Whole Tomato Software
United Kingdom
3287 Posts |
Posted - Jul 10 2018 : 10:19:43 AM
|
VA doesn't parse what's inside #if 0 ... #endif But when it comes to other preprocessor macros, VA parses both the #if and the #else branch. In your example, the #ifndef and #define branch.
What feline offered works for the same reason: both the #ifndef and #else branch will be parsed, while the compiler only sees the first branch. |
|
|
|
MikeGameDev
Junior Member
19 Posts |
Posted - Jul 10 2018 : 3:16:19 PM
|
| Hmm, okay. How can I "fool VA into thinking that the directories contain the Unreal Game engine"? The files I want to exclude aren't modified very frequently. Do I need to follow the steps under "Custom Directories for Non-Microsoft Build Processes" listed in the following link? https://docs.wholetomato.com/default.asp?W213 |
|
|
|
feline
Whole Tomato Software
United Kingdom
18746 Posts |
Posted - Jul 10 2018 : 5:30:42 PM
|
Without a good test case here to try out the idea, this is just an educated guess.
First try and make sure that all of the generated header files are in a single directory tree, since this is the structure we are expecting.
Now add a new dll project to your solution, and make sure it is called "UE4". Add all of the generated header files to this project. You will probably want to exclude the project from the build, just to make sure it does not effect your build process.
Make sure you have the VA options:
VA Options -> Unreal Engine -> Enable support for Unreal Engine 4 = On
When a solution contains a project named UE4 = True
and restart the IDE.
IF this works VA will still have to parse the generated header files once, but after this, should not be so keen to re-parse them. However, if the files are still changing all of the time, I am not sure if this will work as desired. |
zen is the art of being at one with the two'ness |
Edited by - feline on Jul 10 2018 5:34:20 PM |
|
|
|
MikeGameDev
Junior Member
19 Posts |
Posted - Jul 11 2018 : 12:14:26 PM
|
| Hmm, for now it looks like I need to disable VA as it seems to crash my VS. Is it possible that VS is running out of memory? Once VS memory usage hits the 3 GB range things become unstable, VS (the IDE itself) starts notifying me of exceptions. With VA disabled memory usage stays in the 1.2 GB range. |
|
|
|
feline
Whole Tomato Software
United Kingdom
18746 Posts |
Posted - Jul 11 2018 : 3:52:49 PM
|
This does sound like it could be an out of memory crash. The IDE is a 32bit process, so it is capped at a maximum of 4gig of memory, which is more like 3 in reality.
Is this a new problem? You have not mentioned this before.
Doing a rebuild of your VA symbol database MIGHT help here. We do try to keep VA's memory usage to a minimum, so this should not happen, but clearly something is going wrong here. |
zen is the art of being at one with the two'ness |
|
|
|
MikeGameDev
Junior Member
19 Posts |
Posted - Jul 11 2018 : 4:47:36 PM
|
I've been experiencing intermittent crashes, but I just thought it was a VS quirk, I never considered that it could have been VA. Memory usage with VS + dormant VA is still in the 1.2 GB area after a few hours, which is great. No crashes either. Once I enable VA it jumps up to 2.5+ GB after a few seconds. Stock Intellisense is actually holding up pretty well which is surprising.
Are the VA_M.idx and VA_D.idx files in the AppData folder the VA database? Looks like they are, as I can find symbols from the auto-generated header files in them. Currently mine are 130 MB and 90 MB in size, could I manually edit these file to remove some symbols I don't care about?
|
Edited by - MikeGameDev on Jul 11 2018 4:51:00 PM |
|
|
|
feline
Whole Tomato Software
United Kingdom
18746 Posts |
Posted - Jul 12 2018 : 2:40:23 PM
|
This page explains where VA's symbol database is stored, since this varies with the version of the IDE you are using, its not as simple as one given path:
https://wholetomato.fogbugz.com/f/page?W332
Deleting entries manually from the symbol database files is definitely not recommended. I don't actually know what effect this would have, but I would expect problems, since manually deleting sections of any structured data files tends to be a bad idea, unless you can get all of the references, and just the references you want removed.
How many files do you have in your solution?
If you open VA's Open File in Solution dialog (Alt-Shift-O) the title bar contains two numbers. The first number is the number of files currently listed, which changes as you filter the list. The second number is the total number of files in the list, which is normally the number of files in your solution. What is this second number? |
zen is the art of being at one with the two'ness |
|
|
|
MikeGameDev
Junior Member
19 Posts |
Posted - Jul 12 2018 : 2:51:38 PM
|
| The second number is 2353. The files I want to ignore are not in this list, which I believe is a good thing. |
Edited by - MikeGameDev on Jul 12 2018 2:53:01 PM |
|
|
|
feline
Whole Tomato Software
United Kingdom
18746 Posts |
Posted - Jul 13 2018 : 10:19:39 AM
|
This is a very small number of files for this much memory usage, and the parsing times that it seems you are running into.
Can you look in VA's Find Symbol dialog please, and see what the second number is when you set:
Find Symbol dialog -> Show only symbols defined in current solution = On
Find Symbol dialog -> Only classes, structs & namespaces = Off |
zen is the art of being at one with the two'ness |
|
|
|
MikeGameDev
Junior Member
19 Posts |
Posted - Jul 13 2018 : 5:17:24 PM
|
When VS memory usage is at 2.8 GB the number of symbols is 2,048,000.
Note that VA is still parsing. It's those auto generated files that are killing us.
EDIT: The number fluctuates, the highest I've seen was 4,096,000... |
Edited by - MikeGameDev on Jul 13 2018 5:18:41 PM |
|
|
|
feline
Whole Tomato Software
United Kingdom
18746 Posts |
Posted - Jul 14 2018 : 06:48:35 AM
|
Ouch! I had not realised just how massive these auto-generated files were!
Is editing the process that generates them an option? We have a couple of methods for getting VA to ignore the files, IF we can have them edited slightly before VA see's them.
The problem is that VA is designed to try rather hard to find all of the include files, since missing these tends to cause all sorts of problems. But here we have the opposite situation. |
zen is the art of being at one with the two'ness |
|
|
|
MikeGameDev
Junior Member
19 Posts |
Posted - Jul 14 2018 : 8:32:11 PM
|
What did you have in mind? I can make changes to my local copies of the file, but I can't change the generation process. I have tried the "asm" suggestion earlier, but that had an impact on the build process.
The "pre-defining file pre-processor" suggestion I gave earlier seems to be working well with the default VS Intellisense, I guess they don't parse inactive blocks.
I never thought I'd be in a situation where VA worked too well  |
|
|
|
feline
Whole Tomato Software
United Kingdom
18746 Posts |
Posted - Jul 16 2018 : 10:31:20 AM
|
We are considering changes to help VA ignore files or code that is not currently being compiled, but given the out of memory crash, instead of waiting for a change in VA to make this easier, I would like to get file ignoring working now, somehow.
Currently VA is parsing code that has been excluded from the build, which is by design, so that VA can help you with DEBUG or Linux code, for example, while working under Windows. But this means that we cannot use this method here to help VA ignore your code.
Do you know much about why the asm change effected your build process? In theory, since it is surrounded by the #if block, it should have no effect on a C or C++ compiler, but obviously that was not enough to make it safe. I have a variation of this that, in theory, will be safe and will work.
Can you please place the following lines at the top of your generated header files:
#define VA_IGNORE_REST_OF_FILE
VA_IGNORE_REST_OF_FILE
it doesn't matter to much what the #define evaluates to, so long as it won't effect your build process, so perhaps change the #define to a comment?
Now create a new text file called "va_stdafx.h", and make sure that it is in the same directory as your .SLN file. Do not add this file to your solution, VA looks for this file in this location automatically, and if we find it, it is parsed before anything else is parsed, so we can use it to help VA understand confusing code, or in this case, help to hide code from VA.
Now edit the file so it contains the line:
#define VA_IGNORE_REST_OF_FILE _asm {
and make sure that the file ends with a blank line. In the IDE now press the button:
VA Options -> Performance -> Rebuild symbol databases
and restart your IDE. For me, this is causing VA to ignore all code in the header files that start with the #define.
I am assuming that your entire solution directory does not get re-build, thus removing the "va_stdafx.h" file, is this a safe assumption?
This does still leave you having to modify the generated header files before VA see's them, which isn't ideal. |
zen is the art of being at one with the two'ness |
|
|
|
rogerio.ramos
New Member
USA
9 Posts |
Posted - Feb 25 2022 : 2:49:05 PM
|
| Is this still the better way to force VA to ignore code? I'm looking for ways to make VA to skip specific files and directories. |
|
|
|
feline
Whole Tomato Software
United Kingdom
18746 Posts |
Posted - Feb 28 2022 : 05:55:03 AM
|
Are the items you want VA to skip part of your solution, or part of your include directories?
If it is part of your include directories then changing:
VA Options -> C/C++ Directories -> Platform
to Custom, and then editing the include directories to include the ones you want, and exclude the ones you don't want VA to see should do a better job.
If you want VA to ignore parts of your solution to speed up parsing then setting:
VA Options -> Performance -> Parse all files when opening a project = OFF
may well do what you are looking for. |
zen is the art of being at one with the two'ness |
|
|
|
rogerio.ramos
New Member
USA
9 Posts |
Posted - Mar 09 2022 : 3:26:03 PM
|
| The files I want to skip are part of the solution because it is VS solution based on a directory. Under that directory, they are many projects. Some of them are test projects not actively being worked on. They duplicate code, sometimes with a stub implementation just to make them compile or with a minimalist implementation for test purposes. VA ends up showing duplicate definitions for many artifacts. It also include artifacts from those test projects I'm not interested in see. So I want to exclude full directories from the search, even though they show up in the solution because it is a directory based solution. |
|
|
|
feline
Whole Tomato Software
United Kingdom
18746 Posts |
Posted - Mar 10 2022 : 06:22:45 AM
|
Thank you for explaining, that makes quite a bit of sense. We are looking to add support for excluding files and directories when opening a folder, based on the file "settings.json" which the IDE already supports:
case=144472
I don't currently have an estimate for this, but we are hoping to add this soon, since opening directories is becoming more common. |
zen is the art of being at one with the two'ness |
|
|
|
me76
New Member
Canada
7 Posts |
Posted - Jun 22 2023 : 09:50:39 AM
|
quote:
Originally posted by feline
We are looking to add support for excluding files and directories when opening a folder, based on the file "settings.json" which the IDE already supports:
case=144472
The feature was officially announced last year, but never worked for me. I have checked the "Do not parse files excluded by settings.json" box in VA settings, and have tried all possible forms to exclude my_dir directory:
{
"files.exclude": {
"my_dir": true,
"my_dir/**": true,
"my_dir\**": true,
"my_dir\\**": true
}
}
with no luck. my_dir is located at top level in the project, at same level as .vscode directory. Why could it not work? |
Edited by - me76 on Jun 22 2023 09:51:58 AM |
|
|
|
feline
Whole Tomato Software
United Kingdom
18746 Posts |
Posted - Jul 05 2023 : 12:22:08 PM
|
The first exclude line should be working, so just using the file:
{
"files.exclude": {
"my_dir": true
}
}
should work as expected, based on all of my tests.
At the risk of asking a stupid question, you have made sure to place the file into a ".vscode" directory, so you end up with the file:
.vscode\settings.json
as described in the setting name:
VA Options -> Performance -> Do not parse files excluded by .vscode\settings.json (requires restart)
and you have turned this setting on and restarted the IDE? |
zen is the art of being at one with the two'ness |
|
|
|
me76
New Member
Canada
7 Posts |
Posted - Jul 05 2023 : 3:21:42 PM
|
Yes, the file is in .vscode directory. The setting in VA Options is on, and I have restarted Studio.
Our project is a cmake project and doesn't have *.sln file (*.sln and *.vcxproj files are generated by cmake in separate directory) - could it be the reason? |
Edited by - me76 on Jul 06 2023 01:29:26 AM |
|
|
|
Topic |
|
Selectively Excluding Files From Being Parsed
