| Author |
 Topic  |
|
|
foxmuldr
Tomato Guru
USA
414 Posts |
 Posted - Dec 20 2017 : 09:02:40 AM
|
I have a 20+K line .cpp file. It seems like at times it re-parses the entire file before it finds the thing I'm searching for with Ctrl+Alt+F as there is a long delay (15 seconds) before it shows any results. It only does this on this one file. If I go to search from other source files there is no delay.
I am using an older version of VAX, so it may have been corrected in newer versions. I just wanted you to be aware:
License: .. (1-user license) Support ends 2017.02.05
VA_X.dll file version 10.9.2118.0 built 2016.12.12
DevEnv.exe version 14.0.25420.1 Professional
msenv.dll version 14.0.25431.1
Comctl32.dll version 6.10.15063.0
Windows 10 10.0 Build 15063
8 processors (x86-64, WOW64)
Language info: 1252, 0x409
|
|
|
feline
Whole Tomato Software
United Kingdom
19021 Posts |
Posted - Dec 20 2017 : 09:09:57 AM
|
When you are searching from another file, is this file small?
Find References will start to show results as it finds them, and it also searches the current file first, since this is often the file you are most interested in seeing results for. So if the current file is the only very large file, then it will be the only file with a slow initial search.
Since Find References needs to check the scope and syntax, we need to parse and understand the code before doing a find references, which will take a few seconds in a 20+K line file. So if I understand correctly this is the expected behaviour. |
zen is the art of being at one with the two'ness |
 |
|
|
foxmuldr
Tomato Guru
USA
414 Posts |
Posted - Dec 20 2017 : 09:18:44 AM
|
quote:
Originally posted by feline
When you are searching from another file, is this file small?
Small. Other files are not typically larger than 3K lines.
quote:
Since Find References needs to check the scope and syntax, we need to parse and understand the code before doing a find references, which will take a few seconds in a 20+K line file. So if I understand correctly this is the expected behaviour.
Okay. |
|
|
|
feline
Whole Tomato Software
United Kingdom
19021 Posts |
|
|
mario123
New Member
3 Posts |
|
|
foxmuldr
Tomato Guru
USA
414 Posts |
Posted - Mar 19 2018 : 09:47:12 AM
|
The issue is continuing to plague me.
I have an include file which is about 25K lines long, and when I do a Shift+Alt+F search I can see that file is being reparsed by how it pauses during results display. Is it VAX re-parsing it or Visual Studio?
I ask because nothing changes in that file much anymore. But if I reference something that's defined in that file with a find or a Shift+Alt+R refactor, it seems to re-parse the entire file again resulting in about a 10-second delay in completing its re-parse before giving me UI control back. |
Edited by - foxmuldr on Mar 19 2018 09:53:43 AM |
|
|
|
foxmuldr
Tomato Guru
USA
414 Posts |
Posted - Mar 19 2018 : 09:55:22 AM
|
quote:
Originally posted by mario123
@foxmuldr see FastFind
https://marketplace.visualstudio.com/items?itemName=PureDevSoftware.FastFind
Thank you. I appreciate the reply. The description indicates it is a fee-based add-on. I don't mind paying for certain things, but there are facilities and utilities built-in to Visual Studio which are good enough for most of what I do ... and with VAX I have nearly everything else I need. |
|
|
|
feline
Whole Tomato Software
United Kingdom
19021 Posts |
Posted - Mar 19 2018 : 11:22:41 AM
|
VA should only re-parse the file if you actually make a change to the file. Simply scrolling through the file, or doing a find, should not be triggering VA to do a reparse.
If you are doing a Find References or Rename on something involving templates then VA may have to expand some of the templates while parsing, but off the top of my head that is the only exception.
However, if you modify a widely used header file, then the default IDE intellisense parser is likely to do a lot of reparsing, since it will tend to reparse any file that includes this header file. You can check if this is indeed the problem by disabling the IDE intellisense parser via:
IDE tools menu -> Options -> Text Editor -> C/C++ -> Advanced -> Disable Database = True |
zen is the art of being at one with the two'ness |
|
|
|
foxmuldr
Tomato Guru
USA
414 Posts |
Posted - Mar 19 2018 : 12:16:25 PM
|
quote:
Originally posted by feline
VA should only re-parse the file if you actually make a change to the file. Simply scrolling through the file, or doing a find, should not be triggering VA to do a reparse.
Agreed.
quote:
If you are doing a Find References or Rename on something involving templates then VA may have to expand some of the templates while parsing, but off the top of my head that is the only exception.
Unless there is something installed by default by VAX, that's not the case here as I don't use that feature.
quote:
However, if you modify a widely used header file, then the default IDE intellisense parser is likely to do a lot of reparsing, since it will tend to reparse any file that includes this header file. You can check if this is indeed the problem by disabling the IDE intellisense parser via:
IDE tools menu -> Options -> Text Editor -> C/C++ -> Advanced -> Disable Database = True
That is not the case in this case. I can launch Visual Studio 2015, open the solution, open the source file, and do a Shift+Alt+F Find on a symbol and it will take 10-15 seconds. Do it again, same thing. It is always like that.
I can post a video showing what I'm talking about if you'd like to see it. |
Edited by - foxmuldr on Mar 19 2018 12:18:47 PM |
|
|
|
feline
Whole Tomato Software
United Kingdom
19021 Posts |
Posted - Mar 19 2018 : 3:39:54 PM
|
Have you re-mapped your keyboard shortcuts? I ask since by default Alt-Shift-F is "Find References", not "Find References in File", which is likely to make a difference.
What are you doing a find on? A local variable? A parameter? A class member variable? How much of the file is VA going to need to scan and understand to return the correct matches? You are doing a Find References, which requires more work than a simple code match. Also, how many results is VA returning? |
zen is the art of being at one with the two'ness |
|
|
|
foxmuldr
Tomato Guru
USA
414 Posts |
Posted - Mar 22 2018 : 1:47:13 PM
|
quote:
Originally posted by feline
Have you re-mapped your keyboard shortcuts? I ask since by default Alt-Shift-F is "Find References", not "Find References in File", which is likely to make a difference.
No remapping. I am searching for all references to various symbols.
quote:
What are you doing a find on? A local variable? A parameter? A class member variable?
It varies. Typically it's a static function, global variable, or a local variable or parameter. But, it can be anything.
quote:
How much of the file is VA going to need to scan and understand to return the correct matches?
The whole project. The whole project is about 30K lines. The one file is about 26K lines now.
quote:
You are doing a Find References, which requires more work than a simple code match. Also, how many results is VA returning?
All of them. If I initiate the search for something in a small file, it returns the instances of the find quickly for that file, and other small files. It takes a long time to go through the large file.
I would've thought through its previous scan that it would've parsed everything that needed parsed, and then just need to re-parse any changes made. But even in these cases, there are no new changes being made. It is just a raw lookup to track from function into function into function, etc., to find the actual algorithm doing some work.
No code changes still exhibits the same slow behavior.
|
|
|
|
feline
Whole Tomato Software
United Kingdom
19021 Posts |
Posted - Mar 23 2018 : 09:41:06 AM
|
Sorry for my earlier replies, I did not understand the situation properly.
Find References starts by running the find in the current file first, only then moving onto the rest of the solution. This is designed to give you useful results as fast as possible, since the symbol obviously exists in the current file, otherwise you would not have been able to run the find.
Normally this makes sense, since most files are parsed quickly, and quickly return results. However, once we start scanning the solution as a whole, we don't know how long it will be before we find the next file with a result. The bigger the solution, the longer the likely delay.
A 26k line file is not a normal situation. There is no need to parse the file for changes, but this is not about parsing the file for changes. VA knows what symbols and information the file contains from the last time it was parsed, but this is "overview" information. We know the function names and their lines in the file, but we don't store and have in memory every detail of the file. If you do a Goto into this file nothing needs to be re-parsed, so it will be instant, since we will be going to one of the overview symbols.
But a Find References needs to scan the entire file looking for all references to your symbol, and then resolving each of those references to see if the reference is a valid match for your find. We don't need this level of detail about the file after the Find References has finished, so why keep it? Also, to have this detail about all possible symbols in the entire file is to basically compile the file in memory, and keep the details, updating them every time you make an edit and the file is re-parsed. This is not really helpful.
I hope this makes sense and explains what is going on here. |
zen is the art of being at one with the two'ness |
|
|
|
foxmuldr
Tomato Guru
USA
414 Posts |
Posted - Mar 23 2018 : 11:01:16 AM
|
quote:
Originally posted by feline
Sorry for my earlier replies, I did not understand the situation properly.
Find References starts by running the find in the current file first, only then moving onto the rest of the solution. This is designed to give you useful results as fast as possible, since the symbol obviously exists in the current file, otherwise you would not have been able to run the find.
Normally this makes sense, since most files are parsed quickly, and quickly return results. However, once we start scanning the solution as a whole, we don't know how long it will be before we find the next file with a result. The bigger the solution, the longer the likely delay.
A 26k line file is not a normal situation. There is no need to parse the file for changes, but this is not about parsing the file for changes. VA knows what symbols and information the file contains from the last time it was parsed, but this is "overview" information. We know the function names and their lines in the file, but we don't store and have in memory every detail of the file. If you do a Goto into this file nothing needs to be re-parsed, so it will be instant, since we will be going to one of the overview symbols.
But a Find References needs to scan the entire file looking for all references to your symbol, and then resolving each of those references to see if the reference is a valid match for your find. We don't need this level of detail about the file after the Find References has finished, so why keep it? Also, to have this detail about all possible symbols in the entire file is to basically compile the file in memory, and keep the details, updating them every time you make an edit and the file is re-parsed. This is not really helpful.
I disagree. I think it would be very useful. You could create an enhanced "VAX Code Definition Window" with all references instantaneously pulled up in near real-time, certainly as fast as VS's own Code Definition Window.
quote:
I hope this makes sense and explains what is going on here.
It makes sense, but I don't understand why you've designed it that way. Modern CPUs have massive amounts of memory (16GB+ is common now, and memory-mapped files are very fast to maintain a persistent definition across sessions, provided file contents haven't changed).
Once you parse the file you are able to determine context for every symbol from that parse, and store that information making find-all-references almost instantaneous. You should only need to re-parse changed lines, and those could easily back-in to their parent container (member function, static function, namespace, etc.) and have that entire block re-scanned as needed, still something that's reasonably fast.
I was very surprised to see it be so slow since it had already taken some time to scan and parse each file at startup.
I consider it an inappropriate design for a tool like VAX to not utilize that previously-parsed information and make the results available to us more quickly, and without any re-parsing of any kind for symbol searches.
I realize this large file size is atypical, but it would benefit all project sizes were a redesign considered to utilize that prior-parsed information, rather than just discarding it.
It causes me consistent pauses throughout the day when I work on this project, and it's frustrating at each point knowing that because it was previously parsed, it doesn't have to be this way.
|
|
|
|
feline
Whole Tomato Software
United Kingdom
19021 Posts |
Posted - Mar 23 2018 : 12:41:05 PM
|
As I understand it, the IDE's Code Definition window is just going to the definition, the same as VA's Alt-g. This is a very different "code knowledge" to what you are asking for, so sadly you are not comparing two similar things here.
As for memory, the IDE is a 32bit process, so the maximum amount of memory available to the IDE is 4gig, which is more like 3gig in reality. This is a memory limit that some of our users are already running up against. We also get users who are running into problems with the size of VA's symbol database on the hard drive, and are running out of space.
So yes, some more capacity is available, but only sometimes, and not for the bigger solutions we work with.
If you run a Find References for "FooBar->MemberOne()" then this is what VA will be looking for, and looking up. This won't give VA any information about "Fruit->Colour()", so the idea that VA has the information already in memory for other symbols really does not work, unfortunately.
If you want to keep the results around, you can simply clone the Find References Results window. Then the results are kept available, and you can even tell VA to refresh these results. So when it makes sense to keep information in hand and available we do.
The start up time, this is the time taken to parse out the overview information, not all possible information about all symbols. We work to keep this initial parsing stage as fast and light as possible, because again, in larger solutions this delay can be a significant factor. |
zen is the art of being at one with the two'ness |
|
|
|
foxmuldr
Tomato Guru
USA
414 Posts |
Posted - Mar 23 2018 : 2:03:24 PM
|
quote:
Originally posted by feline
As I understand it, the IDE's Code Definition window is just going to the definition, the same as VA's Alt-g. This is a very different "code knowledge" to what you are asking for, so sadly you are not comparing two similar things here.
As for memory, the IDE is a 32bit process, so the maximum amount of memory available to the IDE is 4gig, which is more like 3gig in reality. This is a memory limit that some of our users are already running up against. We also get users who are running into problems with the size of VA's symbol database on the hard drive, and are running out of space.
You can spawn a 64-bit process and communicate with it via named pipes to bypass the 4GB limitation in a 32-bit process. You can shunt the work to background tasks in that way, and receive and process I/O only in your IDE add-on.
quote:
So yes, some more capacity is available, but only sometimes, and not for the bigger solutions we work with.
If you run a Find References for "FooBar->MemberOne()" then this is what VA will be looking for, and looking up. This won't give VA any information about "Fruit->Colour()", so the idea that VA has the information already in memory for other symbols really does not work, unfortunately.
If it doesn't, then there's something awry. The data diagram required to successfully parse source code does back into all parent structures. The tree would be available to know all things about each symbol at each instance use.
quote:
If you want to keep the results around, you can simply clone the Find References Results window. Then the results are kept available, and you can even tell VA to refresh these results. So when it makes sense to keep information in hand and available we do.
The start up time, this is the time taken to parse out the overview information, not all possible information about all symbols. We work to keep this initial parsing stage as fast and light as possible, because again, in larger solutions this delay can be a significant factor.
I can "Rebuild Solution" in five seconds from click to generation of .EXE in a Debug build in Visual Studio 2015. The time it takes for VAX to parse the source file should be sufficient to derive context of every symbol, and maintain it.
It's just one more example of VAX doing only what you guys want to do, and limiting yourself because it's easier to do it the way you're doing than to do it right.
For the record, use of words like "sadly" and "unfortunately" are extremely patronizing in the context you use them above. I am authoring a compiler suite right now and I do know what's required to parse source code into its constituent parts. It's not trivial, but it is doable. It's why I comment on these things as I have done. |
|
|
|
feline
Whole Tomato Software
United Kingdom
19021 Posts |
Posted - Mar 24 2018 : 06:04:59 AM
|
VA does not get to compile any of the code, which is where some of our problems come from. The first big step the compiler gets to run is the pre-processor, which expands all macros, includes the header files, etc. We don't get to do this step, since it substantially re-writes all of the code files.
Yes, in theory, you can map the before and after lines to each other, but this requires that VA ignores all code that is not being compiled. VA is designed to be active on and help with code that is #if'd out, for example debug code in release mode. Also the compiler gets to stop at fatal errors, VA is designed to try and keep going, and do its best on all code past this point, since while you are editing most of the time the code is invalid, so won't compile.
Are you compiling a language that needs a pre-processor? It makes a lot of difference.
While your solution compiles very quickly this is often not the case. For example, how does this idea scale to 300,000 code files? These are the scales some of our users are working at, and as solutions get bigger, different considerations become important. |
zen is the art of being at one with the two'ness |
|
|
|
foxmuldr
Tomato Guru
USA
414 Posts |
Posted - Mar 26 2018 : 09:36:56 AM
|
I had occasion on Friday to switch from VS 2015 to VS 2010 for another reason, and in VS 2010 the find was nearly instantaneous.
I then went in and refactored the code so that related algorithms are grouped into their own #include files, so rather than having a single 26K line source file, I now have about 10 source files each 2K to 3K lines. Now the VAX Shift+Alt+F find takes about 2 seconds in VS 2015. In Visual Studio 2010 it's still nearly instantaneous.
Any ideas why it's so much slower in VS 2015?
Also, in VS 2015 the VS-built-in "Find all references" function took about 3 seconds the first time to find a global memory variable sparsely used, and subsequent finds were nearly instantaneous. Repeated on other static functions and global variables the finds were then also nearly instantaneous after the first one.
--
Rick C. Hodgin
|
Edited by - foxmuldr on Mar 26 2018 09:38:54 AM |
|
|
|
foxmuldr
Tomato Guru
USA
414 Posts |
Posted - Mar 26 2018 : 09:41:44 AM
|
quote:
Originally posted by feline
Yes, in theory, you can map the before and after lines to each other, but this requires that VA ignores all code that is not being compiled. VA is designed to be active on and help with code that is #if'd out, for example debug code in release mode. Also the compiler gets to stop at fatal errors, VA is designed to try and keep going, and do its best on all code past this point, since while you are editing most of the time the code is invalid, so won't compile.
I like that design. I've found it very useful. I remember requesting a function a long time ago that would allow VAX to be able to find #define tokens that are not currently or actively in use, such as have been commented out, or are in another #ifdef block that's not active in this particular compile.
quote:
Are you compiling a language that needs a pre-processor? It makes a lot of difference.
I'm using a C++ compiler, but compile mostly C-like code with very few C++ features in use, though I do use a few. I have "use precompiled headers" turned off, so it's doing a full recompile each time.
--
Rick C. Hodgin
|
Edited by - foxmuldr on Mar 26 2018 09:56:38 AM |
|
|
|
feline
Whole Tomato Software
United Kingdom
19021 Posts |
Posted - Mar 26 2018 : 11:15:39 AM
|
I don't see why changing the IDE would have such an effect on the speed of the Find References. Trying this here, triggering Find References in a 23,000 line file, I am getting a very similar speed of response in both VS2010 and VS2015.
If this was the identical file in both cases then I am not sure why the speed difference, but it almost sounds like some of the speed difference is down to the IDE, and that suggests the IDE's intellisense parser, but that is just a guess, especially since so far I am not seeing the same effect. |
zen is the art of being at one with the two'ness |
|
|
|
foxmuldr
Tomato Guru
USA
414 Posts |
Posted - Mar 26 2018 : 2:44:42 PM
|
quote:
Originally posted by feline
I don't see why changing the IDE would have such an effect on the speed of the Find References. Trying this here, triggering Find References in a 23,000 line file, I am getting a very similar speed of response in both VS2010 and VS2015.
If this was the identical file in both cases then I am not sure why the speed difference, but it almost sounds like some of the speed difference is down to the IDE, and that suggests the IDE's intellisense parser, but that is just a guess, especially since so far I am not seeing the same effect.
I'm not sure what it is. I am also observing an issue in Visual Studio 2015 when I single-step debug locally that it takes about 1 second between F10 or F11 presses to single-step line-by-line. In Visual Studio 2010 it is nearly instantaneous.
I am using Visual Studio 2010 and the Shift+Alt+F results always pop up within a second or so, which is highly usable.
I'm current on my service packs and what not for VS 2015. I'll try un-installing it and re-installing it.
--
Rick C. Hodgin
|
|
|
|
feline
Whole Tomato Software
United Kingdom
19021 Posts |
Posted - Mar 26 2018 : 2:50:00 PM
|
If you go to the dialog:
IDE tools menu -> Extensions and Updates...
and find Visual Assist in the list of installed extensions, and disable VA, which will require an IDE restart, does this stop these delays when stepping through debugging one line at a time? |
zen is the art of being at one with the two'ness |
|
|
|
foxmuldr
Tomato Guru
USA
414 Posts |
Posted - Mar 28 2018 : 6:00:52 PM
|
quote:
Originally posted by feline
If you go to the dialog:
IDE tools menu -> Extensions and Updates...
and find Visual Assist in the list of installed extensions, and disable VA, which will require an IDE restart, does this stop these delays when stepping through debugging one line at a time?
No. It's looking more and more like it's some issue with my local install.
|
|
|
|
feline
Whole Tomato Software
United Kingdom
19021 Posts |
Posted - Mar 29 2018 : 10:31:49 AM
|
To try and help you work out what is causing the problem with your local install, you can try running the command:
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\devenv.exe" /RootSuffix SpeedTest
which will create a new, default profile, and run VS2015 with this profile. This lets you test the IDE with a default profile, to see if its the profile or your machine in general that seems to be the factor.
To return to your default profile, just launch the IDE without the RootSuffix command line switch. |
zen is the art of being at one with the two'ness |
|
|
|
foxmuldr
Tomato Guru
USA
414 Posts |
Posted - Mar 30 2018 : 4:34:16 PM
|
quote:
Originally posted by feline
To try and help you work out what is causing the problem with your local install, you can try running the command:
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\devenv.exe" /RootSuffix SpeedTest
which will create a new, default profile, and run VS2015 with this profile. This lets you test the IDE with a default profile, to see if its the profile or your machine in general that seems to be the factor.
To return to your default profile, just launch the IDE without the RootSuffix command line switch.
Thank you.
I resinstalled VS2015 and tried as you indicated. It operates at the same speed on all. I tried closing every window in the debugger, and disabling every extension and even uninstalling most of them. Same results. I tried running as Administrator and regular user. Same results.
I'm at a loss.
VS2010 remains speedy so I have a work-around.
I will try VS2017 and see how it works. I also have VS2008 installed but I have to build a project to all of the VS2010 settings to go backwards.
Thank you for your help. |
|
|
|
foxmuldr
Tomato Guru
USA
414 Posts |
Posted - Mar 30 2018 : 10:10:04 PM
|
quote:
Originally posted by foxmuldr
I will try VS2017 and see how it works. I also have VS2008 installed but I have to build a project to all of the VS2010 settings to go backwards.
VS2017 is exactly as slow. I don't know why. There must be something installed on this machine which is affecting Visual Studio 2015 and later versions. It's actually unusable at this point.
VS2008 also runs as VS2010 at full speed. Neither VS2008 or VS2017 have VAX installed. |
|
|
|
feline
Whole Tomato Software
United Kingdom
19021 Posts |
Posted - Mar 31 2018 : 07:57:45 AM
|
As a first step, try disabling the IDE intellisense parser. This can have a significant effect on the IDE performance. Beyond this though, you will need to try tracking system resource usage, and talk to Microsoft directly about this, since it seems safe to say its nothing to do with VA.
IDE tools menu -> Options -> Text Editor -> C/C++ -> Advanced -> Disable IntelliSense = True |
zen is the art of being at one with the two'ness |
|
|
|
foxmuldr
Tomato Guru
USA
414 Posts |
Posted - Apr 02 2018 : 08:13:17 AM
|
quote:
Originally posted by feline
IDE tools menu -> Options -> Text Editor -> C/C++ -> Advanced -> Disable IntelliSense = True
Same speed with IntelliSense disabled. I will contact Microsoft. |
Edited by - foxmuldr on Apr 02 2018 08:30:04 AM |
|
|
|
feline
Whole Tomato Software
United Kingdom
19021 Posts |
Posted - Apr 02 2018 : 10:24:16 AM
|
| Normally intellisense is the cause of performance slowdowns, so that's a little unexpected. Hopefully they can help you to get to the bottom of this. |
zen is the art of being at one with the two'ness |
|
|
| |
Topic |
|
|
|
Slow find all references on large files
