Memory Leak?

[Guest Buzby]

Hi there,

 

I am having a few problems with regards to the stability of my VoiceGuide installation now that it has been in production use for c. 6 months.

 

The problem seems to be that after a few days of runtime my system runs out of memory as a result of the amount being used increasing steadily until the system can no longer cope. It is quite a powerful machine, twin Xeon processors, 2.5Gb RAM, 200Gb RAID-5 disk etc.

 

I have investigated memory/resource usage using Perfmon and noticed that the number of process handles used by VG seems to increase continuously, only being reset if VG is stopped and restarted. Right now it is using 44400 handles, compared 100-600 used by each of the other processes.

 

I can only assume this number of handles is a result of my extensive use of VBScript modules in my VG application. An average phone call will lead to in excess of 100+ VBScript modules being executed. Each VBScript module carries out a number of COM calls into my Lotus Domino 7 database.

 

So, I'm guessing that this is where the leakage is occuring. I don't expect anyone to debug my software, but I just wondered if you could give me any pointers for making sure that all memory and handles are released as each VBScript module executes.

 

Best wishes.

[SupportTeam]

Which version of VoiceGuide are you using?

 

(If it's not the latest then please update to latest version as our tests here are done mainly with the latest versions only)

 

We don't see any handles usage type problems which you describe on our test systems - you can see some load test results here:

http://voiceguide.com/forums/index.php?showtopic=3710

In the load test results provided, there are a couple of screenshots on which you can see the total number of handles used by the whole system at beginning of load test and after 5000 calls are made and received by the system. The number of handles was essentially unchanged.

 

I can only assume this number of handles is a result of my extensive use of VBScript modules in my VG application. An average phone call will lead to in excess of 100+ VBScript modules being executed. Each VBScript module carries out a number of COM calls into my Lotus Domino 7 database.

Guess the problem could be caused by what is called by VBSCripts...

The test script in the load test uses a VBScript module as well, so looks like the problem would not be caused by the process which invokes the VBScript itself.

So, I'm guessing that this is where the leakage is occuring. I don't expect anyone to debug my software, but I just wondered if you could give me any pointers for making sure that all memory and handles are released as each VBScript module executes.

Do you create any objects in VBscript which should then be released using the "Set x = Nothing" type command?

[Guest Buzby]

Hi many thanks for your reply so far.

 

I am using the very latest version of VG for Dialogic (6.0.3107) across all of my servers, so no problems there.

 

Do you create any objects in VBscript which should then be released using the "Set x = Nothing" type command?

 

How critical is this? I'm creating lots of objects, and I have used this command across all my VBScripts, however, I have not white-box tested every case to make sure the Set x=Nothing command runs for every object that was created regardless of the outcome of the script.

 

Interesting, I've just had a look at the VGMulti.exe task in 'Sysinternals - Process Explorer' (an excellent Task Manager replacement) and found that the vast majority of handle usage points to 'Non-existent Process (123456)'. There are thousands of these handles all with different IDs. When I investigate these process IDs they indeed point to previous WScript.exe tasks that have since completed. Does this mean that by not doing a 'Set x = nothing' for every single object created that VG will not release the handle when each WScript.exe instance terminates?

 

If 'Set x = nothing' is critical to keeping handle count, and thus memory allocation, low, then I may need to do a proper analysis of all my VBScript modules because there is a lot of code in there!

[SupportTeam]

Does this mean that by not doing a 'Set x = nothing' for every single object created that VG will not release the handle when each WScript.exe instance terminates?

The objects created in VBScript should be released by Windows' VBScript interpreter/engine upon VBScript completion. Whether this actually happens for all the different objects is another story. There are many comments out there from other VBScript programmers which mention that doing a 'Set x = nothing' is a good idea...

 

How to you end the VBScripts, do you use the Run_ResultReturn COM function or do they end in a different way?

 

Attached .exe has some checks when doing handle allocation within VoiceGuide itself, maybe sometimes the handles in VG do not get released properly depending on how the VBScript ends...

 

Please try using the attached .exe on your system and see if the handle counts on your system still increment over time.

 

Could you also please .ZIP up some .vgm traces from your system after you have used the new .exe for a while (few dozen calls or so) and post them here.

VgMulti_6.0.3110.zip

[Guest Buzby]

Hiya,

 

I have run the new 6.0.3110 exe for a while and attach the VGM trace below. Sorry about the size of the file, our server creates quite a bit of traffic!

 

From what I can tell, the handle count is now staying a bit more consistent, although there is still some 'non-existent' ones showing. VG seems to be tidying these up a bit more effectively now, albeit not perfectly.

 

I hope the attached trace helps, and look forward to hearing your reply.

 

Best wishes.

0508vgm.zip

[SupportTeam]

Thanks for the trace. It let us see where the leak was happening on this system.

 

The .exe provided previously should be limiting the number of old handles open by the system to at most one per line.

 

The .exe attached here should be even better at reducing the number of outstanding old handles at any one time.

 

If you could post the trace form this new .exe as well then we could confirm how well the handles are cleaned up on this system.

VgMulti_6.0.3111.zip

[Guest Buzby]

Hi again,

 

Unfortunately we have had to revert to 6.0.3110, as I think we are having stability troubles with 6.0.3111.

 

6.0.3111 seemed to run fine until our server started getting busy with lots of phone calls. Then, randomly, we started receiving lots of 'WScript.exe has generated an error and is being closed down' messages, with each call terminating suddenly.

 

I can't say 100% sure whether this was the case before 6.0.3111, however, we have managed approximately 2000 genuine calls (taking 5000 minutes of time) with 6.0.3110 and it still runs successfully now, whereas 6.0.3111 could only managed a couple of hundred calls before it started falling over.

 

My rough stab at the possible cause surrounds the 'handle checking' code you have implemented - I wondered whether it is at all possible whether the checking code could cause the WScript.EXE handles to be shut down prematurely; i.e. whilst they are still running on an active line. Would this cause an error such as this?

 

Best wishes.

[SupportTeam]

v6.0.3111 tries to close and free the handle it uses to track the VBScript task immediately after RunResult COM function is called on the line.

 

We'd not anticipate that affecting the VBScript task itself, but looks like that it may have some side effects.

 

As v6.0.3110 puts an upper limit on the number of open handles, then using v6.0.3110 should not result in runaway handle usage.

In v6.0.3110 Handles are only closed when a new VBScript is started, which means that you will see some handles pointing to non-existent tasks, but there should never be more of these handles then there are lines themselves.

 

Let us know if you come across any issues when using v6.0.3110

[Guest Buzby]

OK, thanks for your quick reply. I think there's several places in my VBScripts where I call the Run_Result command then continue to do other processing afterwards. This would obviously affect things.

 

6.0.3110 seems fine in practice, and as you say seems to stop the increase in handle usage at an upper limit. There are still handles pointing to non-existent processes, and there were definitely more than lines (i.e. we have 16 lines, but there were over 250 handles pointing to non-existent processes) however as long as the system doesn't keep filling up and crashing, we are happy! I will update you after I have allowed the machine to run for a few more days and see whether the handle count is increasing.

 

NB. I note that there's a version 6.0.3113 floating around in the forums, so I hope the problems from 6.0.3111 don't find their way into future versions!

[SupportTeam]

The changes made for v6.0.3111 will not be included in future releases.

[SupportTeam]

If you could do a screenshot showing the handles' IDs and the trace(s) from VG since .exe was restarted then we could probably identify the actual handles that are causing problems...

[Guest Buzby]

Hello again,

 

Sorry to resurrect an old topic, but the handle leak has been niggling away at me for a while and I was hoping to resolve it once and for all!

 

Basically the latest versions of VG (we're running 6.0.3168) are pretty good at handle usage, and this topic fixed some earlier issues pretty well. However, there is still a small leak which causes our server headaches after a few weeks of running (or a few days if there is a high volume of calls).

 

The latest advice was to take a screenshot and zip up some trace files. However, as our server generates lot of traffic during the day, it is unfeasible for me to shut the server down, switch on the traces, do some test calls, then switch of the traces again.

 

So instead, I have done a test late at night which involved placing a call into the system and letting it run for a while. Sure enough after a few minutes of activity, we have a 'Non-existent process' listed in Process Explorer. If I were to leave the server running, over a period of a few days there will be a few thousand of these clogging up the system. The only way to resolve this is to bounce VG.

 

So, with the picture and trace files relating to this non-existent process attached, over to you!

vgtrace.zip

[SupportTeam]

It looks like we were able to identify the cause of the handle leak. The leak was happening when a VBScript was ran before call was answered. (Handle was used to track VBScript progress and VG was loosing track that there was an existing open handle at call answer time).

 

Attached .exe should fix this problem. Please update system with attached .exe and see if this resolves handle leaks for you

 

Note that under some circumstances there may be handles created by VG which may point to non-existing process, but their number should be upper-bound limited to no more then twice the number of lines controlled by VG.

 

Thanks for the screenshot and traces - both were needed to track this one down.

VgMulti_6.0.3181.zip

[Guest Buzby]

Many thanks, received 1000+ calls today and handles for VGMulti.Exe seem to have stabilised at around 750. Will watch over the next few days but the fix appears to be working as I would normally expect a big handle increase after this kind of loading over 24 hours.

 

Very many thanks once again. I think my list of issues is now closed, so I'm going to put my feet up for a few days, hope you do too!!!

[SupportTeam]

Glad to hear it's working fine now.

 

How many handles do you see reported when VG is first started and immediately after taking the first call?

 

The number of handles allocated from after first call is finished is what should be upper bound to no more then 2 per line...