VG v6 Exe Crashes (goes Into Microsoft Debugger)

[LTympanick]

Running VG for Dialogic v6.03342

 

Running PRI-ISDN 23 lines with Dialogic 240-JCT System 6.x drivers...

 

Running on a Windows 2003 server with 2GB memory...

 

Runs fine until something happens that makes VG want to go into the Windows debugger...When this happens, my system goes down and my customers aren't very happy when they hear all circuits busy because the system isn't up at all. Of course when this happens, I don't just lose one line, I lose the entire system.

 

Please observe the log files I have...vgm & tw. I had them at log level 2 because I wasn't expecting any problems, but my system was very busy this past weekend, all 23 lines going at certain times and therefore the likelyhood of something bad happening unfortunately increased and unfortunately came to fruition....HOWEVER and this is important, the first time it crashed at 06:58 in the morning, there wasn't more than 10 lines or so active as far as I can tell.

 

Otherwise the system has been up & running handling about 300 calls a day just fine....for many days now...there is just something that will happen at some point that kills it and for the life of me i can't figure out what that may be...

 

One thing I do know is that if I have calls in the outgoing call queue and I try to start VG, it will crash right away upon any attempt to restart it...Not sure if that is in any way related to why it crashes in the first place but that is something I observed...you will notice in the log files that an attempt to restart the system at 0725am initially caused it to crash again until I made the calls in the queue delayed until 7:30 so I could get the system back up & running.

 

If you look in the VGM file for "VoiceGuide for Dialogic" string you can see the several times i had to reboot....i have also attached my tw log file in 2 parts to see if that could possibly help as well.

 

Of course this is a very difficult problem because it is almost impossible to reproduce unless you can give me a clue as to what might possibly be going on....Any advice would be greatly appreciated.

 

I can't figure out how to see the release notes on the upgraded version to see if it would potentially fix this problem. Also, since I'm an actual customer, if I did wish to download the upgrade, is there a way for me to do it without having to fill out the form every time?

 

Thanks.

0915vgm.zip

0915tw1.zip

0915tw2.zip

[SupportTeam]

system has been up & running handling about 300 calls a day just fine....for many days now...

Do you do an automated restart on this system ?

 

We'd probably recommend a regular restart of the Windows server itself. Weekly is most common, but nightly would not hurt either.

 

One thing I do know is that if I have calls in the outgoing call queue and I try to start VG, it will crash right away upon any attempt to restart it...

We'll have a look at this scenario closer on our test systems here.

 

Is this system used for only outbound calls, or is there a mix of inbound and outbound?

 

Can you post the script which is used?

 

What is the log level set to ins VG.INI? Please ensure the log levels are set to 10, we can then get fuller information in the traces.

[LTympanick]

Why is it necessary to have to reboot the system every day or week? That just seems lame to me. I don't see how to automate rebooting as one has to wait for the Dialogic drivers to settle in before you can start the app. Are you saying there is a way to do that? Seems like a pretty risky strategy to me.

 

Besides, rebooting once a day wouldn't have helped the 2nd two crashes after my first one. I rebooted the system after the first one and a few hours later, I was down again. With a busy system, the problem seems to trigger more readily so that rebooting would not be an effective solution for me. All it would mean is that I can count on my system dying when it is needed most.

 

Both inbound & outbound calls are made from this system. I only do outbound calls from lines 13 - 23 but inbound calls could be coming in on any of the 23. All lines however are dedicated to this system. Nothing else happens on them except for calls handled by VG.

 

I have uploaded the script for both inbound & outbound calls. Right now, inbound far outweighs outbound in terms of frequency like by about 100 to 1.

 

Also please comment on the extensive ERROR messages in tw2. Can you tell how those started?

 

I know those eventually led to a freeze a while later after a previous restart.

 

Thanks.

3S_HotLine_Inbound.vgs

3S_HotLine_Outbound.vgs

[SupportTeam]

We have been unable to reproduce any crashes on our test systems as of yet.

QUOTE

Also please comment on the extensive ERROR messages in tw2. Can you tell how those started?

These error messages just show that VoiceGuide was not retrieving the Dialogic driver level messages (after it VG crashed at around 10:23:26).

Do you have any screenshot of the errors that appear on the screen when the problem occurs?

Does the debugger that comes up lets you save any traces or report files?


Another user has reported a similar crashing problem to your with a recent version (http://voiceguide.com/forums/index.php?showtopic=5165) and v6.0.3362 seems to have fixed this issue for them, but I'm not sure if these problems are the same.

Could you please update to v6.0.3362 and see if problems persist.

[LTympanick]

I agree with you that this other problem sounds a lot like mine. I will attempt the upgrade to v6.0.3363 and let you know how it goes. Thanks.

[LTympanick]

Haven't had a chance to upgrade to latest v6, still running v6.0.3342.

 

However, I was able to get a crash again on 3342...Here's the sequence of events:

 

 

Had just rebooted server to install Windows upgrades.

 

Fire up VG.

 

Make a call. Appears to get in funky state in the post call processing routine where I add a call accounting record to my database. At least that appears to be the last thing logged (see 1011vgm.txt)

 

I make another call onto the same line and VG crashes upon this 2nd call, seemingly before even answering the phone. Never get any more log information. Still showing only the last activity of adding a call accounting record upon the end of the last call.

 

Have also attached the access violation dialog box and the window inside the Microsoft debugger showing the stack trace.

 

Hope this helps to give a clue...

 

p.s. Now when I start up VG again, after logging out of previous session in Windows and logging back in again, I'm able to make more than one call without it crashing....I mentioned before that I thought I needed to have a call in the call queue for this to happen but it appears I was able to see this crash without any call in the call queue.

 

Again, while this crash happens with just a single call, the system will now be up & running just fine until the system gets really busy and then I will potentially see it crash again but I can't be sure that is the exact same circumstance.....

 

AcessVioDiag.bmp

1011tw.txt

1011vgm.txt

[SupportTeam]

The stack trace shows that the error happened well inside the Dialogic drivers themselves, - the stack does not show any VoiceGuide related DLLs/EXEs - just a stack of libsrlmt.dll and libisdnr4.dll - both of which are Dialogic DLLs.

 

First off could you please update to latest version of VG and confirm if you are still seeing the problem with this new version.

 

Please set the logging to a higher level. The current traces do not contain much information as the logging level has been set too low. Please set logging levels to '10'.

 

Do you have anything else running on this system?

 

When installing the new version of VG could you please reinstall the SR drivers as well.

[LTympanick]

Okay,

 

I have now upgraded to VG Dialogic 6.0 v3377.

 

I have upgraded Dialogic Drivers to 6.0 v171. I couldn't get v3377 to work with Dialogic 6.0v178 so I dialed those drivers back to v171.

 

There are 2 instances where I can guarantee a crash, even with this latest version.

 

1. The first time I make a call into VG after re-booting. I then allow it to crash, i.e go into the Microsoft debugger. I subsequently restart VG and then it will be fine.

 

2. If there are calls in the call queue, that will be a sure fire way to get it to crash, right out of the starting blocks as well. I see something in the logs about "Operation is not allowed when the object is closed." Attempting any inbound call under this condition will precipitate the crash although I think it's not really the inbound call but the attempt at an outbound call before the system is ready that causes it to crash right out of the starting blocks.

 

This is the case that concerns me because I believe it has a relation to when I'm seeing the system crash when the system is busy. The system can be running along just fine handling a bunch of simultaneous calls but I think there is a timing problem when one has inbound & outbound calls attempting to go out on the same lines. I think this is what makes my system crash when it gets real busy and 20 or more lines are in use. It's just a guess on my part but case 2 above makes me think this simulates this situation of a busy system attempting to make outbound calls because in both cases, the system appears to be trying to make outbound calls possibly before the lines that just handled an inbound call are ready to perform an outbound call on the same line for example. I need to overlap inbound & outbound on the same set of lines because I don't have enough lines to dedicate to each function. I need to be able to have the same lines perform both inbound & outbound calls.

 

I have attached the log 10 log files for your perusal.

 

Any help would be greatly appreciated.

0212vgm.zip

[SupportTeam]

Does the debugger show that the error still happening in the Dialogic drivers?

 

Can we have a look at this machine? Please email support@voiceguide.com with your details and a link to this thread.

 

Have you tried separating the inbound and outbound lines to confirm that this problem is caused by a conflict between an incoming call arriving on the same channel as the outbound call is placed?

 

What Dialogic card is used on this system? Which version of Windows is used on this system now?

 

The "rsCallQueADO_FetchComplete (no error) pRecordset related err=Operation is not allowed when the object is closed" errors. probably have no relation to this problem, as this error just indicates that the system was unable to retrieve any data from the external OutDial call source. From what external database are the outbound calls loaded?

[LTympanick]

I am using a Dialogic D240JCT-T1 card.

 

Running on Windows Server (Standard Edition) 2003R2 Service Pack 2.

 

Yes, I would be happy to have you take a look at this machine.

 

What kind of details do you need in the e-mail? I have a LinkSys router so I may have to play with the config a little to allow inbound access to the machine. Is that what you mean. Please explain exactly what info you need to connect which is what I presume you are talking about.

 

Yes, I still believe the debugger would show it's in some low level Dialogic libraries when it crashes but I haven't confirmed that this time around. The symptoms are identical to how they have been in the past so I presume nothing has changed.

 

Again, this problem only really impacts me when my system is really busy which is when I am most likely to see the system go down unexpectedly. The best I can do to reproduce is to have something in the call queue which is set up on my SQL server 2005 with the following CONNECT string in my VG.ini

 

;Line to connect to CallQue in SQL DB HotLineSystem rather than using .MDB that comes with VG

DbConnectString_ADO=Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=HotLineSystem;Data Source=HOTLINE4;Application Name=Outbound Dialing Queue

 

This seems to work fine under normal operation. However, if calls are in the queue prior to me attempting to start VG, then VG will die upon initial startup. My only hope is that whatever makes it crash in this circumstance may be related or the same thing as what happens when my system gets really busy.

 

Please advise on further info you need and how you can look at my system.

 

Thanks.

 

[SupportTeam]

Easiest way to setup system access for us is to use the logmein.com service.

 

Please set this up on the server and the email us the logmein.com login details. In the email include a link to this support forum thread so that we can link up the email with the issue.

 

 

[LTympanick]

Hi again, sorry for the long delay in response...

 

Okay, I have set up a logmein.com account. Please re-read this entire thread to get a history of what I am dealing with here...Random crash...and I do mean random...VG log of no use since crashes seem to occur at truly random points in time with no real pattern emerging...even to the point where we appear to be in the dialogic drivers when the crash occurs...

 

However, I have new information that has put all my previous observations into question. For example, I have not been able to reproduce the issue where I thought VG crashed simply because calls were in the queue upon start up...or because the system was making both outbound and inbound calls using the same lines....That did not happen to me this past weekend....Nor did my machine crash at all this weekend where VG handled over 10,000 calls, both inbound & outbound, in mostly a 10 hour window over the 2 days....

 

I have a network setup and typically in the past I would monitor my VoiceGuide production machine using Windows Terminal Services (WTS) remote console mode to view my production VG machine from another computer. This is typically how I am monitoring my VG system while it is running full out with VoiceGuide handling many calls that can come in on a busy day such as this past weekend when I handled 5,000 calls on both Saturday and Sunday.

 

To my amazement, for the past several weeks, and this past weekend especially when the system was really busy, the system did not falter on me even once. It ran flawlessly the entire time....No crashes whatsoever...not even once.

 

The difference this time? This time I did not use Microsoft Windows Terminal Services to monitor my main VG machine. I simply monitored it with a local monitor and keyboard...this is the only real change I made since my last post...could WTS possibly be an issue???

 

I remember how you've mentioned in the past with a 24 line system, you had to create the monitor program because displaying states within VG itself would cause VG to crash on a more than 4 line system....so you created the monitor program...is there any possible way this monitor program or VG itself or the Dialogic Drivers could be having a conflict with WTS so that after a certain point in time it could cause VG to crash??? Could you imagine a scenario where the dialogic drivers could have a problem with WTS???

 

Again, I ran with 10-20 lines active for MANY hours at a time this past weekend, with not one crash, the first time this has happened since I've owned VG which is now about a year...Of course my system is not this busy every day but it hasn't crashed for weeks now, since I stopped monitoring it remotely using WTS. Typically I have only been able to get the system to crash when it is very busy but it's only ever happened when I was watching remotely using WTS which was my typical MO prior to this past weekend. In some repsects it lends credence to the theory that it's more likely to possibly run into a WTS conflict when more monitor activity is going on with a busy system???

 

Would greatly appreciate any feedback and I have set one of my non-VG machines with logmein.com just in case you would still like to remotely monitor...In that case, we'd have to use WTS to see the VG machine. Friday May 9 (US) of this week 3:00-500 EST might be a good time to check in because I expect to be busy due to rain in this area....Saturday morning May 10, EST 7:00 - 10:00 EST may also be a good time as I expect this to be busy as well....I can give you my US telephone number as well. I can also do Skype calls....

 

Please let me know if any of this information allows you to reproduce the random crash problem in your world first...Use MTS to access your VG machine and then get it real busy with 10-20 lines active for as long as you can and see if it crashes into the debugger at some random point. This is what has been happening to me for the past year but that all changed this weekend when I stopped using MTS...Coincidence????

 

If not, maybe we can coordinate a time when you can remotely log in as time zone differences may be an issue...

 

Thanks...would love to get to the bottom of this once and for all......

 

[SupportTeam]

Guess the only way to better see what happened would be to get the DrWatson traces from the system. These should indicate at which point in the program the system crashed and it could better point as to whether Windows Terminal Services played a role in this or not.

 

VoiceGuide v7 offers better traceability of any issues as it runs managed code. It is also designed to run larger systems more efficiently, so we encourage users of 20+ channels to use VoiceGuide v7 instead of v6.

[LTympanick]

OK.

 

Ugh, upgrading to System 7 is going to cost me a fortune (at this point more than I originally paid for v6) and I still see another post from Miva

(Miva's post) that makes me worried that upgrading may not solve my crash problem anyway since she is running into it with Dialogic 6 drivers under system 7 as well.

 

You now say that I should have never installed Dialogic SR6 drivers in the first place because v6 works with SR5.1 drivers only (honestly who would have figured) . OK, it might have been nice if someone said that a long time ago but whatever, I could try to go back to SR5.1 drivers if that's an issue and it looks like it might be.

 

I think Dialogic has an issue in their SR6 drivers when it comes to outbound calling. From my monitoring of the system of when it crashes, it does appear almost every time that I can recall, several outbound calls were being placed while handling many simultaneous inbound calls on other lines but the system only appears to ever have an issue when outbound calls are in the picture. Unfortunately, logs are useless because the system crashes and nothing useful gets logged. (We've tried looking at those...no help).

 

So I'll be happy to go back to SR5.1.1 drivers since they might be more reliable that Dialogic's new ones. In six months I'll give you an update on how things have gone since just so you could put this to bed once and for all.

 

Thanks for all your help...I really mean it...I'm sure this has been as frustrating for you as it has been for me.

 

 

 

 

 

 

[SupportTeam]

I think Dialogic has an issue in their SR6 drivers when it comes to outbound calling.

There are no issues that we are aware of. Also, we have test systems here doing literally millions of inbound/outbound calls with SR6.0 drivers on 100+ line systems and there are no unresolved issues outstanding. Many customer are also running large outbound systems based on SR6.0+VG v7 and there are no outstanding issues reported by people running that configuration.

 

The Miva report that you refer to was reported yesterday and full traces have not even yet been supplied so it's not yet possible to comment on what has happened there, but we can see that no exceptions are generated by either the drivers or VoiceGuide v7 and VoiceGuide continues to run. Also the issue seems to be related to two-line recording and not outbound calling.

 

So I'll be happy to go back to SR5.1.1 drivers since they might be more reliable that Dialogic's new ones.

SR5.1.1 are not more reliable then SR6.0. The SR6.0 has had many more years of development put into it by Dialogic and we recommend using SR6.0 drivers (with VG v7) on cards that can use SR6.0

[LTympanick]

Yes, but I'm running VG v6 not v7.

 

All I know is that my system intermittently crashes (and not in recoverable way, try going into the Microsoft Debugger) and you have said it's happening in the Dialogic drivers...see your own posts above...

 

Also, you just got finished telling me I'm running the wrong drivers for VG v6, that I should be using SR5.1.1.

 

In this post My other recent thread, the last post you made is quoted as follows:

 

Note that VoiceGuide v6 is meant to be used with Dialogic SR5.x series of drivers, and VoiceGuide v7 is meant to be used with Dialogic SR6.x series of drivers (the drivers for use with VG v7 are provided on our Downloads page). I think that you are already using SR6.x drivers on your system so v7 would be the VG version to use.

 

Well, I do like your product but there is no way in hell I can justify spending another $4,000 on top of the $3,000 I have already spent to simply upgrade to v7. I pleaded with sales but they insist on playing hardball, so guess what, you have priced me out of the market.

 

So, given I have to stay with v6, once and for all, which drivers should I be using?

[SupportTeam]

With VoiceGuide v6 you should be using SR5.1.1 drivers.