Voiceguide Crashing

[Guest Jonah]

VoiceGuide is crashing in the middle of calls and the service shuts down. I just brought it live today on a new server. Please let me know if you need any more info. I have also emailed support with the log files.

[SupportTeam]

Did the problem start occurring when you changed server?

 

Do you have a screenshot or a transcription of the error report from the Windows Event Log?

 

Please .ZIP up the vgEngine and ktTel traces and post them here.

 

Also please confirm which version of Windows is used and which version of the Dialogic drivers is used.

 

[Guest Jonah]

We have been developing with VoiceGuide7. We have had it running on this server during development and testing and it seemed to perform just fine. Today is the first day I've had it running with a full call load (12000 records). I am unable to post logs here as they are too large. I will attempt to email them to support.

 

System seems to hold stable for an hour or more - then it crashes. A complete system reboot is required to make VoiceGuide work again -- simply restarting the service doesn't do the trick. Crashed 2 times on me today.

 

OS: Windows Server 2003 Standard with SP2

Dialogic is 6.0 PCI

[SupportTeam]

Do you have a screenshot or a transcription of the error report from the Windows Event Log?

 

Which version of VoiceGuide v7 are you using (first few lines of vgEngine trace will tell us that) the and what is the SU update version of the SR6.0 ? Is it SU184 or some other version?

 

I assume that you are doing outbound calls, correct?

 

Are you using VoiceGuide's default database (VistaDB) as the data source or have you setup another database to use as the OutDial store (SQL Server or MySQL etc)

 

When emailing traces please .ZIP them up first. This will reduce the size dramatically.

[Guest Jonah]

I have emailed the screenshot of the event log as well as all of the logs - you should receive them shortly

 

VoiceGuide_7.0.9_090218.exe

 

Yes, mostly outbound but some inbound

 

We are using SQL Server

 

 

[SupportTeam]

And what is the SR6.0 Build Number? (in Dialogic DCM goto Help menu and select 'About').

 

The event log screenshot received by us (and attached here by us) shows that the error is occurring in a Dialogic driver library: libgc.dll

 

So maybe this is because incorrect/old version of Dialogic SR6.0 drivers is installed on this system.

[Guest Jonah]

Here is a screenshot.

Doc1.doc

[SupportTeam]

Looks like the Dialogic drivers installed on this server are SR6.0 SU90. These are very old drivers (they are 4 years old) and not the drivers that are meant to be used with VoiceGuide v7. VoiceGuide is compiled against a specific Dialogic driver version, and that is why you must use that same Dialogic driver version on your system.

 

Please download the Dialogic drivers that we have link to next to the the VoiceGuide v7 download on our Downloads page. They are SR6.0 SU184.

 

At the same time you may as well download the current v7 from the Downloads page as well - it is newer then the version that you are using.

 

Then to update the Dialogic drivers do the following:

 

1. Backup your scripts and config files.

2. stop the VoiceGuide and Dialogic services and uninstall Dialogic and VoiceGuide.

3. restart system

4. install the SR6.0 SU184 drivers.

5. restart system

6. install VoiceGuide (best to install latest version you just downloaded).

7. restore the config files and scripts etc, reapply the license and restart VoiceGuide service.

 

 

(the screenshot that you have provided is attached below)

[Guest Jonah]

ok, I was unable to uninstall the old drivers so a complete system rebuild was nesassary. Anyway, installed new Dialogic Drivers and the latest version of VoiceGuide from the download page. Now VoiceGuide will not even start up. I will include logs. Please respond as quickly as possible as this is classified as a 'high' level server issue and we have already lost a day. Thank you!

Doc1.doc

0514_ktTts.zip

[SupportTeam]

Can you setup remote access to this machine using www.logmein.com or VNC or similar? We can then log into that machine and see what is happening on it. Please email login details to support@voiceguide.com and include a link to this support forum thread in the email.

[Guest Jonah]

Thanks for the quick response.

 

I can't enable remote access as this server is housed behind a VPN firewall. Are you available on some sort of chat such as google? I can get you live info that way.

[SupportTeam]

Also please check whether the right ISDN protocol has been selected for this T1 line in the DCM.

 

In the ktTel traces we see:

 

214002.526 6612 WARN gc_OpenEx ... returned linedev<=0. Check DCM to confirm ISDN protocol has been selected on this trunk.

 

Are you able to receive calls on this line when using Dialogic's ISDIAG testing app?

 

 

[SupportTeam]

Please make some tests using ISDIAG to make sure the Dialogic drivers have correctly installed themselves are and are correctly configured and the card itself is working with the T1 plugged into it. ISDIAG testing procedure is outlined here: http://www.voiceguide.com/vghelp/source/ht...igisdnsetup.htm

[Guest Jonah]

I had the ISDN Protocol wrong. You guys are good! That's why I like VoiceGuide. So now that I've completed the upgrade, I will attempt the outdial campaign again. I'll let you know if that fixed the original issue.

 

Thanks!

[SupportTeam]

OK, thanks for letting us know that this is what the immediate problem was. Let us know how you go with the outdial campaign now that the SU184 drivers are used on the system.

[Guest Jonah]

It's been running for a couple weeks now with no errors. However, it crashed again today leaving all of the lines in what looks like a 'hung' state. I'll send the log files to support as they are too big to post here.

[SupportTeam]

Could you please post a screenshot of the error logged by Windows in the Windows' Event Log Viewer.

(or the .ZIP of full DrWatson log if available)

 

[Guest Jonah]

Screenshot attached. Let me know if you need anything else.

err.doc

[SupportTeam]

This looks to be a different issue.

 

We had a look at the traces and we can see that on this system you are not running VoiceGuide scripts, but are using external application to issue your own commands to the Dialogic card directly. Essentially you are using the VoiceGuide Dialer to manage the outbound dialing for you and the once the number is dialed you are using the WCF interface to essentially directly call Dialogic API.

 

VoiceGuide exposes some of the Dialogic API, and this allows for some advanced operations be performed, but calling Dialogic API functions incorrectly or at incorrect times will lead to problems, and looks like this is what we are seeing here.

 

From trace we can see that calls were started on 23 of the 24 lines at 9am, and many calls were made, but by 12pm all of the outbound calling lines were in error state, effectively blocked from making further calls. The state of all lines was "Dialing" :

 

0602_1201_vgEngine.txt:123759.980 10 dial state 1:Null,2:Dialing,3:Dialing,4:Dialing,5:Dialing,6:Dialing,7:Dialing,8:Dialing,9:D

ialing,10:Dialing,11:Dialing,12:Dialing,13:Dialing,14:Dialing,15:Dialing,16:Dial

i

ng,17:Dialing,18:Dialing,19:Dialing,20:Dialing,21:Dialing,22:Dialing,23:Dialing,

 

Looking at the last call on line 23 we can see that the reason why the line was placed in this permanent "Dialing" state is because the external application issued a Play_Start command before the dialing was completed. Such calling of Dialogic API causes an error in the Dialogic drivers and blocks the line. As you can see Dialogic drivers issued no more events for this line, and the VoiceGudie's own internal error recovery was disabled for this line as VoiceGuide detected that another application is now controlling the line and when this is detected then VoiceGuide will not interfere with external apps.

 

Leaving multiple lines in error state could lead to unpredictable behavior in the Dialogic drivers and this may well have been the cause of the error that eventually resulted in bringing down the entire app.

 

There are also a lot of other errors logged by VoiceGuide relating to communicating with the external app - an extract of those errors is attached. Any external WCF apps should be designed in such a way that the WCF connection is stable - and we do not get constant callback connection errors that we can see here. Over 5000 WCF callback errors were caused in VoiceGuide between 9am and 11:30am - and it looks like errors were caused by external app not maintaining the WCF connections it set up.

 

It looks like you have been using this T1 system to make a fair few calls already (about 150,000?), so maybe some recent changes in the app have resulted in a bug in your app - the bug being that the external app is not waiting for Human/Machine detection to complete before starting the play - but this free support forum is not really intended to provide support in debugging user's own external applications. If you chose to use COM/WCF to call Dialogic APIs directly then please contact sales@voiceguide.com to setup a support agreement that will allow us to handle support enquiries related to this system in future.

 

110114.509 6 67 dial StartOutgoingCall start tel[3033430536] live[C:\VoiceGuide\VGBones7\VGScripts\LWC.vgs] am[C:\VoiceGuide\VGBones7\VGScripts\LWC.vgs] fax[] onNotAnswered[C:\VoiceGuide\VGBones7\LWC\EXE\LWC.exe $RV_LINEID LWC $RV[OutDial_Result] $RV[RepositoryID] $RV_CALLEDNUMBER] onRetriesEnd[] rv[[RepositoryID]={75963}] CampaignName[6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23] options[<CallerID>8669870010</CallerID>] ID=146870

110114.509 6 67 ls set Dialing

110114.509 6 67 answering machine detection enabled. sCallProgressOption = [DX_PAMDOPTEN]

110114.509 6 67 rv clear all

110114.509 6 67 rv add [RepositoryID]{75963}

110114.509 6 67 rv add [OutDial_RetriesLeft]{-1}

110114.509 6 67 dial making call LineState settings: tel[1 3033430536] live[C:\VoiceGuide\VGBones7\VGScripts\LWC.vgs] am[C:\VoiceGuide\VGBones7\VGScripts\LWC.vgs] fax[] noAnswer[C:\VoiceGuide\VGBones7\LWC\EXE\LWC.exe $RV_LINEID LWC $RV[OutDial_Result] $RV[RepositoryID] $RV_CALLEDNUMBER] retriesEnd[] OnHangup[] rv[[RepositoryID]={75963}[OutDial_RetriesLeft]{-1}] options=[<CallerID>8669870010</CallerID>]

110114.509 6 67 state Dialing 13033430536

110114.509 6 67 t timer set 30 sec : EV_TIMEOUT_MAKECALL_NORESPONSE

110114.509 6 67 ls set Dialing

110114.509 6 67 rvns add [OutDial_Result]{NOTCONTACTED}

110114.509 6 67 ktTel_MakeCall_Enqueue([13033430536],DX_PAMDOPTEN, 30,0,0,8669870010,<CallerID>8669870010</CallerID>) call

110114.509 6 67 qTel add cmd_MakeCall [0,0,30,0,0][8669870010|<CallerID>8669870010</CallerID>|13033430536|DX_PAMDOPTEN|]

110114.509 7 67 qTel run cmd_MakeCall

110114.524 7 67 ev generic: ktTel_Completion 10000 0,Completion_MakeCall, crn=28c036a, 42730346,0,0,13033430536,<CallerID>8669870010</CallerID>,<result>ok</result><crn>42730346</crn>

110114.524 7 67 qScr add evktTelCompletion 0 Completion_MakeCall

110114.524 6 67 qScr run evktTelCompletion Completion_MakeCall, iActionID=0, crn=0[42730346|0|0|0|0][13033430536|<CallerID>8669870010</CallerID>|<result>ok</result><crn>42730346</crn>|||]

110114.524 6 67 OnCompletion_ktTel_MakeCall sRet=[<result>ok</result><crn>42730346</crn>]

110114.524 6 67 ls set Dialing

110114.524 6 67 ScriptState set Running_Dialing, (called from OnCompletion_ktTel_MakeCall)

110114.524 6 67 state Dialing 13033430536...

110114.540 17 67 ev CallState GCEV_DIALING, crn=28c036a, iEvent=2 ,16,0,16,,,]. vgEngine v7.0.3415.28902 (2009-05-08 15:03:23.86)

110114.540 17 67 qScr add evCallState 2 GCEV_DIALING

110114.540 6 67 qScr run evCallState GCEV_DIALING, iActionID=0, crn=28c036a[16|0|16|0|0][|||||]

110114.540 6 67 zWcfEventSinksSubscribed sink.CallbackPort.CallState call

110114.540 6 67 ls set Dialing

110114.540 6 67 yLineStateAppPov: Dialing crn=28c036a,ev=2,GCEV_DIALING,0, sParam1=

110114.540 6 67 ls set Dialing

110114.571 13 rem Line_State 67 = [Dialing]

110114.571 13 67 wcf yLineStateAppPov=[Dialing]

110114.571 13 rem Line_State 67 = [Dialing]

110114.571 13 67 wcf yLineStateAppPov=[Dialing]

110114.665 13 rem Line_State 67 = [Dialing]

110114.665 13 67 wcf yLineStateAppPov=[Dialing]

110114.665 13 rem Line_State 67 = [Dialing]

110114.665 13 67 wcf yLineStateAppPov=[Dialing]

 

 

 

110118.696 13 67 rem Play_Start C:\VoiceGuide\VGBones7\SystemFiles\1SecondSilence.vox

110118.696 13 67 qScr add cmdPlayWavTts 0

110118.696 6 67 qScr run cmdPlayWavTts , iActionID=0, crn=0[8|0|0|0|0][C:\VoiceGuide\VGBones7\SystemFiles\1SecondSilence.vox|||||]

110118.696 6 67 PrepareAndPlayWavAndTts start iModuleIdx=8, iPlayFileIdx=0, bMakeTtsFile=True, sSoundFileIs=C:\VoiceGuide\VGBones7\SystemFiles\1SecondSilence.vox

110118.696 6 67 PrepareAndPlayWavAndTTS sPlayFile=C:\VoiceGuide\VGBones7\SystemFiles\1SecondSilence.vox

110118.696 6 67 state [Run Program 1] Playing dtmf (C:\VoiceGuide\VGBones7\SystemFiles\1SecondSilence.vox) using PlayStart

110118.696 6 67 t timer clear (force=False)

110118.696 6 67 qTel add cmd_PlayStart [0,0,0,0,0][||||]

110118.696 6 67 play start ok, playId=12359

110118.696 7 67 qTel run cmd_PlayStart

 

err.zip