Voip Line Busy With "<Offered>:cid....." Status

[iTime]

Hello,

 

Our IVR server with VoIP stops responding in about 30 minutes after restart or after some calls are made, showing "<offered>:cid..." status (Please see the attached screen shot.)

We have installed patch for a fix from a previous issue, but I am not sure if that is the cause of this problem.

 

Would you take a look at the log and let me know?

Thanks.

VG_Down.zip

[iTime]

Adding to the comment above: I found out that this problem happens when a machine calls to our IVR Server.

 

I noticed that the number in "cid" above was a tele-marketer's number thinking, so tried to call to this number from our another IVR server with VG.

The problem happened again: the status monitor shows the first one got busy and stuck on "<offered...>" status then VG attempts to answer with the following port until all the 4 ports get stuck.

 

So, it looks like our IVR server cannot handle when a inbound call is made from a automated machine.

Your advice is appreciated.

 

Thanks.

[SupportTeam]

It looks like you are using a pre-release v7.4.0 on this system, and HMP 3.0 SU347.

 

There was a bug in this version (related to HMP using same CRN for different calls). This should now be fixed in this version of VG:

 

[old link removed]

 

Please note that above is still a 'pre-release' version of VG v7.4

 

If you encounter issues with this new version please post traces as before.

[iTime]

The provided patch above has fixed the issue, thank you.

 

But we get another problem after applying the patch. I am not sure if this was caused by the new patch, but I am posting under this ticket for now.

For some users, VG does not response to they key pressing.

095007.065  6  19   6 state [PlayWelcomeGetEmpNumber] Number Input
095007.065  6  19   6       PrepareAndPlayWavAndTts start iModuleIdx=1, iPlayFileIdx=1, bMakeTtsFile=True, sSoundFileIs=
095007.065  6  19   6       PrepareAndPlayWavAndTTS sPlayFile=WAV Files\Welcome_01.wav,WAV Files\Welcome_02.wav
095007.065  6  19   6 state [PlayWelcomeGetEmpNumber] Playing wav (WAV Files\Welcome_01.wav,WAV Files\Welcome_02.wav)
095007.065  6  19   6       FindSoundFile start [WAV Files\Welcome_01.wav]
095007.066  6  19   6       FindSoundFile start [WAV Files\Welcome_02.wav]
095007.066  6  19   6 t     timer clear (force=False)
095007.066  6  19   6       q_tel +     cmd_PlayStart 730695 ,C:\Program Files (x86)\VoiceGuide\Scripts\IntelliTime\WAV Files\Welcome_01.wav,C:\Program Files (x86)\VoiceGuide\Scripts\IntelliTime\WAV Files\Welcome_02.wav []
095007.066  6  19   6       play  start ok, playId=730695
095007.066  7  19   6       q_tel run   cmd_PlayStart 0 00:00:00.0005000 max:2|00:00:00.0115015
095014.850 17  19   6 ev    CallState GCEV_DISCONNECTED, crn=8000006, iEvent=0 ,16384,0,64, s1:, s2:, s3:]. build_date: 2013-09-17 11:22:38.88
095014.851 17  19   6       q_scr +     evCallState 0 GCEV_DISCONNECTED
095014.851  6  19   6       q_scr run   evCallState sCode=[GCEV_DISCONNECTED] iActionID=0, crn=8000006 [16384|0|64|0|0][|||||] 00:00:00 max:1|00:00:00.0005001
095014.851  6  19   6       callstate crn_event=8000006 0|GCEV_DISCONNECTED state=16384|Disconnected calldirection=0 gcCallState=64 [||], ScriptState=LS_GETNBRS_PLAYWELCOMEMSG
095014.851  6  19   6       LineEvCallState L1_current=Connected crn_event=8000006,ev=0,GCEV_DISCONNECTED,0, sParam1=
095014.851  6  19   6       LineEvCallState LINECALLSTATE_DISCONNECTED begin
095014.851  6  19   6       L1_set Disconnected (iState=LS_GETNBRS_PLAYWELCOMEMSG)
095014.851  6  19   6       in LINECALLSTATE_DISCONNECTED iState=LS_GETNBRS_PLAYWELCOMEMSG
095014.851  6  19   6       HangupCall (LineEvCallState_DISCONNECTED) crn_in=8000006, L1=Disconnected, L2=Running_Normal, lPlayId=730695, lRecId=0
095014.851  6  19   6       ccrn_connected=0, crn_NewIncomingCall=0, crn_CurrentMakeCall=0, crn_InProcessOfReleasing=8000006
095014.852  6  19   6       after hangup script: [C:\Program Files\VoiceGuide\Scripts\IntelliTime\PostHangUp_Hiring.vgs]
095014.852  6  19   6       FindFullPathToScript_VgsOrVgp start [C:\Program Files\VoiceGuide\Scripts\IntelliTime\PostHangUp_Hiring.vgs]
095014.852  6  19   6       after hangup script: [C:\Program Files (x86)\VoiceGuide\Scripts\IntelliTime\PostHangUp_Hiring.vgs]
095014.852  6               LoadedVgs_Find C:\Program Files (x86)\VoiceGuide\Scripts\IntelliTime\PostHangUp_Hiring.vgs at idx=4
095014.852  6  19   6       After Hangup script started (from LineEvCallState LINECALLSTATE_IDLE handler)
095014.852  6  19   6       L2_set Running_AfterHangup  (called from StartAfterHangupScriptIfDefined)
095014.852  6  19   6 t     timer clear (force=False)
095014.852  6  19   6       RunModule start Run VB Script, [ProcessFinishUpHiring], iModuleIdx=126, previous: vgm=1, vgs=4:1
095014.853  6  19   6 t     timer set   30 sec : EV_TIMEOUT_HANGUP
095014.853  6  19   6       QueueUserWorkItem scrmod_RunVbs.RunModule call
095014.853  6  19   6       HangupCall exit after starting afterhangup script
095014.853  4  19   6 state [ProcessFinishUpHiring] type: Script_VBScript, iRunWait=0

This is one of the example (I hope I picked the correct part). The user mentioned he provide his Employee ID, but IVR kept asking so he just hung up.

I believe there are several cases like that in the attached log. (so I excluded a possibility that it could have been related with a broken telephone on user's end.)

 

I also found out that VG service failures have been reported into Windows Event viewers several times for last 24 hours, so I have included the most recent failure logs.

 

Thanks.

 

 

VG_NoResponseToKeyPress.zip

[iTime]

Adding to my previous post, I found bunch of Application errors related with "vgIVRService.exe" in Windows Event Viewer and it looks as if the same problem persists from my previous post (http://voiceguide.com/forums/index.php?showtopic=10365&hl=).

 

I noticed the VG service stops whenever this happens and restarts because I set "Restart" option in the Service property.

 

Would you be able to look at this issue as well? It is a critical issue because it is happening almost every time when there are multiple users calling in at the same time.

I am including the Event logs and VG logs.

 

Thanks.

EvtView_CrashLog.zip

[SupportTeam]

For some users, VG does not response to they key pressing.

 

Could be a problem with the phone, or maybe a poor quality SIP connection.

 

Can you reliably replicate the problem when calling from certain phone?

 

To confirm we would need a WireShark trace that captures the incoming VoIP call (both SIP and RTP).

 

Please make a capture of problem call and save as .pcap file. Then .ZIP up trace and post it here.

[iTime]

OK, I will ask our customers to reproduce the symptom and try to record the Wireshark trace.

 

But in the mean time, Can you take a look at the Application failure case that I had posted followed by the thred above?

It is still happening almost evey hour and it is an urgent case for our customer. Let me know if you have not received the collection of Event Log, then I will repost.

 

Thanks.

[SupportTeam]

Regarding the faulting errors: the error when the fault occurs is always the same:

Faulting module name: MSVCR90.dll, version: 9.0.30729.4940, time stamp: 0x4ca2ef57
Exception code: 0xc0000417

Can you tell us more about how this system is set up?

OS appears to be: Microsoft Windows Server 2008 R2. Is there anything else apart from Windows+HMP+VoiceGuide installed on this system?

Is Widows running on a normal physical machine, or is it a virtual machine?

 

HMP used on this system is 3.0 SU328 - which is an older version no longer made available for download from Dialogic.

Can you please try updating system to latest version of HMP as well.

18th:

0821 - 11:20 pm
0824 - 11:23 pm
1408 - 5:07am (19th)
1411 - 5:10 am (19th)
2330 - 2:29 pm (19th)
2338 - 2:37 pm (19th)

19th:

0813 - 11:12 pm
0955 - 12:45 am (20th)
1157 - 2:56 am (20th)
1210 - 3:09 am (20th)
1238 - no traffic on system for 23 minutes (system restart without stopping service first?)

[SupportTeam]

Are you able to collect more information about the exceptions that you are seeing on the system?

 

Please see:

 

http://blogs.technet.com/b/yongrhee/archive/2010/12/29/drwtsn32-on-windows-vista-windows-server-2008-windows-7-windows-server-2008-r2.aspx

 

http://msdn.microsoft.com/en-us/library/bb787181%28VS.85%29.aspx

[iTime]

Can you tell us more about how this system is set up?

OS appears to be: Microsoft Windows Server 2008 R2. Is there anything else apart from Windows+HMP+VoiceGuide installed on this system?

Is Widows running on a normal physical machine, or is it a virtual machine?

 

It is running on a VM Environment. This server is in a hosted environment. (You helped us resolve TTS issues with this VM on: http://voiceguide.com/forums/index.php?showtopic=10063&hl=)

The Server has nothing much: an Anti-virus and Wireshark are most notable applications others than HMP, VG and ATT Natural Voice.

I checked the log and this error started showing since Sep. 10th, when I reported the error, and I believe I have installed "VoiceGuide_7.4.0_130910" as suggested in this thread: http://voiceguide.com/forums/index.php?showtopic=10383&hl=

 

I will try to update HMP driver as suggested soon.

 

I also include Windows Error report generated yesterday and today.

 

Please keep me posted.

 

Thanks.

 

WinReport.zip

[SupportTeam]

We would recommend removing the Anti-Virus software for now as well. Just to remove another variable.

 

Are there any errors in Dialogic's RTF logs?

 

We have been testing the 130917 pre-release on our HMP testing systems here and we have not seen any issues that you encounter.

Test servers have Windows installed on physical machines - not VMs. Standard test setup size is 120 simultaneous calls going through an IVR script.

 

We have set up a test system with HMP SU323 - which seems closest to SU328 what you have been running and for last 12 hours there has been no issues with 120 simultaneous SIP calls all the time going through that system (each individual call 0-30 sec)

 

Maybe it is a better idea to install SU323 on your system as well to match, instead of going to latest SU version.

[iTime]

We would recommend removing the Anti-Virus software for now as well. Just to remove another variable.

 

This is a hosted environment and we do not have control on the anti-virus software, but I assure you it has been OK since June until last week, so I would strike out the possibility from it.

 

I have updated the HMP driver to SU347 that is available on Dialogic site (as you recommended) and attached rtf logs and VG logs.

So far there is no VG error reported on the Event viewer, but this is because it is late night on the customer's site, thus few users.

 

I will keep you posted after this weekend.

In the mean time, please take a look at the logs and let me know your findings.

 

One weird thing I can tell is a status message showing on Status Monitor: 3rd port shows "hanging up...[Line Status wcf]", which I never saw before. (I clicked the hang-up button, but it remains)

Call duration is marked "0" and the "End Call Date" is 9/20 21:52:05.

 

Thanks for your help again.

 

VG_HMPLogs.zip

[iTime]

Any update on this issue?

I have updated the HMP driver to the most recent available on Dialogic site (SU347) last Friday, but VG is still crashing.

I attach the log of VG, Dialogic and Windows error reporting after the update of the driver.

As you can see, VG stops abruptly and restarts more often now. You can see several logs got created in 5 to 10 minutes of interval because it crashed and restarted.

I also add a screen shot, which shows that several lines got "stock" and never gets released unless I click "hangup" button from the Status monitor.

 

This is a production environment, so if you could help me resolve this issue soon, that will be great.

 

Thanks.

VG_CrashLogs.zip

[SupportTeam]

Can you please install HMP SU323 in this system and post traces as before showing system startup and restarts.

 

SU323 can be downloaded from Dialogic HMP downloads page.

[iTime]

Please find the attached logs based on your request.

It was generated under VG pre-release (130917) with HMP su323.

The log is between 9/25 7:30 PM and 9/26 11:00 AM and it shows several times of crashes as usual.

There was a big dump file (about 40MB) generated by Windows event, but could not attach here.

 

I hope you could find the cause of the crash.

 

Thank you again.

VG130917__HMP323_Logs.zip

[SupportTeam]

There was a big dump file (about 40MB) generated by Windows event,

 

You should now be able to attach a 40MB file to a post. Please .ZIP up file before posting it.

[iTime]

Here it is.

Let me know if you need something else.

 

Thanks.

Edited 09/29/2013 01:57 AM by SupportTeam
supplied attachment deleted after downloading

[iTime]

While posting the attachment with the dump file, the screen disappeared once without any success message.

Please let me know if you have not received the big attachment.

Thanks.

[SupportTeam]

Thank you for providing the .mdmp and .hdmp files. These allowed us to see quickly where the issue was.

 

The attached dumps showed the problem was related to trace logging in ktTtsSapi.dll - module that handles Text-to-Speech, and creates the MMDD_ktTts.txt trace logs.

 

The code area in question was fixed, and the version with the fixes is available for download from:

 

[old link removed]

 

Please note that above is still a 'pre-release' version of VG v7.4

 

If you encounter issues with this new version please post traces as before, including the .mdmp and .hdmp dump files.

[iTime]

I have installed the new patch, but VG still crashes.

 

I attach the logs including Win report and dumps as before.

I also noticed that I had to uninstall and reinstall in order to apply the patch: just stopping the service and applying patch on the previous version did not change the version information on the log file.

 

Thanks.

VG_Crash_130929.zip

[SupportTeam]

We'll look at the traces.

 

just stopping the service and applying patch on the previous version did not change the version information on the log file.

 

If the VoiceGuide service is stopped and all VoiceGuide programs are exited and the files still can't be replaces with the newer versions by the installer, then some other process is blocking access to those files. This maybe related to the issues that you are seeing on your system.

[SupportTeam]

Can you please set the ktTel log level in VG.INI to 0. ie:

[Log]
ktTel=0

and restart the VoiceGuide service.

Please advise if you are still seeing any issues after that change, and if you do then please post traces as before, including the .mdmp and .hdmp dump files.

[SupportTeam]

Above workaround should work for the version on pre-release 7.4 currently installed on your system.

 

If you would like to re-enable ktTel tracing then please update system to this version:

 

[old link removed]

[iTime]

The provided patch finally fixed the issue.

 

We have had our customer install the patch and been monitoring for about a week until now, and there is no Windows dump, error, or abrup stops of VG service.
However, FYI, I still had to fully uninstall and reinstall VG rather than simply installing on top of the previous version.

 

Thanks for the fix.

[SupportTeam]

Thanks for letting us know that this issue is now fixed.

 

 

Regarding installation:

Was the installation done while logged in as Administrator?

The installer should stop the VoiceGuide service, and once the service is stopped the files should be able to be overwritten with new versions.

[iTime]

Yes, I was an Admin, and installed by clicking "Run as an Administrator", but did not stop the service manually as I believe that is the part of installation.

I will try to stop manually and reinstall next time.

 

Thanks.