Vg Service Crashes

[Guest donamenband]

Dear voiceguide Support,

 

Here's what happens the voiceguide service crashes.

 

--A window pops up asking me whether or not I wish to debug using MS visual studio.

--The voiceguide tray icon suddenly appears with that red X on it, indicating that the service has stopped.

--Inbound callers hear the phone ring twice (as usual) but after that, they hear silence.

 

I then start the voiceguide service using the tray icon and everything works normally once again.

 

This last time was the third time it happened. This presents serious reliability issues.

 

Please see the attached logs, which were collected immediately after the crash (before the VG service was re-started.)

 

Please advise.

 

Thank you.

logs.zip

[SupportTeam]

It looks like the problem is with the Text To Speech (TTS) engine that VoiceGuide is told to use.

 

Looks like the TTS engine crashes/stops and does not process TTS requests issued to it. The crash advice you see on the screen would be from the TTS engine.

 

VoiceGuide itself is still running.

 

Traces show VoiceGuide is still answering calls - but caller is not hearing anything as the TTS engine is not creating any sound files.

 

This is why caller hears nothing after 2 rings. VoiceGuide has answered the call.

 

 

Which TTS engine is used on this system?

Are you able to pre-record the parts of your playback data instead of using TTS?

You would probably be able use Say Number modules to speak back information, instead of TTS, as looks like the information that TTS is used to speak to caller is mostly numbers and times.

 

 

 

170818.574 6 7 4 state Ring 1

170824.589 6 7 4 state Ring 2

170824.589 6 7 4 answering the call (picking up the line)

 

170825.152 21 7 4 state [CheckAreaCode] type: Script_VBScript, iRunWait=1

170825.777 16 7 4 rem Run_ResultReturn [success] 7

 

170825.792 21 7 4 state [PreWelcome] type: Script_VBScript, iRunWait=1

170826.495 16 7 4 rem Run_ResultReturn [success] 7

 

170826.495 6 7 4 [Localhost] VGMTYPE_PLAY run start

170826.495 6 7 4 tts generate wait ()

 

170841.824 17 7 4 ev Dialogic 134,TDX_CST, crn=0, 0,0,0,DE_LCOF,,

170841.824 17 7 4 state Hanging up... [Loop Current Off (non-xfer vgm)]

[Guest donamenband]

1-

For TTS I'm using mostly NeoSpeech with a little bit of AT&T Natural voices.

Are you able to see which which of those two engines is causing the problem?

Are you able to see what text was passed to the TTS engine when the crash occurred? (Maybe there are certain strings that cause it to crash.)

 

2-

Unfortunately, I can't be without TTS.

 

3-

Please take into account that the following observations support the idea that the crash is coming from voiceguide and not from the TTS engine:

• The voiceguide tray icon appears with an X after the crash.
  
• As soon as I click "start" in the voice guide tray icon, everything works once again.
  
• I didn't take a screenshot, but I seem to remember that the debug popup said "voiceguide". I'm sure I would have noticed if it had said "neospeech" or "natural voices" isntead.
  

4-

By default, the "voiceguide IVR" service in "Windows Component Services" has "First failure" set to "Take no action". Is this intentional?

Is it possible that changing this to "restart the service" would solve the problem?

[SupportTeam]

For TTS I'm using mostly NeoSpeech with a little bit of AT&T Natural voices.

Are you selecting the TTS engi using XML tag in the TTS stream? or another method?

 

 

From ktTts trace we can see that this TTS call never returns:

 

153249.136 5724 fn TtsToWavFile (iLineId=3, zsTextXml=<?xml version="1.0"?> <voice required="Name=Paul;"> Here are the listings for <pitch middle="-2">Brooklyn</pitch>.<silence msec="100"/>It's now 8:32. Latest<pron sym="sh ax m aa">Shema</pron><pron sym="m ah g ax n ax v r ah m">Magen Avraham</pron> Degrees, was at 8:26. Latest<pron sym="sh ax m aa">Shema</pron><pron sym="m ah g ax n ax v r ah m">Magen Avraham</pron> Fixed, will be at 8:42. Latest<pron sym="sh ax m aa">Shema</pron>Graw and<pron sym="b ao l ih t ao n y ax">Baal Hatanya</pron>, 9:18. Latest<pron sym="sh aa k r ih s">Shachris</pron>, 10:33. <rate speed="-3">midday</rate>, 1:01. earliest <pron sym="m ih n h aa">Mincha</pron>, 1:39. <rate speed="-2">sunset.</rate>, 8:27. <silence msec="400"/>nightfall 3 stars, 9:16. nightfall 72 minutes, 9:39. listings for <pitch middle="-4">tomorrow</pitch>. dawn degrees, 3:52. dawn fixed, 4:24. earliest tolis and <pron sym="t t f ih l ih n">Tfillin</pron>, 4:37. <rate speed="-4">sunrise</rate>, 5:36 and 46 seconds. Latest<pron sym="sh ax m aa">Shema</pron><pron sym="m ah g ax n ax v r ah m">Magen Avraham</pron> Degrees, 8:26. . (len=2198), zsFname=C:\Program Files\VoiceGuide\temp\tts_3_2.wav, iWavFormat=1, iSpeed=100, iVolume=100, lParam1=0, zsParam1=)

153249.136 5724 TtsToWavFile wrote all data to task idx 2

153249.136 5724 TtsToWavFile lPreviousSemaphoreValue 0 (max 1 owr 0)

153249.136 4528 1 WaitForNewTask returned

153249.136 4528 1 task to process : 2

153249.136 4528 tts 1 task 2 setup output file

153249.136 4528 1 task 2 file exists C:\Program Files\VoiceGuide\temp\tts_3_2.wav

153249.136 4528 1 task 2 deleted existing file.

153249.136 4528 1 task 2 AssignFormat => 0

153249.136 4528 1 task 2 BindToFile C:\Program Files\VoiceGuide\temp\tts_3_2.wav => 0, 0x0

153249.136 4528 1 task 2 SetVoice => 0, 0x0

153249.136 4528 1 task 2 SetOutput => 0, 0x0

 

 

We can see that VoiceGuide continues to answer calls for another 2 hours or so, but with TTS no longer working.

 

If you can captures the screenshot of the crash report we can better see where the crash occurs. Most likely it would have been brought on by the original TTS issue in some way.

 

Looks like starting the VoiceGuide service again, and having VoiceGuide perform TTS initializations lets TTS work again, but these lengthy initializations is not something that should be done on a regular basis.

 

Does the TTS engine comes with some command line utility which can be used to generate the WAV file? This command line utility cna then be called from VoiceGudie's Run Program module.

 

 

Is it possible that changing this to "restart the service" would solve the problem?

Wouldn't hurt to do this, but this would not revolve the original problem - with VoiceGuide service running, but not getting any response from the TTS.

[Guest donamenband]

The crash just happened again, so we now have some more information.

 

In reply: Yes, the TTS voice is selected using <voice required="Name=VoiceTitle;"> tags.

(Voices in use are: AT&T Mike, AT&T Charles and Neospeech Paul. The default voice in vg.ini is set to Mike.)

 

Please find screenshots in the attached "files.zip". Included is also the disassembly code obtained from visual studio at the time of the exception.

 

I have also attached traces.zip. (It is estimated that the crash occurred between 5.00 and 5.20pm)

 

This time it happened again that callers heard the phone ring twice followed by silence.

 

Unlike last time, this time the tray icon showed the service to be running even after the crash.

(I am assuming this change is due to the fact that I had set "First failure" to "restart the service" in "Windows Component Services". I had hoped that this setting would solve the problem automatically but it clearly did not.)

 

In any case, we have confirmed beyond a doubt that the service in which the exception occurs is "VgIvrService.exe".

 

It need not be stated that if the exception were handled by terminating the current call and reinitializing TTS, that would still be better than not handling the exception at all.

 

Thank you.

files.zip

traces.zip

[SupportTeam]

Screenshots show error occurred in AcaTtsSapi5.dll.

 

Screenshots were taken at 17:46:19 onwards, and show that VoiceGuide service is still running and main VoiceGuide service was not restarted before that time. This can be seen from VoiceGuide Line Status Monitor (we can still see status on port 2 that was set at 17:11:44), and the ktTel and vgEngine traces that show calls continued being processed.

 

VoiceGuide service is still working and ready to answer calls etc.

 

ktTts trace shows that TTS generation problems start at 17:11:45. Cant say when the AcaTtsSapi5.dll error occurred as screenshots do not show this, but it was probably soon after 17:11:45. You may be able to get the timestamp from the Windows Event Log.

 

The reason why vgIvrService.exe can be seen in error screenshots is because vgIvrService.exe stated the TTS task on which the error occurred, but error on that TTS task did not affect VoiceGuide operation itself. Just all TTS requests after 17:11:45 were not processed.

 

 

 

VoiceGuide service was restarted about 10 minutes after taking the screenshots, at 17:54:33. I assume that restart was done by you?

 

 

170338.093  3520  fn    TtsToWavFile (iLineId=3, zsTextXml=<?xml version="1.0"?>
<voice required="Name=paul;">

...

(len=932), zsFname=C:\Program Files\VoiceGuide\temp\tts_3_2.wav, iWavFormat=1, iSpeed=100, iVolume=100, lParam1=0, zsParam1=)
170338.093  3520        TtsToWavFile wrote all data to task idx 50
170338.093  3520        TtsToWavFile lPreviousSemaphoreValue 0 (max 0 owr 0)
170338.093  4216          1   WaitForNewTask returned
170338.093  4216          1   task to process : 50
170338.093  4216  tts     1   task 50 setup output file
170338.093  4216          1   task 50 file exists C:\Program Files\VoiceGuide\temp\tts_3_2.wav
170338.109  4216          1   task 50 deleted existing file.
170338.109  4216          1   task 50 AssignFormat => 0
170338.109  4216          1   task 50 BindToFile C:\Program Files\VoiceGuide\temp\tts_3_2.wav => 0, 0x0
170338.109  4216          1   task 50 SetVoice  => 0, 0x0
170338.109  4216          1   task 50 SetOutput => 0, 0x0
170339.578  4216          1   task 50 speak returned. now waiting
170339.687  4216          1   task 50 generation completed
170339.687  4216  r       1   task 50 speak to file closed
170347.265  3520  fn    TtsToWavFile (iLineId=3, zsTextXml=<?xml version="1.0"?> <voice required="Name=paul;"> Step one. Set default <pitch middle="-4">location</pitch>. (len=222), zsFname=C:\Program Files\VoiceGuide\temp\tts_3_3.wav, iWavFormat=1, iSpeed=100, iVolume=100, lParam1=0, zsParam1=)
170347.265  3520        TtsToWavFile wrote all data to task idx 51
170347.265  3520        TtsToWavFile lPreviousSemaphoreValue 0 (max 0 owr 0)
170347.265  4216          1   WaitForNewTask returned
170347.265  4216          1   task to process : 51
170347.265  4216  tts     1   task 51 setup output file
170347.265  4216          1   task 51 file exists C:\Program Files\VoiceGuide\temp\tts_3_3.wav
170347.265  4216          1   task 51 deleted existing file.
170347.265  4216          1   task 51 AssignFormat => 0
170347.265  4216          1   task 51 BindToFile C:\Program Files\VoiceGuide\temp\tts_3_3.wav => 0, 0x0
170347.265  4216          1   task 51 SetVoice  => 0, 0x0
170347.265  4216          1   task 51 SetOutput => 0, 0x0
170348.703  4216          1   task 51 speak returned. now waiting
170348.765  4216          1   task 51 generation completed
170348.765  4216  r       1   task 51 speak to file closed


171144.343  4   3   2 state [PreWelcome] type: Script_VBScript, iRunWait=1


171145.125  3520  fn    TtsToWavFile (iLineId=3, zsTextXml=<?xml version="1.0"?> <voice equired="Name=Paul;">  (len=104), zsFname=C:\Program Files\VoiceGuide\temp\tts_3_1.wav, iWavFormat=1, iSpeed=100, iVolume=100, lParam1=0, zsParam1=)
171145.125  3520        TtsToWavFile wrote all data to task idx 52
171145.125  4216          1   WaitForNewTask returned
171145.125  4216          1   task to process : 52
171145.125  4216  tts     1   task 52 setup output file
171145.125  4216          1   task 52 file exists C:\Program Files\VoiceGuide\temp\tts_3_1.wav
171145.125  4216          1   task 52 deleted existing file.
171145.125  4216          1   task 52 AssignFormat => 0
171145.125  3520        TtsToWavFile lPreviousSemaphoreValue 0 (max 0 owr 0)
171145.125  4216          1   task 52 BindToFile C:\Program Files\VoiceGuide\temp\tts_3_1.wav => 0, 0x0
171145.125  4216          1   task 52 SetVoice  => 0, 0x0
171145.125  4216          1   task 52 SetOutput => 0, 0x0
171228.781  3520  fn    TtsToWavFile (iLineId=7, zsTextXml=<?xml version="1.0"?> <voice equired="Name=Paul;">  (len=104), zsFname=C:\Program Files\VoiceGuide\temp\tts_7_1.wav, iWavFormat=1, iSpeed=100, iVolume=100, lParam1=0, zsParam1=)
171228.781  3520        TtsToWavFile wrote all data to task idx 53
171228.781  3520        TtsToWavFile lPreviousSemaphoreValue 0 (max 0 owr 0)
171259.296  3520  fn    TtsToWavFile (iLineId=7, zsTextXml=<?xml version="1.0"?> <voice equired="Name=Paul;">  (len=104), zsFname=C:\Program Files\VoiceGuide\temp\tts_7_1.wav, iWavFormat=1, iSpeed=100, iVolume=100, lParam1=0, zsParam1=)
171259.296  3520        TtsToWavFile wrote all data to task idx 54
171259.296  3520        TtsToWavFile lPreviousSemaphoreValue 1 (max 1 owr 0)
171321.625  3520  fn    TtsToWavFile (iLineId=7, zsTextXml=<?xml version="1.0"?> <voice equired="Name=Paul;">  (len=104), zsFname=C:\Program Files\VoiceGuide\temp\tts_7_1.wav, iWavFormat=1, iSpeed=100, iVolume=100, lParam1=0, zsParam1=)
171321.625  3520        TtsToWavFile wrote all data to task idx 55
...
174518.500  3520  fn    TtsToWavFile (iLineId=7, zsTextXml=<?xml version="1.0"?> <voice equired="Name=Paul;">  (len=104), zsFname=C:\Program Files\VoiceGuide\temp\tts_7_1.wav, iWavFormat=1, iSpeed=100, iVolume=100, lParam1=0, zsParam1=)
174518.500  3520        TtsToWavFile wrote all data to task idx 69
174518.500  3520        TtsToWavFile lPreviousSemaphoreValue 16 (max 16 owr 0)


174549.812  4168   7 ev    TDX_SETHOOK (SetHook Completed)
174549.812  4168   7       set hli->pCTelProxyhli (in TDX_SETHOOK). current hli->pCTelProxyhli=091398F8, pTelClientEvents=091398F8
174549.812  4168   7       TDX_SETHOOK - is onhook
174549.812  4168   7 r     Dialogic  TDX_SETHOOK 135 (0 0 0 DX_ONHOOK  )


175433.718  3464           ------------------------------------------------------------------------------
175433.718  3464           ktTel_SR60 DLL v7.3.1, created: Jun 19 2012, 22:38:52
175433.718  3464           start at 0715 175433.718
175433.718  3464           ------------------------------------------------------------------------------

[SupportTeam]

AcaTtsSapi5.dll seems to be an Acapaella TTS DLL.

 

Do you have Acapella TTS installed on this system? If yes then you should uninstall it and ensure that AcaTtsSapi5.dll is no longer present on your system.

[Guest donamenband]

Screenshots were taken at 17:46:19 onwards...Cant say when the AcaTtsSapi5.dll error occurred as screenshots do not show this, but it was probably soon after 17:11:45.

Service had been down for about 40 minutes when screenshots were taken. So, yes, this makes sense.

 

VoiceGuide service was restarted about 10 minutes after taking the screenshots, at 17:54:33. I assume that restart was done by you?

I thought I restarted VG immediately after I finished taking the screenshots at 17:49, but perhaps it is possible that I waited 5 minutes before restarting.

 

Looks like starting the VoiceGuide service again, and having VoiceGuide perform TTS initializations lets TTS work again, but these lengthy initializations are not something that should be done on a regular basis.

I understand your argument that these initializations should not be done on a regular basis, but perhaps they can be done automatically whenever a TTS task times out. For example, you could say that if VG does not get a response from TTS within 30 seconds, TTS should automatically be re-initialized. If your analysis is correct, this feature would spare all subsequent inbound calls.

[SupportTeam]

Do you have Acapella TTS installed on this system? If yes then you should uninstall it.

 

If you do not have Acapella TTS installed on this system then are you able to determine which TTS engine the file AcaTtsSapi5.dll was installed by?