VoiceGuide unresponsive, large vgStatusMonitor log file

[Distant DBA]

Hi. Occasionally we find our VoiceGuide becomes unresponsive and we get a massive vgStatusMonitor log file:

154646.552 Task_Display_List No Response (Timeout)

154646.574 This request operation sent to net.tcp://localhost:7134/PortStatus/ did not receive a reply within the configured timeout (00:00:02).  The time allotted to this operation may have been a portion of a longer timeout.  This may be because the service is still processing the operation or because the service was unable to send a reply message.  Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.

154646.667 Task_PollAndDisplay_List_Task_ConnectToService_GeneralExceptionObject reference not set to an instance of an object.

154646.771 Task_Display_List No Response (CommunicationException)

154646.791 The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.

154646.792 Task_PollAndDisplay_List_Task_ConnectToService_GeneralExceptionObject reference not set to an instance of an object.

154646.792 Task_Display_List No Response (CommunicationException)

154646.793 The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.

154646.794 Task_PollAndDisplay_List_Task_ConnectToService_GeneralExceptionObject reference not set to an instance of an object.

154646.795 Task_Display_List No Response (CommunicationException)

154646.795 The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.

154646.796 Task_PollAndDisplay_List_Task_ConnectToService_GeneralExceptionObject reference not set to an instance of an object.

The above happened at line 15 in the log file and those three lines repeated to 38 million lines of text / 4.4GB of log file until we rebooted PC. VoiceGuide itself was unresponsive so we could not stop/start the service. A reboot of the host PC does the trick, but a permanent solution would be preferable as we normally only check our system once a day.

• VoiceGuide v7.6.42
• Windows 10 Pro version 21H2
• Dialogic D4PCI-U card

Thanks for any assistance you can provide.

Gary

[SupportTeam]

Could you please post an extract from the ktTel and vgEngine traces that are around the the time when the vgStatusMonitor log started to fill with the error messages.

Also:

As you are using Windows 10 and not a Server class version of Windows you must ensure that Windows 10 never goes to "Sleep" or "Hibernation".

"Hybrid Shutdown" or "Fast Boot" is also not supported and must be disabled.

These settings can be set on the "Settings > System > Power & sleep" screen.

Dialogic drivers do not handle Windows' "Sleep"/"Hibernation"/etc.

[SupportTeam]

In the just released v7.6.45 of VoiceGuide the Line Status Monitor app should no longer rapidly fill it's vgStatusMonitor log with the error messages that you observed if a similar type of issue occurs.

If you can forward the ktTel and vgEngine trace files as per previous post we can then better see what could be the possible cause of the problems that you encountered.

If you see any errors in Windows' Event Viewer app from around the time VoiceGuide encountered issues please post those as well (screenshots are ok).

[Distant DBA]

Many thanks for your support so far. Attached are requested log files, truncated. Preceding sensitive data in original files has been deleted from attached copies. Remaining sensitive data has been replaced with XXX's and YYY's

I noted a memory error in attachments. Checking system today, memory usage is 54% and steady. System is 64-bit Windows 10 with 8GB RAM. VgIvrService (32-bit) process is using 165.8MB

Below is the text lifted from Event Viewer. Thanks again for your assistance so far.

Gary

 

Event viewer details for two events on 29th September at 15:46

Error    29/09/2022 15:46:44    .NET Runtime    1026    None
--------------------------------------------------------

Faulting application name: vgIvrService.exe, version: 7.6.42.0, time stamp: 0x6247a89b
Faulting module name: KERNELBASE.dll, version: 10.0.19041.1889, time stamp: 0xdc0d8494
Exception code: 0xe0434352
Fault offset: 0x0012ca42
Faulting process ID: 0x1d14
Faulting application start time: 0x01d8c71d9942de0e
Faulting application path: C:\Program Files (x86)\VoiceGuide\vgIvrService.exe
Faulting module path: C:\WINDOWS\System32\KERNELBASE.dll
Report ID: d919b60a-8e26-4715-8049-ad7ec4745953
Faulting package full name: 
Faulting package-relative application ID: 

Error    29/09/2022 15:46:55    Application Error    1000    (100)
-------------------------------------------------------------

Application: vgIvrService.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.OutOfMemoryException
   at System.Runtime.Serialization.DataContractSerializer.ReadObject(System.Xml.XmlDictionaryReader, Boolean)
   at System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter+PartInfo.ReadObject(System.Xml.XmlDictionaryReader, System.Runtime.Serialization.XmlObjectSerializer)
   at System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.DeserializeParameterPart(System.Xml.XmlDictionaryReader, PartInfo, Boolean)
   at System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.DeserializeParameter(System.Xml.XmlDictionaryReader, PartInfo, Boolean)
   at System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.DeserializeParameters(System.Xml.XmlDictionaryReader, PartInfo[], System.Object[], Boolean)
   at System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.DeserializeBody(System.Xml.XmlDictionaryReader, System.ServiceModel.Channels.MessageVersion, System.String, System.ServiceModel.Description.MessageDescription, System.Object[], Boolean)
   at System.ServiceModel.Dispatcher.OperationFormatter.DeserializeBodyContents(System.ServiceModel.Channels.Message, System.Object[], Boolean)
   at System.ServiceModel.Dispatcher.OperationFormatter.DeserializeRequest(System.ServiceModel.Channels.Message, System.Object[])
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.DeserializeInputs(System.ServiceModel.Dispatcher.MessageRpc ByRef)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(System.ServiceModel.Dispatcher.MessageRpc ByRef)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(System.ServiceModel.Dispatcher.MessageRpc ByRef)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage41(System.ServiceModel.Dispatcher.MessageRpc ByRef)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(System.ServiceModel.Dispatcher.MessageRpc ByRef)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(System.ServiceModel.Dispatcher.MessageRpc ByRef)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(System.ServiceModel.Dispatcher.MessageRpc ByRef)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(System.ServiceModel.Dispatcher.MessageRpc ByRef)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(System.ServiceModel.Dispatcher.MessageRpc ByRef)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(System.ServiceModel.Dispatcher.MessageRpc ByRef)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean)
   at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(System.ServiceModel.Channels.RequestContext, Boolean, System.ServiceModel.OperationContext)
   at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(System.ServiceModel.Channels.RequestContext, System.ServiceModel.OperationContext)
   at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(System.IAsyncResult)
   at System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(System.IAsyncResult)
   at System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(System.IAsyncResult)
   at System.Runtime.AsyncResult.Complete(Boolean)
   at System.ServiceModel.Channels.TransportDuplexSessionChannel+TryReceiveAsyncResult.OnReceive(System.IAsyncResult)
   at System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(System.IAsyncResult)
   at System.Runtime.AsyncResult.Complete(Boolean)
   at System.ServiceModel.Channels.SynchronizedMessageSource+ReceiveAsyncResult.OnReceiveComplete(System.Object)
   at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(System.Object)
   at System.ServiceModel.Channels.SocketConnection.FinishRead()
   at System.ServiceModel.Channels.SocketConnection.OnReceiveAsync(System.Object, System.Net.Sockets.SocketAsyncEventArgs)
   at System.ServiceModel.Channels.SocketConnection.OnReceiveAsyncCompleted(System.Object, System.Net.Sockets.SocketAsyncEventArgs)
   at System.Net.Sockets.SocketAsyncEventArgs.OnCompleted(System.Net.Sockets.SocketAsyncEventArgs)
   at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSuccess(System.Net.Sockets.SocketError, Int32, System.Net.Sockets.SocketFlags)
   at System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)

0929_0000_vgEngine.txt

0929_ktTel.txt

[SupportTeam]

Traces suggest that VoiceGuide and .NET handled that exception and continued on answering calls after these errors.

Windows' Event Viewer logged the errors that occurred within .NET itself at 15:46:44 & 15:46:55 - the "System.OutOfMemoryException" error and its related general .NET exception.

VoiceGuide traces show the "System.OutOfMemoryException" error raised within VoiceGuide at 15:45:57 and 15:46:43

However the traces show that at 15:47:27 VoiceGuide detects another incoming call and answers it, plays the "Welcome Message.wav" to caller and then starts a VBScript that successfully reads in contents of a file, and then successfully calls a Web Service and reads in the data returned from that Web Service, and then moves on to a GetNumbers module that plays "enter card number.wav". All of that is done with no errors.

The vgEngine provided stops at that point (at 15:47:34) as does the provided ktTel trace so not sure what happened afterwards, but it does show that the system functioned after the errors logged in Windows' Event Viewer.

In vgEngine trace as part of the "Run VBScript" module logging VoiceGuide logs the following line at 15:47:34 :

154732.407  31                     systemstatus : AvailablePhysicalMemory=2767 MB, currentProcess.HandleCount=495114

Which shows there is over 2GB of physical memory available, but the "Handle Count" does seem a bit high.

 

Did the vgEngine and ktTel traces stop logging at at 15:47:34 or do they continue on and show the 15:47:27 call being handled?

What are the last lines in the 0929_0000_vgEngine.txt trace, and in the ktTel traces before it shows the system restart?

 

Please also note that for business applications we recommend using a 'Server' grade version of Windows, not Win10.

[D.H.]

Hi, we are using Voiceguide v.7.6.40, recently encounter the same problem above, disk full because of too large  vgstatusmonitor log file, size of several hundred G. I want to confirm that we can simply replace the vgIvrService.exe of v.7.6.40 by vgIvrService.exe of v.7.6.45, need not resintall the whole Voiceguide again, correct? Is there any other files of newer versions need to be replaced?

[SupportTeam]

Not possible just change the .EXE

The .DLLs on system must be the ones made for use with that .EXE

Only way to change to different version is to run the installer for the new version. There is no need to fully uninstall the previous version before running the installer, just stop the VoiceGuide service and exit all the VoiceGuide programs before running the installer for the new version.

[D.H.]

Thanks for your reply. 1. May I know the approximate release date of version 7.6.45? As we don't want to upgrade to the newer versions too frequently, so as to avoid anything unexpected happen which cause service interruption. If the difference between version 7.6.40 and version 7.6.45 is not big, we may suggest customer to hold it for the moment.

[SupportTeam]

v7.6.45 was released on 6th October 2022.