System.outofmemoryexception Errors

[Sebastian]

Hi,

 

Today we had a situation that VG stopped for no reason. In Windows logs I found:

 

System:

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" /> 
  <EventID Qualifiers="49152">7034</EventID> 
  <Version>0</Version> 
  <Level>2</Level> 
  <Task>0</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x8080000000000000</Keywords> 
  <TimeCreated SystemTime="2015-04-22T16:23:19.871927900Z" /> 
  <EventRecordID>17042</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="720" ThreadID="5816" /> 
  <Channel>System</Channel> 
  <Computer>XXX</Computer> 
  <Security /> 
  </System>
- <EventData>
  <Data Name="param1">VoiceGuide IVR</Data> 
  <Data Name="param2">1</Data> 
  </EventData>
  </Event>

Application:

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name=".NET Runtime" /> 
  <EventID Qualifiers="0">1026</EventID> 
  <Level>2</Level> 
  <Task>0</Task> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2015-04-22T16:21:05.000000000Z" /> 
  <EventRecordID>826243</EventRecordID> 
  <Channel>Application</Channel> 
  <Computer>SISTI_HP_21_VG</Computer> 
  <Security /> 
  </System>
- <EventData>
  <Data>Aplikacja: vgIvrService.exe Wersja architektury: v4.0.30319 Opis: proces został przerwany z powodu nieobsłużonego wyjątku. Informacje o wyjątku: System.OutOfMemoryException Stos: w System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*) w System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)</Data> 
  </EventData>
  </Event>

and:

 <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Application Error" /> 
  <EventID Qualifiers="0">1000</EventID> 
  <Level>2</Level> 
  <Task>100</Task> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2015-04-22T16:21:07.000000000Z" /> 
  <EventRecordID>826244</EventRecordID> 
  <Channel>Application</Channel> 
  <Computer>SISTI_HP_21_VG</Computer> 
  <Security /> 
  </System>
- <EventData>
  <Data>vgIvrService.exe</Data> 
  <Data>7.4.5.0</Data> 
  <Data>54fdfdcd</Data> 
  <Data>KERNELBASE.dll</Data> 
  <Data>6.1.7601.18409</Data> 
  <Data>53159a86</Data> 
  <Data>e0434352</Data> 
  <Data>0000c42d</Data> 
  <Data>734</Data> 
  <Data>01d076c467910452</Data> 
  <Data>C:\Program Files (x86)\VoiceGuide\vgIvrService.exe</Data> 
  <Data>C:\Windows\syswow64\KERNELBASE.dll</Data> 
  <Data>93ccf1c7-e90b-11e4-8d3a-40a8f04b9c0b</Data> 
  </EventData>
  </Event>

Please help me to find the problem. Our server configuration is:

- Intel Core i3 3.30Ghz

- 6GB RAM

- Windows 7 64b

 

In attachment I send log files.

vg-logs.zip

[SupportTeam]

Error cause was:

 

System.OutOfMemoryException

 

at:

 

System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)

System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*

 

 

vgEngine traces shows a lot of these System.OutOfMemoryException errors occurring on this system for up to half an hour before the crash – with vgEngine log showing that up to a third of the 1600 database connections made between 6:00PM and 6:21PM did not complete due to the OutOfMemory error, and pretty much every VBSCript module was not able to run as system was unable to instantiate a new ClearScript VBScriptEngine object.

 

Traces show that VoiceGuide handles the errors and continues on with the calls, but looks like some 3^rd party module (PostgreSQL data provider?) did not handle this exception.

 

Traces also show that VoiceGuide continued working for over 2 minutes after 6:21PM. So not sure if this was the actual error that caused the crash or (more likely) there was some other memory issue that stopped the system from working.

 

ktTel traces also show errors when trying to malloc 4.6MB sound data buffers.

 

 

The .hdmp and .mdmp files would identify the module that caused the crash, but what needs to be resolved here is what is causing your system to run out of memory.

 

Looks like you are using a PostgreSQL ODBC driver on this system and it looks like others reported that this driver is responsible for memory leaks. Please see: https://github.com/pocoproject/poco/issues/152 and http://pocoproject.org/forum/viewtopic.php?f=12&t=2471&p=10369#p10369

 

 

Were you by chance able to observe the memory levels on the system when the system performance started to be degraded (began to have issues connecting to the database, not running VBScripts)?

 

Suggest monitoring memory allocations on this system when you begin to notice OutOfMemoryException errors in future and determine what is responsible for the memory usage.

 

This app may be of use: https://technet.microsoft.com/en-us/library/ff700229.aspx