Vgservices_Wcf Demo App Monitor Limitation

[Guest Jonah]

Hi, I am using the Demo app to monitor events on lines. When I pass in a -1 I get events for all lines. However, I need to see the events more instantaniously and when there are more than about 10 lines running, there is a significant delay.

 

As a fix I have attempted to run 46 instances of the demo app and pass in the line number that I want it to monitor (1-46) but they start crashing after about 10 to 12 instances. Is there a limit on the number of lines that can be monitored individually? If so, is there any other way to monitor events of just an individual line?

 

Let me know if you need log files or anything.

 

Thanks!

[SupportTeam]

There should not be much of a delay. WCF events would be passed to external applications pretty promptly.

 

How did you establish that there is a delay on your system? Did you confirm that this is a delay in the WCF events actaully arriving or do you just see delays in the events getting processed by your app?

 

Can you post some traces etc which document this?

 

 

Having a separate event monitoring app for each line would not be the recommended approach, but still you should not see the WCF events receiving component crashing. I assume that you have modified the app significantly to implement your own functionality. Were you able to establish what part of the code causes the crash? Can you post some screenshots of the crash report etc?

[Guest Jonah]

Here are the logs...containing...

 

1. Started VoiceGuide and called into the system (line 1) with only 1 line running - events received in timely manner

 

2. Started a number of additional lines all pegging a Voice Mailbox. Repeated step one and this time the events were delayed significantly.

 

I will attach some code as well that show how I monitor the demo app. As you can see, I do not pass any events to my app unless they are keypresses or voicefile triggers or call hangup events.

 

Please let me know if you need additional info.

 

Thanks!

0513_ktTts.zip

VGCSInterface.zip

VG Event Monitor.zip

[SupportTeam]

Traces show that some external application is repeatedly polling the Line_State state in tight loop (using the remoting interface). Traces show this is done is continuously in a tight loop without waiting in between successive calls.

 

Such polling of state is not desirable and is more the likely causing excessive load on VoiceGuide engine.

 

Any external applications that use VoiceGuide's WCF, Remoting or COM interfaces should be designed in such a way that they do not have tight loops such as these.

 

 

In place where WCF events are fired to subscribed we see that 3 applications are subscribed to receive events and the events are forward onto all 3 subscribed applications immediately.

 

eg:

 

152903.266 17 7 3 ev CallState GCEV_DROPCALL, crn=280002e, iEvent=2 ,1,0,32,,,]. vgEngine v7.1.3699.20074 (2010-02-16 11:09:08.70)

...

152903.266 6 7 3 zWcfEventSinksSubscribed sink.CallbackPort.CallState call

152903.266 6 7 3 zWcfEventSinksSubscribed sink.CallbackPort.CallState call

152903.266 6 7 3 zWcfEventSinksSubscribed sink.CallbackPort.CallState call

 

why do you say that the events are delayed? How much are they delayed by? How do you measure this delay?

[Guest Jonah]

I took the loop-polling of Line_State out of my application and it didn't seem to make a difference. I am attaching the logs. At one point during the call to line 1, I press "1-2-3-4-5-6-7-8-9" and it takes about 10 seconds before VoiceGuide sends the events for those digit presses and they come a few seconds apart. I can see other events coming in on other lines between them. I can tell it's delayed because I am pressing the digits and I have an event handler in my app that pops the event to the screen as soon as it comes in.

 

Please let me know if you need further information.

 

Thanks!

0513_ktTts.zip

[SupportTeam]

The traces confirm that VoiceGuide issues the WCF events for DTMF tones in same way as the CallState WCF events, and as the events are sent on the same WCF channel it is puzzling why some events are arriving at your application promptly and some appear delayed.

 

Microsoft's WCF Service Trace Viewer would let you trace the actual events as they go through the WCF stack on your system.

 

Google "WCF Service Trace Viewer" to get more info on this.

 

 

We will do a test setup on our systems here to see if we can see any such behaviour.

[Guest Jonah]

I think ALL of the events are delayed. It's just that the DTMF tones are easier for me to differentiate because I know when I press them. Let me know if you are able to set up a test system for this. If you need to see more of my code, please let me know. Works great for just a few lines but I am about to launch a campaign with about 90k records in about a month on all 46 of my lines and in it's current state, it will fail.

 

Basically with the code I send you earlier (The modified version of WCF Demo and the VB.Net app) I use the VB app to monitor the WCF_Demo app (text changed) and then you can just write all the events to the screen. While you are doing this, send 12-15 simultanious calls to a voicemail (or somewhere) - they should all play several voicefiles - and then call in to one of the lines. The events should be delayed. Please let me know if you need more info or code to duplicate this.

 

Thanks!

[SupportTeam]

We have setup a test here on one of our servers to verify how promptly the events received by VoiceGuide (DTMF/Callstate/etc) are forwarded to external user applications which listen to WCF events broadcast from VoiceGuide.

The test on our system showed that the delay between the Dialogic card reporting the event and the WCF listener receiving an alert is about 50ms, and rarely is it more then 100ms.

All the log files generated by the test are attached below as well as the WAV file played to \the system by the caller, and the WCF events listener application used - you can use these to replicate the test if you wish.


Our test system was running the standard 'Credit Card Payment' demo application that ships with VoiceGuide and using an E1 ISDN trunk (with 30 channels).

We sent calls into that system, with the calling system playing a sequence of DTMF tones - which resulted in test system progressing though the Credit Card payment script.

For this test we generated 1000 calls over 30 channels - so each channel got 33 or 34 calls one after another.

The events listener is based on the demo one that ships with VoiceGuide, but this one has been modified to save events from each port in a separate log file - so that events received by each port can be clearly seen.


You can look at timestamps of events as saved by the WCF listener, and the timestamps in the ktTel trace of when the event arrived from the Dialogic card.

Would recommend you use the WCF demo application supplied with Voiceguide to first test WCF event reception on your system and build the system up from there. If you have any issues with WCF events reception please show appropriate logs documenting the problems encountered.

Not sure how your external applications are designed, and analyzing them is outside the scope of this forum, but we would recommend that you only have one application doing the WCF connection and only that application doing all the events receiving for all the ports on the system. Starting a new application each time a new call comes in, with each application doing its own WCF connection to VoiceGuide just for the length of the call is not the recommended approach for systems with a larger number of ports.



The VoiceGuide logs are 8MB .ZIPed. They can be downloaded here: [old link removed]
other files attached below.

0520_VgWcfEvents.zip

cctest_all_2.zip

Demo_vgServices_WCF_Source.zip

[SupportTeam]

Also:

 

From log files it can be seen roughly 1800 events were fired for each port over the 16 minute period it took for the 1000 calls to be made on this system.

 

So on average there was about 54,000 events over 17*60=1020 seconds, which is about 53 events per second - but if we were to further analyse the event flow we would probably regularly see event volume of 100 events per second - especially at the beginning of test when credit card data was entered at more or less the same time on all 30 channels

 

NB. the CPU usage on the test machine that received the calls and had the WCF event sink test app running on it stayed at <5% usage mark all the time while the calls on 30 channels were received. (System CPU is Intel Quad core Q8200 @ 2.33MHz, 2GB RAM)

[Guest Jonah]

Thank you very much for your detailed response. I appreciate you taking the time to duplicate this on your system. I will strip my app down to the basics and build from there. If I locate the problem on my end I will post it back to this forum.

 

Thanks!

[Guest Jonah]

I changed the way I use the WCF inteface to only use one instance of it to monitor all the lines. This seems to have taken care of the issue. It seems to be working very well most of the time but on occasion (maybe once every couple days) the WCF interface stops receiving events. I am not sure what is causing this to happen. It seems to lose its connection with the service. This causes all the lines to basically go into a frozen state. Shutting down and relaunching the WCF Interface fixes the issue but the calls that were live when it shut down are all ended prematurely. I have attached the log file that I believe contains the appropriate timeframe. It was on 6/25 at approx 6:33 PM.

 

Is there a point where the WCF Demo app stops listening to events? Please let me know if you need more information.

0625_1859_vgEngine.zip

[SupportTeam]

Is there a point where the WCF Demo app stops listening to events?

No. The WCF communications channels should just stay up, unless there is some error that occurs that results in it disconnecting.

 

If you think that the disconnection occurs after a couple of days then maybe try restarting your app daily at a time when there are no calls on system (eg: 4am)

 

The trace starts at 18:59:50, so it does not capture the 6:33PM time.

[Guest Jonah]

Sorry, I attached the wrong one - try this one. I actually do restart it if a call comes in and there have not been any events for at least 5 minutes. So it does get restarted several times per day.

0625_1826_vgEngine.zip

[SupportTeam]

Looked through the trace from 18:33:00 to 18:34:00 and all looked fine with system making calls and running scripts and responding to WCF API calls.

 

So there does not look anything out of the ordinary in the trace, and the WCF connection looks like is maintained.

 

Can you tell us about what specific timestamp or part of the trace you would like us to look at?

[Guest Jonah]

I will monitor my system and next time it happens I can try to give you an axact timestamp.