VoiceGuide needs restart once VoIP is re-established?

[iTime]

Hello Support,

I have a question related with restarting VG service.

Most of our customers' IVR servers are subscribing VoIP service (HMP). One issue is that when the VoIP service provider performs certain maintenance or repairs, the service would be unavailable, yet the service is still NOT available even after the line is recovered unless VG Service is restarted.

If you see the log (attached), it shows abnormal activities begins at 2/18 11PM, and continues until we restarted on 19th around 9:20AM. 

230000.607  44                     stats BackupStaticsIntoBins zStatsIvrCall completed
230000.608  22                     ex    stats_ivr_call table not present. Please update tables in VoiceGuide database
230000.610  22                     ex    stats_ivr_call table not present. Please update tables in VoiceGuide database
230000.611  22                     ex    stats_ivr_call table not present. Please update tables in VoiceGuide database

 

During this period of time, we get warning email (below) from the VoIP provider. (The subscription option our customer is taking is "unlimited" so, the content of the mail is not true, but it seems that is how they detect the problem as of.)

A call from 785806XXXX to 785380XXXX was rejected on trunk SC_1010004_IPTG because the limit of calls has been reached and completing this call would exceed the limit.

 

So, my question is, is there a way that VG knows its service needs to be restarted in these kinds of cases so that it, or we, can get the VG service restarted?

Thanks for your help in advance.

 

 

VG_Downtime.zip

[SupportTeam]

How are the VoIP trunks routed to the system now? Are they routed to IP address, or is HMP set to 'Register' itself with the VoIP provider in order to receive calls?

The traces supplied do not show system startup so we cannot see that setting.

If you could capture the WireShark trace of SIP communications on the system happening before and after restart then this would assist in giving complete picture of what is occurring.

(in WireShark use this for "filter": sip)

 

The "stats_ivr_call table not present" is unrelated. On your system you will see that message every half an hour when the system is operating as well. It just means that the statistical reporting information is not saved on your system as it looks like you must be using an external database as VoiceGuide's own database, and the table stats_ivr_call has not been created in that database. if you do not need statistical historical records then this log entry can be ignored.

[iTime]

Quote

How are the VoIP trunks routed to the system now?

<VoIP_Registration>
<Display>TopekaIVR</Display>
<Protocol>SIP</Protocol>
<RegServer>ld01-06.fs.fusionconnect.net</RegServer>
<RegClient>1234567890@192.168.100.93</RegClient>
<LocalAlias>1234567890@192.168.100.93</LocalAlias>
<Expires>180</Expires>
</VoIP_Registration>

We use your Config.xml file to register to the trunk based on the registration/authentication information the service provide (FUSION) provided.

And please find the attached WireShark trace that shows Start, Stop and Restart of VG Service. (I could not simulate SIP trunk disconnection as that is not under our control.)

 

 

VG_Downtime_WireShark.zip

[SupportTeam]

WireShark Capture shows that on this system HMP is issuing REGISTERs to the SIP service provider on a regular basis, at short intervals.

If your SIP provider goes offline again please run WireShark to see if those REGISTERs are still being issued. 

It is possible that when your SIP provider goes offline their response to a REGISTER request is: "403 - Incorrect Authentication". In which case no more REGISTER requests would be sent out by HMP any more, and hence the system would not REGISTER when the SIP provider comes back up again.

Also, after a number of unanswered REGISTER requests the Dialogic HMP SIP stack may stop sending them.

Can you please post the ktTel trace file for the day when the outage occurred? Can you advise the time when the outage occurred so that we know where to look in the log file?

 

Direct IP trunk with no ongoing authentication would avoid this issue.

[iTime]

Please find the attached; we suspect the disconnection happened around 11PM on 18th, and the KtTel records are not available during that time.

OK, it is good to know Static IP works better for this. We will consult with the provider if that is option.

Thanks.

VG_Log_3.zip

[SupportTeam]

ktTel from 18th shows last call arriving at 22:55:26 and ending at 22:56:10

No errors were raised by HMP to VoiceGuide for rest of the day to indicate that any issues were encountered, so from VoiceGuide's point of view there were no issues to address...

[iTime]

Yes, that is the problem that neither HMP nor VG noticed disconnection, thus no need to attempt re-initiate the connection on this end.

We will try to speak with the VoIP provider regarding the Static IP option.

Thank you.

[iTime]

I have one more thing that is most likely related with this issue.

We got a call from another customer from ours this morning, saying their IVR server was not responding. We checked the log and it looks really similar with this.

So, could you verify this is the same case as the above that there is no sign of error, but VG or HMP driver could not re-initiate the VoIP connection?

I can see the last call before the problem was 2/19 06:51 PM, and we got a report from the customer that "IVR is down and restarted VG Service around at 2/20 06:57 AM.

Please see the attached and let us know. We could not acquire Wireshark of the moment of the problem for this one either BTW (we had to fix the problem ASAP).

 

VG_Log_2020-01-18-20.zip

[SupportTeam]

In this case we can see in the ktTel log that HMP reported an issue with SIP Registration on Feb 19th, at 1 minute past midnight:

028 000100.508 28376         ev    GCEV_SERVICERESP (board device)
029 000100.508 28376               GCEV_SERVICERESP ResultInfo: gcValue=1285(0x505|GCRV_INTERNAL|event caused internal failure) gcMsg=[Event caused internal failure] ccLibId=8 ccLibName=[GC_H3R_LIB] ccValue=[0x96||] ccMsg=[IPEC_REG_FAIL_internalError] additionalinfo=[]
030 000100.508 28376         WARN  IPEC_REG_FAIL_internalError : internal IP CCLib error encountered while attempting to formulate an outgoing REGISTER request.
031 000100.508 28376         WARN  **********************************************
032 000100.508 28376         WARN  SIP REGISTER Failed
033 000100.508 28376         WARN  **********************************************

 

The Dialogic RTF log is not showing us much more. Only entry for that time is:

02/19/2020 00:01:00.533  30732        5228 gc_h3r                  ERR1         sip_register.cp:5977  !     0 ! SipRegAOR::handleRegistered(reason=-1) - RvSipRegClientGetReceivedMsg(0xc394c48) failed: 0

 

so not detailed information what caused the error, but this error can be caused by VoIP provider not responding to SIP REGISTER messages.

In this case, as there is an error raised, there should be a way of restarting the Register process automatically after some delay. Will look into this and advise.

 

Are you able to obtain from the original system the ktTel logs and Dialogic RTF logs for February 19th? And the Dialogic RTF logs for Feb19th as well? Please post those if possible. Perhaps the issue on that system occurred on 19th and not 11pm on 18th?

[iTime]

Thanks.

Per your request, I have collected ktTel, VG Logs, and RTF logs of the original system from 19th to 20th (until current time).

I now can see "WARN SIP REGISTER Failed" message and "sip_register" error on ktTel and RTF logs as you mentioned.

However, the "sip_register..." errors on RTF log are still present on 20th log, yet the system has been behaving OK on 20th (today) so far without any issues. (So maybe "SIP REGISTER FAILED" error on KtTel shows the true error, but RTF log may not?)

 

One more thing I would like to address was logged on "0219_1049" log. We restarted the VG Service at 10:49AM to re-establish the connection to VoIP service, but then had to restart later again with the reason being (we had found out later) the VG service started as "Unregistered version." 

We had to restart the whole server to correct this. I am just mentioning to let you know that It never happened in the previous version.

Thanks again for all the inputs and help. 

104941.016  18               lic   0 VMWARE[HMP[0727749C1B21_LK055912]] code=Y02C2
104941.016  18               lic   0 ignore as uid not on system  VMWARE[HMP[0727749C1B21_LK055912]] code=Y02C2
104941.017  18               reg   *********************************************************************************
104941.017  18               reg   UNREGISTERED EVALUATION VERSION, lines: 30 id: VMWARE[HMP[0727749C22A3_LK055912]]
104941.017  18               reg   *********************************************************************************
104941.027  18                     OpenChannels start
104941.027  18                     OpenChannels set ktTel_LineOpen loop start

 

VG_TPK_ORI.zip

[iTime]

Sorry for multiple postings, but I found another server that is in the same situation.

This one is from our Dev. Environment, subscribing to the same VoIP service. (I can tell now the service provide had performed global maintenance on their servers/trunks without informing us for the last couple of days...)

I can see Same Register failure, and it is a bit late, but turned on Wireshark today before restarting the VG service.

As you suspected, there is no attempt of "REGISTER" on Wireshark; no SIP or RPT activities at all in the log.

Once I restarted, The VG service came back to normal, issuing "Register" request by the interval.

Thank you.  

 

373 011200.710  5528         ev    GCEV_SERVICERESP (board device)
374 011200.711  5528               GCEV_SERVICERESP ResultInfo: gcValue=1285(0x505|GCRV_INTERNAL|event caused internal failure) gcMsg=[Event caused internal failure] ccLibId=8 ccLibName=[GC_H3R_LIB] ccValue=[0x96||] ccMsg=[IPEC_REG_FAIL_internalError] additionalinfo=[]
375 011200.711  5528         WARN  IPEC_REG_FAIL_internalError : internal IP CCLib error encountered while attempting to formulate an outgoing REGISTER request.
376 011200.711  5528         WARN  **********************************************
377 011200.711  5528         WARN  SIP REGISTER Failed
378 011200.711  5528         WARN  **********************************************
379 011200.711  5528               extension event - not procesed

 

IVR_DMO_NoSvc.zip

[SupportTeam]

We will see if we can replicate the loss of registration responses case and see how VoiceGuide can be made to automatically recover. Will advise in next few days.

 

Regarding the registration issue: both the licenses for VMWARE[HMP[0727749C1B21_LK055912]] and for VMWARE[HMP[0727749C22A3_LK055912]] should be in the license file. Looking at the licensing records we can see that both were supplied. Looks like license for UID VMWARE[HMP[0727749C22A3_LK055912]] is currently not in the license file.

 

 

[iTime]

Quote

We will see if we can replicate the loss of registration responses case and see how VoiceGuide can be made to automatically recover. Will advise in next few days.

Thank you. It would be really great to have this ability on the VG Service. We will be looking forward to your solution.

 

We will try to add the other ID to the License file for this.

Thanks.

 

[iTime]

Hello VG Support,

Any update on this issue?

We are looking forward to hearing good news from you soon.

Thank you.

 

[SupportTeam]

The SIP REGISTER reconnect/re-register upon SIP service dropout is implemented in VoiceGuide v7.6.21

v7.6.21 should be available for general release sometime next week. It will be downloadable from the VoiceGuide Downloads page.

 

[iTime]

That's great!

Thanks for the update.

 

[iTime]

Hello,

I have been checking your Download page for the new version of VG, but have not seen it updated.

Any news?

[SupportTeam]

V7.6.21 has not yet been released. There are some other features that are getting included in that version and more work is still needed on those other features.

[iTime]

Ok, so we expect some delay.

Do you have a rough ETA?

Thanks.

[SupportTeam]

Advice is 'in about a week'.

[iTime]

OK, since now we know you guys are still "active," which is really nice, we will keep checking your Download page.

Hope you all stay safe.

Thank you!

[iTime]

OK, we found the new patch is now available on the download page.

We will test and let you know if there is any issues.

Thank you a lot again!