VoIP calls are not sent to a specific 'line'. There are no 'lines' or 'channels' like on physical systems.
The task of sending out the 'busy' is usually done by the PBX (for both traditional and VoIP systems).
The PBX checks to see if it can send the call to the destination and if it cannot as there already is the max number of calls to that destination (or all physical lines/channels are already used by other calls) then the PBX sends out the 'busy', and the end system (which is VoiceGuide in this case) is not even notified that there was a call.
If you cannot set the '2 calls max per extension' limit on the Avaya then the checks of how many calls are currently connected to the particular extension would need to be made on the VoiceGuide system, and if the incoming call the '3rd simultaneous' call then VoiceGudie can just end that call without answering it.
You need to set up your script to "start without answering the call" and then in the script check how many other calls to the same extension are currently on system. If 2 calls to the same extension currently exist on system then just hangup the call (ie. branch in script to the Hangup module). The call would be ended without ever being answered.
Keeping track of how many calls are connected to each extension can be done by using a Global Result Variable for each extension that holds the current number of calls. This variable would be checked and incremented in the script, and then decremented when call ends. The decrementing should be carried out in a "On Hangup" script.