The Ducks took the lead on 17 minutes when Chris Bangura skilfully rounded Stortford keeper Micky Desborough to score from a tight angle and it was 2-0 to the Ducks 14 minutes later when Rob Paris out jumped Desborough to head home his second goal in successive home matches.
The visitors didn't let their heads drop however and were back in the game on 35 minutes when Trevor Paul took advantage of slack defending and lobbed Adam Wheeler from relatively close range.
Matters were level on 38 minutes when Wheeler was well beaten by a superb 25 yard free kick from Freddie Hyatt which David Beckham would have been proud of.
Early in the second half Vinnie John missed a great chance to give Stortford the lead before Greg Williams restored Aylesbury's lead with a superb goal which saw him take the ball across the face of the area, past four defenders and hammer it into the bottom left corner of the net to send the home support into a frenzy.
The three points were not to be Aylesbury's though as in the 4th minute of injury time Vinnie John got the better of Chris Boothe to fire home from close range despite more than a hint of a foul.