Early on, a Barber cross was headed just wide by Talboys, but moments later United countered when Ollie Stanbridge released Kieran Gallagher and his low angled drive was diverted for a corner by Blake.
It took United time to get into gear and Gary Phillips did well to block a Robson shot. Talboys then struck an upright and Newbery headed wide from close range.
But against the run of play United took the lead through Lee Cook with a superb piece of skill on 31 minutes. From just outside the box, Cook flicked the ball over a defender and smashed his shot past a stunned Blake.
Two minutes into the second half Gallagher clipped over a cross and a Neil Catlin header put United 2-0 up.
On 64 minutes Newbery waltzed through unchallenged to pull a goal back, and a double save by Phillips retained United's slender lead. He did well to parry an Elverson cross-shot and did even better to tip over the close range follow-up by Denys.
After 83 minutes Gallagher cut-in from the left side and squared the ball back to Cook who lashed the ball past Blake to score his second and Aylesbury's third.
Just before time Denys reduced the arrears and there were three anxious minutes before the home fans were able to celebrate a hard-fought and memorable victory. Memorable certainly for Cook who will remember his first senior United goal for a long time.