Chris Coleman's men, just three days before their FA Cup quarter-final with Chelsea, had the best of the first half but failed to take the chances they created.
Freddy Eastwood forced keeper Paddy Kenny into a scrambled save before Jordan Henderson wasted a gilt-edged opportunity when he fired wired after being put through by Robbie Simpson.
But United, who had looked impressive enough without really creating many chances, broke the deadlock on the stroke of half-time.
Leigh Bromby headed goal wards and keeper Keiren Westwood pushed the ball clear, but a linesman ruled that it had crossed the line.
The keeper raced 45 yards to protest and was promptly booked.
But if the home side had a burning sense of injustice in the second half it didn't show as, eight minutes after the interval, Chris Morgan rose between Scott Dann and Stephen Wright to head home a second.
The veteran defender had been unlucky in the first half when Westwood had pulled off a superb save to keep the effort at bay.
After Morgan's goal it looked as if Coventry were down and out but Coleman made a couple of changes and suddenly they picked up the pace.
They earned their reward when Aron Gunnarsson launched a long throw in the 70th minute and Dann was able to jump unchallenged and headed calmly down.
Henderson then blasted over after a scramble when he should have done better, but the Blades did well to limit Coventry's chances.
Worryingly for Coleman, highly-rated Dan Fox had to be replaced with a knee injury at half time and sub Clinton Morrison limped off at the end.