Smith put his spot-kick to the top left corner after Blades keeper Paddy Kenny had given his side the advantage by saving the visitors' first penalty.
Kenny immediately turned villain by hitting the next penalty against the right post and the Hornets converted all their remaining spot-kicks with substitute Paul Shaw also missing for United.
The Blades' regular penalty-taker, Jack Lester, had been substituted in extra time.
To compound the hosts' misery manager Neil Warnock faces a touchline ban after being sent to the stands for his reaction after referee Mark Clattenburg denied strong claims for a penalty in injury time.
It is a shame the teams could not emulate his passion as a lively opening to the game soon subsided and it started to resemble a practice match which sent the crowd into a stupor long before the end.
It was hard to believe a lucrative cup tie with one of the Premiership big boys potentially awaited the winners as the lack of urgency and the seeming inability of either team to construct productive moves was quite alarming.
The teams had signalled their initial attacking intentions with both sides finding plenty of space in front of a sparse crowd.
Watford broke upfield with Bruce Dyer who found Ashley Young six yards out and his first-time shot was inches wide with Blades keeper Kenny beaten.
After just nine minutes Nick Montgomery gathered the ball at the edge of the area from Paul Thirlwell.
He advanced a couple of yards before unleashing a shot across goal which beat Hornets keeper Richard Lee but came back off the post before being hacked away for a corner.
Watford boss Ray Lewington rested forwards Danny Webber and Heidar Helguson and it was difficult to see how a goal would materialise with highly-rated youngster Hameur Bouazza looking to be their most likely outlet.
The game was crying out for a spark of creativity and a minute before half-time Kevan Hurst, on for the injured Michael Tonge, ran at the Hornets defence before unleashing a shot which found its way through to Andy Gray unmarked 12 yards out.
He had time to pick his spot but the Blades' top scorer was thwarted by Lee who pushed the ball around the post for a corner.
The first action of the second half saw Bouazza's shot from the right-hand side of the area beat Kenny but go a yard wide of the left post.
The Blades thought they had gone ahead in the 53rd minute when Leigh Bromby's long throw was flicked on by Chris Morgan and Lester dived to head in at the far post but he was adjudged to be offside.
The Blades fans were then clamouring for a goal after Lester's deflected lob had Lee backpedalling but he managed to narrowly prevent the ball from crossing the line.
The ever-busy Lester was inches away from a goalscoring chance in the 72nd minute when Lee just managed to get a hand to the ball as the pair met in mid-air.
Seconds after the home side were appealing for a penalty for handball Dyer was released by Bouazza and he almost squeezed the ball in at the near left-hand post but Kenny just managed to get a hand on it.
The home side were pressing to the end and the ball was hacked away following a scramble in the box before Warnock was sent to the stands after Clattenburg denied his side a penalty when Jay DeMerit appeared to handle Jon Harley's cross.
In the first period of extra time Gray put a header inches wide when he had a good chance from six yards out and the same player floated a header inches over from Bromby's pass.
With just one minute to go DeMerit put a free header over the bar when he could well have won the tie for the Hornets without the need for a penalty shoot-out.