Andy Liddell scored the winning spot-kick after Kenny had saved from Teddy Sheringham and Marlon Harewood.
The result sent United to Highbury for a fifth round tie on Saturday and piles the pressure on West Ham boss Alan Pardew.
Sheringham's 63rd minute penalty, after Chris Morgan had been sent off for a foul on Harewood, equalised Liddell's eighth minute free-kick.
Despite three league defeats on the trot at Bramall Lane, United swept into the lead when Liddell powered home from 20 yards, low to Stephen Bywater's left, after Carl Fletcher had fouled Paul Shaw.
Andy Gray had United's best chance to double the lead when he anticipated Malky Mackay's weak back header on 20 minutes, but his lob was easy for Bywater.
Anton Ferdinand came to West Ham's rescue on 29 minutes blocking Gray's goalbound effort after Phil Jagielka had pulled the ball back into a crowded penalty area.
Ferdinand again had to block a great effort on 37 minutes following Leigh Bromby's cross.
Sheringham then rounded Kenny after Danny Cullip's slip, but when he cut back inside, his shot was deflected over.
Michael Tonge beat Ferdinand for pace and saw his shot saved as United pressed in the second half.
The game took a turn West Ham's way on 62 minutes when Morgan was sent off for a second bookable offence after hauling down Harewood ten yards from goal. Sheringham slotted the spot kick home.
Hayden Mullins missed the best chance in extra time, heading Carl Fletcher's cross over the bar.
Gray and Jon Harley scored for United in the shoot-out, while teeanger Mark Noble was the Hammers' only scorer.