Holloway, who steered the Pilgrims to seventh in the Coca-Cola Championship table, quit on Wednesday after just one full season at Home Park and was named as Leicester's new manager 24 hours later.
But the Pilgrims gave caretaker-boss Tim Breacker, Holloway's former assistant, a perfect start with Halmosi side-footing the visitors into a 24th-minute lead.
The Hungary international held his nerve after Sylvan Ebanks-Blake had played him through one-on-one with goalkeeper Ian Bennett to convert into the bottom corner from 12 yards.
The win lifts Plymouth into the play-off places, while United's steady climb up the table following back-to-back wins was checked by their first league defeat in seven matches.
Before Plymouth's opener, Breacker saw Barry Hayles, Ebanks-Blake and Halmosi all fire in long-range shots, while the Blades had to wait until the 21st minute before their side's first effort when Danny Webber's 25-yard drive flew wide.
United, who have only lost once at home this season, struggled to carve out any chances, with Nick Montgomery's two speculative efforts well off target.
Blades boss Bryan Robson had the luxury of naming an unchanged side for the third match running.
Robson's side were unbeaten in their last six league matches, but did not look like making it three successive wins as Ebanks-Blake saw his venomous shot blocked on the line by home defender Phil Bardsley just before the break.
Keith Gillespie volleyed narrowly over the crossbar from 20 yards for the home side soon after the restart, but Plymouth still held the upper hand.
Ebanks-Blake was denied a second for Plymouth by the woodwork when his 30-yard shot hit the post, while David Norris volleyed just wide.
Robson responded with a double substitution in the 58th minute, Michael Tonge and Billy Sharp replacing Gillespie and David Carney respectively.
But still it was Plymouth who pressed. Midfielder Lee Martin fired in a shot that forced a fumbling save from Bennett.
The Blades slowly began to make in-roads with Tonge prompting most things from midfield, but they still struggled to create clear-cut scoring chances.
Gary Naysmith went close to hauling United level when he volleyed from close range after Tonge's corner was half-cleared, but Paul Connolly blocked his goalbound effort.
Chris Armstrong lashed in a shot from 25 yards that Plymouth goalkeeper Romain Larrieu did well to hold.
But despite throwing everything at the visitors in the closing stages, United were unable to find a way through, with the Pilgrims determined to end a turbulent week on a high note.
The Blades slipped to their first home defeat since going down 2-1 to Southampton at the end of September.