O’Donnell took the lead on three separate occasions, only for the world No3 to respond each time with breaks of 91, 94 and 52.
However, O’Donnell, who registered three half-century breaks of his own, held firm in the decider to record his first-ever victory over the former world champion.
Defending champion and former world champion Kyren Wilson made it through, dispatching Ben Mertens 4-0 in a comfortable progression to round two.
Luca Brecel also advanced with a 4-1 victory over Lei Peifan, while John Higgins beat Liam Highfield.
For O’Sullivan, ranked 13th in the world, the loss marked a disappointing return to domestic competition.
He had not featured at a UK tournament since his narrow 13-12 second-round defeat to Higgins at April’s World Championship, and was appearing at the Northern Irish Open for the first time since 2022.
