Pulled door handle off when frozen!, how do I refit it ?

Did not realise door was frozen solid gave door handle the usual tug to open after unzapping the remote from the key. nothing, so I gave the handle a harder pull only to be left with the door handle in my hand. It dosent look broken just off from it's fixings at either end. How do I refit it ?, I can't see a way into removing the door panels from inside the door either. Currently accessing the car from the passenger side, what a pain! help!