Do I have to take down the drywall before install Hardi-Backer board? Or just install on top of the drywall?

when you say adhesive I hope you mean thinset. You should spread it on the hardibacker and back butter the tile. place it in immediately. don't let it dry first. work in small areas at a time.

If the current drywall is solidly installed you can screw the backboard right on the drywall by driving screws right into the existing frame. You should not have to put a layer of thinset and let it dry. Using a 1/2 inch notched trowel should give you the coverage and adhesion you desire. I would install the stone over several days. If you put to much up in one day you have a higher potential for it to sag before drying.