Will the cooler continue to dispense water even when the power is off (as during a power outage)?

Yes but not Hot Water

My guess is ... only a limited amount. The pump needs to pull the water up from the 5 gallon container - it can't do so without power. So the only water you'll be able to access is the water that's already in the heat/cool compartments.

Yes, the dispenser will be able to dispense water in the event that electricity is not available. The dispenser only requires electricity to heat and cool the water.