Which/what frequency/protocol should be chosen?

The Hub and the extender have the capability to use both zigbee and zwave. The device will automatically select the appropriate frequency. Iris does however seem to prefer Zwave devices over zigbee, as zigbee devices are known to have less range than zwave on iris, as well as lesser support.

ZigBee and Z-Wave are two different short range wireless communication standards. The Iris Hub has circuitry that allows it to "talk" in both standards simultaneously. The Iris Hub creates Zigbee and Z-wave networks in your home - similar to your home area WiFi network. As such, these networks will have a certain range unique to your property. The Iris Range Extender will help make the range of these Zigbee and Z-Wave networks larger.