Will this work on connecting an old 5" diameter plastic pipe to 4" pvc?

No, You would have to reduce the 5" to 4" first with a solvent-weld reducer. This compresses only enough to seal on the designated pipe size. It's fairly stiff rubber.

This flexible coupling will not work for your purposes based on the size of the pipes you have given. However, the RCC100 (item # 23438) will work for you. The inner diameter on the bigger side measures 5 1/4", so it will be big enough for your 5" pipe. Please remember that the flexible couplings are meant for drain, waste and vent applications only with little to no water pressure.