can you use the auger to dig up tree roots

No. When it hits roots it is going to stop.

You will probably break your arms. Once any handheld auger hits a big tree root, the bit stops turning instantly, and the handle takes all of the force. With a one person auger, all of the torque is held in place by you.

Sorry, but an auger probably won't help with the roots. The auger blades just loosen dirt and channel it upwards. It cannot actually cut through roots. It may break through smaller roots, but if the roots are large enough to disturb the fence, they are probably too large for an auger to break through. Other options may be the use of a jackhammer with wide blade or reciprocating saw once you expose the roots. You might check with your insurance company to see if the damage would be covered and then have a company with the proper tools take care of the issue. In some areas, damage on your property caused from trees on the neighbor's property must be paid for by the neighbor that owns the tree.