Can this be used on the transom of a boat? If so, what setting (hard wood, soft wood, wall)?

if the transom is made of wood, determine the species and use either hardwood or softwood depending on that. If its made of fiberglass use building materials mode. these meters work by measuring conductivity so if a material that is conductive is measured it will not read accurately.