Add AncestryInvalid zero value for ancestry errors

CheckAncestry returned the literal 0 alongside every error, which equals AncestryReachable. A caller that ignored the error would read a failure as a provable fast-forward. Make the zero value a distinct AncestryInvalid sentinel and return it explicitly on every error path.