Do I owe my son forgiveness for having an affair with my woman?

You said it yourself..you didn't want or need him, you only had an obligation as a mother. He's never really been a son to you, and has since caused trouble once he was part of your life. It would be pointless and a pain in the neck. Forget you even have a kid, he can clearly look after himself.