Is this considered adultery? or unethical?

Yes, it is adultery, because he is married. I strongly suggest that you get away from him. Don't work here. I don't care if they pay well, it's not worth it. There's too much potential for a big mess. Get the hell away.