I am having the same problem with new v12 installation on self-hosted server. There are several services on the same server, will all of them I was able to configure email without any issues, except Erpnext, which constantly is giving “Incoming email account not correct” when trying to configure email domain.
I guess the code that is doing domain test is in apps/frappe/frappe/email/doctype/email_domain/email_domain.py and the code looks a bit strange.
The code is trying to establish connection to email server, however, it does not supply any user name and password for this. Some low security servers might be responding to such request, however, most of the email servers would refuse such a connection, and test will fail.
It is not obvious why such a domain test is done at all. Usually the test would involve sending test email from the email server to make sure that one can access email server. So, there should be possibility to enter user name and password.
It looks like this feature is designed to prevent users to configure own email server, and without email, any erp system is useless.