I finally realized the key to getting self signed certificates to work with Mail.app after much reading between the lines. The trick is to make sure the Common Name value in your certificate matches the mail server domain name.
To quickly generate a self signed cert, follow this quick guide (there are many ways to do it).
When it asks for your common name (it may say YOUR NAME), put in the fully qualified domain name of your mail server. i.e., mail.example.com, NOT example.com, which was my previous problem. Obviously, give this certificate to your mail server before proceeding.
Next, follow these instructions on importing your certificate from this Apple support page.
Note that if you view the info for the imported cert in the Keychain application, at the bottom you can set the trust settings. If not set, set the value to 'always trust'.
After that, Mail.app should stop asking you to OK the connection to your mail server.
Leave a comment