WHMCS> Setup> General Settings > Mail> Mail Type

Mail Type: SMTP
SMTP Port: 25/587
SMTP Host: mail.mydomain.com
SMTP Username: user@mydomain.com
SMTP Password: Password
SMTP SSL Type None: (None - SSL - TLS)