[Feature/Bug] Fully support SMTP for self-hosting (Frontend UI missing fields + Backend global fallback ignores SMTP)

[Bartolumiu]

When self-hosting Intlayer, the current implementation strongly enforces vendor lock-in to Resend, despite documentation and UI elements suggesting custom SMTP is supported.

During a deep dive into configuring an external SMTP relay (Oracle Cloud / Postfix), I managed to identify several blocking bugs across the frontend UI and backend email service logic.

1. Frontend Bug: Organization Settings UI Missing SMTP Fields

In the dashboard (/organization), when navigating to "Email Configuration" and selecting the "Use a custom mailer" checkbox, the UI provides a toggle between "Resend" and "SMTP".

• The Bug: Clicking the "SMTP" tab does not render the SMTP input fields (Host, Port, User, Password). The DOM continues to display the <h4 ...>Resend Configuration</h4> header and the resendApiKey input field.
• Silent Failure: Clicking "Save Email Configuration" does nothing. No network request is dispatched. It appears the frontend schema validation (likely Zod) intercepts the submission because provider is set to smtp but the required SMTP fields are physically missing from the DOM, causing a silent validation crash.

2. Backend Bug: System Emails Hardcoded to Resend Fallback

System-level emails (like Password Resets and Magic Links) do not pass an organizationId to the sendEmail function. When organizationId is null, the backend completely ignores any .env SMTP variables and hardcodes a fallback to Resend.
In apps/backend/src/services/email.service.tsx (Lines ~1285-1312, end of the file inside "sendEmail" method):

const mailerConfig = organizationId
    ? await resolveOrganizationMailer(organizationId)
    : null;

try {
  if (mailerConfig?.provider === 'smtp') {
    await sendViaSmtp(mailerConfig, {{ /* ... */ });
  } else {
    // BUG: Unconditional fallback to Resend if organizationId is null.
    // Ignores process.env.MAIL_PROVIDER and process.env.MAIL_SMTP_* entirely.
    const apiKey = mailerConfig?.resend?.apiKey ?? process.env.RESEND_API_KEY;
    const resend = new Resend(apiKey);

    // ...
  }

  logger.info(`Email sent ${type} to ${to}`);
} catch (err) {
  logger.error(err);
}

Because NODE_ENV=production relies on this fallback for non-org emails, it is currently impossible to use SMTP for user sign-ups or password resets.

3. Backend Bug: Silent Crash on Email Validation Errors

If the email transporter fails to initialize (e.g. forcing MAIL_PROVIDER=smtp via .env but failing the if block, or using an RFC-invalid MAIL_FROM string like Hello | Intlayer <email@...> without quotes), the error handler drops the actual exception.

• The logs instantly print error: undefined in the same millisecond as info: sending reset password email.
• The server does not crash, but the email even it entirely aborted with no helpful stack trace.

Steps to Reproduce

1. Spin up the intlayer-selfhost Docker image.
2. Provide valid MAIL_SMTP_* variables in the .env and omit RESEND_API_KEY.
3. Attempt to request a password reset via the UI.
4. Observe the error: undefined in the container logs and no SMTP network attempt.
5. Log into the dashboard, go to Organization Settings, and click the "SMTP" tab under Email Configuration. Observe the missing input fields.

Proposed Solution / Feature Request

1. Fix Conditional Rendering in UI: Wire up the frontend components in the Organization Settings so the SMTP tab actually renders the inputs for smtp.host, smtp.port, smtp.user and smtp.password.
2. Global SMTP Fallback: Modify email.service.tsx so that if organizationId is null, it checks process.env.MAIL_PROVIDER (or similar) to determine if it should initialize Nodemailer using global process.env.MAIL_SMTP_* variables instead of blindly defaulting to Resend.
3. Better Error Handling: Improve the try/catch block in sendEmail to log the actual err.message instead of undefined when Nodemailer or Zod throws a synchronous setup validation error.

Environment & Debugging Setup

• Deployment: Docker (aymericzip/intlayer-selfhost)
• OS: Ubuntu Server 24.04.4 LTS
• Relevant Docs: The pl and en-GB (among other languages) documentation incorrectly lists MAIL_SMTP_HOST as valid global variables, whereas the base en docs removed them.

To isolate these bugs, bypass TooManyRedirects proxy loops, and trace the undefined error, the following configuration was used. Notice the build-args workaround required to inject a custom domain, and the sidecar SMTP relay used to test the backend's email validation logic:

docker-compose.yml

services:
  intlayer:
    build:
      context: ./intlayer-src
      dockerfile: docker/selfhost/Dockerfile
      args:
        - VITE_BACKEND_URL=${BACKEND_URL}
        - VITE_DOMAIN=${DOMAIN}
        - VITE_SITE_URL=${APP_URL}
        - VITE_IDE_URL=${APP_URL}
      container_name: intlayer
      restart: unless-stopped
      env_file:
        - .env
      volumes:
        - intlayer-data:/data
      networks:
        - proxynet
        - internal

  # Sidecar relay to isolate Intlayer's internal SMTP validation logic
  smtp-relay:
    image: boky/postfix:latest
    container_name: intlayer_smtp_relay
    restart: unless-stopped
    networks:
      - internal
    environment:
      - RELAYHOST=${OCI_SMTP_HOST}:${OCI_SMTP_PORT}
      - RELAYHOST_USERNAME=${OCI_SMTP_USER}
      - RELAYHOST_PASSWORD=${OCI_SMTP_PASSWORD}
      - ALLOW_EMPTY_SENDER_DOMAINS=true

volumes:
  intlayer-data:

networks:
  proxynet:
    external: true
  internal:
    internal: true

.env

# Application URLs
APP_URL=https://intlayer.example.com
NEXT_PUBLIC_APP_URL=https://intlayer.example.com
BACKEND_URL=https://api.intlayer.example.com
INTLAYER_BACKEND_URL=https://api.intlayer.example.com
BETTER_AUTH_URL=https://api.intlayer.example.com
NEXT_PUBLIC_BACKEND_URL=https://api.intlayer.example.com
DOMAIN=example.com

# SMTP Diagnostic Configuration
MAIL_PROVIDER=smtp
MAIL_SMTP_HOST=smtp-relay
MAIL_SMTP_PORT=25

# Fed to Intlayer to prevent 'error: undefined' validation crashes, while sidecar handles actual external auth.
MAIL_SMTP_USER=dummy_user
MAIL_SMTP_PASSWORD=dummy_password
MAIL_FROM=accounts@example.com

# Reverse Proxy Fixes
NODE_TLS_REJECT_UNAUTHORIZED=0
BETTER_AUTH_TRUST_PROXIES=true

[aymericzip]

Hey @Bartolumiu

Thanks for reporting

Just as a note, the self hosting is actually not officially released, it will come with the v9, so all improvements are welcome

Custom SMPT is implemented, but not tested

I'm traveling actually, and preparing the v9 so I invite you to submit a PR to implement needed changes if you wanna contribute

Otherwise I will try to implement it next week

[aymericzip]

The SMTP set up is configurable in the /organization dashboard