Handle lines longer than 1000 bytes: error "bad mail input format"

[blueyed]

Currently dma chokes on mail input which contains lines longer than 1000 bytes, because of the buffer being used.
When it does not find a newline at the end, it adds one, but then assumes/requires this to be the last line.

The error is "bad mail input format".

This might happen when using dma to handle web application errors during development / local setup, and the included debug information contains huge environment etc settings (e.g. "LS_COLORS").

I think that it would be saner to not require RFC2822 compliance, but accept the mail nonetheless.

The code in question: https://github.com/corecode/dma/blob/master/mail.c#L376

[corecode]

Do you have a suggestion how to deal with longer lines? Something that
does not complicate the code?

On 01/11/2014 07:00 AM, Daniel Hahler wrote:

Currently dma chokes on mail input which contains lines longer than
1000 bytes, because of the buffer being used.
When it does not find a newline at the end, it adds one, but then
assumes/requires this to be the last line.

The error is "bad mail input format".

This might happen when using dma to handle web application errors
during development / local setup, and the included debug information
contains huge environment etc settings (e.g. "LS_COLORS").

I think that it would be saner to not require RFC2822 compliance, but
accept the mail nonetheless.

The code in question:
https://github.com/corecode/dma/blob/master/mail.c#L376

—
Reply to this email directly or view it on GitHub
#18.

[blueyed]

The easiest approach would probably be to increase the buffer length, from 1000 bytes to maybe 64kB (65536).
It would/could then still fail in the same way, but it's much more unlikely.

Or, just do not add a newline (and effectively aborting / cutting the input) and keep reading in 1000 byte chunks.

[corecode]

The RFC is absolutely clear on this issue:

Each line of characters MUST be no more than 998 characters

and
However, there are so many implementations that (in compliance with
the transport requirements of [RFC5321
http://tools.ietf.org/search/rfc5321]) do not accept messages
containing more than 1000 characters including the CR and LF per line,
it is important for implementations not to create such messages.

It is clear that we can not forward any message with more than 1000
characters per line. What do you suggest as alternative? Insert a
linebreak after 998 characters?

On 01/11/2014 11:15 PM, Daniel Hahler wrote:

The easiest approach would probably be to increase the buffer length,
from 1000 bytes to maybe 64kB (65536).
It would/could then still fail in the same way, but it's much more
unlikely.

Or, just do not add a newline (and effectively aborting / cutting the
input) and keep reading in 1000 byte chunks.

—
Reply to this email directly or view it on GitHub
#18 (comment).

[blueyed]

Well, I think that dma is not creating the message, but only transports it.

The bug is really with the app that hands it over to dma.

Inserting newlines can be very problematic, if the message is meant to contain (encoded) data, and the result may fail to parse afterwards.

Imagine an automated message, with "key=value" pairs on separate lines, where it may result in "key=verylongvalue\r\nstillvalue".

However, breaking lines after 998 bytes is still far better than rejecting the mail altogether.

I can imagine having a configuration option (in dma.conf) for this: RFC2822_SPLITLINES.
If set, CRLF gets inserted after 998 bytes, otherwise the lines would get bypassed unchanged.

[corecode]

It is not optional to forward messages with more than 998 characters per
line: it is forbidden. Do you know how other MTAs handle this? I only
see two options: break line, or reject (bounce).

On 01/12/2014 06:01 PM, Daniel Hahler wrote:

Well, I think that dma is not creating the message, but only
transports it.

The bug is really with the app that hands it over to dma.

Inserting newlines can be very problematic, if the message is meant to
contain (encoded) data, and the result may fail to parse afterwards.

Imagine an automated message, with "key=value" pairs on separate
lines, where it may result in "key=verylongvalue\r\nstillvalue".

However, breaking lines after 998 bytes is still far better than
rejecting the mail altogether.

I can imagine having a configuration option (in dma.conf) for this:
RFC2822_SPLITLINES.
If set, CRLF gets inserted after 998 bytes, otherwise the lines would
get bypassed unchanged.

—
Reply to this email directly or view it on GitHub
#18 (comment).

[blueyed]

Then breaking the line is better than rejecting the mail.

Postfix has a line_length_limit option, with a default of 2048 (http://www.postfix.org/postconf.5.html):

Upon input, long lines are chopped up into pieces of at most this length; upon delivery, long lines are reconstructed.

It sounds like this will effectively pass through long lines?!

[bcoca]

@blueyed, IIRC that limit is for internal processing across Postfix apps and filters, for actual smtp this is the setting:
http://www.postfix.org/postconf.5.html#smtp_line_length_limit

[corecode]

Good catch. Any suggestions how we should handle this?

On 03/26/2014 08:24 PM, Brian Coca wrote:

@blueyed https://github.com/blueyed, IIRC that limit is for internal
processing across Postfix apps and filters, for actual smtp this is
the setting:
http://www.postfix.org/postconf.5.html#smtp_line_length_limit

—
Reply to this email directly or view it on GitHub
#18 (comment).

[bcoca]

I think rejection is OK, but maybe add more info to the message, 'bad mail input format' (I just ran into this) is a bit too succinct, knowing that its a line length error helps pinpoint the issue much faster. Also, quoting RFC helps convince programmers change their code vs demand that the MTA just accept their emails.

[corecode]

alternatively we could force a linebreak when accepting the mail, like
postfix does when sending?

On 03/26/2014 08:47 PM, Brian Coca wrote:

I think rejection is OK, but maybe add more info the the message, 'bad
mail input format' (I just ran into this) is a bit too succinct,
knowing that its a line length error helps pinpoint the issue much
faster. Also, quoting RFC helps convince programmers change their code
vs demand that the MTA just accept their emails.

—
Reply to this email directly or view it on GitHub
#18 (comment).

[blueyed]

The Postfix approach seems sensible.
To me everything is better than rejecting the mail.

quoting RFC helps convince programmers change their code

I agree. There could be a warning instead of the current error.

Given the use case of dma (for personal or lightweight use in containers) I do not think it should be too strict (i.e. rejecting the mail).

[bcoca]

I disagree, I use dma because it it simple and lightweight and doesn't try to do complex stuff like line splitting with character or armor encodings. If it accepts mail and it then gets rejected by another mailer, it creates more issues than it solves.

I am a fan of fail early and fail loudly, if you start accepting emails and push the problem downstream you never have an incentive to fix the actual issue, the program composing non compliant smtp messages.