Bad rfc822 field name? (fwd)

hi all,

i have done some further investigation of my problem and it would appear
that pine may be the culprit. i was hoping some other people may have an
opinion on this, and an appropriate way to deal with.

my suggestion would be to check for presence of : on the line before we
assume its part of a new header line?---------- Forwarded message ----------
Date: Fri, 23 Nov 2001 12:25:35 +1000 (EST)
From: matt carter matt@iseek.com.au
To: Jesse Vincent jesse@bestpractical.com
Subject: bad rfc822 field name?

hey jesse,

i found this problem still in 2.0.9 with long subject lines and it has me
baffled but i think ive got it nailed.

webrt:/var/log/rt# cat rt.log.6758.1
Bad RFC822 field name ’ Needed’
at /opt/local/packages/rt2/lib/RT/Template.pm line 297
Bad RFC822 field name ’ Needed’
at /opt/local/packages/rt2/lib/RT/Template.pm line 297
webrt:/var/log/rt#

actual subject was

-snip-
Subject: (mercury) remove dialup account for barry nillsen lawyers no longer needed
-snip-

it looks like rt seems to chop the subject when it gets to a certain
length and i’m getting 1 error per template. in the above there is
two erors because i have oncreate notify owner, notify group.

whats interesting is if i display the ticket the title bar at the top says

-snip-
RT/iseek-track: Ticket #540 (mercury) remove dialup account for barry nillsen lawyers no longer needed
-snip-

which is a complete subject yet when you actually look at the ticket
history you can see that the word “Needed” which caused rt to barf has
been excluded from the subject line.

so in one sense it’s fine, but in another it’s broken so i definately know
i’m not going crazy :slight_smile:

-snip-
History
Display mode: [Brief headers] [Full headers]
Fri Nov 23 11:24:17 2001
matt - Ticket created
[Reply] [Comment]

     Date: Fri, 23 Nov 2001 11:24:14 +1000 (EST)
     From: matt carter <matt@iseek.com.au>
     To: <support@iseek.com.au>
     Subject: (mercury) remove dialup account for barry nillsen

lawyers no longer

-snip-

Further investigation shows there is a new line in the subject alright.

mysql> select Subject from Tickets where id = 540;
| Subject
|
| (mercury) remove dialup account for barry nillsen lawyers no longer
needed |
1 row in set (0.00 sec)

mysql>

You can see below that quite a number of tickets have the same problem.

mysql> select count(Subject) from Tickets where Subject like “%\n%”;
| count(Subject) |
| 18 |
1 row in set (0.01 sec)

mysql>

On further investigation I found this :

cat r

From: matt@iseek.com.au
To: support@iseek.com.au
Subject: a long line with no new lines in it that i definately know and
was not long enough try again

some content

cat r | sendmail -t

works fine. i get my auto reply.

-snip-
Date: Fri, 23 Nov 2001 12:17:24 +1000
From: support support@iseek.com.au
To: matt@iseek.com.au
Subject: [iseek-track #548] a long line with no new lines in it that i
definately know and was not long enough try again

Thank you for emailing the iseek Communications Support Team.
-snip-

if i cut and paste the exact same subject into pine, and send it.

-snip-
Date: Fri, 23 Nov 2001 12:17:09 +1000 (EST)
From: matt carter matt@iseek.com.au
To: support@iseek.com.au
Subject: a long line with no new lines in it that i definately know and was not long enough try again
-snip-

i get

Bad RFC822 field name ’ Not Long Enough Try Again’
at /opt/local/packages/rt2/lib/RT/Template.pm line 297
Bad RFC822 field name ’ Not Long Enough Try Again’
at /opt/local/packages/rt2/lib/RT/Template.pm line 297

so this appears to be a combination of two problems:

  1. pine potentially? inserting a \n in the outbound message

  2. rt not checking for the existence of : before we assume it’s a header.

thoughts?

–matt

Was Re: [rt-users] bad rfc822 field name? (fwd)

i have done some further investigation of my problem and it would appear
that pine may be the culprit. i was hoping some other people may have an
opinion on this, and an appropriate way to deal with.

my suggestion would be to check for presence of : on the line before we
assume its part of a new header line?

Looking at this old problem (which we’re getting at the moment, which
throws mh for a merry dance), it seems to be in RT::Template being rather
silly about headers ( and I’m a tad busy to go and force Jesse to write ‘I
will not write a mail-parser without understanding mail standards’ several
thousand times :wink: )

RFC822 cites:

 3.1.1.  LONG HEADER FIELDS

    Each header field can be viewed as a single, logical  line  of
    ASCII  characters,  comprising  a field-name and a field-body.
    For convenience, the field-body  portion  of  this  conceptual
    entity  can be split into a multiple-line representation; this
    is called "folding".  The general rule is that wherever  there
    may  be  linear-white-space  (NOT  simply  LWSP-chars), a CRLF
    immediately followed by AT LEAST one LWSP-char may instead  be
    inserted.  Thus, the single line


    can be represented as:

                JJV@BBN

[more examples]

         The process of moving  from  this  folded   multiple-line
    representation  of a header field to its single line represen-
    tation is called "unfolding".  Unfolding  is  accomplished  by
    regarding   CRLF   immediately  followed  by  a  LWSP-char  as
    equivalent to the LWSP-char.

So, pulling a sample out of our hat, we have a multi-line subject header
as follows:

  received either, from your service, from a service that you host,
  solicitoring replies to your service. In my country, this kind of
  email is illegal. please investigate.

Line 292 of RT::Template builds up the MIME::Entity based on a
split(/\n/). Ok, but theres no joining of header lines that have been
folded.

So, if we change the loop to read:

if ($headers) {
my $prevkey = undef;
foreach $header (split(/\n/,$headers)) {
if( $header =~ /^\s+/ && defined( $prevkey) ){
# This is a continuation.
# Although implied in the docs, “-1” as the index to add doesn’t
# seem to work as wanted. Use replace.
$self->{‘MIMEObj’}->head->add( $prevkey, $self->{‘MIMEObj’}->head->get( $prevkey ) . $header );
# Note that technically we should be doing get($prevkey) . “\r\n” . $header
# and then calling unfold() as the above could lead to strange
# numbers of spaces.
}else{
(my $key, my $value) = (split(/: /,$header,2));
chomp $key;
chomp $value;
$self->{‘MIMEObj’}->head->fold_length($key,10000);
$self->{‘MIMEObj’}->head->add($key, $value);
# Cache the key in case it continues.
$prevkey = $key;
}
}
# Unfold all the headers if still needed.
$self->{‘MIMEObj’}->head->unfold();
}

Applying this change to RT::Template results no more complaints about bad
RFC822 headers :wink:

Other portions of the RT code base also exhibit certain confusion about
folded headers.

                         Bruce Campbell                            RIPE
                                RIPE 41                             NCC
                   Jan 14-18, Amsterdam                      Operations

I believe that a fix for this was already in cvs for 2.0.12. (In my defense,
I don’t think I wrote that code :wink: And I beleve everywhere else that we’re
dealing with headers, we’re using Mail::Internet or MIME::Head,
rather doing it myself.On Wed, Jan 16, 2002 at 04:49:25PM +0100, Bruce Campbell wrote:

Was Re: [rt-users] bad rfc822 field name? (fwd)

On Wed, 28 Nov 2001, matt carter wrote:

i have done some further investigation of my problem and it would appear
that pine may be the culprit. i was hoping some other people may have an
opinion on this, and an appropriate way to deal with.

my suggestion would be to check for presence of : on the line before we
assume its part of a new header line?

Looking at this old problem (which we’re getting at the moment, which
throws mh for a merry dance), it seems to be in RT::Template being rather
silly about headers ( and I’m a tad busy to go and force Jesse to write ‘I
will not write a mail-parser without understanding mail standards’ several
thousand times :wink: )

RFC822 cites:

 3.1.1.  LONG HEADER FIELDS

    Each header field can be viewed as a single, logical  line  of
    ASCII  characters,  comprising  a field-name and a field-body.
    For convenience, the field-body  portion  of  this  conceptual
    entity  can be split into a multiple-line representation; this
    is called "folding".  The general rule is that wherever  there
    may  be  linear-white-space  (NOT  simply  LWSP-chars), a CRLF
    immediately followed by AT LEAST one LWSP-char may instead  be
    inserted.  Thus, the single line

        To:  "Joe & J. Harvey" <ddd @Org>, JJV @ BBN

    can be represented as:

        To:  "Joe & J. Harvey" <ddd @ Org>,
                JJV@BBN

[more examples]

         The process of moving  from  this  folded   multiple-line
    representation  of a header field to its single line represen-
    tation is called "unfolding".  Unfolding  is  accomplished  by
    regarding   CRLF   immediately  followed  by  a  LWSP-char  as
    equivalent to the LWSP-char.

So, pulling a sample out of our hat, we have a multi-line subject header
as follows:

Subject: here is a sample of SPAM Filth and Depravity that I have
received either, from your service, from a service that you host,
solicitoring replies to your service. In my country, this kind of
email is illegal. please investigate.

Line 292 of RT::Template builds up the MIME::Entity based on a
split(/\n/). Ok, but theres no joining of header lines that have been
folded.

So, if we change the loop to read:

if ($headers) {
my $prevkey = undef;
foreach $header (split(/\n/,$headers)) {
if( $header =~ /^\s+/ && defined( $prevkey) ){
# This is a continuation.
# Although implied in the docs, “-1” as the index to add doesn’t
# seem to work as wanted. Use replace.
$self->{‘MIMEObj’}->head->add( $prevkey, $self->{‘MIMEObj’}->head->get( $prevkey ) . $header );
# Note that technically we should be doing get($prevkey) . “\r\n” . $header
# and then calling unfold() as the above could lead to strange

numbers of spaces.

  }else{
    (my $key, my $value) = (split(/: /,$header,2));
    chomp $key;
    chomp $value;
    $self->{'MIMEObj'}->head->fold_length($key,10000);
    $self->{'MIMEObj'}->head->add($key, $value);
    # Cache the key in case it continues.
    $prevkey = $key;
  }
}
# Unfold all the headers if still needed.
$self->{'MIMEObj'}->head->unfold();

}

Applying this change to RT::Template results no more complaints about bad
RFC822 headers :wink:

Other portions of the RT code base also exhibit certain confusion about
folded headers.


Bruce Campbell RIPE
RIPE 41 NCC
Jan 14-18, Amsterdam Operations


rt-devel mailing list
rt-devel@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-devel

http://www.bestpractical.com/products/rt – Trouble Ticketing. Free.

Ok. I was wrong. the patch is not yet in the cvs copy,
but is noted at http://fsck.com/rt2/Ticket/Display.html?id=1058

I can’t actually recall what I was waiting on, but I’ll look harder
at it when I get back from Slovenia and Austria next week.

-jOn Wed, Jan 16, 2002 at 04:49:25PM +0100, Bruce Campbell wrote:

Was Re: [rt-users] bad rfc822 field name? (fwd)

On Wed, 28 Nov 2001, matt carter wrote:

i have done some further investigation of my problem and it would appear
that pine may be the culprit. i was hoping some other people may have an
opinion on this, and an appropriate way to deal with.

my suggestion would be to check for presence of : on the line before we
assume its part of a new header line?

Looking at this old problem (which we’re getting at the moment, which
throws mh for a merry dance), it seems to be in RT::Template being rather
silly about headers ( and I’m a tad busy to go and force Jesse to write ‘I
will not write a mail-parser without understanding mail standards’ several
thousand times :wink: )

RFC822 cites:

 3.1.1.  LONG HEADER FIELDS

    Each header field can be viewed as a single, logical  line  of
    ASCII  characters,  comprising  a field-name and a field-body.
    For convenience, the field-body  portion  of  this  conceptual
    entity  can be split into a multiple-line representation; this
    is called "folding".  The general rule is that wherever  there
    may  be  linear-white-space  (NOT  simply  LWSP-chars), a CRLF
    immediately followed by AT LEAST one LWSP-char may instead  be
    inserted.  Thus, the single line

        To:  "Joe & J. Harvey" <ddd @Org>, JJV @ BBN

    can be represented as:

        To:  "Joe & J. Harvey" <ddd @ Org>,
                JJV@BBN

[more examples]

         The process of moving  from  this  folded   multiple-line
    representation  of a header field to its single line represen-
    tation is called "unfolding".  Unfolding  is  accomplished  by
    regarding   CRLF   immediately  followed  by  a  LWSP-char  as
    equivalent to the LWSP-char.

So, pulling a sample out of our hat, we have a multi-line subject header
as follows:

Subject: here is a sample of SPAM Filth and Depravity that I have
received either, from your service, from a service that you host,
solicitoring replies to your service. In my country, this kind of
email is illegal. please investigate.

Line 292 of RT::Template builds up the MIME::Entity based on a
split(/\n/). Ok, but theres no joining of header lines that have been
folded.

So, if we change the loop to read:

if ($headers) {
my $prevkey = undef;
foreach $header (split(/\n/,$headers)) {
if( $header =~ /^\s+/ && defined( $prevkey) ){
# This is a continuation.
# Although implied in the docs, “-1” as the index to add doesn’t
# seem to work as wanted. Use replace.
$self->{‘MIMEObj’}->head->add( $prevkey, $self->{‘MIMEObj’}->head->get( $prevkey ) . $header );
# Note that technically we should be doing get($prevkey) . “\r\n” . $header
# and then calling unfold() as the above could lead to strange

numbers of spaces.

  }else{
    (my $key, my $value) = (split(/: /,$header,2));
    chomp $key;
    chomp $value;
    $self->{'MIMEObj'}->head->fold_length($key,10000);
    $self->{'MIMEObj'}->head->add($key, $value);
    # Cache the key in case it continues.
    $prevkey = $key;
  }
}
# Unfold all the headers if still needed.
$self->{'MIMEObj'}->head->unfold();

}

Applying this change to RT::Template results no more complaints about bad
RFC822 headers :wink:

Other portions of the RT code base also exhibit certain confusion about
folded headers.


Bruce Campbell RIPE
RIPE 41 NCC
Jan 14-18, Amsterdam Operations


rt-devel mailing list
rt-devel@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-devel

http://www.bestpractical.com/products/rt – Trouble Ticketing. Free.

Ok. I was wrong. the patch is not yet in the cvs copy,
but is noted at http://fsck.com/rt2/Ticket/Display.html?id=1058

The patch referenced above ignores folded headers by only taking the first
line of a multi-line header. The other lines would seem to vanish into
the ether.

I can’t actually recall what I was waiting on, but I’ll look harder
at it when I get back from Slovenia and Austria next week.

Possibly that :wink:

I’ve found that this conflicts in some weird way with my work on #1187
(only with folded headers), so it might bite other bits as well. Hrm,
think I need some sleep. (And the latest bad-singer version of Don’t Cry
For Me Argentina
is getting annoying)

                         Bruce Campbell                            RIPE
                                RIPE 41                             NCC
                   Jan 14-18, Amsterdam                      Operations

Ok. After looking at it harder myself, I think that was why I wasn’t
just applying that patch. I believe that what you’ll see checked in is
code that just uses MIME::Parser to do all this for us, as we should
have had it in the first place. . (Though I need to make
sure it deals properly with cases where there are no headers.)

-jOn Wed, Jan 16, 2002 at 10:59:12PM +0100, Bruce Campbell wrote:

On Wed, 16 Jan 2002, Jesse Vincent wrote:

Ok. I was wrong. the patch is not yet in the cvs copy,
but is noted at http://fsck.com/rt2/Ticket/Display.html?id=1058

The patch referenced above ignores folded headers by only taking the first
line of a multi-line header. The other lines would seem to vanish into
the ether.

I can’t actually recall what I was waiting on, but I’ll look harder
at it when I get back from Slovenia and Austria next week.

Possibly that :wink:

I’ve found that this conflicts in some weird way with my work on #1187
(only with folded headers), so it might bite other bits as well. Hrm,
think I need some sleep. (And the latest bad-singer version of Don’t Cry
For Me Argentina
is getting annoying)


Bruce Campbell RIPE
RIPE 41 NCC
Jan 14-18, Amsterdam Operations


rt-devel mailing list
rt-devel@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-devel

http://www.bestpractical.com/products/rt – Trouble Ticketing. Free.

Ok. After looking at it harder myself, I think that was why I wasn’t
just applying that patch. I believe that what you’ll see checked in is
code that just uses MIME::Parser to do all this for us, as we should
have had it in the first place. . (Though I need to make
sure it deals properly with cases where there are no headers.)

Oh! RT::Interface::Email ParseMIMEEntityFromSTDIN isn’t calling
$head->unfold(). That could be making a difference.

After applying the patch below (and removing my previous one to
RT::Template), I don’t get any warnings. The subject is also stored in
the SQL as an expected really really long line, without any \n’s in there.

257a258,260
>
>     # Unfold the headers
>     $head->unfold;


                         Bruce Campbell                            RIPE
               Systems/Network Engineer                             NCC
             www.ripe.net - PGP562C8B1B                      Operations

matt carter wrote:

Further investigation shows there is a new line in the subject alright.

The message handling code is not handing header continuation
correctly. At a guess, it’s assuming a continuation line
starts with a tab, when in reality a space (what pine seems
to insert) is also valid (from memory.)

Hmm. Off to look. :slight_smile: