[Exim] Spamassassin and transport failure

[John Horne]

Hello,

I have been looking to use spamassassin (version 2.20) on our central
mailhubs. I have run it successfully on my linux PC with Exim 4.04. So I
thought I'd move it onto one of the mailhubs - a Sun Solaris 8 box with Exim
3.36.

It all seems to work okay for messages submitted from the mailhub itself
(i.e. I log onto the mailhub and invoke:
exim ***@tracy.csd.plymouth.ac.uk
('tracy' is the name of the system). The message headers show the
received protocol as 'spam-scanned'. However, if I return to my work PC and
send a message so that it goes via 'tracy', then I get an error returned:

The following address(es) failed:

***@tracy.csd.plymouth.ac.uk
Child process of spamcheck transport returned 2 from command:
/usr/local/exim/bin/exim

The following text was generated during the delivery attempt:

------ ***@tracy.csd.plymouth.ac.uk ------

An error was detected while processing a file of BSMTP input.
The error message was:

554 Unexpected end of file

The SMTP transaction started in line 0.
The error was detected in line 4.
0 previous messages were successfully processed.
The rest of the batch was abandoned.
554 Unexpected end of file
Transaction started in line 0
Error detected in line 4

==========================================

It then gives a copy of the original message. I installed spamassassin
according to the web site:
http://dman.ddts.net/~dman/config_docs/exim3_spamassassin.html

I am using spamd which was started as root.

The relevant bits of the Exim configure file are:

# Spam Assassin (transport)
spamcheck:
driver = pipe
command = /usr/local/exim/bin/exim -oMr spam-scanned -bS
transport_filter = /usr/local/bin/spamc
bsmtp = all
home_directory = /tmp
current_directory = /tmp
user = exim
group = mail
no_return_path_add
log_output
return_fail_output
prefix =
suffix =

# Spam Assassin (director)
spamcheck_router:
no_expn
no_verify
domains = tracy.csd.plymouth.ac.uk : localhost
local_parts = jhorne
condition = "${if and { {!def:h_X-Spam-Flag:} \
{!eq {$received_protocol}{spam-scanned}}} {1}{0}}"
driver = smartuser
transport = spamcheck

-------------------

I have searched the mailing list archives and can find similar problems when
using the spamcheck.pl script (?), but no real solution. I can find no-one
having this problem when using spamd. I should add that for the trusted
users 'exim' is included, so I can see no reason to have a problem with the
pipe. The error seems to indicate that it is a problem with running the exim
binary, but I have no idea what it is - why does it get an unexpected end of
file? Because local messages work okay I can't run exim with '-d9' since it
won't show any problem (actually I have just done that and I'll check the
output, but it looks okay and no indication of what may cause a problem
remotely).

Very strange. Anyway, as usual these things happen on a Friday afternoon :-)

If anyone has any ideas about this then I'd be grateful.


John.

------------------------------------------------------------------------
John Horne, University of Plymouth, UK Tel: [phone]
E-mail: ***@plymouth.ac.uk
PGP key available from public key servers

--

## List details at http://www.exim.org/mailman/listinfo/exim-users Exim details at http://www.exim.org/ ##

[John Dalbec]

Post by John Horne
Hello,
I have been looking to use spamassassin (version 2.20) on our central
mailhubs. I have run it successfully on my linux PC with Exim 4.04. So I
thought I'd move it onto one of the mailhubs - a Sun Solaris 8 box with Exim
3.36.
It all seems to work okay for messages submitted from the mailhub itself
('tracy' is the name of the system). The message headers show the
received protocol as 'spam-scanned'. However, if I return to my work PC and
/usr/local/exim/bin/exim
An error was detected while processing a file of BSMTP input.
554 Unexpected end of file
The SMTP transaction started in line 0.
The error was detected in line 4.
0 previous messages were successfully processed.
The rest of the batch was abandoned.
554 Unexpected end of file
Transaction started in line 0
Error detected in line 4
==========================================
It then gives a copy of the original message. I installed spamassassin
http://dman.ddts.net/~dman/config_docs/exim3_spamassassin.html
I am using spamd which was started as root.

What flags did you start spamd with? I think you need -F 0 at a
minimum. Otherwise spamd will add an mbox-style "From " line.
John

Post by John Horne
# Spam Assassin (transport)
driver = pipe
command = /usr/local/exim/bin/exim -oMr spam-scanned -bS
transport_filter = /usr/local/bin/spamc
bsmtp = all
home_directory = /tmp
current_directory = /tmp
user = exim
group = mail
no_return_path_add
log_output
return_fail_output
prefix =
suffix =
# Spam Assassin (director)
no_expn
no_verify
domains = tracy.csd.plymouth.ac.uk : localhost
local_parts = jhorne
condition = "${if and { {!def:h_X-Spam-Flag:} \
{!eq {$received_protocol}{spam-scanned}}} {1}{0}}"
driver = smartuser
transport = spamcheck
-------------------
I have searched the mailing list archives and can find similar problems when
using the spamcheck.pl script (?), but no real solution. I can find no-one
having this problem when using spamd. I should add that for the trusted
users 'exim' is included, so I can see no reason to have a problem with the
pipe. The error seems to indicate that it is a problem with running the exim
binary, but I have no idea what it is - why does it get an unexpected end of
file? Because local messages work okay I can't run exim with '-d9' since it
won't show any problem (actually I have just done that and I'll check the
output, but it looks okay and no indication of what may cause a problem
remotely).
Very strange. Anyway, as usual these things happen on a Friday afternoon :-)
If anyone has any ideas about this then I'd be grateful.
John.
------------------------------------------------------------------------
John Horne, University of Plymouth, UK Tel: [phone]
PGP key available from public key servers
--
## List details at http://www.exim.org/mailman/listinfo/exim-users Exim details at http://www.exim.org/ ##

--

## List details at http://www.exim.org/mailman/listinfo/exim-users Exim details at http://www.exim.org/ ##

[Troy Engel]

Hi John,

I had the same exact problem, and there was no real answer ever posted
to the problem. I saw a few posts on SAtalk list that there were/are
"known problems" with perl < 5.6.0 (I am using redhat 6.2, perl 5.005),
so my *theory* is that it's some obscure dumb perl problem.

For our installation, I solved it with the following steps (I also
started with the same configs you did, on dman's webpage):

1) upgrade to Exim4.04 -- the same *type* of error still happened, but
it was much more informative and gentler to deal with. The upgrade was
a little rocky, but the guys here on the list helped me through it. yay.

2) after watching the logs, I noticed that only *local* mails caused
this weird BSMTP error 2 problem. So (again with help fromt he lists) I
altered the smapcheck_router 'condition' line to ignore all local,
inter-domain email:

condition = "${if and { {!def:h_X-Spam-Flag:} {!eq
{$received_protocol}{spam-scanned}} {!eq {$received_protocol}{local}}
{!eq {$sender_address_domain}{$domain}} } {1}{0}}"

3) move spamcheck_router: to *above* system_aliases int he exim.conf
file. This, coupled with #2, causes things to be scanned or skipped
before the aliases are expanded, which cuts down on re-scanning the same
email sent inbound to multiple people (like staff and project group
aliases, etc).

We've now been running sucessfully for 3 days now without a single BSMTP
error 2 anymore, and spamassassin is doing it's thing. Performance is
also damn good now that local mails are skipped.

hope this helps,
-te

Post by John Horne
Hello,
I have been looking to use spamassassin (version 2.20) on our central
mailhubs. I have run it successfully on my linux PC with Exim 4.04. So I
thought I'd move it onto one of the mailhubs - a Sun Solaris 8 box with Exim
3.36.
It all seems to work okay for messages submitted from the mailhub itself
('tracy' is the name of the system). The message headers show the
received protocol as 'spam-scanned'. However, if I return to my work PC and
/usr/local/exim/bin/exim
An error was detected while processing a file of BSMTP input.
554 Unexpected end of file
The SMTP transaction started in line 0.
The error was detected in line 4.
0 previous messages were successfully processed.
The rest of the batch was abandoned.
554 Unexpected end of file
Transaction started in line 0
Error detected in line 4
==========================================
It then gives a copy of the original message. I installed spamassassin
http://dman.ddts.net/~dman/config_docs/exim3_spamassassin.html
I am using spamd which was started as root.
# Spam Assassin (transport)
driver = pipe
command = /usr/local/exim/bin/exim -oMr spam-scanned -bS
transport_filter = /usr/local/bin/spamc
bsmtp = all
home_directory = /tmp
current_directory = /tmp
user = exim
group = mail
no_return_path_add
log_output
return_fail_output
prefix =
suffix =
# Spam Assassin (director)
no_expn
no_verify
domains = tracy.csd.plymouth.ac.uk : localhost
local_parts = jhorne
condition = "${if and { {!def:h_X-Spam-Flag:} \
{!eq {$received_protocol}{spam-scanned}}} {1}{0}}"
driver = smartuser
transport = spamcheck
-------------------
I have searched the mailing list archives and can find similar problems when
using the spamcheck.pl script (?), but no real solution. I can find no-one
having this problem when using spamd. I should add that for the trusted
users 'exim' is included, so I can see no reason to have a problem with the
pipe. The error seems to indicate that it is a problem with running the exim
binary, but I have no idea what it is - why does it get an unexpected end of
file? Because local messages work okay I can't run exim with '-d9' since it
won't show any problem (actually I have just done that and I'll check the
output, but it looks okay and no indication of what may cause a problem
remotely).
Very strange. Anyway, as usual these things happen on a Friday afternoon :-)
If anyone has any ideas about this then I'd be grateful.
John.
------------------------------------------------------------------------
John Horne, University of Plymouth, UK Tel: [phone]
PGP key available from public key servers
--
## List details at http://www.exim.org/mailman/listinfo/exim-users Exim details at http://www.exim.org/ ##

--
Troy Engel, Systems Engineer
Hockey. Kinda like Figure Skating in a War Zone.


--

## List details at http://www.exim.org/mailman/listinfo/exim-users Exim details at http://www.exim.org/ ##

[John Horne]

Post by Troy Engel
I had the same exact problem, and there was no real answer ever posted
to the problem. I saw a few posts on SAtalk list that there were/are
"known problems" with perl < 5.6.0 (I am using redhat 6.2, perl 5.005),
so my *theory* is that it's some obscure dumb perl problem.

The mailhub has perl version 5.6 installed:
perl -V

Post by Troy Engel
For our installation, I solved it with the following steps (I also
1) upgrade to Exim4.04 -- the same *type* of error still happened, but
it was much more informative and gentler to deal with. The upgrade was
a little rocky, but the guys here on the list helped me through it. yay.

Can't do that at the moment. Probably a couple of months when all the
students have gone home.

Post by Troy Engel
2) after watching the logs, I noticed that only *local* mails caused
this weird BSMTP error 2 problem. So (again with help fromt he lists) I
altered the smapcheck_router 'condition' line to ignore all local,

Nope, I get the opposite. The local mail is fine, it's the non-local mail
that's the problem.

Post by Troy Engel
3) move spamcheck_router: to *above* system_aliases int he exim.conf
file. This, coupled with #2, causes things to be scanned or skipped
before the aliases are expanded, which cuts down on re-scanning the same
email sent inbound to multiple people (like staff and project group
aliases, etc).

Already done.

Post by Troy Engel
What flags did you start spamd with? I think you need -F 0 at a
minimum. Otherwise spamd will add an mbox-style "From " line.

Yup, already done that. spamd starts with '-d -x -F 0'.

I'll keep looking at it, but anymore suggestions would be welcome.

John.
--
John Horne, University of Plymouth, UK Tel: [phone]
E-mail: ***@plymouth.ac.uk
PGP key available from public key servers

--

## List details at http://www.exim.org/mailman/listinfo/exim-users Exim details at http://www.exim.org/ ##

[John Horne]

Post by John Horne
/usr/local/exim/bin/exim
An error was detected while processing a file of BSMTP input.
554 Unexpected end of file

Okay, I've got this solved :-) I can't debug exim remotely, but of course the
transport is invoking exim for me, so I changed my exim configure to say:

command = /usr/local/exim/bin/exim -oMr spam-scanned -bS -d9

Restart exim. I then sent a message from my work PC and all the debugging was
returned as part of the error message. I received (the relevant bit):

SMTP<< MAIL FROM:<***@plymouth.ac.uk>
SMTP<< RCPT TO:<***@tracy.csd.plymouth.ac.uk>
SMTP<< DATA
search_tidyup called
ld.so.1: /usr/local/bin/spamc: fatal: libgdbm.so.2: open failed: No such
file or directory

LOG: 4 MAIN
unexpected EOF while reading SMTP data (after header) from exim
Handling error in batched SMTP input
An error was detected while processing a file of BSMTP input.

The problem was that spamc wasn't finding the libgdbm libraries. The
libraries were installed into /usr/local/lib, and I have that path in the
/etc/profile file, so I'm a little lost as to why it had a problem (must be
the way the process is forked? I'll have to read a bit more about pipe
transports). Anyway, I created soft links of the libraries in /usr/lib, and
it then started to work okay :-)

John.
--
John Horne, University of Plymouth, UK Tel: [phone]
E-mail: ***@plymouth.ac.uk
PGP key available from public key servers

--

## List details at http://www.exim.org/mailman/listinfo/exim-users Exim details at http://www.exim.org/ ##

[j.linn]

My problems is when the system runs out of resources, the time to scan
takes longer and longer. Yesterday is was failing on a few 100 messages
but I think I needed more tday. In the spamd log it shows by the time to
scan going from 0 or 1 sec to 20 to 100 to .... and finally it breaks the
timeout on [B]SMTP.

The error mesage is givben below with d9 as you suggested.

I use Exiscan v1 patched and modified so I can force it to limit the
number of exims it spawns (and then spamd's). It now fails at 50 but works
fine at 5.

I have also included the message size in the transport condition so as not
to even call spamd if the message is too large. I expect that the SA
whitelist would be better in the EXIM condition.

However the fact remains that it can fail and start bouncing messages is
what concerns me. I cannot find anything about pipes failing but that is
the crux of the problem now.


QUESTION: does anyone know how to trap errors in PIPES and hold the mail
for a retry?

John Linn
Date: Wed, 22 May 2002 14:16:21 +0100
From: Mail Delivery System <Mailer-***@abdn.ac.uk>
To: ***@abdn.ac.uk
Subject: Mail delivery failed: returning message to sender

This message was created automatically by mail delivery software (Exim).

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

***@mailtest.abdn.ac.uk
pipe delivery process timed out

The following text was generated during the delivery attempt:

------ ***@mailtest.abdn.ac.uk ------

Exim version 3.33 debug level 9 uid=7951 gid=2000
Berkeley DB: Sleepycat Software: Berkeley DB 3.1.17: (July 31, 2000)
Caller is an admin user
Caller is a trusted user
user name "JAL" extracted from gecos field "JAL, for e-mail support"
originator: uid=xxxx gid=yyyy login=XXXX name=JAL
sender address = NULL
smtp_setup_msg entered
SMTP<< MAIL FROM:<***@abdn.ac.uk>
SMTP<< RCPT TO:<***@mailtest.abdn.ac.uk>
SMTP<< DATA
search_tidyup called
LOG: 4 MAIN
SMTP data timeout (message abandoned) on connection from local process

------ This is a copy of the message, including all the headers. ------

Return-path: <***@abdn.ac.uk>
Received: from mailhub2.abdn.ac.uk ([139.133.7.24])
by mailtest.abdn.ac.uk with esmtp (Exim 3.33 #4)
id 17AVr7-0004kj-00
for ***@mailtest.abdn.ac.uk; Wed, 22 May 2002 14:08:41 +0100
Received: from sysa.abdn.ac.uk ([139.133.7.110] helo=sysa)
by mailhub2.abdn.ac.uk with esmtp (Exim 3.33 #4)
id 17AVq8-0006hh-00
for x-***@abdn.ac.uk; Wed, 22 May 2002 14:07:40 +0100
Date: Wed, 22 May 2002 14:07:37 +0100 (BST)
Message-Id: <***@sysa>
From: ***@sysa.abdn.ac.uk
To: x-***@abdn.ac.uk
Subject: test11
X-Scanner: exiscan *17AVq8-0006hh-00*0wW7I.eyFE.*



--

## List details at http://www.exim.org/mailman/listinfo/exim-users Exim details at http://www.exim.org/ ##

[dman]

--
On Wed, May 22, 2002 at 02:37:50PM +0100, j.linn wrote:
| My problems is when the system runs out of resources, the time to scan
| takes longer and longer.

How about looking at the deliver_queue_load_max option?

| Yesterday is was failing on a few 100 messages but I think I needed
| more tday. In the spamd log it shows by the time to scan going from
| 0 or 1 sec to 20 to 100 to .... and finally it breaks the timeout on
| [B]SMTP.

I managed to do that once :-). I had just brought my system up and
was trying to transfer the mails that had landed in my inbox at school
to my own machine. With ~900 messages arriving as fast as formail and
sendmail could transfer them my system load average neared 30. Oops.
After a couple of minutes spamd got through all the messages and the
load went down. (Then I realized I messed up the formail command and
hosed the messages. I started the process over again, but after
setting the load_max option.)

-D

--

"640K ought to be enough for anybody" -Bill Gates, 1981

GnuPG key : http://dman.ddts.net/~dman/public_key.gpg

--
[ Content of type application/pgp-signature deleted ]