Exim + Dovecot deliver overquota problems

[Joan]

Good morning,

We are having an issue with a dovecot + exim setup.
We are using dovecot's deliver for the mail delivery with the pipe driver:

dovecot_transport:
driver = pipe
command = /usr/lib/dovecot/deliver -d $local_part@$domain -f $sender_address
user = vmail
group = vmail
envelope_to_add = true
return_path_add = true
delivery_date_add
log_output
temp_errors = 64 : 69 : 70: 71 : 72 : 73 : 74 : 75 : 78

In dovecot.conf I have enabled the quota_full_tempfail, so the deliver
script fails when the final recipient is overquota.
quota_full_tempfail = yes

This is working without problems with all the mail except for the
overquota recipients. On those recipients I get:
Child process of dovecot_transport transport
returned 75 (could mean temporary error) from command: /usr/lib/dovecot/deliver

I would like this to be returned as a 550 error (or at least a 450
temp error), but I get the standard success message:

helo example.com
...
250 Accepted
data
354 Enter message, ending with "." on a line by itself
jlk
.
250 OK id=1NMgHz-0004VH-Jw


Thanks for your help

[Nikita Koshikov]

On Mon, 21 Dec 2009 12:21:41 +0100

Post by Joan
Good morning,
We are having an issue with a dovecot + exim setup.
driver = pipe
user = vmail
group = vmail
envelope_to_add = true
return_path_add = true
delivery_date_add
log_output
temp_errors = 64 : 69 : 70: 71 : 72 : 73 : 74 : 75 : 78
In dovecot.conf I have enabled the quota_full_tempfail, so the deliver
script fails when the final recipient is overquota.
quota_full_tempfail = yes
This is working without problems with all the mail except for the
Child process of dovecot_transport transport
returned 75 (could mean temporary error) from command: /usr/lib/dovecot/deliver
I would like this to be returned as a 550 error (or at least a 450
helo example.com
...
250 Accepted
data
354 Enter message, ending with "." on a line by itself
jlk
.
250 OK id=1NMgHz-0004VH-Jw
Thanks for your help

You need to check user's quota on routing stage, not transport.

I have wrote perl script for my setup that checks existence of maildirsize file and counting quota value for mailbox on smtp-processing stage, thus if user overquota exim's router will return error and you can treat it as 4xx or 5xx response.

Here is example of exim router:

quota_check:
driver = redirect
domains = +local_domains
allow_defer
allow_fail
condition = ${if \
eq{${perl{check_quota}{$local_part}{$domain}}} {no}\
}
data = :fail: Account is under quota //5xx error
#data = :defer: Account is under quota //4xx error

Post by Joan
--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Joan]

Post by Nikita Koshikov
You need to check user's quota on routing stage, not transport.

It'd be much better, because I would allow me to mix overquota and
no-overquota recipients.
But the problem in my case is that I am using some hashing to
distribute the users' mailboxes, dovecot can handle this easily
because it supports some interesting stuff
(http://wiki.dovecot.org/Variables)
With a perl script I could also do that, I would like only if it's the
last resort (too much overhead I would think)

Post by Nikita Koshikov
I have wrote perl script for my setup that checks existence of maildirsize file and counting quota value for mailbox on smtp-processing stage, thus if user overquota exim's router will return error and you can treat it as 4xx or 5xx response.
   driver = redirect
   domains = +local_domains
   allow_defer
   allow_fail
   condition = ${if \
       eq{${perl{check_quota}{$local_part}{$domain}}} {no}\
       }
   data = :fail: Account is under quota //5xx error
   #data = :defer: Account is under quota //4xx error

[Phil Pennock]

Post by Joan

Post by Nikita Koshikov
You need to check user's quota on routing stage, not transport.

It'd be much better, because I would allow me to mix overquota and
no-overquota recipients.
But the problem in my case is that I am using some hashing to
distribute the users' mailboxes, dovecot can handle this easily
because it supports some interesting stuff
(http://wiki.dovecot.org/Variables)

Exim has built-in quota support, if you let it do the maildir delivery,
and the paths can be constructed using a wide selection of variables,
which certainly seem to cover the variables relevant to storage-time
selection in the list you provide (obviously pid of connection and that
it's IMAP won't be supported, but you won't be storing based on that).

Exim also has a nice selection of expansion operators to perform things
like hashing, substring selection, and more. See section 11.6
("Expansion operators") in the Spec.

The biggest issue here is likely to be an operational one, how do you
make sure your hashing in the Dovecot server remains in sync with the
hashing in the Exim server. And that's solvable with presubmit checks.

-Phil

[Ken Price]

Post by Joan

Post by Nikita Koshikov
You need to check user's quota on routing stage, not transport.

It'd be much better, because I would allow me to mix overquota and
no-overquota recipients.
But the problem in my case is that I am using some hashing to
distribute the users' mailboxes, dovecot can handle this easily
because it supports some interesting stuff
(http://wiki.dovecot.org/Variables)
With a perl script I could also do that, I would like only if it's the
last resort (too much overhead I would think)

Post by Nikita Koshikov
I have wrote perl script for my setup that checks existence of
maildirsize file and counting quota value for mailbox on

smtp-processing

Post by Joan

Post by Nikita Koshikov
stage, thus if user overquota exim's router will return error and you

can

Post by Joan

Post by Nikita Koshikov
treat it as 4xx or 5xx response.

Forgive my late entry into this thread, but I've been driving myself crazy
for the last 12 hours trying to figure out the exact same problem. ACL,
routers, etc, etc. To paraphrase (and sum up): How to prevent backscatter
from either Exim or Dovecot LDA by rejecting email in the SMTP session.
And I finally found the answer. It's so simple that I wanted to shoot
myself for all the time spent trying to work out a solution.

Answer: You can't. (Huh? What?)

And this is why. Firstly, overquota scenarios aren't meant to be
permanent errors. By definition, they're temporary. And not just
temporary as in defer temporarily, I mean INTERNALLY temporary. Queued by
Exim. The user is meant to log back in and clean things up, right? So
immediately, forget about throwing a 5XX/4XX back to the sending mail
server. Your mail server is meant to queue it, and wait til your slack
user makes some space in his account. So does this mean you can't handle
this in the SMTP session? Not exactly.

Have I lost you yet? Probably, cause I was lost too. Look, in the most
simple Exim configuration you can make sure every user maps directly to a
maildir (and the corresponding maildir++ maildirsize file).

***@domain1.com --> /usr/local/mail/domain1.com/jimsmith/Maildir/

Can we process overquota scenarios in the SMTP session for these basic
1-to-1 recipients? YES! Oh sweet simplicity, YES! BUT. Who among us
only has this simple setup? If you do, then you're not my audience and
just stop reading. In the real world, don't we also have an alias
"***@domain1.com" that points to "***@domain1.com"? Maybe we also
have domain1.NET which is a domain alias of domain1.com. MAYBE
"***@domain1.com" is also a forward to his
"***@fullcompanyname.com" account also residing on the server. So,
again, I ask you, "Do you really think you can gracefully check and
deny/defer based on the RCPT in the SMTP session?" When the RCPT is
"***@domain1.com" which forwards to "***@domain1.com" which forwards
to "***@fullcompanyname.com". Where the only account with a maildir
and a corresponding quota is "***@fullcompanyname.com"? How many
layers of recursion are necessary? You have to check this for every
incoming RCPT, so how many CPU cycles are you wasting? How complex is this
ACL Macro? How inefficient?!

I have multiple Exim MTA's sharing a maildir store over NFS. I modestly
have several thousands of users and many hundreds of domains. I *highly*
customized a Vexim configuration and made it sing ... until the quota
issue. Since I use Dovecot, I moved to Dovecot's LDA/Deliver to handle the
last leg of message delivery. I now use Dovecot's Dictionary Quota (MySQL
backend) to store quota information. This allows me to run a cronjob to
search for over quota accounts and identify those who may simply be lazy,
and those who may be subject to backscatter attacks, and disabling accounts
if necessary. (You can also do this with maildir++ quotas, but that's a
lot of file IO compared to a simply SQL query).

All in all, the solution isn't with fancy ACL conditions or Macros, and
certainly not in the routers because by then (as previously mentioned in
this thread) the message is already accepted and you're throwing
backscatter (or null routing). The solution is taking a step back and
solving the problem outside the MTA, and rethinking policy.

I apologize for the lengthy and opinionated response, but if it saves
anyone the frustration I went through, then it was worth it. :-)

-Ken Price

[Mike Cardwell]

Post by Ken Price
Can we process overquota scenarios in the SMTP session for these basic
1-to-1 recipients? YES! Oh sweet simplicity, YES! BUT. Who among us
only has this simple setup? If you do, then you're not my audience and
just stop reading. In the real world, don't we also have an alias
have domain1.NET which is a domain alias of domain1.com. MAYBE
again, I ask you, "Do you really think you can gracefully check and
deny/defer based on the RCPT in the SMTP session?" When the RCPT is
layers of recursion are necessary? You have to check this for every
incoming RCPT, so how many CPU cycles are you wasting? How complex is this
ACL Macro? How inefficient?!

It's much easier than you think. All you have to do is this in each of
the routers which rewrites the address:

address_data = $local_part@$domain

Then after you've done a "verify = recipient" in your rcpt acl you'll
find that $address_data contains the final address.
--
Mike Cardwell : UK based IT Consultant, LAMP developer, Linux admin
Cardwell IT Ltd. : UK Company - http://cardwellit.com/ #06920226
Technical Blog : Tech Blog - https://secure.grepular.com/blog/
Spamalyser : Spam Tool - http://spamalyser.com/

[Ken Price]

On Tue, 05 Jan 2010 09:25:14 +0000, Mike Cardwell

Post by Mike Cardwell
It's much easier than you think. All you have to do is this in each of
Then after you've done a "verify = recipient" in your rcpt acl you'll
find that $address_data contains the final address.

Mike, I love you.

I spent too many hours with my head buried in crap yesterday - over
thinking the solution - and completely forgot about the routers being used
for "verify". Argh. The 2 minute configuration change would have saved me
hours. Now it works like a charm for either deferring or denying during
the SMTP session. Now, if only I can get Dovecot's Deliver/LDA to mimic
Exim's quota_is_inclusive behavior and I'll be a happy man.

Thanks!
-Ken Price

[W B Hacker]

Post by Ken Price
On Tue, 05 Jan 2010 09:25:14 +0000, Mike Cardwell

*snip*

Post by Ken Price
...... Now, if only I can get Dovecot's Deliver/LDA to mimic
Exim's quota_is_inclusive behavior and I'll be a happy man.
Thanks!
-Ken Price

Ken,

'If at first you don't succeed... cheat!'

- Provide a fudge-factor to the measured quota (available less current message
size or available less max-incoming message size) ... to whichever daemon needs
the adjustment in order to stay out of trouble. ELSE, actually allow a modest
over-qouta of 'enough' space to render the difference insignificant in daily use.

CAVEAT: Not tested.

With heaviest-IMAP users at under 3 GB growth per annum, and upgrading HDD about
every 3 years for Preventive Maintenance, the rate of growth in mass-market HDD
sizes has so far outstripped the rate of growth in user storage.

Here, anyway, the admin time would cost more than the larger HDD by two orders
of magnitude, if not three.

Your Mileage Obviously Varies

;-)

Bill

[Kirill Miazine]

Post by Ken Price

Post by Mike Cardwell
It's much easier than you think. All you have to do is this in each of
Then after you've done a "verify = recipient" in your rcpt acl you'll
find that $address_data contains the final address.

Mike, I love you.
I spent too many hours with my head buried in crap yesterday - over
thinking the solution - and completely forgot about the routers being used
for "verify". Argh. The 2 minute configuration change would have saved me
hours. Now it works like a charm for either deferring or denying during
the SMTP session.

Mike's solution is smart!

However, if there are other people who might manage your Exim system,
you should definetly make a short comment in your routers setting
address_data that address_data assignment should not be touched as
$address_data will be used in ACL.