Monthly Archives: March 2008

tuning mailservers: go for the low-hanging fruit first

Enough about spammers, let’s tackle tuning a mail-server. After installing munin on a couple of boxes I quickly found some low-hanging fruit to pick. My mail-servers use mysql for many things. A couple of my mysql-servers had their query-cache disabled, which seems default for the linux-distro they use. If your pdns- and postfix-box is glued together with mysql you might have a problem on your hands. Many queries are repeated a number of times. Not caching them leads to many longer-running queries, which leads to a larger number of parallel queries, which leads to more memory-use and more parallel tasks. Your box wil eat memory, processes and file-handles. Load will  increase, throughput will suffer, and within days you’re close to deciding that your cluster need another box. 

Because of the weekend I decided to tune whatever I could, and the results are amazing, even without going deep into high-end tuning. Just picking some low-hanging fruit was sufficient. Before the weekend the mail-server in question was always swapping, had a load of > 15, consumed most of it’s CPU-time waiting for disks to catch up, and was slow in a most unpleasant way. 2 hours later it’s mostly idle, I’ve got memory to spare and the box feels like a Xeon should. Snappy.

The biggest improvement was made by enabling various caches in mysql. Pdns makes loads of queries, and caching those is always good. Some customers get loads of mail, and caching those also helps. After tuning a bit I got a query cache hit-rate of 90%, which quickly translates to 1/10th of the disk-accesses for mysql I had before. Less waiting on disk-access means less parallel queries, so I could lower that too, saving lots of memory, processes and file-handles.

Next up: check your syslog-configuration. Mine was full of doubles, by which I mean things logged in two different files. If you haven’t done anything to syslogd.conf and just installed packages this might be the case. I found out my predecessors did even worse. Every POP3 and IMAP log-in was written to courier.log, mail.log, messages.log and secure.log, which would count as quadruple logging. Realise that *every* log-in incurs four write-actions to logfiles instead of one, and you’ll see why checking makes sense. 

For my tuning I found to more or less overlapping tools to quickly analyze your mysql. The first is mysqltuner which is a perl script that will analyze a running mysql instance and give some advice. The second is tuning-primer.sh which is slightly more verbose, and explains why you should set certain values and offers links to explanations. Both will offer you a certain degree of insight which might be hard to find otherwise. For instance: Many people over-allocate resources to their mysql-server. You might be tempted to allocate 500 parallel connections. But you probably didn’t realise that each connection will require memory and that you thus can allocated more than physical memory. Which might bite you when your machine is heavily loaded and part of your running processes will end up in swap, slowing your machine to the point of unusability. 

The effects of adding query-cache to your mysql

strange behaviour from spammers

That sounds almost to corny to be taken serious. Let me explain. I’m tuning
my mailservers. Due to a sharp increase in spam during the last week things
started to creak and squeek. So turn up my logging, and start with the
low-hanging fruit first. More on that in a future post.

But browsing through logfiles I noticed a strange pattern of behaviour from
certain hosts:

Mar 29 18:57:27 mailfallback1 postfix/smtpd{14421}: NOQUEUE: reject:
RCPT from xxx-218-222-201.adsl.terra.cl{201.222.218.xxx}: 504 5.5.2
{usuario-10c7392}: Helo command rejected: need fully-qualified hostname;
from={knlgresqgef@ xxxxxxxx.com} to={az795113.1731@xxxxxxxx.nl} proto=ESMTP
helo={usuario-10c7392}
Mar 29 18:57:27 mailfallback1 postfix/smtpd{14421}: NOQUEUE: reject: RCPT
from xxx-218-222-201.adsl.terra.cl{201.222.218.xxx}: 504 5.5.2
{usuario-10c7392}: Helo command rejected: need fully-qualified hostname;
from={knlgresqgef@ xxxxxxxx.com} to={az9871@ xxxxxxxx.nl} proto=ESMTP
helo={usuario-10c7392}
Mar 29 18:57:27 mailfallback1 postfix/smtpd{14421}: NOQUEUE: reject: RCPT
from xxx-218-222-201.adsl.terra.cly201.222.218.xxx}: 504 5.5.2
{usuario-10c7392}: Helo command rejected: need fully-qualified hostname;
from={knlgresqgef@ xxxxxxxx.com} to={aza80@ xxxxxxxx.nl} proto=ESMTP
helo={usuario-10c7392}

and 8 more like that from that specific IP. After getting a reject (its HELO
hostname doesn’t resolve) it tried reusing the 11 open sessions for sending
mail to another account, and another, before closing the connection.
Obviously the zombie at the other end doesn’t parse responses.

Looking for more I found many thousands of identical patterned attemps from
as many different hosts over the last couple of hours. This seems to have
started a week ago, give or take a day. Always a relative fresh zombie,
though about half of them already made it to some kind of DNSBL.

Useless waste of bandwidth. Spammers stupid.

for google: multiple connect, mailserver, zombie, spammer, attack

waarom je itunes niet op random moet zetten

Je valt van het ene uiterste in het andere:

20080320001959

Landrush numerieke domeinnamen

Om m’n irritatie over de gang van zaken bij de landrush van de numerieke domeinnamen in Nederland van wat substantie te voorzien heb ik even met wat gegraaf wat getallen op een hoop gegooid.

Ik heb gekeken naar de domeinnamen 0.nl tot 99999.nl, met 00001.nl enzo erbij. Dat zijn in totaal 111110 unieke domeinnamen. Daarvan zijn 11245 namen daadwerkelijk geregistreerd, het grootste deel tussen 0000 en 9999. Van die 11245 zijn er 10096 bij twee partijen terecht gekomen. 90% van de geregistreerde ‘name-space’ is in handen van twee partijen. De een heeft een lange reputatie als domeinnaam-kaper en de ander schept luidkeels op over hoe slim hij het wel niet aangepakt heeft. In totaal zijn er 108 deelnemers in geslaagd een domeinnaam te registreren in het stukje ‘name-space’ waar ik naar gekeken heb.

verdeling.jpg

Met 108 deelnemers in een veld van 11245 domeinnamen verwacht je een redelijk neutrale verdeling van toegewezen domeinnamen. Niet iedereen zal voor ‘alles’ gegaan zijn, maar specifieke namen willen, en dus enkele puntjes scoren, andere deelnemers gaan voor alle netnummers van Nederland of andere lange reeksen en scoren tientallen of zelfs een paar honderd punten. Maar de verdeling zoals die nu tot stand gekomen is, is in mijn ogen een duidelijk teken van fraude.

(20:44: iemand wees me op de verwisseling sunrise/landrush. Die fout heb ik de afgelopen maanden al duizend keer gemaakt. Maar: aangepast.)