[Asrg] Re: Asrg Digest, DNSBL BCP v.2.0

[gep2 at terabites.com]

On Fri, 2 Mar 2007 20:03:11 -0500 (EST)
  Daniel Feenberg <feenberg at nber.org> wrote:
> 
> 
> On Fri, 2 Mar 2007, gep2 at terabites.com wrote:
> 
>> As you know, in a recent thread I commented on what a 
>>LOUSY solution 
>> IP-address-based blacklists are, in general.
>>
>> Part of the problem is that it is a VERY blunt 
>>instrument, especially for 
>> companies which operate a large network from behind a 
>>NAT router.
>>
>> Well, I've recently been personally involved in trying 
>>to put out just such a 
>> fire at one of my consulting clients.
>>
>> The client company is an oil and gasoline distributor. 
>>Each evening the 
>> software system I built and maintain for them sends 
>>E-mails and fax-by-e-mail 
>> price updates to their customers.  It also automatically 
>>generates and routes 
>> invoices, credit memos (for example, the amounts of the 
>>'pay at the pump' 
>> credit card transactions they handled the day before) to 
>>their customers. 
>> They probably send something approaching a thousand such 
>>business-critical 
>> E-mails and fax-by-E-mails per day.
>>
>> All works well until about a week and a half ago, one of 
>>their computers 
>> managed to get infected by some kind of exploit (despite 
>>every machine in the 
>> company having and running Symantec Antivirus software 
>>which is updated 
>> daily).  We found the problem within 24 hours and 
>>managed to clean it all up, 
>> (hopefully!) but meanwhile at least one or two E-mail 
>>blacklists (including 
>> XBL) flagged the IP address of the company's router, and 
>>starting on the 21st 
>> some of their E-mails stopped getting delivered due to 
>>the blacklisting.  By 
>> Saturday, as more ISPs picked up on the new list, E-fax 
>>(who processes and 
>> delivers their fax-by-e-mail transmissions) suddenly 
>>started bouncing even 
>> their outgoing fax attempts (even though their systems 
>>had been clean, as 
>> best we can tell) for at least two or three days 
>>already.
>>
>> Businesses which count on reliable AND TIMELY e-mail 
>>transmissions simply 
>> cannot have their E-mail transmissions blocked "en 
>>masse" like this.
>>
>> Not being able to issue credits, deliver invoices, and 
>>send price updates 
>> (and in the oil and gas business, prices change daily) 
>>is a monumental burden 
>> on a company which is guilty of nothing more than being 
>>a victim, just like 
>> so many other companies and individuals have been (and, 
>>doubtless, will 
>> continue to be).
>>
>> Worse, from a Internet strategic standpoint, the dangers 
>>of this kind of 
>> blunt-instrument blocking of E-mails for AN ENTIRE 
>>COMPANY just because ANY 
>> ONE computer within their network is infected (and it 
>>could even be an 
>> infected notebook computer carried in from home and 
>>connecting to the office 
>> wireless LAN) will force more companies to insist on 
>>MORE DANGEROUS separate, 
>> routable IP addresses for each machine in their 
>>company.... so that blocking 
>> will affect JUST ONE of their machines and not put the 
>>whole company out of 
>> business.  Ultimately, this "solution" just puts 
>>dramatically more (and very 
>> unwanted) pressure on the limited IP address space, and 
>>INCREASES the 
>> likelihood of getting the company's machines infected 
>>since now their 
>> machines each have a real, routable IP address that can 
>>be attacked by 
>> scanners and other exploits originating from outside the 
>>company.
>>
>> It has taken me personally much of the last week to try 
>>to mitigate the 
>> problems for my consulting client... managing 
>>communications with the
> 
> <snip>
> 
> Can I ask why it was so difficult? Could you not change 
>the IP address of the MTA easily? 

No, because:

1)  They use several outgoing mail servers, at least 
several of which are behind their NAT router (and thus all 
have the same single IP address, from the Internet side).

2)  Their "vanity domain" mail server (outside their 
inhouse LAN) is "protected" by the same kind of 
braindamaged IP-address-based blacklist 
("emaildefenseservice", but let's not name names... :-/) 
and thus won't accept their outgoing mail messages either, 
since they come from the company's blacklisted IP router.

3)  Ditto for E-fax, the E-mail-to-fax gateway that the 
client company uses.

4)  Changing the IP address of the company's NAT router 
could only be done by their ISP/telephone company, since 
the IP address belongs to (and is set by) the phone 
company.

> Perhaps your customer had only one? 

Only one NAT router, yes.

> Perhaps you weren't aware that was 
>possible? 

I'm very aware of what is POSSIBLE, but you still have to 
ACHIEVE that and it necessarily involves not just a number 
of other companies (not all of which provide competent 
support 24/7) but also reconfiguring a lot of relatively 
technical software within the victim company itself.  For 
companies like my consulting client, which does not have 
inhouse IT staff, this is at best a challenge.

>Certainly XBL doesn't list an entire address 
>block on day one, 

It listed the company's NAT router, and they have only one 
IP address (well, two actually... their 'modem' and the 
NAT router behind it).  But listing either one of those is 
equivalent.

>so I assume that the infected machine 
>was using the companies own smtp server as a smarthost 
>and that is why the entire company was inconvinienced. 

No.  It was sending (apparently) using its own SMTP 
sending engine, but behind the (single) NAT router and 
therefore from the Internet side was indistinguishable (by 
IP address, anyhow) from any other E-mails coming from 
within the entire company, from any of their inhouse 
outgoing mail servers.

>Another possibility for the company is to use the ISP 
>smarthost for its mail. 

The ISP's "smarthost" would have doubtless blocked the IP 
address as well, due to them using braindead 
IP-address-based blacklisting!!  :-(

>It is quite rare for ISP MTAs to 
>be blacklisted, though it does happen.  I understand that 
>getting off blacklists is slow and difficult to speed up, 
>but why is it the only solution?

It isn't, of course, but none of the "solutions" are 
obvious or necessarily easy for an affected copmany to 
implement, especially depending on how deeply embedded 
their mail sending architecture is within their 
applications.

> I also refer you to 
>http://www.nber.org/sys-admin/smarthost.html where I list 
>some commercial smarthosts. While not free, the cost 
>would be similar to an hour or two of typical consulting 
>rates, for a year of smarthosting, by which time the 
>original IP address would be off the blacklists.

What I eventually did was to arrange an emergency external 
host (NOT using a braindead XBL blacklist!) that the 
company's inhouse systems could use as an upstream MTA, 
thus changing the IP address seen by the people they need 
to reach.

But having the original IP address "off the blacklists" is 
only a temporary solution, only until (inevitably) one of 
the company's machines is eventually infected again, and 
the whole insane Keystone Kops episode will inevitably 
play out again.

> Postings such as yours have more weight if they include 
>the relevant IP address.

The CONCEPT here is what matters;  the specific IP address 
involved in this SPECIFIC incident really isn't important.

But if you think you can shed additional insight by 
knowing it, the IP address is 66.73.52.194 (their router) 
and 66.73.52.193 (their "modem" or whatever they connect 
through).  FWIW, that address is probably going to change 
anyhow because even before this incident, they have 
arranged to change their ISP connectivity provider and I 
expect their router IP address will change as a result.

> Daniel Feenberg
> feenberg isat nber dotte org

Gordon Peterson
http://personal.terabites.com
1977-2007  Thirty year anniversary of local area 
networking