For the recently tuned in, there's been an occasional noise from
me over the past couple of months that the load over the FML has
been growing to the point where I/my company can't afford the
communications costs.  I've started to do some things about this.
John Rosloot asked what an exploder is, so I thought I'd take the
opportunity to elaborate on what's going on.
 
There are a number of ways that the costs can be reduced.
 
One approach was to improve our transmission speeds (our provider
charges by time, not bytes).  Given the characteristics of the traffic,
I've been able to improve the throughput of UUCP to our provider
by almost a factor of two (from 800-900 to 1700 characters per second).
It just took me a while to certify that it was reliable enough.
For the cognoscenti, I moved back to "f" protocol from "g".
 
Another approach is that of batched mail (BSMTP et. al.).  This
is attractive, but this requires cooperation (and work) from my
provider - who is understaffed, and I'd rather not bug them with
something this fundamental now that also reduces their income.
I should point out, though, that this, at best, gives a factor
of 2 or so.
 
The next approach is through the use of subscriber list exploders.
That's what the majority of this message is about.  The transfer
rate improvement plus the exploder will decrease costs by a factor
of 10 to 20.
 
But first, a message from our sponsors: (;-)
 
***************** IMPORTANT to ANON Users ******************
 
I would prefer not to put the anon users on a list exploder in the USA,
where they could, unlikely of course, be subject to legal action or some
sort of technical extract-attack.  Neither of which is possible on
ferret.ocunix.on.ca (non-internet/secure/out-of-jurisdiction).
 
Those subscribers who're in ferret-free zones and do not want their
addresses on a server, should contact me (preferably to ferret-request)
and I'll ensure that they're not sent to the server.  Do this only
if you're in a ferret-free-zone (to include dorms with no-pet rules
in regions that do allow ferrets) - each one of these costs money.
 
*********************************
 
An exploder is a mechanism by which a single mail message coming
in is "exploded" out to a large number of people.  The term is
usually applied when the message is generated on one machine and
sent to an alias on another machine which aliases (via one
mechanism or another) it to some large number of addressees.
[Though, "cnews" also has an exploder in it to do multi-casting.]
 
Really large mailing lists generally don't want to do all of the
message generation on one machine.  For example: most of the
mail transfer agents ("MTA", such as sendmail, smail etc.) handle
large "To" lists very badly.  In my particular case, I've had
to recompile my MTA a couple of times to increase the addressee
limits.  Sendmail is notorious for REALLY bad behaviour with large
To lists - we're talking system crashes, SMTP lockups, lost mail,
choked routers, looping bouncebacks, network lockup, the works.
Pull-the-plug time and destructive recovery procedures.
 
One way of avoiding this to acquire the help of a number of other friendly
systems (our exploders).  The subscription list is subdivided in some fashion
(geographically, or simply by counts) and a piece is given to each of
these friendly machines.
 
Then, the moderator sends a copy of the message to each of the
friendly machines, who, in turn, "explode" it out to the set of
people in their chunk of the subscribers list.
 
In our case, the LISTSERV at CUNY is the friendly site.  I will
be giving it the FML subscription list, minus a few addresses
that I can better handle here, such as geographically local ones,
gateways (ie: mail-to-news), anonymous users into the dubious position
of being subject to legal action), and a few others.
 
At present, I have ~210 subscribers.  That works out to ~60 copies
of each issue being uucped out (my mailer tries to optimize by folding
multiple recipients into single messages).  Really large issues
can cost as much as $10 or more to send: calculations show that
it costs me (actually, my company) about 1 dollar per 5-8K bytes
of issue size.
 
With the exploder in operation, I will be sending to approximately
20 subscribers (~5-7 UUCP copies) including the one that goes to the
exploder.  The exploder will handle the remaining 180-190 subscribers.
 
My UUCP load, and therefore costs, will therefore go down by a factor
of 10 or so.  At least on the sending side.  The receiving side
won't change, but we can consider it to be roughly equivalent to
1 copy of the issue plus as much as 1 copy per bounce that occurs (many
MTAs truncate the return mail on a bounce to minimize costs).
 
I'll also save quite a bit of CPU in the mail routing department, and
will remove the suspense of wondering whether *this* time smail will
hit its limit again and woof its cookies ;-)
 
********************************
 
A number of people have offered financial assistance.  For which I'm
quite appreciative.  However, CUNY running the exploder makes that
unnecessary.  Besides, trying to pay for the load would cause more
accounting headaches than it would be worth - corporate income
tax and all that crap.
 
[Posted in FML issue 0457]