for INFO NATION
On the weekend of July 6th and 7th, 1996, many customers at Winternet found that they couldn't dial in, or if they could, that almost no services at winternet.com were up and running. In an unusual and still unexplained move, two of the owners of Winternet, Kevin Mathison and Michael Frankowski, fired Mike Horwath, Winternet's popular systems administrator on July 5th. By the next day, practically the entire staff was gone, and basic services were not working, causing disruption not only for Winternet's customers, but also for other Internet providers with direct connections to Winternet.
According to the log of an irc session between Mathison and several ex-Winternet staffers, Mathison said the change in management had been in the works for at least two weeks. Yet Mike Horwath says that his dismissal came without warning.
Official statements from the current owner/managers say that the entire staff quit; however, all other accounts of the events say that Kevin Mathison walked through the office, personally telling all employees on hand that they were fired. Employees who were not in the office on the 5th were left wondering whether they were still employees of Starnet Communications, the parent company of Winternet.
There is not much agreement between the official statements from Winternet's new management and the accounts of other people who were there. Mike Horwath told as much of his story to InfoNation as the possible legal wrangling to come would allow:
"I walked into the office of Winternet about 12:45pm...
"I noticed alot of new people sitting around the technical support office. I later learned that these people had already been hired, before any board meeting, or any meeting whatsoever. To this day, I don't know who 70% of them are.
"When I entered my office, I was suprised to find Kevin Mathison digging through my desk, a definite break of any trust I had for him...I told him that I wanted to meet with Mr. Frankowski and him as soon as possible, and we agreed to use the conference room.
"Mr. Mathison, Mr. Frankowski, Jim Nelson (from MinnNet), and myself went down to the conference room to talk. [Jim Nelson was contacted for this story, but did not reply.] As soon as Kevin started talking, I decided I needed to have a lawyer present. I stated so and walked out of the room. At that time, Mr. Mathison said that if I was going to call a lawyer, that we would not have the meeting. I said 'fine'.
"While walking back to the Winternet office, I asked my friend to call my lawyer and ask him to come down to meet me, or offer some advice. As I walked back into the Winternet offices, Mr. Mathison tells me that I am fired, as well as my whole staff, in front of Chad Trost and the new Winternet staff members...I contested his firing by explaining that no board meeting had occured and his statement back was that it did not matter, that I was fired.
"Kevin Mathison asked Doug McIntyre for the root password, I then told Doug to not answer him and told Kevin he could ask me...As I was walking down the hall to my office, I told him that I would give him the root passwords when I was leaving. At that time, he screamed (very loudly) to 'take the system down, now!', and repeated it to make sure he was heard. Some unknown people went into the equipment room at that time and powered down many pieces of equipment and started unplugging power cords.
"One of the *new* employees went into the equipment room and started pulling all of the ethernet cabling (speculation, from reading the logs, there were many NFS errors and password errors while this was going on - assumption that they had pulled the cabling from the hubs and switches).
"(Later, I noticed they had also pulled all of the modem racks a couple feet away from their resting positions, stretching cabling until it was taught...)
"Again, Kevin started harassing Doug McIntyre to give him the root password, and at that time, I yelled from my office to stop harassing Doug and to come and harass me. I had to repeat myself a couple of times and then Kevin came down to my office...This continued off and on for the rest of the time I was in the office.
"As soon as I was finished with packing up my desk, I went into the computer room with Mr. Frankowski and worked with him to change the root passwords on all of the core UNIX servers.
"I was never asked for anything else, not router passwords, not configuration information on the routers, terminal servers, modems, mail servers, nothing at all.
"Later, we left the building; myself, my friend, Doug McIntyre, and Chad Trost, one of my employees at the time. We found out later that night that the everyone, including myself, from the old staff was now blacklisted from entering the building."
As for Mathison and Frankowski's version of the events, neither of them responded to e-mailed requests for information. The original announcement by Winternet's new management read as follows:
"On Friday, July 5, 1996, Mike Horwath was removed from day to day operations by the board of directors. He remains an owner and retains his position on the board of directors. The Winernet [sic] employees were given the opportunity to continue their employment with us, but elected to leave with Mr. Horwath..."
Later, Thomas Keeley of the new Winternet staff posted an update to Usenet:
"In case there is anyone who doesn't know, as of Friday, Mike Horwath's employment with Winternet was terminated. This may be confusing at first so I'll try to explain.
"Mike Horwath, Michael Frankowski, and Kevin Mathison are all equal shareholders of Starnet, the mother company of Winternet. These three make up the board of directors. Winternet is the public access branch of Starnet, if you will.
"From what I understand, Mike Horwath was relieved of the day to day administration of winternet for failing to follow mandates of corporate concern handed down by the board of directors. Although it may not be evident, the failure to follow these mandates were increasingly jeopardizing Winternet's hopes of a future.
"Please let me interject a personal opinion here. I agree completely that Mike Horwath was and is one of the best admins I know and no matter what the personal feelings, he will be missed here.
"Secondly, it is true that the former crew has quit. After being offered the chance to stay on with Winternet, each of them declined. And again, they will be missed.
"For further information on any of the above subjects, I recommend mailing direct to Kevin Mathison (kmathson@winternet.com) or Michael Frankowski (mfrankow@winternet.com) and asking them...
"Down time...Unfortunately, yes, we've had our share and then some. A few hours after Mike Horwath was terminated, several of our key component machines suddenly, and unfortunately, went down. Our primary router dropped and erased it's routing tables. Also our authorization server went down as well as several frame relays. Unfortunately, since the new admins were only given some of the root passwords, life becames very hectic as we were required to obtain root passwords before we could apply changes and corrections to the system and so the simple 15 to 30 minuet project became hours of very difficult resetting.
"We currently believe that we have reenstated our network and our lines to both MR-Net and Alternet..."
A more pithy response was given by Kevin Mathison to ex-Winternet employees on irc: "To tell the truth, in this case, would get the company sued and we'd like to give a truthful answer, so none was better."
The responses from customers were scathing. Those who were lucky enough to get a dialup or who had alternate accounts posted dozens of complaints to the Usenet group mn.general, as did other ISPs trying to network with Winternet. Apart from the second-guessing and venting, there was incredulity that one of the best technical people on the Internet in Minnesota, and his entire staff, had been replaced. The "new" Winternet's technical problems were listed in agonizing, embarrassing detail:
"What kind of a haphazard excuse is it, even if the
authentication server had been having problems earlier... "Mike
couldn't fix it, so don't expect us to be able to." is what I am
hearing. These FOOLS are rebooting machines by POWER-CYCLING
them! Power-cycling! This is truly technical idiocy to the
highest degree when dealing with UNIX workstations..."
(belgo@belgo.com)
"You disconnected 2 NAP peering sessions when you unplugged
MCIA, without notice. You violated, and are currently still in
violation of 3 peering agreements:
IAXS <-> Winternet
Vector <-> Winternet
Vector <-> IAXS
"I realize you are totally fucking clueless, but these NAP
peering sessions take load off of your core links, and our
highly saturated upstream providers. By doing so you denied
service to your customers, and mine, not something I take
lightly...This resulted in an immediate net load increase on
your full core links of around 300-400kbps on the outbound side,
dumbasses. Mike earned my respect through knowledge, hard work,
and dedication to his network, and customers. You guys appear to
be majorly clueless on an increasing scale in all three of these
areas. At your current rate I have no doubt that the value of
Winternet will be reduced to < $50 in short order."
(dan@iaxs.net)
"Odd how most every piece of equipment developed "problems"
shortly after the junta...none of the mpls* set of USR modems
can get to the authorization stuff, so anyone trying to log into
the 2001 bank has to try repeatedly until they get one of the u*
bank of modems."
(mtpins@winternet.com)
"Earlier this week, after finally getting a useable USR port, I
logged into subzero. I then tried to run complex programs
(like, "ls" and "who") and was rewarded with a 'not enough
memory' error."
(kbrint@geeks.org)
"Most users cant send email, now it looks like we can't use
electronic news. The web is slow, prob because we are losing
half are packets going anywhere. I forsee a dim future for
Winternet."
(tenney@plains.nodak.edu)
"After several days of flakey, intermittent service [July 11th]
mail is dead in the water. If other postings about the situation
are correct, it's so broken that my mail and your mail is
disappearing off the face of the planet without notification."
(gfs@toad-hall.com)
"I am one of 2 sites holding massive amounts of their mail
(Interactive.net is the other). I now hold over 3675
messages(47.7 meg) of Winternet mail. We probably have your
mail...and this is why:
"Jul 11 13:20:46 alyssa.iaxs.net sendmail[23157]: HAA12605:
to=<fin@winternet.com>, delay=06:06:23, xdelay=00:00:00,
mailer=smtp, relay=absolut-zero.winternet.com., stat=Deferred:
Connection refused by absolut-zero.winternet.com.
"I have some queued mail for Winternet that is over 15 hours
old...I hereby suggest that you guys
figure out what is wrong before this puts you, and your
customers out of business."
(dan@iaxs.net)
"Whats worse, one of our MX entries (interactive.net) just sent
50 megs of our mail to the big sendmail in the sky because our
mail server was refusing connections for so long."
(ahupp@winternet.com)
"What makes me question their claims about this being a purely
"business" decision was a quote by one of the pinheads stating
Mike was a great tech guy but couldn't handle the business
end...so, they lock out their great tech guy and replace the
staff rather than simply installing someone to be
the "business guy" and let mike and his staff continue to
provide good service."
(murmur@winternet.com)
"This is completely unacceptable. Even putting aside the
politics there is a serious problem. The fact that they
completely hosed the service without explanation and warning,
when they knew exactly what they were getting in for and chose
to do it anyway means that the new owners are
not interested in customer service..."
(sdevore@barr.com)
"...if the majority owners expected resistance from Mike, then
they should've made sure they were capable of handling the
consequences of their manner of firing him. This doesn't seem
to be the case either."
(kaszeta@kenai.me.umn.edu)
Bil MacLeslie of Vector Internet Services, who are in the same building with Winternet, told InfoNation about the tech problems they have faced:
"Winternet first removed a direct ethernet link between the
Vector network and the Winternet network without notice. Vector
had to change their routing to restore connectivity. Winternet
then somehow disabled the Regional Area Peering Point (RAP)
shared between Infinity Access, Vector and Winternet. This is a
T1 Frame network between various ISPs designed to offload
traffic from their main connections that is destined between the
providers. The RAP is in it's second month of usage and is
heavily used (offloading Usenet alone is worth the costs of the
PVC). It took a day or longer to restore the RAP when Dan Maus
over at Infinity reamed Winternet out for their foolish move.
This is still not totally correct, we accessed the RAP via the
ethernet link, thus our traffic still travels upstream out a T1
to MR.net. We are requesting the new management honor the
existing contract for the ethernet link with Winternet but we
have had little communication as of yet.
"The third issue appeared when Winternet re-booted their router
in order to gain access to the router and then flushed the
config. This caused a routing problem with their upstream
provider (MR.net) which promptly moved them to static routes.
Doing so removed the route to Vector on the Winternet side, once
again disabling routing. We notified them of this problem and
they rectified it within a few hours.
"As you can imagine, I'm tired of this...These outages were
totally avoidable and show a level of incompetance the
management needs to overcome. If this were a startup ISP, the
likely hood of them surviving would be very slim. I am waiting
to see what blunders they do next."
Mathison and Frankowski apparently had little to do with the day-to-day operations of Winternet; they didn't seem to know who exactly was working for them. Mike Horwath recalls an incident that was seen by several former Winternet employees: "Kevin pointed his finger at [a customer], and yelled 'You're fired.' She responded that she did not work there. He then stated, 'I don't care. Get out.'..."
Hints have been scattered around by Mathison and others about alleged financial irregularities while Horwath was in charge of operations. In a cagey irc session that was arranged between Kevin Mathison and several ex-Winternet employees, Mathison asked whether the "company" had paid for pizza or movie nights for the employees, and whether anyone in that session had signed checks without authorization.
However, in one of his few public statements on Usenet, Kevin Mathison said, cryptically, "It's not about money". For his part, Mike Horwath told InfoNation that he has never received a written explanation of his dismissal as sysadmin, nor any list of what things he might have done wrong. In a Usenet post, Horwath said "for the 2 weeks prior to the fiasco, they had full accounting records and almost all of the receipts and invoices dating back 2 1/2 years."
Was the new Winternet staff trying to blame their many technical problems on Mike Horwath? Al Iverson felt moved to post a retort to Usenet: "I would like to point out that the staff member accused Mike of tampering with Winternet equipment on Friday -- well, he must be pretty damn fast because he was at the movies all evening, then he was at Perkins until the wee hours of the morning. I doubt the accuracy of their claims."
However, Horwath does appear to have sent a message to a mailing list for admins at other servers on the EFNet irc network, informing them of the changes and technical problems at Winternet. In response to the situation, irc.winternet.com was dropped from EFNet. While this left Winternet customers without a local irc server, it may have been a wise security move for EFNet, given the many administrative problems and oversights at Winternet since then.
On July 12th, Mike Horwath accepted a sysadmin position with Vector Internet Services--right down the hall from his former spot at Winternet. Bil MacLeslie told InfoNation that over 100 former Winternet customers had signed up for accounts with Vector since the weekend of July 5th--65 of them on July 12th alone. As for Horwath, he thanked his supporters on the Internet, both for their outpourings on Usenet and for their loyalty as customers. "People are just phenomenal sometimes, and this is a perfect example of what someone can do to build a community and gain support by just being nice. It feels good to be appreciated."
Winternet is at:
http://www.winternet.com
Vector Internet Services is at:
http://www.visi.com
Al Iverson has put together several pages of information and
links on the Winternet changes. Formerly at Winternet, they are
now at:
http://www.visi.com/~rad/wnet.html
Mike Horwath can be mailed at:
drechsau@geeks.org
Charles Gimon teaches an Intro to the PC class at the English Learning Center in South Minneapolis. He can be reached at gimonca@skypoint.com.
Back to my net writings.
Back to my home page.