|
|
PCI Late News
In an effort to keep all our users better
informed, we have implemented this log. We will log system outages and system
upgrades. Note - PCI's normal maintenance window is from 2 a.m until 6 a. m.
every Wednesday. You might experience unreliable connections during this
maintenance time.
- 3/23/2008 04:00 - 04:10. Massive power outage in North Colorado Springs takes out some wireless.
- 2/24/2008 09:50 - 12:24. Hardware failure on a core T-1 router. Replaced hardware. This outage affected PCI's
phones since our VOIP goes on one of the affected T-1s.
- 3/31/2007 07:30 - 11:00. Wireless customers on the UCCS tower are down due to a cell-phone company disrupting power
without notifying PCI.
- 2/20/2007 10:45 - 14:45. A PCI core router broke and had to be replaced.
- 6/22/2006 - 6/23/2006 - Several PCI email servers blocked by email servers using SpamCop. A PCI
customer was spamming on 6/21/2006 and 6/22/2006, probably a result of a virus.
- 4/27/2006 21:00 to 23:20 - Web upgrade causes outage for all PHP web pages.
- 4/23/2006 22:00 - Move to new domain web host machine. Upgrades to PHP.
- 4/19/2006 - Move to new domain web host machine cancelled due to earlier outage.
- 4/19/2006 - 11:40 to 14:30 upstream routing issues caused an extended outage.
- 12/26/2006 - 1/29/2006 off and on email blocking by Verizon.net. Verizon automatically blocks any email server that
blocks them. Meanwhile, they allow SPAMming. PCI's email servers have intermittently blocked
Verizon servers due to the large amount of emails sent to invalid PCI email addresses. PCI has put checks
in place to never block Verizon email servers, no matter how much SPAM they send.
- 1/8/2006 00:30 - 01:30 Email outage as server restored to original location.
- 1/7/2006 13:21 - 15:00 power outage at main POP takes out email again.
- 1/6/2006 21:00 email services restored.
- 1/6/2006 18:30 to 21:00: PCI's data and phone lines were severed. Our Mail Server is unreachable
and some wireless is down.
- 1/3/2006 For no apparent reason, Verizon.net has been blocking PCI email for several days.
They have agreed to remove the block (within 72 hours) with no other explanation.
- 5/31/2005 Email server problems all day until around 15:30. Rebooting the main email server solved the problems.
- 5/6/2005 evening. Lightning storms have wreaked some havoc with Wireless Towers.
- 1/26/05 to 2/6/05 SPAM Filtering has had periodic outages -- especially from 2:30 am to 6:00 am.
Software changes have been made and additional hardware is on the way. We apologize for any
inconveniencs and appreciate your patience.
- 1/18/05 Core router outage between 20:48 and 21:40 caused temporary outage.
- 1/7/05 some emailboxes had incoming email rejected for 'unknown user' between 4:15 and 4:40 pm.
- 12/8/04 to 12/13/04 Norton Antivirus updates caused those clients who updated to have trouble
checking email. On 12/13/04 Norton LiveUpdate included a Core Element update that appears to
fix the problem.
- 11/4/04 Large power Outage in Colorado Springs takes down UCCS wireless site.
- 10/27/04 SPAM Filtering outage during much of the day has been corrected.
- 6/26/04 Wireless towers on all parts of town down due to lightning. Several key pieces of equipment at 3
towers damaged. As of 09:48 6/27, all but the Falcon area have been restored.
- 6/6/04 Equipment problems have cause Wireless outages on the East side of Colorado Springs and Falcon area.
- 6/3/04 Denver technical support number (303-695-6178) is not working. We are working with Qwest to get
this fixed. Please use the Toll-Free number 1-(800) 859-3452 or 1-(877) DIALPCI.
- 5/10/04 Filtered Email redelivery was delayed over the weekend. It has been fixed and all
email marked for redelivery since 5/7 will be resent.
- 4/26/04 00:00 to 08:30 am some dialup logins failed.
- 4/12/04 10:30 to 11:30 am PCI's email server is down.
- 4/7/04 6:30 am to 08:30 am PCI's email servers for checking email were down.
- 3/13/04 AOL blocked 1 of PCI's outgoing email servers based on complaints they have received. One PCI
customer's account has been suspended for sending SPAM to AOL on 3/12/04. It appears to have been a virus/worm,
but the account will not be unsuspended until an investigation is complete. AOL should be removing
the block soon.
- 1/28/04 AOL blocked 2 of PCI's 3 outgoing email servers based on complaints they have received.
They have agreed to remove the blocks sometime today. They
refuse, however, to share with us any evidence that led to the blocking. In the future,
AOL has agreed to forward complaints to us before taking what we feel is unjustified action.
- 12/30/03 17:00 to 18:00 - Core PCI router in Colorado Springs melted down and had to be rebuilt. Outage
affected Colorado Springs DSL, email, home pages, Colorado Springs dialup, and Colorado Springs
dedicated circuit customers.
- 12/15/03 to 12/17/03 11:30 am - Verizon Internet has lost its mind and is incorrectly interpreting normal
operation of our email server's 'bouncing' email back to them as SPAM. We are working with
them to get this situation resolved.
- 12/16/03 0900-1100 News server down for scheduled maintenance.
- 11/13/03 Upgraded our email checking server (POP3) at 5:30 pm. Unfortunately, it uses a
slightly different method of identifying emails. So, those customers who 'leave email
on the server' may suddenly 'get' a bunch of old emails.
- 11/7/03 Added SMTP Authentication to all email servers. This allows
email clients to use a userid/password authentication if so configured. Mainly to be used by
customers sending email from networks foreign to PCI, but may be used by any customer.
- 11/6/03 PCI's main mail server melted down between 8:30 am to 12:30 pm. All email delivered to
backup MX was delivered by 9:00 pm that night.
- 9/24/03 Qwest working on lines to email server. Outage at 10:40 am.
- 9/20/03 Entire North side of Colorado Springs had power outage 1:00 am to 3:30 am. Email and
personal web pages were down.
- 8/3/03 Colorado Springs DSL down from 8:37 pm until 10:45 pm due to a Qwest circuit outage.
- 4/21/03 redirect of http://pcisys.net to http://www.pcisys.net had a problem
where part of the URL was dropped. Fixed this morning.
- 4/18/03 11:00 pm - Restructure home directories. Mail is out.
- 4/16/03 DNS resolution of pcisys.net changed. Any hard-coded web pages
with http://pcisys.net should be automatically forwarded to http://www.pcisys.net.
- 4/4/03 - Old version of FormMail.pl (Matt's Script Archive) discovered. It
had been used to send SPAM until now.
- 3/28/03 - Bad modem found and fixed. Had been causing some busy signals in
Colorado Springs
- 3/27/03 - Bad cable replaced on mail server. 30 minute outage from 6:00
pm.
- 7/23/02 - Hardware failure on main mail server interrupted email checking
and outgoing email from 9:20 pm until 12:00 pm.
- 7/4/02 - Denver DSL down from 6:00 pm to 7/5/02 2:30 am. Problem isolated
to a bad DS-3 coaxial cable between ICG and Qwest. Cable replaced.
- 7/3/02 - Denver DSL down from 11:30 pm to 7/4/02 9:00 am. Problem cleared
during isolation testing.
- 6/30/02 - Denver DSL down from 1:30 am to 8:30 am. Problem cleared during
isolation testing.
- 6/12/01 - PCI service was intermittent because of a core router problem.
- 3/27/01 - PCI's service was degraded from 1000 to 1130 hours becuase of a
Denial of service attack.
- 1/19/01 - Mail Server upgrade between 1 am and 4 am.
- 1/8/01 - UUNET Backbone outage in Denver between 5:40 pm and 6:00 pm
- 11/01/00 - Work continues on the new
PCI dialup page which can
be viewed by following the link.
- 7/25/00 - Colorado Springs power outage causes disconnects and login
problem for about 20 minutes in the afternoon.
- 7/21/00 - Denver "dead air" and "number has been disconnected" caused by
ICG Communications switch misprogramming. Problem began around 5 pm and ended
at 11 pm. Fix actions required several equipment resets during
trouble-shooting.
- 4/12/00 - US West major fiber cut causes fast busy signals in parts of
Denver.
- 4/10/00 - Upstream UUNet Internet access out from 8:30 pm to 12:40 am
because of AT&T equipment problem.
- 4/9/00 - PCI's network attacked. Fix action temporarily affected routing
around 10 pm.
- 3/15/00 - ICG removed the phone number 303-418-0006 between 4:00 pm and
8:45 pm.
- 2/28/00 - US West cable cut in Colorado Springs causes busy signals for
many customer.
- 8/19/99 - Highlands Ranch MegaBits outage caused by problem in US West
Central Office.
- 6/3/99 - Colorado Springs fast busy problem on 457-0008 is corrected.
- 4/19/99 - Denver Internet access cut from 1:00 pm to 4:30 pm due to AT&T
equipment problems on the UUNet T-3.
- 4/6/99 - 4/7/99 - US West busied our Denver Technical Support Line all day
on Tuesday and Wednesday morning. They fixed the problem Wednesday morning at
10:45.
- 1/12/99 - 10:00 to 10:45pm Routing problems on equipment serving the
457-0008 number effected users.
- 1/07/99 - 7:30pm to 8:45pm PCI's overall performance was degraded due to a
ping flood attack. We are working with our upstream providers to minimize the
effects of future attacks.
- 12/02/98 - 3:30am to 6:00am Denver Outage (During normal maintenance
period) for installation of new phone lines.
- 11/13/98 - US West fiber cut causes one-way trunking (Our modems can't
hear your modem) for calls from the Littleton Central Office to the US West's
Denver Main.
- 10/20/98 - New equipment in Denver caused temporary network problem around
1300.
- 10/20/98 - Denver - US West has messed up the translations for access
number 303-743-0465 (Some callers receiving message "this number no longer in
service"). We will be taking these lines out of service because of US West's
inability to keep them working properly. If you have been using this access
number, please switch to 303-418-0005.
- 10/19/98 - Denver added additional phone lines to the K56flex/ISDN access
number.
- 10/9/98 - Denver users receiving "all circuits busy" message and fast
busies. ICG/USWest notified of problem. PCI servers are NOT busy -- local
phone switches are not forwarding traffic. Resolved at 1230 on 10/9/98.
- 10/6/98 - Denver ICG disables correct rolling on main access number
303-418-0005 -- resolved 10/6/98
- 10/5/98 - Denver Equipment upgrade to accomodate new circuit.
- 10/2/98 - Denver ICG installs circuit on short notice to replace circuit
cut by US West.
- 9/29/98 - Denver US West cuts T-1 (24 phone lines) and said they couldn't
fix it for 6-7 business days.
- 9/28/98 - Denver has routing problem for several minutes around 7:30 pm.
Equipment reset.
- 9/11/98 -Denver loses 24 to 48 X2 phone lines for several hours due to an
ICG problem.
- 9/4/98 - Denver - Additional phone lines installed and operational.
- 9/2/98 - Denver - 24 additional phone lines scheduled -- DELAYED by phone
company until next week.
- 9/1/98 - Short Routing Problem in Denver this morning.
- 8/26/98 (4:40 pm) - Phone lines restored. TCG has coordination problems,
apparently.
- 8/26/98 (3:45 pm) - Denver phone lines down again (418-0004, 4180006).
Repair re-opened.
- 8/26/98 (2:30 pm) - Phone lines restored. TCG (another phone company)
removed circuits by mistake.
- 8/26/98 (8:30 am) - Some Denver phone lines down (418-0006, 418-00004).
ICG working as critical repair.
- 7/29/98 - Denver phone lines restored.
- 7/28/98 - Denver loses 24 phones lines on Tuesday night.
- 7/23/98 - Denver US West cable cut repaired. Englewood/Dry Creek
translations fixed.
- 7/21/98 - Denver US West cable cut in Northern Denver (Boulder,
Louisville, Gunbarrel, and Lafayette) causing fast busy problems on ALL phone
calls from these areas.
- 7/21/98 - Translations between ICG and US West are incorrect for
Englewood/Dry Creek. US West is working this issue. Result is fast busies out
of Englewood/Dry Creek.
- 7/1/98 - Power outage at TCG facility causes problems for Denver access
numbers around midnite and early morning.
- 6/29/98 - ICG/US West have routing problems out of their Aurora switch.
They are beginning work today.
- 6/23/98 - US West/ICG running diagnostics to determine reason for fast
busy in parts of Denver. PCI has capacity, but some customers are receiving
busy signals from the US West switch.
- 6/19/98 - News Server outage between 10:45 and 12:00.
- 6/19/98 - Denver and Colorado Springs K56Flex access number upgraded to
support V.90 (Lucent and USR/3COM)
- 6/2/98 - Denver X2 Access number upgraded to V.90.
- 3/10/98 - Phone company switching problems caused intermittent connection
problems on the Colorado Springs 457-1746 phone number between 22:30 and
00:30.
- 3/10/98 - Fast busy in Aurora caused by ICG/US West upgrade. Scheduled
resolution 3/13/98 (Friday).
- 3/5/98 - Electrical Problem at CS Downtown POP knocked out service on all
phones lines from 20:37 - 21:20
- 2/19/98 - New POP open in Denver. Current X2 lines will roll to new POP.
- 1/16/98 - New phone lines added in Denver. The 418-0005 access number was
messed up from 0700 to (so far) 1030. We are working with them on getting it
back. Please use access number 418-0150 for now.
- 12/31/97 - Colorado Springs. US West has hosed up some of our dialup ISDN
lines. If you are having trouble dialing in to ISDN, please use access number
457-1746.
- 12/20/97 New 33.6 phone line in Colorado Springs, 457-1869 please give it
a try.
- 11/20/97 - Minor changes made to time online calculations so ISDN
customers see only ISDN time.
- 11/20/97 - Denver US West translated several Denver prefixes wrong
(including 721). This has caused some fast busies (meaning all circuits are
busy out of US West).
- 11/12/97 - Denver. New phone lines added. We are taking new signups again.
- 10/31/97 - Denver. User's in Highlands Ranch area are getting dead air or
fast busy signal when they call the 418-xxxx numbers. This outage has been
called in to US West and ICG for resolution. As of 11/3/97, US West still has
not completely resolved the issue.
- 10/24/97 - Denver has stopped taking new signups while we await additional
phone lines. We have not been able to get a firm installation date from the
phone company. We are below our 10-1 user to modem ratio.
- 10/7/97 - Denver. Power Outage downtown took out the phones all over
Denver this morning.
- 9/26/97 - Denver. Routing instability in upstream provider fixed at 5:15
pm. If you had trouble reaching some web destinations today, please try again.
- 9/23/97 - Denver and CS Portmaster's upgraded to new OS. Fixed disconnect
problems with US Robotics Sportster modems.
- 9/15/97 - Denver ISDN lines goofed up by US West. Problem is on-going.
- 9/8/97 - CS announces K56Flex Modem support (access number 457-1746).
- 9/8/97 - CS will shutdown current X2 access number (591-0817) in favor of
a different access number with many more lines.
- 9/8/97 - CS. Temporary access number 447-8775 will be deactivated. The new
access number is 457-1746.
- 9/5/97 - ICG (our phone company for digital lines) had an outage on 9/5/97
from 9:02 am until 10:00 am. The Denver phone numbers 418-0004, 418-0005, and
418-0006 were affected.
- 8/28/97 - Denver announces K56Flex Modem support.
- 8/25/97 - Denver. US West's Sullivan Central Office was broken since from
Saturday (8/23) until Monday (8/25) and has denied Aurora customers access to
numbers 418-0004, 0005, 0006, and 0007. US West knew about the problem
Saturday and refused to open a trouble-ticket until Monday morning.
- 8/20/97 - Denver. Expected Upgrade (on 8/26 or 8/27) to replace equipment
on 418-0006 and 418-0007. Will enable Rockwell/Lucent 56K and ease some
connection problems.
- 8/19/97 - Denver. New phone lines are not rolling correctly. Emergency fix
order put in to the phone company. There are actually plenty of lines with
additional access number 418-0007.
- 8/18/97 - Denver. Some disconnects today (10:00 am to 4:30 pm) on 418-0006
as we moved the new digital phone lines to their final configuration.
- 8/18/97 - More analog lines cut on access number 743-0465. This will be
the last cut on these lines. Please try the new access numbers.
- 8/6/97 - PCI (Colorado Springs) lost 2 phone lines in the administration
office -- We believe it is US West's newest attempt to put us out of business.
We apologize for any busy signals when calling for tech support. US West has
promised to fix it by 7:00 pm Thursday (8/7/97).
- 8/5/97 - New digital phone lines installed to support PRI (ISDN) and
dialup (418-0006).
- 8/5/97 - There will be periodic disconnects this week as we move phone
lines to different equipment. We are sorry for the inconvenience, but these
changes will result in better service.
- 8/4/97 - New Analog (14.4 - 33.6) Access Number in Denver (418-0005)
- 7/31/97 - X2 technology available in Denver (Access 418-0004)
- 7/29/97 - Mail server moved from Jerry to Kramer. There were a few outages
on Thursday (7/30) morning.
- 7/19/97 - Recent problems with the Internet? Check the
USA
Today Report about the problems.
- 7/1/97 - DARKEST DAY IN PCI HISTORY UUNET (PCI's Backbone
to the Internet) was down across the country for over 7 hours. This made it
impossible for anyone to connect to the internet. As well, Jerry (PCI's email
and web server) went down for approx. 30 minutes. We are sorry for any
inconvience these problems may have caused.
- 6/25/97 - Denver has added 7 more phone lines.
- 6/23/97 - The phone companies have yet to deliver the lines previously
promised on the 12th of June. The new promised date is the 24th of June. We
are sorry for any inconvenience this may have caused.
- 6/22/97 - Denver is still waiting for the ISDN lines originally promised
for the middle of this month. The new promised date for ISDN is the 10th of
July. Please accept our apologies for the inconvenience.
- 5/13/97 - PCI introduces X2 technology in Colorado Springs. 24 lines have
been reserved for this new technology.
- 5/13/97 - Denver has installed 8 new phone lines. There was a temporary
outage to install the new modems. Sorry for the inconvenience this may have
caused.
- 5/12/97 - Our router died last night causing users to be able to see only
our webpage and not allowing them out on the web. We are researching the
cause, we apologise for any inconvenience.
- 5/2/97 - Denver has installed 26 new phone lines.
- 4/21/97 - Denver experienced a 6 hour down time. We are sorry for any
inconvience that this may have caused.
- 4/10/97 - Upgraded our system security which will now prevent email only
accounts from logging on to our system. Ensure that you are not trying to log
in to PCI with an email only account, you will be rejected. You will still be
able to check your email on the email only account when dialing in with the
primary logon account.
- 4/07/97 - Frontpage Extensions are still being worked on.
- 3/26/97 - System Maintenance being performed in the wee hours tomorrow
morning.
- 3/26/97 - New ISDN Lines are finally being installed in the Springs, they
should be up and running in a couple of days.
- 3/21/97 - Newman the newserver is acting strange and is not responding, we
are working on the problem.
- 3/20/97 - Newman the newserver will be down for several hours beginning at
10:30am. We are performing a software upgrade to better serve our customers.
- 3/18/97 - Colorado Springs has added 11 new phone lines to the 447-8775
access number. This will lessen the possibility of busy signals during the
primetime hours. We are below our promised 10-1 user to modem ratio. We will
be adding additional lines in two weeks. We apologize for the inconvenience.
- 3/13/97 - Denver has added second access number (743-0072)
- 3/5/97 - Denver has added 16 new phone lines to better serve you.
- 2/13/97 - The New email program that PCI has installed has caused erratic
problems. These problems are being resolved. We are sorry for any inconvience.
- 2/13/97 - Denver has added 8 new phone lines to better serve our Denver
PCI customers.
- 2/10/97 - Colorado Springs opens new POP server. This adds the new access
number 447-8775 and adds approx 32 new phone lines for customer use.
- 1/28/97 - Jerry and Estelle suffered brain fade today causing connectivity
problems, seems they have been failing for the last two days causing long
login times. Both were rebooted and appear to be functioning normally again,
we will continue to monitor for further problems.
- 1/28/97 - New Phone Lines from USWEST not installed as expected at our COS
location. User sign on patterns have changed over the last two weeks requiring
new lines. We are still slightly below our promised 10-1 user to modem ratio,
yet it is still not enough. We have refused new signups in COS to relieve the
burden. We apologize for the inconvenience.
- 1/27/97 - Projected opening date of new downtown Colorado Springs POP
- 1/20/97 - ISDN Service down. Hardware failure expected up 1/21
- 1/19/97 - Colorado Springs main T-1 loses connection for 30 minutes
- 1/14/97 - PCI upgrades memory in Main server
- 1/14/97 - PCI adds 20 new phone lines in Colorado Springs
- 1/10/97 - PCI adds 16 new phone lines in Denver
- 1/8/97 - PCI upgrades News Server
- 12/20/96 (Friday, again) - Frank and Estelle decided not to let any of our
users on the system. A few whacks and they are back in shape.
- 12/13/96 (Friday the 13th) - Our news server went offline for a few
minutes today, our System Administrators were quick to respond and had us back
up in no time at all.
- 12/12/96 - PCI had a technical glitch this morning which affected the mail
server and the PCI Systems Homepage, we apologize for the inconvenience.
- 12/12/96 - Denver Phone service has been erratic due to work being done by
USWest, please accept our apologies for the temporary service disruptions.
- 11/19/96 - PCI adds 30 new phone lines to serve you better
- 10/30/96 - PCI adds 16 new phone lines to serve you better
- 10/28/96 - Microsoft FrontPage Server Extensions Added
- 10/28/96 - Home Page server upgraded to Apache 1.1.1
- 10/3/96 - 20 more new phone lines here at PCI
- 10/1/96 - PCI welcomes Robert Stark, the newest member of the PCI
Technical Support Team
- 9/16/96 - DENVER is now Open!!!!
- 8/29/96 - We are now offering ISDN!
- 8/27/96 - 8/29/96 US West has switching problem again causing access
numbers to respond with "number disconnected" error message. Some users
experienced difficulty, we are very sorry for any inconvience.
- 8/27/96 - PCI added 9 new phone lines today.
- 8/13/96 - Apparently, US West had a switching problem Wednesday evening.
Several of our access numbers responded with a 'number disconnected' message.
The outage lasted only about 20 minutes. Apologies to those affected.
- 8/7/96 - Upgrade to one of our terminal servers - Estelle.
- 8/1/96 - Added 37 new phone numbers. 572-0499 is a new access number.
- 8/1/96 - PCI added a new tech-support staff member. Stephen Lane comes to
us from Oregon.
- 7/20/96 - Implemented comprehensive rules and regulations
|