System Outage Issue on August 14, 2018

  • 1
  • Question
  • Updated 2 years ago
At approximately 12pm PST on August 14th, the operations staff at the Technology Center were made aware that some colleges experienced login errors that prevented them from connecting to the application forms. The staff began researching the problems and were seeing both the problem and normal operations at the same time. Ultimately, service was fully restored at approximately 2pm Pacific after several of the application nodes were restarted.

The team is still conducting their post-mortem review, however preliminary analysis indicates that the outage was caused by a significant spike in login attempts that ultimately resulted in a disruption in connections to the database. We are still looking at the data, but believe this could be related to a spam attempt during a peak usage time. 

In the short term, we are increasing all available system resources to prevent this from occurring in the future.  We are also looking to take advantage of Amazon's auto-scale features so that we can grow available servers on demand. 

One additional note: We also received a call yesterday from a college that coincidentally encountered an error while trying to login to another CCC application through the SSO proxy. This was an isolated issue that was completely unrelated to the outage. There was no connection between yesterday's outage and the SSO proxy. 

We apologize for the service interruption and look forward to a successful busy season with no more service outages.

Thank you.
Photo of Patty - OpenCCCApply Product Manager

Patty - OpenCCCApply Product Manager, Official Rep

  • 310 Posts
  • 15 Reply Likes

Posted 2 years ago

  • 1
Photo of Esau Tovar

Esau Tovar

  • 2 Posts
  • 0 Reply Likes
Hi Patty. There is another outage now and the Tech Center phone number is unresponsive. I have tried 4 times now and after about 4-5 minutes either someone answers and the call gets disconnected, or there is no answer at all, the call just terminates. Can you please post an update? Thank you.

Esau
Photo of Merrie Wales

Merrie Wales, Official Rep

  • 960 Posts
  • 57 Reply Likes
Esau,
What is the URL for the site you are saying is unresponsive?  And what is the phone number you are calling?
(Edited)
Photo of Merrie Wales

Merrie Wales, Official Rep

  • 960 Posts
  • 57 Reply Likes
Esau,
We can not locate any system that is down for CCCApply.  Please share what URL you are trying to access. 

Thank you for bringing the issue with the Helpdesk phone line to our attention.  Additionally, I have more information about the Student Helpdesk line.  The Technology Center is working with the Student Helpdesk vendor to resolve wait time issues for incoming calls.  As of a few minutes ago, the wait time issue was resolved, and we will continue to monitor these wait times, and work with our vendor to improve caller experience.
(Edited)