RESOLVED - All Services are back to normal
Updates
BOSSDesk Incident Report
On the day of March 15th, we experienced a partial outage, affecting all customers of BOSSDesk.
Timeline: (all times listed in EST)
March 15th
Around 10:00am - One of our technicians detected cases of Heroku timeout errors across all instances of BOSSDesk, and notified the engineering team.
Around 10:52am - Engineering determined the Cause to be an email processing loop triggered by a new edge case not previously tested for. Engineering forcibly halted the loop and removed the conditions that caused this occurrence. System resumed normal functionality.
Around 12:22 - Customers reported getting “Connection closed” when attempting to access the Site. Issue was quickly identified as an unrelated issue stemming from one of our cloud hosting providers, Heroku.
Around 12:44 - Heroku restored normal operations
Around 4pm - Customers started to report Error 502: Bad Gateway. Issue was again traced to an error at Heroku.
Around 4:30pm - Heroku stabilized and normal service was again restored. Heroku is still investigating the cause of the issue.
Root Cause:
The outage was actually 3 different unrelated outages that occurred throughout the day. The first outage was caused by a rare edge case in BOSSDesk email processing, which was identified as an email misconfiguration. The 2nd and 3rd outages were caused by issues at one of our providers, Heroku, and more information can be found on
https://status.heroku.com/incidents/1973
https://status.heroku.com/incidents/1974
Resolution and Recovery:
We were able to stop the original loop causing the 1st outage, and Heroku resolved their service issues internally.
Corrective Measures:
While we are unable to prevent further issues from the Heroku side, We will be issuing a patch to prevent recurrences of the Email processing issue in the future. As of now the service is stable.
BOSSDesk Incident Report
On the day of March 15th, we experienced a partial outage, affecting all customers of BOSSDesk.
Timeline: (all times listed in EST)
March 15th
Around 10:00am - One of our technicians detected cases of Heroku timeout errors across all instances of BOSSDesk, and notified the engineering team.
Around 10:52am - Engineering determined the Cause to be an email processing loop triggered by a new edge case not previously tested for. Engineering forcibly halted the loop and removed the conditions that caused this occurrence. System resumed normal functionality.
Around 12:22 - Customers reported getting “Connection closed” when attempting to access the Site. Issue was quickly identified as an unrelated issue stemming from one of our cloud hosting providers, Heroku.
Around 12:44 - Heroku restored normal operations
Around 4pm - Customers started to report Error 502: Bad Gateway. Issue was again traced to an error at Heroku.
Around 4:30pm - Heroku stabilized and normal service was again restored. Heroku is still investigating the cause of the issue.
Root Cause:
The outage was actually 3 different unrelated outages that occurred throughout the day. The first outage was caused by a rare edge case in BOSSDesk email processing, which was identified as an email misconfiguration. The 2nd and 3rd outages were caused by issues at one of our providers, Heroku, and more information can be found on
https://status.heroku.com/incidents/1973
https://status.heroku.com/incidents/1974
Resolution and Recovery:
We were able to stop the original loop causing the 1st outage, and Heroku resolved their service issues internally.
Corrective Measures:
While we are unable to prevent further issues from the Heroku side, We will be issuing a patch to prevent recurrences of the Email processing issue in the future. As of now the service is stable.
← Back