A customer reported an issue with their tenancy via email. The customer success team created a support request on behalf of the customer, and immediately escalated the incident with the Development team.
A fix was issued and the incident was closed at 12:50 PM (AEST).
The ability to login with username over the email is a new feature that was rolled-out as an option to all tenants, using a configuration variable.
The last production deployment overrode the configuration for a specific set of customers and set the default behaviour back to email instead of username, which prevented learners from using their username.
The incident lasted 3 hours from discovery to remediation.
On Tuesday morning, students of a tenancy reported the validation errors. While it was still possible to login, the message was confusing to users and could prevent some of them from accessing the platform.
Some students were also having trouble resetting their password since the pop up box asked for email instead of the username, which some younger students may not have access to.
The Customer Success team created a support request on behalf of the customer, and immediately escalated the incident with the on-call support engineer who investigated the issue, identified it as a regression, then applied a fix involving a re-deployment of the applications(~2hrs), and then re-tested in Production.
The QA team updated their automated testing scripts for all future deployment to prevent releasing the pipeline if an incorrect configuration item is detected against the source definitions for each customer using them.
The Scrum master and Lead Development Engineer also implemented additional process to ensure that customers configurations are not modified by developers directly.