What happened:
In the morning of Dec 9, we deployed new changes to improve the security for Google Docs for edge cases such as:
- multiple teachers collaborating in a single class and copying a Google Doc assignment that contains Google Docs they were not originally an owner of.
- when the teacher-on-record of a class changes or a substitute teacher needs control over Google Doc assignments
However, to better support this logic, some migration of data was required so that we could properly track on our side who the "correct" owners of these Docs are. This process did not complete overnight as expected due to unforeseen speed issues with Google's API. Once school was underway, any Google Doc assignments not yet migrated to our new data structure caused the error message seen above.
Who it impacted:
This issue impacted any operation that would affect the Google permission of that file, namely student submissions and unsubmit operations. Students saw this error when trying to submit any assignment that included a Google Doc, and Teachers also encountered this if they tried to submit/unsubmit on behalf of a student.
Final resolution:
All data was successfully migrated by 9:25PM PST. In all, this incident lasted from 4:00AM to 9:25PM on Dec 9, 2021.
Our team sincerely apologizes for the extended time it took to migrate. We are making adjustments to design techniques to avoid this type of incident in the future.
Fortunately, there was no data loss during this time and students were able to access and edit their Google Docs assignments in eKadence. It was only the final submit process that had an error. There was no other disruption to services.