Live Before & After

From: alice@company.com To: team@company.com Subject: Deploy pipeline failing again Hi team, The deploy pipeline for the staging environment has been failing consistently since last night. The error seems to be related to the database migration step. Specifically, it's failing on the "add_user_preferences" migration with a foreign key constraint violation. I've checked the migration file and the referenced table "user_settings" does exist, but the column types don't match. The migration expects a UUID but the actual column is an integer. Can someone take a look? Best, Alice Senior Backend Engineer --- On Mon, Jul 14, 2026 at 3:45 PM, Bob <bob@company.com> wrote: > Hi team, > > I noticed the same issue this morning. The staging database was restored > from a production backup that had a slightly different schema. The > "user_settings.id" column was changed from integer to UUID in production > two weeks ago (see PR #1423) but the staging restore used an older backup. > > The fix should be to re-run the staging restore with the latest production > snapshot, or update the migration to handle both types. > > Best, > Bob > DevOps Lead > > --- > On Mon, Jul 14, 2026 at 10:30 AM, Charlie <charlie@company.com> wrote: >> Hi team, >> >> Actually I think the issue is different from what Bob is describing. >> The migration "add_user_preferences" was created last week and it assumes >> the column type is UUID. But the staging database was restored from a >> backup taken before the UUID migration ran. >> >> There are two possible fixes: >> 1. Restore staging from a more recent backup (after the UUID migration) >> 2. Create a new migration that handles the column type mismatch >> >> I'd recommend option 2 since it's safer and doesn't require waiting for >> a new backup to complete. >> >> Also, we should add a check in our CI pipeline to verify that migrations >> are compatible with the staging schema before deploying. >> >> Best, >> Charlie >> Staff Engineer >> >> DISCLAIMER: This email and any attachments are confidential and intended >> solely for the addressee. If you are not the intended recipient, please >> notify the sender immediately and delete this email from your system. >> Unauthorized use, disclosure, or copying is strictly prohibited. >> >> --- >> On Sun, Jul 13, 2026 at 8:15 PM, Diana <diana@company.com> wrote: >>> Hi team, >>> >>> I've been looking into this from the database side. The issue is that >>> our backup rotation policy only keeps weekly snapshots, and the UUID >>> migration ran between snapshot cycles. So the automatic restore process >>> picked the closest snapshot which was from before the migration. >>> >>> I'm updating the backup policy to take snapshots before and after every >>> schema migration. This will prevent this class of issues in the future. >>> >>> For now, I've manually patched the staging database to add a UUID column >>> alongside the existing integer column. The deployment should work now. >>> >>> Best, >>> Diana >>> Database Administrator >>> >>> DISCLAIMER: This email and any attachments are confidential and intended >>> solely for the addressee. If you are not the intended recipient, please >>> notify the sender immediately and delete this email from your system. >>> Unauthorized use, disclosure, or copying is strictly prohibited. >>> >>> --- >>> On Sun, Jul 13, 2026 at 5:00 PM, Eve <eve@company.com> wrote: >>>> Hi team, >>>> >>>> Just to add to the discussion - I think the root cause is deeper than >>>> just a backup issue. Our migration framework doesn't handle column type >>>> changes gracefully. When we changed user_settings.id from integer to >>>> UUID, the migration itself succeeded, but any migration that references >>>> that column assumes the new type. If a restore happens between those >>>> two migrations, everything breaks. >>>> >>>> I propose we add a database schema version check at the start of every >>>> deployment that compares the expected schema against the actual schema >>>> and aborts with a clear error if they don't match. >>>> >>>> This would have caught this issue before any migrations ran. >>>> >>>> Thoughts? >>>> >>>> Best, >>>> Eve >>>> Platform Architect >>>> >>>> DISCLAIMER: This email and any attachments are confidential and intended >>>> solely for the addressee. If you are not the intended recipient, please >>>> notify the sender immediately and delete this email from your system. >>>> Unauthorized use, disclosure, or copying is strictly prohibited. >>>> >>>> --- >>>> On Sat, Jul 12, 2026 at 11:00 AM, Frank <frank@company.com> wrote: >>>>> Hi team, >>>>> >>>>> I'm not directly involved in the staging issue, but I wanted to share >>>>> something from our production environment. We had a similar issue last >>>>> quarter when the "add_billing_fields" migration ran on a database that >>>>> was missing a previous migration. The fix was to add a dependency check >>>>> in the migration runner. >>>>> >>>>> Our migration runner now checks that all previous migrations in the >>>>> dependency graph have been applied before running any new migration. >>>>> This would have caught the staging issue as well. >>>>> >>>>> I've attached the PR for reference: #1456 >>>>> >>>>> Best, >>>>> Frank >>>>> Infrastructure Engineer >>>>> >>>>> DISCLAIMER: This email and any attachments are confidential and intended >>>>> solely for the addressee. If you are not the intended recipient, please >>>>> notify the sender immediately and delete this email from your system. >>>>> Unauthorized use, disclosure, or copying is strictly prohibited. >>>>> >>>>> --- >>>>> On Fri, Jul 11, 2026 at 4:30 PM, Grace <grace@company.com> wrote: >>>>>> Hi team, >>>>>> >>>>>> Just a quick update - I've confirmed that the fix Diana applied >>>>>> (adding the UUID column alongside the integer column) is working >>>>>> in staging. The deployment completed successfully and all tests pass. >>>>>> >>>>>> However, I noticed something odd - the "add_user_preferences" migration >>>>>> also creates a new table called "user_preferences" that references >>>>>> user_settings.id. Because the column type was ambiguous during the >>>>>> migration, the foreign key constraint was created without specifying >>>>>> the column type, which could cause issues down the line. >>>>>> >>>>>> I've created a follow-up ticket to clean this up: TICKET-4242 >>>>>> >>>>>> Best, >>>>>> Grace >>>>>> QA Lead >>>>>> >>>>>> DISCLAIMER: This email and any attachments are confidential and intended >>>>>> solely for the addressee. If you are not the intended recipient, please >>>>>> notify the sender immediately and delete this email from your system. >>>>>> Unauthorized use, disclosure, or copying is strictly prohibited. >>>>>> This email message has been scanned for viruses and malware. >>>>> >>>>> Please consider the environment before printing this email. >>>>> >>>> IMPORTANT NOTICE: Our company has moved to a new email system. Please >>>> update your contacts accordingly. Old email addresses will stop working >>>> after August 1st, 2026. >>>> >>> This communication contains information that is confidential and may also >>> be privileged. If you are not the intended recipient, please notify us >>> immediately. >>> >> IMPORTANT: For urgent deployment issues, please use the on-call Slack >> channel (#deploy-urgent) instead of email for faster resolution. >> > Thanks everyone for the input. Let's implement the schema version check > that Eve proposed and also fix the specific migration. Alice, can you > take the lead on the migration fix? > > Alice - please confirm if you've seen this message. > > Best, > Bob