R3 Data Recovery Logo
? Security House, Windsor Street, Sheffield S4 7WB · ☎️ 0800 999 3282
Tap To Call: 0800 999 3282
R3 Data Recovery Logo
Problem with your hard drive or other data storage device? Contact us for immediate help and assistance!
CLICK HERE NOW

GitLab backups fail

Source-code hub GitLab.com is in meltdown after experiencing data loss as a result of what it has suddenly discovered are ineffectual backups.

On Tuesday evening, Pacific Time, the startup issued a sobering series of tweets we’ve listed below. Behind the scenes, a tired sysadmin, working late at night in the Netherlands, had accidentally deleted a directory on the wrong server during a frustrating database replication process: he wiped a folder containing 300GB of live production data that was due to be replicated.

Just 4.5GB remained by the time he canceled the rm -rf command. The last potentially viable backup was taken six hours beforehand.

That Google Doc mentioned in the last tweet notes: “This incident affected the database (including issues and merge requests) but not the git repos (repositories and wikis).”

So some solace there for users because not all is lost. But the document concludes with the following:
So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.

The world doesn’t contain enough faces and palms to even begin to offer a reaction to that sentence. Or, perhaps, to summarise the mistakes the startup candidly details as follows:

- LVM snapshots are by default only taken once every 24 hours. YP happened to run one manually about 6 hours prior to the outage
- Regular backups seem to also only be taken once per 24 hours, though YP has not yet been able to figure out where they are stored. According to JN these don’t appear to be working, producing files only a few bytes in size.
- SH: It looks like pg_dump may be failing because PostgreSQL 9.2 binaries are being run instead of 9.6 binaries. This happens because omnibus only uses Pg 9.6 if data/PG_VERSION is set to 9.6, but on workers this file does not exist. As a result it defaults to 9.2, failing silently. No SQL dumps were made as a result. Fog gem may have cleaned out older backups.
- Disk snapshots in Azure are enabled for the NFS server, but not for the DB servers.
- The synchronisation process removes webhooks once it has synchronised data to staging. Unless we can pull these from a regular backup from the past 24 hours they will be lost
- The replication procedure is super fragile, prone to error, relies on a handful of random shell scripts, and is badly documented
- Our backups to S3 apparently don’t work either: the bucket is empty

Making matters worse is the fact that GitLab last year decreed it had outgrown the cloud and would build and operate its own Ceph clusters. GitLab’s infrastructure lead Pablo Carranza said the decision to roll its own infrastructure “will make GitLab more efficient, consistent, and reliable as we will have more ownership of the entire infrastructure.”

At the time of writing, GitLab says it has no estimated restore time but is working to restore from a staging server that may be “without webhooks” but is “the only available snapshot.” That source is six hours old, so there will be some data loss.

Last year, GitLab, founded in 2014, scored US$20m of venture funding. Those investors may just be a little more ticked off than its users right now.

“On Tuesday, GitLab experienced an outage for one of its products, the online service GitLab.com,” a spokesperson for the San Francisco-based biz told The Register in an email, adding: “This outage did not affect our Enterprise customers.”

“We have been working around the clock to resume service on the affected product, and set up long-term measures to prevent this from happening again,” the spinner said. “We will continue to keep our community updated through Twitter, our blog and other channels.”

Meanwhile, the sysadmin who accidentally nuked the live data reckons “it’s best for him not to run anything with sudo any more today.”

Posted In: Backup | News

What Our Customers Say About Us

R3 Data Recovery is an excellent emergency data recovery service that I would have no hesitation in recommending to anyone.

From the initial telephone conversation I had with them I was extremely impressed by their implicit understanding and sensitive handling of my stress and panic, believing I had possibly lost 10 year’s worth of invaluable documentation, imagery and archiving from my highly respected research held on a USB stick.

They worked efficiently and conscientiously to retrieve data urgently for me and made provision for me to receive it and all in time for an important PP presentation, the success of which was crucial to my career.

They were mindful in keeping me informed of their progress with the recovery process, hence putting my mind at rest, and they accomplished this all within a working day/evening. They worked hard and consistently until the job was completed successfully and made themselves available for me to talk to during out-of-work hours. They transferred all the remaining data that was not urgent onto a USB stick and returned it in the post to me, thus very satisfactorily completing an exemplary and outstanding service.

Adrian Howells (Performance Artist and Honorary Research Fellow ) - University of Glasgow

What Does A REAL Data Recovery Lab Look Like?
Need Quick Help?
Fill out our no-obligation diagnostics form below to get a FREE consultation or callback.

Our data recovery engineers can diagnose the majority of faults remotely for free.


  • Technical Details

    You don't need to select the following; however, please provide a brief description of the problem in the "Other information" field.

  • This field is for validation purposes and should be left unchanged.
FREE No Obligation Data Recovery Consultation Request
Corrupted Data? Speak to our expert team on 0800 999 3282 or click here for help!

About our Company

R3 logo
R3 Data Recovery Services are designed to meet the needs of businesses and home users requiring a fixed cost, highly successful data recovery solution.

R3 Data Recovery is rated (4.6) by 294 customer reviews on Reviews.co.uk

Quick Links

Latest NewsCareers at R3Area CoveragePrivacy PolicyCookies PolicyTerms & ConditionsTestimonials

Latest Tweets

Get In Touch

Security House, Windsor Street, Sheffield S4 7WB
0800 999 3282
enquiries@r3datarecovery.com
Information Commissioner's Office Certificate
© 2004 – 2018 · R3 Data Recovery Ltd · Company No.05033569 · VAT No. GB119444414
twitterfacebookpinterestyoutube-playgoogle