All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We’ll see if that changes over the weekend…

  • kadotux@sopuli.xyz
    link
    fedilink
    English
    arrow-up
    104
    arrow-down
    2
    ·
    edit-2
    4 months ago

    Here’s the fix: (or rather workaround, released by CrowdStrike) 1)Boot to safe mode/recovery 2)Go to C:\Windows\System32\drivers\CrowdStrike 3)Delete the file matching “C-00000291*.sys” 4)Boot the system normally

    • StV2@lemmy.world
      link
      fedilink
      English
      arrow-up
      64
      arrow-down
      4
      ·
      4 months ago

      It’s disappointing that the fix is so easy to perform and yet it’ll almost certainly keep a lot of infrastructure down for hours because a majority of people seem too scared to try to fix anything on their own machine (or aren’t trusted to so they can’t even if they know how)

      • HaleHirsute@infosec.pub
        link
        fedilink
        English
        arrow-up
        69
        arrow-down
        1
        ·
        4 months ago

        They also gotta get the fix through a trusted channel and not randomly on the internet. (No offense to the person that gave the info, it’s maybe correct but you never know)

        • kadotux@sopuli.xyz
          link
          fedilink
          English
          arrow-up
          15
          arrow-down
          1
          ·
          4 months ago

          Yeah, and it’s unknown if CS is active after the workaround or not (source: hackernews commentator)

        • letsgo@lemm.ee
          link
          fedilink
          English
          arrow-up
          8
          ·
          4 months ago

          True, but knowing what the fix might be means you can Google it and see what comes back. It was on StackOverflow for example, but at the time of this comment has been taken offline for moderation - whatever that means.

        • ColeSloth@discuss.tchncs.de
          link
          fedilink
          English
          arrow-up
          3
          ·
          4 months ago

          Meh. Even if it bricked crowdstrike instead of helping, you can just restore the file you deleted. A file in that folder can’t brick a windows system.

      • NaibofTabr@infosec.pub
        link
        fedilink
        English
        arrow-up
        50
        ·
        4 months ago

        This sort of fix might not be accessible to a lot of employees who don’t have admin access on their company laptops, and if the laptop can’t be accessed remotely by IT then the options are very limited. Trying to walk a lot of nontechnical users through this over the phone won’t go very well.

        • AccountMaker@slrpnk.net
          link
          fedilink
          English
          arrow-up
          17
          ·
          4 months ago

          Yup, that’s me. We booted into safe mode, tried navigating into the CrowdStrike folder and boom: permission denied.

          • Cryophilia@lemmy.world
            link
            fedilink
            English
            arrow-up
            11
            ·
            4 months ago

            Half our shit can’t even boot into safe mode because it’s encrypted and we don’t have the keys rofl

            • Oderus@lemmy.world
              link
              fedilink
              English
              arrow-up
              1
              ·
              4 months ago

              If you don’t have the keys, what the hell are you doing? We have bitlocker enabled and we have a way to get the recovery key so it’s not a problem. Just a huge pain in the ass.

      • thehatfox@lemmy.world
        link
        fedilink
        English
        arrow-up
        32
        ·
        edit-2
        4 months ago

        Might seem easy to someone with a technical background. But the last thing businesses want to be doing is telling average end users to boot into safe mode and start deleting system files.

        If that started happening en masse we would quickly end up with far more problems than we started with. Plenty of users would end up deleting system32 entirely or something else equally damaging.

        • Ookami38
          link
          fedilink
          English
          arrow-up
          7
          ·
          4 months ago

          I do IT for some stores. My team lead briefly suggested having store managers try to do this fix. I HARD vetoed that. That’s only going to do more damage.

      • r00ty@kbin.life
        link
        fedilink
        arrow-up
        15
        ·
        4 months ago

        It might not even be that. A lot of places have many servers (and even more virtual servers) running crowdstrike. Some places also seem to have it on endpoints too.

        That’s a lot of machines to manually fix.

        • StV2@lemmy.world
          link
          fedilink
          English
          arrow-up
          1
          ·
          4 months ago

          That is unfortunate but also leads me to a different question

          Why do people like windows server? I’ve had to use it a couple of times for work and although it’s certainly better than just using the desktop windows it’s so heavy compared to running something like Debian

          In our case, the fact we were using windows server actually made it a worse experience for customers aswell because the hardware was not up to it (because budget constraints) so it just chugged and slowed down everything making it a terrible experience for everyone involved (not to mention how often it’d have to be rebooted because a service wouldn’t restart)

        • Oderus@lemmy.world
          link
          fedilink
          English
          arrow-up
          1
          ·
          4 months ago

          You can do it over the phone. I just did a few dozen this morning and it was relatively easy.

          • wreckedcarzz@lemmy.world
            link
            fedilink
            English
            arrow-up
            4
            ·
            4 months ago

            “yes, now open the file explorer. No, that’s internet explorer… Yes, with the files. Now go to this pc… No, I know you are at this pc, but the entry on the left. No that’s the keyboard. On the screen. Where it says this pc, on the left. The left. The left. … That’s the start menu. Okay, let’s try this a different way. On the keyboard, press the windows key and r. No, simultaneously. The windows key is the one with the flag. Yes. R. As in Romeo. Yes I know a window appeared, very good. Now type c colon backslash windows backslash system 32… Yes like the numbers. No, that’s a semicolon. Yes. Shift. On the keyboard. Simultaneously. And another backslash drivers. Click OK. What error? Why did you type that after the colon? It needs to go at the end. Yes, the end. Yes. Yes. Now click OK. What error? Read the text you typed to me. Why didn’t you delete the semicolon? Yes. Yes. What error?! AHHHHHHHHHHHHHHHH”

            yeah, sometimes that’s just not an option…

      • Grandwolf319
        link
        fedilink
        English
        arrow-up
        2
        ·
        4 months ago

        I wouldn’t fix it if it’s not my responsibly at work. What if I mess up and break things further?

        When things go wrong, best to just let people do the emergency process.

    • cheeseburger@lemmy.ca
      link
      fedilink
      English
      arrow-up
      45
      ·
      4 months ago

      I’m on a bridge still while we wait for Bitlocker recovery keys, so we can actually boot into safemode, but the Bitkocker key server is down as well…

    • WagnasT@lemmy.world
      link
      fedilink
      English
      arrow-up
      13
      ·
      4 months ago

      Man, it sure would suck if you could still get to safe mode from pressing f8. Can you imagine how terrible that’d be?

      • a_postmodern_hat@lemmy.world
        link
        fedilink
        English
        arrow-up
        14
        ·
        4 months ago

        You hold down Shift while restarting or booting and you get a recovery menu. I don’t know why they changed this behaviour.

      • Ookami38
        link
        fedilink
        English
        arrow-up
        8
        ·
        4 months ago

        That was the dumbest thing to learn this morning.

    • resin85@lemmy.ca
      link
      fedilink
      English
      arrow-up
      2
      ·
      4 months ago

      Not that easy when it’s a fleet of servers in multiple remote data centers. Lots of IT folks will be spending their weekend sitting in data center cages.