It was to talk about “team restructuring”
Randomly got a message from one of my reports asking what this “Mandatory Team Meeting” was on his calendar. I hadn’t been invited, but it was our whole company shutting down ¯\_(ツ)_/¯
Hey, that happened to me, too!
I got scheduled for a mandatory meeting with 1 hour notice. During lunch.
I asked my boss what it was. He didn’t know either. I joked that it was us being shut down.
Sure enough, 1 hour later we were both writing LinkedIn recommendations and helping each other find jobs after it was announced that our whole studio was being shut down by corporate and myself plus all my coworkers were all now jobless.
A former coworker of mine once learned that his company was shutting down because the office was raided by FBI agents who seized all the computers, servers and company documents. Everybody sat around in the empty office for a little while and then went home, and nobody ever got paid or heard from the company ever again. Even the tax documents at the end of the year didn’t get sent out.
What did he have to do without the tax documents?
I don’t know exactly. There’s some process you go through with the IRS to find out what you were paid and how much was withheld in situations like this where the company just goes out of existence suddenly. The IRS has all that info anyway because companies withhold and submit the taxes (at least while they still exist).
Well, they’re supposed to, anyway.
I at least had the cathartic experience of being told “hey we need to shut down EVERYTHING before 7pm because that’s when the email will turn off, so log into every service you know we use and delete it all.” And then I spent the next couple hours clicking every delete button I could.
K8s clusters? Delete. Prod DB? Delete. Prod DB backups? Delete. S3 buckets? Delete. Cloudflare account? Delete.
It was actually kinda fun.
This sounds therapeautic
Oof, how did it end up going?
The company shut down.
To shreds you say? How are the investors holding up?
To shreds, you say?
But the company’s not supposed to shut down.
Is that not typical?
Of course it’s not typical! Ordinarily, companies don’t just shut down. I want to make very clear that this is not the normal state of affairs!
So what’s the minimal employee requirements for a company?
Oh, I suppose … one?
Well, one, I suppose.
That’s not very typical, I’d like to make that point.
Chance in a million
The front fell off.
The front fell off?
Yeah that’s not very typical, I’d like to make that point.
(The idea’s that the company tanked, just like this tanker did.)
Still one of the greatest pieces of cinematics ever produced, and I’ll die on this hill.
Random team meeting on the first Friday after I got hired. “Telltale has lost it’s funding and everyone is being let go”. Fun week.
Bitwise industries did that to us but it was an optional meeting. Like a kick to the gut.
Worst part is they stole our 401ks.
¯\(ツ)/¯\ you dropped this
Companies are often insane. I’m working in one who has this one guy build a super complicated architecture, because he don’t know aws. So instead of just using a message queue on aws, he is building Java programs and tons of software and containers to try and send messages in a reliable way. Costs the company huge money, but they don’t care, since he is some old timer who has been there for like 10 years and everyone let’s him do what he wants.
No vendor look-in with his solution though.
It’s a different form of lock-in since it’s just his creation. When he leaves, all of this will be very hard to maintain and the company will probably rebuild it all on aws.
I have been bringing this up but they say that it’s too late to change direction now (they are afraid to upset the guy).
But I’m looking on the bright side. I get to learn a lot of stuff I otherwise I wouldnt if this was a single managed aws service. I’m bringing in terraform and instead of just putting a message queue there, I need to spin up entire architectures to run his ec2 instances with all the apps and everything required to make things work.
Takes months… So for me it’s fun. I don’t have to pay for it. But companies are crazy. :)
Yeah great but what about when he dies and nobody else knows how it works? I’ve had to deal with that more than once (creator of Blackboard and creator of IP Office Contact Center, when they died so did the product)
I personally always try to engineer away from cloud services. They cost you ridiculous amounts of money and all you need is documentation afterwards. Then it can be easier and faster than AWS or GC
You’re the guy 1984 was talking about…
Got to agree with @[email protected] here, although it depends on the scope of your service or project.
Cloud services are good at getting you up and running quickly, but they are very, very expensive to scale up.
I work for a financial services company, and we are paying 7 digit monthly AWS bills for an amount of work that could realistically be done with one really big dedicated server. And now we’re required to support multiple cloud providers by some of our customers, we’ve spent a TON of effort trying to untangle from SQS/SNS and other AWS specific technologies.
Clouds like to tell you:
- Using the cloud is cheaper than running your own server
- Using cloud services requires less manpower / labour to maintain and manage
- It’s easier to get up and running and scale up later using cloud services
The last item is true, but the first two are only true if you are running a small service. Scaling up on a cloud is not cost effective, and maintaining a complicated cloud architecture can be FAR more complicated than managing a similar centralized architecture.
I worked in operations for a large company that had their own 50,000 sq ft data center with 2000 physical servers, uncountable virtual servers, backup tape robots, etc… Their cooling bill would like to disagree with your assessment about scaling. I was unpacking new servers regularly because, when you own you own servers, not only do you have to buy them, but you have to house them (so much rented space), run them, fix them, cool them, and replace them.
Don’t get me wrong, I’ve also seen the AWS bill for another large company I worked for and that was staggering. But, we were a smaller tech team and didn’t require a separate ops group specifically to maintain the physical servers.
If you really need the scale of 2000 physical machines, you’re at a scale and complexity level where it’s going to be expensive no matter what.
And I think if you need that kind of resources, you’ll still be cheaper of DIY.
You are paying aws to not have one big server, so you get high availability and dynamic load balancing as instances come and go.
I agree its not cheaper than being on prem. But it’s much higher quality solutions.
Today at work, they decided to upgrade from ancient Ubuntu version to a more recent version. Since they don’t use aws properly, they treat servers as pets. So to upgrade Ubuntu, they actually upgraded Ubuntu on the instance instead of creating a new one. This led to grub failing and now they are troubleshooting how to mount disks etc.
All of this could easily be avoided by using the cloud properly.
That could be avoided by using on prem properly, too. People are very capable of making bad infrastructure whether on prem or cloud.
Yep. Virtualization is not a unique selling point of the cloud, despite the benefits of it seeming to be one of the largest selling points.
I used to work on an on premise object storage system before, where we required double digits of “nines” availability. High availability is not rocket science. Most scenarios are covered by having 2 or 3 machines.
I’d also wager that using the cloud properly is a different skillset than properly managing or upgrading a Linux system, not necessarily a cheaper or better one from a company point of view.
where we required double digits of “nines” availability
Do you mean 99% or 99.99999999%? Because 99.99999999% is absurd. Even Google doesn’t go near that for internal targets. That’s 1/3 of a second per year of downtime. If a network hiccup causes 30s of downtime, you’ve blown through a century of error budget. If you’re talking durability, that’s another matter, but availability?
For ten-nines availability to make any sense, any dependent system would also have to have ten nines availability, and any calling system would have to have close to ten nines availability or it’s not worth ten nines on the called system.
If the traffic ever goes over TCP/IP, not even if it ever goes over the public internet, if it ever goes over Ethernet wires, ten nines sounds like overkill. Maybe if it stays within a mainframe computer, but you’d have to carefully audit that mainframe to ensure that every component involved also has approx ten nines.
If you mean 2 nines availability, that’s not high availability at all. That’s nearly 4 days of downtime a year. That’s enough that you don’t necessarily need a standby system, you just need to be able to repair the main one within a few hours if it goes down.
Sorry, yes, that was durability. I got it mixed up in my head. Availability had lower targets.
But I stand by the gist of my argument - you can achieve a lot with a live/live system, or a 3 node system with a master election, or…
High availability doesn’t have to equate high cost or complexity, if you can take it into account when designing the system.
We have on prem and do all our upgrades by burn the OS and move the data, with the exception of the hypervisor OS (which has a pretty resilient bulk self upgrade built in, and we have a burn-the-OS plan documented for if they do crash). Even system file corruption of a random pet server? New VM and reattach the data disk. Need high availability? Throw F5 or HAProxy at the problem (assuming L7 protocol support).
Both cloud and on prem can work equally when done right. The most important part is to understand that both have different types of cost (human, machine, developer) and to make the right choice based your/your customer’s needs and any applicable laws or regulations about data locality. And yeah, sometimes one will be better for someone and not someone else.
Seven figures of cloud engineering can’t solve stupid, but neither can seven figures of datacenter. This isn’t some Sith/Jedi concept where you have hard definitions of dark and light or good and evil - though sometimes both will see each other as the enemy, and they are in a way competitors.
Yup, if your solution is not cloud agnostic you’ve fucked up.
Being cloud-agnostic also means additional cost/complexity.
Sometimes the only way to win the game is by not playing it.
I would argue that most cloud native services existed in their standalone forms way before public clouds made their own versions. For example there are loads of message queue systems that are just as easy to incorporate and are cloud agnostic, some of them are FOSS. Sure you can reinvent the wheel but in most cases something like RabbitMQ will work OK depending on the use case. Having cloud vendor lock in is where cost catches up with you. Complexity is arbitrary since there are ways to make anything overcomplicated.
RabbitMQ is more expensive on AWS than e.g. SNS/SQS. It’s not a coincidence, you’re trading lock-in for a cheaper price.
The increased complexity comes from the fact you will need some components which exist in either managed, but vendor lock-in form, or you need to spin them up / managed yourself.
I didn’t look at the username, so this came across as an underserved Orwell-referencing insult. Lol
Accusing him of being O’Brian or something.
One time I rewrote an Azure function to make it slightly more efficient. The cost savings were ~$50k /yr. Cloud services have their place but it is amazing how quickly the costs can spiral out of control.
Nah, unless you have a super popular app or suck down bandwidth I rarely find the costs of AWS exceed the labor of a good sysadmin.
Plus, if the cloud service goes down, you don’t need to worry about your service being out as well
old timer
10 years
😬
Didn’t you know, anyone that stays at a company more than 18 months is old…
Hey, I just hit 18 months, almost to the day! …but was at the previous job 23 years lol. Good to know I’m back to old timer status.
What the company likes about the old timer is that because he has been there for 10 years, he will likely be there for the next 10 years to support the complicated system he is creating now. If a younger team member creates something using a modern approach, there is the risk they will leave in a years time and no one knows how the system works.
No one knows how to use a well documented, publicly available service? No, I’d argue that no one knows how to use a private, internal only, custom solution.
That because you’re an engineer (I assume). The people signing off on these kinds of projects don’t know enough themselves, so they go to someone they trust (the old timers) to help them make the decision. The old timers don’t keep up with new tech, so we keep reinventing the wheel.
“keeping up with new tech” is often just re-inventing the wheel. If it isn’t broke, and can still be maintained, then why break it because you like the flavor of the week?
In this case, we’re talking about the OPs example of someone implementing a complex message passing architecture in Java instead of using an off the shelf solution. There are devs with 20+ years at the same company who don’t know the basics of networking/cloud, because they haven’t improved their technical skills much in those 20 years and instead focused on corporate politics. Those are the people who tend to gets asked for advice from upper management.
So he’ll rip an even bigger hole, when he is retiring because the company never bothered to get a new solution running. Then they get a hydra of legacy code that is poorly documented and probably using some old hacks based on even older forum posts, nowhere to be found again.
Oh god. I do a lot of PowerShell scripting at my place, and less than half my team is proficient in it. My co-workers who are almost never write comments in their scripts. Meanwhile, if it’s anything that will live longer than ~5 manual runs, I spemd more time on comments and documentation than scripting.
That effort is valued, but I’m shocked that my team isn’t more aware of the need for documentation. We literally experienced the “bus factor” situation a few years ago.
We have someone at my company who has been here for 30 years that gets to do whatever he wants basically - but what he builds is great. He doesn’t even have a BS degree or anything related, he started as a paralegal who wanted to make his life easier, and has built several iterations of the software that the entire company uses. He’s now my boss, running the data engineering and science department and I gotta say that he’s genuinely great. The only bad things I’ve run across that he’s built are things that he explicitly told management were meant to be just a quick bandaid fix to a problem to buy time for a full fledged solution… and they kept it as the full fledged solution. The stuff still works, it’s just awful to make updates or change to
There is nothing more permanent than a temporary solution.
That describes two out of the four jobs I’ve had in my career lol
It pays the bills though so what can I say, the tech I actually want to work with is what personal projects are for lol
That is probably what used to be required. Have you told him about the message queue?
old timer = being paid little
That doesn’t mean he or his fuck ups are free.
A bad architecture means slower development, more bugs, less reliability. All of which cost money.
Quick and dirty as they like to say
“Team restructuring” is so much fun, you never know what you’re going to get.
Your boss’s boss now reports to a slightly different VP? Everyone is getting fired? No way to know which it’s going to be, until the end of the meeting.
Bit let me first say that these are difficult times, and we’re proud of this team.
It’s like I’m there
Pizza?
If nobody is getting fired HR won’t be there
Best case: The harasser on your team has been fired
Worst case:
The harasser onyour team has been firedThe harasser has been promoted.
Spoiler: OP works for Blizzard.
Worster case: You’re the harasser and the police are on the way.
Rollercoaster: Your disgusting behavior stems from deep psychological issues and your arrest is what set the chains in motion to get mental help. You come out a better person.
And then you make everyone pay
I had sm vague inspiration for a meme inspired by this thought actually
Its a you get/i receive meme with a mentally ill homeless guy
We receive mild crime and aggression. They receive food and a temporary roof in jail.
With all the taxes spend on hostile architecture, emptying and clean frequently occupied places general police costs youd think it be cheaper and safer to just help humans in need whenever they need.
It is cheaper, but it suffers two problems:
-
it is work to implement without guaranteed success.
-
while maintained and working people will complain about the cost without recognizing the benefit. See: Brexit
The succes is that we are trying to be a responsible society making sure that people are treated in a human way with there most basic needs met instead of normalizing shitting the homeless.
I very much do agree on your second point though, building a society is hard and complicated and its near impossible for a citizen to overview all the dynamics to how their taxes benefit all or are being misused.
I do think we should try to do better though. Whats the point of organized society if not trying to improve standards and quality of life for all. Maybe it takes ubi for people to stop worrying about money and more about people, maybe we find an entirely different solution but we have to do something.
-
This was just a “team restructuring” but I was scanning the invite list to see if there was a name missing.
I was going to say who got restructured
That happened to me. I noticed a vague Monday morning meeting when I logged on. Checked with my team to see if they knew what it was about and no one knew. Supervisor was MIA on slack. Just before it starts we got a group text from him that essentially said, “what the fuck. I’m so sorry guys. I’m not allowed to speak or I’m immediately fired”
I checked the invite list and, sure enough… VP of department, VP of HR, my supervisor, and my small team. I instantly knew we were all fired.
Joined the meeting a few minutes early and it was just my teammates all wondering out loud what’s going on. They’re all pretty young. Couldn’t help but blurt out, “nice knowing yall…”
Supervisor texts me with “please don’t, we’ll grab a drink right after this”
The cool executives log and blah blah blah your team is getting shuttered thanks bye.
We did get drinks at 9:30 in the morning.
Oh and my supervisor quit a month later, right after he got the end of year bonus. I don’t blame him. Good dude. He helped a lot of the team secure other jobs in the industry within 3 months
That’s not just a supervisor that’s a proper leader.
We got reorganized last month. Scrapped almost all the projects we were working on and fired 1/4 of the workforce (mostly sales and support staff). On the plus side, I’m still employed and I’ve been able to use the last month to catch up on personal shit while the higher ups figure out what they want to spend money on next. On the down side, the new project I’m assigned to sucks and is never going to be successful. At least I don’t think it will.
But, as long as the paychecks keep rolling in…
As someone who has been there before, time to get that CV up to date, get any linkedin stuff sorted, and use that free time to start browsing the jobs market so that you’re ready.
I once didn’t take that advice and promptly got dumped on my backside without the last month’s pay because the whole thing had folded (I got paid a few months later through the liquidators, but that didn’t help get my rent paid when I needed it!)
First thing I did was update my resume. I “job hopped” a bit over the last few years, out of necessity, and I’ve only been at this job for a year. I’ll probably try to stick it out for a while. It’s also a brand new tech stack to me so as far as I’m concerned I’m getting paid to pad my resume.
Edit: also I work for an unprofitable division of a much larger profitable company so not as much risk of the entire place folding.
Sounds like you’re prepared then!
They won’t for much longer. Assigning you to a crappy project may just be a way to get you to quit. If you don’t, you may be in the next round of layoffs.
You may not want to work somewhere that has no direction anyway.
This sounds like something an Overwatch employee would say for the 3rd time.
How about a meeting an hour after our daily standup and it’s the CTO and CIO saying “ok is everyone on the call?”
I was telling myself “oh crap, the company is gonna shut down”
5 minutes later “So as of this very moment, everyone stop working. The company is officially closed as of this meeting.”
We were a start up essentially. Made it almost 9 years with ups and downs. This all happened at the end of July. At least got 3 month severance and insurance covered. I do start a new job Tuesday.
Well, you jinxed it. Can’t blame them.
I did, but for not what you think :D
I do start a new job Tuesday.
Congrats!
Is this what happens when you start a start-up instead of starting a business?
Next Friday is Hawaiian shirt day
So, you should ask yourself, with every decision you make “is this good for the company?”
Good point. Hawaiian shorts too. Full ensemble.
They doubled the mandatory flare. Again.
So how the daily go?
The HR rep is the real red flag. The other two could be there for exciting announcements too.
expired
The HR rep is the real red flag.
Yep. I knew our company’s death-march lawsuit of IBM had reached me when my boss immediately asks “Elaine, are you on the call?” I took the news stoically enough, so much that they asked whether I was okay, and something like the 137th layoff for this company is my only one so far.
For me it was a random 15 minute 1 on 1 with the director that appeared out of nowhere on my calendar.
I checked his calendar in outlook and saw like 8-10 15 minute meetings scheduled back to back. I immediately knew something was up. 5 minutes later my coworker called me almost sobbing saying he had been cut. I knew the deal. 15 minutes later another coworker called and said he had been cut too. Sure enough, my time came too.
It sucked but, I found another job before my last day and managed to pocket the 6 weeks of severance.
I had a very similar thing happen to me. The oddity was that I had just signed an offer letter with another company the week before and I gave my two weeks notice to my boss, but that message hadn’t traveled up the pipe yet. So my one-on-one with a director was basically
Director: “Half you team was let go, but your job is safe!” Me: “Cool. You know I’m leaving next week, right?” Director: awkward blank stares
I really wish I had been laid off. Saved someone else their job and I would have gladly taken that severance pay on the way out the door.
Haha, I had actually been wanting to leave, but I had a feeling that a layoff might happen and I wanted that severance. So, I spent all my spare time interview prepping and waiting. Sure enough, we were laid off and I was primed to interview immediately.
This exact thing happened to me. They were canceling our project. :(
Luckily none of us lost our jobs. We all just got assigned to different projects/teams.
Scrum meeting now a corporate war room.
If it’s after 4 on a Friday and your VPN has just kicked you out, start leveraging those external contacts HARD.
Ouch
