Story time

I am the CTO of a southern european governmental outfit with 300 employees of which 250 is developers. We make software for financial institutes within the EU. I have been here for over 10 years and have made it my pride achievment to convince the key tech players of the EU to move to GNU/Linux from Windows.

Fast forward (backwards) one year ago and about 1 million euros in external consultant cost to migrate our development environment from Windows to GNU/Linux. Everything is working fine and the developers are happy. Remember, this is a public governmental team that are payed for by the citizens tax money. The transition also included procurement of new hardware. I ended up buying about 300 Lenovo workstation and an additional 200 laptops for all of my developers. I consulted the Ubuntu Desktop certified hardware list and invested about 1.5 million euros in brand new computers. They all had new Intel i7 CPU's and Nvidia GPU's.

This monday, not a single developer could log in to their computer. We use Ubuntu 16.04 LTS. All workstations and laptops gets automatic updates (basically a apt-get update && apt-get upgrade) nightly to ensure security and feature updates. The login screen in Unity just popped right back up after a login. We started investigating this and Google told us that this is the famous "login loop". I contafted external Linux consultants that said that we just have to work from the console until Canonical fixes the faulty update to some driver. Since everything we do is based on Eclipse, this is not an option.

Again. Since monday, a team of about 250 developers are still not able to do anything in the office. Some have started working from home instead on their own computers. We bleed money everyday and 2.5 million euros do not seem like a good investment right now. Oh and I also lost my job.

Thank you Linux on the desktop.
Never. Ever. Again.

>enterprise projects
>ubuntu
nigger why didn't you use centos

Do you think we will trust you?

250 developers + (You) were unable to figure it out?

>apt-get update && apt-get upgrade) nightly


Fake.
Just undo it.

You should probably rewrite it in Rust.

literally fucking this
and also
>not having backups
>not saving all your work-related files to a shared file server so that everyone can access them if they just bring their laptops to work
>mfw 251 people haven't thought of just using their own computers at work while letting tech guys work on reverting desktops back to the previous update

> I am the CTO of a southern european governmental outfit
> I consulted the Ubuntu Desktop certified hardware list
> This monday, not a single developer could log in to their computer
> I contafted external Linux consultants
> I also lost my job.

What are you crying about? Clear case of severe incompetence.
So you consult linux experts after spent millions and after that shit broke. Based on a google search you spent millions on software and hardware?
If this is not fake, you totally deserved to lose your job.

OP here. We have backups of our site files. We use a paid git remote repository as our main storage depot for our source code. We *can* still work on our own laptops and from home. That is not the issue. EU directives about how and when you are allowed to work on security restricted software is quite hard.

>government males bank software
why

But you're trolling? It's not true that gov give responsibility to such incompetent people?

That is your opinion and I respect it. It took me about three years to form the transition project and see it through. I worked with some of the self-proclaimed best in the industry. When I say "I" in my story, most of it is me delegating responsibility to contractors, EU staff or inhouse staff.

You would be surprised if I told you how bad it really is and how extremely competent I am compared to most of my peers.

Okay, I'll believe that this thread isn't bait for a moment.
> They all had new Intel i7 CPU's and Nvidia GPU's.
Are you aware of Nvidia's infamously bad linux drivers? Did that come up during 3 years you devised that transition plan?

> 300 Lenovo workstation and an additional 200 laptops
Subtle thinkpad reference, still bait. Why lenovo? American corporations buy Dell, isn't that more reputable?

Yes. But Canonical ensured us that the open source driver is safe compared to the closed sourced official driver from Nvidia.

Lenovo because all the developers wanted Thinkpads and Thinkstations.

>baiting this hard

>canonical pushes a broken update
>this is somehow incompetence on the receiver's end
eat shit

cool blog bro

>Yes. But Canonical ensured us that the open source driver is safe compared to the closed sourced official driver from Nvidia.

Do the hardware you ordered come with optimus? It's easy to toggle the onboard intel graphics chip and nvidia GPU on Windows, but near impossible on Linux.

Using canonical is incompetence. I never had such an issue with my very stable Arch GNU/Linux distribution.

Thank you.

>nightly auto-upgrades
what the actual fuck ?
This has got to be bait. No one that got into a position of controlling 300 professionals could be this stupid.

Wow what a great evolution of the thinkpad general thread meme where you give out free laptops.

This is what Canonical recommended us to do, to be able to provide LTS and security. Who am I to go against the recommendation of the creator of the product?

Tech consultants tried to remove the GPU's and start from the onboard VGA. BIOS booted up fine but the graphical system did not even load. It just dropped us into a console.

Hmmm why not RedHat?

>install Lubuntu
>use this trash heap of an OS for a month, just glad to have a Unix-like development environment
>apply updates and, at some later date, reboot
>black screen and coil whine
>can't switch to any of the ttys
>just black, forever
>same in (recovery mode)
Very stable OS with well tested packages.

Well, in hindsight, that is a very good question. We took the bait and bought into it. I would not do it again.

>All workstations and laptops gets automatic updates (basically a apt-get update && apt-get upgrade) nightly to ensure security and feature updates.

>NOT DOING ROLLING SOFTWARE UPDATES
>IN TWENTY FUCKING SEVENTEEN

DO YOU THINK THOSE STORIES YOU HEAR ABOUT ENTERPRISES RUNNING OLD SOFTWARE IS BECAUSE OF INCOMPETENCE?

NEVER UPGRADE ANYTHING UNLESS

1. YOU ARE SURE YOU CAN ROLL IT BACK
2. YOU HAVE TESTED IT.

THIS SHIT COULD HAVE EASILY HAPPENED ON WINDOWS.

You should never install feature updates, only security ones. I’m sorry to hear about your situation but if microsoft had a similar bug, would you say “windows, never again!”. Probably not, you would get mad and then accept it and go about your business, like most companies do. That’s the issue with GNU/Linux, ONE bug and it’s “linux is broken!!!”, but windows can literally do anything and that’s okay, right?

Okay that's the final straw, it's a laptop.
You can't really "remove the GPU" unless you desolder the graphics chip from the motherboard.

Nice bait thread. I hope you get bad dreams tonight, a thread died because you posted this shit.

Are you for real? Ofcourse I was refering to opening up a desktop computer in a tower chassi. I did not even think to mention this because no one would missread that so badly.

I've never worked a project based office job in my life, and even I know that it's a bad idea to allow automatic updates on a system that cannot afford downtime.

>This is what Canonical recommended us to do,

In an enterprise situation you only have yourself to blame. As a CTO you should understand the buck stops at you. If Canoical didn't provide emergency support in this situation where you were down for 5 days and losing 2.5Euro/day, you got fucking fleeced and your legal team should be fired as well. Your team hasn't been able to work since *Friday*. You should have rolled back the updates or redeployed the entire infrastructure on *Tuesday* from a known working configuration.

In all honesty though, when it comes to developer hardware, Macbooks + OS X with VPN or BeyondCorp is better. Thats what Google uses, and are you going to argue that Google is doing it wrong?

>only sync "security updates"
>get a shitload of the faggot developers' userless churn anyway
>"hey why does this program look different now???"
>"hey why isn't this opening my files???"
>"hey this program's crashing now"
modern software development is SHIT

Alright, OP here. I got to go. Thanks some of you for your support and fuck off everyone that insulted my competence based on being the receiver of the brown end of the stick.

Then it makes 0 fucking sense why you'd buy laptops with Optimus chips when they're infamously bad and Nvidia's laptop drivers glitch out even on Windows.

Besides the fact that this story is completely fake, if your developers exclusively used eclipse, why would you have them switch to Linux?

See

>Things that never, ever happened: The OP
Anyone with half a brain cell would force employees to use either a Mac or a Windows machine and have their development environment be POSIX compatible. Your CI/CD should obviously be Linux based, preferably OpenSuse.

>ubuntu
That's what you get

>using ubuntu
kek
try red hat next time

Canonical's buggy junk instead of RHEL/SLES or CentOS? Possitively retarded, thanka for the chuckle.

>not a single person on staff said "hey why aren't we using RHEL instead of Ubuntu"
maybe you deserved to be fired

>not using debian net iso
>not installing xfce
>not using debian in general

no

>2017
>unironically using linux for development

fuck debian and the broken packages they roll out

FUCK

Number of serious companies who use Debian or Ubuntu for anything but client machines: 1
Number of serious companies that use RHEL or CentOS: 1000+

>All workstations and laptops gets automatic updates (basically a apt-get update && apt-get upgrade) nightly to ensure security and feature updates.
Nightly updates are _not_ opsec.
Download patches and recompile if you trust the patch author; otherwise, do it inhouse.

Stupid retard, security patches are 99% of your updates and should always be 99% of your updates, the rest are done per application ONLY IF NEEDED

Your other issue is Cannonical, so your update

This is like falling for the linux meme, but, like, a million times stronger. I have no reaction image for this.

>using shitbuntu
I'm a winfag and even I know you should have used debian stable, redhat or centos.

Honestly Niggerbuntu does more bad to Linux than good.

Works on my machine you fucking brainlet.

What could you expect from a distro that was created by nigger africans and unironically uses a nigger word for its name?

It was a mistake for the FSF to let it attain such market presence.

Isn't the FSF against it? AFAIK stallman hates it.

RMS is always right.

Linux Foundation has more control than FSF, they should have seriously stopped it at the first sign, as if ubuntu* emails not making a single upstream patch wasn't fucking indicative enough of its destructiveness to the community.

>created by nigger africans
proof?

Fake, gay and should've ran with OpenSUSE, with Btrfs you can roll back easily.

>All workstations and laptops gets automatic updates (basically a apt-get update && apt-get upgrade) nightly to ensure security and feature updates

I thought you were suppose to test updates on a few test computers to make sure nothing breaks and then role them out?

Mark Richard Shuttleworth (born 18 September 1973) is a South African entrepreneur who is the founder and CEO of Canonical Ltd., the company behind the development of the Linux-based Ubuntu operating system.[2]

But he is white.

Literally born and educated in Africa

But he is white, not a nigger (black).
Are you memeing me?

What's your point? He was born, raised and educated by niggers, also his fucking product has nigger in the name.

Dousing poop in bleach doesn't make it not-poop

Remember to use real OS THAT JUST WORKS YES ITS WINDOWS

Ah, you are memeing then.

He was raised and educated by boers, which are white people from dutch descent, his product has a nigger name because of new age cuck brainwash. Do you even know what the apartheid is? Are you one of those discord guys trying to meme niggers?

You sound like a discord fag for using ubuntu

I don't use shitbuntu, I'm just calling your bullshit off for your wewuzkangs bullshit.

The answer is to use something that is not broken; i.e. CentOS

Stop using nightly updates retard

What's wrong with Ubuntu?

It has a nigger name, it's botnet, it's reddit. Do you need anything else to drop it?

More importantly it's buggy as shit.

Near impossible? Literally all it takes is installing the nvidia propietary driver, going to the nvidia control panel and toggling a menu option. I'd read that it was nearly impossible but I decided to try to do it today to get more battery life and it literally took me less than a minute to switch to the Intel graphics. I's not as easy to switch between the two modes on the fly when you start a gpu intensive app like in windows since switching required logging out but it worked just fine.

I could have told you from my limited experience to not rely on linux it is a fucking meme. youre relying on people who don't develop for money but on the goodness of their hearts.

>using debshit or anything based on debshit
>ever

it's not even suitable for single user workstations, retard
they still use autistic shit like dpkg and apt/aptitude that makes pacman/portage look equally professional
and their package maintainers are constantly sperging over freedom for "ethical' reasons instead of legal reasons like normal person

use redhat like everyone else

>NEVER UPGRADE ANYTHING UNLESS
>2. YOU HAVE TESTED IT.
This right fucking here. I only work at a lowly community college, but we test out EVERY fucking update before putting it in the field. Every month we hook up our test-bed machine, update everything, then see if anything has broken. That's how we found out that one Win10 update broke some features in Word, instead of breaking Word for all of our users. That's how we found out the latest video drivers wouldn't work with dual monitors instead of dropping all of our users to a single monitor.

This is BASIC sys-admin shit. OP is LARPing or completely incompetent.

Or you can avoid this shit by just allowing only security patches.

>Having updates roll out on all machines every night
What is the point of this? If you want to keep machines up to date, you should do weekly security updates as they roll out or if a critical update is needed get it as soon as possible once stability is ensured.
And only do other updates when it's mission critical such as something affecting your functionality. And if you are going to roll out updates, you need to first run them on a test server/network/machine(s) to ensure everything is working as intended and things don't break left and right, AND OBVIOUSLY YOU HAVE BACKUPS AND A WAY TO ROLL BACK TO PREVIOUS VERSIONS.

The fact that you even suggest that a multi million dollar investment had a basic failure in organization like this makes me think that the people at the head of the organization of the project (whom I assume includes you) are grossly incompetent.
Blame Linux for your incompetence some more. No matter what people say here regarding whichever distribution you chose or should have used, this is a management flaw 100%.

What should I use that's stable and just as easy?
I use Ubuntu Mate and it's comfy.
What bot nets and glitchy?

Use debian or manjaro.

So for 10 years things have been running flawlessly save this one fuck up?

Just think of all the fuckups you avoided by not using Windows.

what do you suggest they do, write a patch on some god forsaken linux project? thats like a car garage telling its customers to fix their own engines, they drive it so they should know how to fix anything on it

I used Manjaro, it's awesome but an update killed my install and I'm too much of a brainlet to get my printer to work, otherwise I really like it.

I'll give Debian a shot. How about Fedora?

Fedora is fine too.

If you're looking at pure stability there's no beating RHEL/Cent/SLES and Slackware.

Everything else is pretty much a joke besides custom distros

Except that it is all software...

One thing I don't understand; why is Ubuntu so bad?
Fedora's from Red Hat (a major corporation).
Is it which company's the least crappy, or is Cananical just shady?

t. Happy Ubuntu Mate user

and? it still takes time, setup, and a learning curve before you can do anything in someone else's code

have you ever tried to open just any random unknown code repo and written a 100% good patch just like that? get real

I had the login loop with a Nvidia gpu. When I switched to an old amd gpu it worked right away

and by setup i mean tools and environment

Nice meme

>American corporations buy Dell
Not OP, I've got a Dell business laptop at work that's a few years old now, an E6540. The cooling is such a piece of shit that it can't really sustain any 100% load without throttling at all. Even at ~60% load it hits 85C. Pretty fucking shit for a business machine, I expected much better from a computer which is meant for work, it's not supposed to be some thin'n'light joke meant to browse Facebook.

Sure, but then my users bitch about "I have this feature on my computer at home, why don't we have it here?"
>BECAUSE WE DON'T NEED IT
Then I have to hear whining from teachers and staff and the fucking DEAN about this crap. So we do full updates for everything every month. Then we get to sit around and figure out what Win10 decided to install behind our backs, turn back on, or delete without permission. I still don't know why Win10 feels the need to uninstall RSAT every single time we do a major feature update.

Is this a new pasta?
if not then good job Rajesh, seems like you earned your Bing points

>I am the CTO of 300 employees of which 250 is developers.
>decide to move to linux.
>people loved linux.
>The transition also included procurement of new hardware.
>I bought 500 computers with nearly identical hardware.

>All workstations and laptops gets automatic updates and some update broke everything.
>I lost my job.

This is why you have an IT department.
We are only 30 developers in my company and we have an IT department.
Obviously this is not their only job, but part of their job is:
Make sure computers are updated, make sure computers are ordered, make sure if something breaks or doesn't work, people can get help.
Running a script that automatically changes the software on a system can go wrong, you should know that.
If this was the only way you could think of to keep everyone updated, why not have a single test computer that updates before you apply them to everyone?
Or even better, maintain your own mirror of all the software you use and have a one day delay so you can catch critical errors like this.

Now assuming none of this foresight were possible and you needed to "rescue" the situation once 500 computers are unable to show any graphics.
Partition the drive so everything they used before is now on a partition you mount to the home partition.

This should take you ~ 30 minutes which means it would take you ~ 30 days if you go and do this one by one on every system yourself 8 hours a day.
Now if you have any developers on the team, you might have one of them write a script to make this either faster or automatic.
Now if you also have more than one usb drive, and that would also help you.
HOW THE FUCK DID YOU SPEND A WEEK SITTING ON YOUR ASS?

Ubuntu has gone to shit. I installed it some time ago when I built a new machine and it broke after less than a few days of usage. Switched to Arch and it's been running to this very day.