I worked for a $2 billion company in the US. It was outsourcing software development to an Indian company. I designed a new command for managing a hardware device. The essence of my design was to gather information about configuration from a database. It was generic - it wasn't dependent on the hardware model. The team in India wanted to hard-code for each hardware model. More money to make if a new model were introduced. My design was eventually used, after significant pushback. The software was still in use 20 years later. I should point out that it isn't just outsourcing to other countries. Outsourcing to big domestic consulting firms regularly results in catastrophes.
I will give consulting firms one, small, bone. They're often brought in by companies which have some horribly outdated process or another and asked to deal with a problem for the least amount of money. When often the problem is no one wants to actually fix the real process problem. As an internal engineer, you're used to the problems and know what the real requirements are. Meanwhile, the consulting firm is given a set of crappy requirements that are very specific and are not forward thinking at all. Plus, sub-contractors are used to working with insane clients. For example, on one job the customer demanded we use their ticketing system. To create a ticket for a feature or defect, I had to fill out a PDF and E-mail it to someone. That person would then click the "new" button I wasn't allowed to and manually assign the issue a number. The ticketing system assigned a number and prominently displayed it, but they did not like it, so used their own number in all the paperwork.
So based on the difference between what thlse HCL employee resumes claimed vs reality the 737 Max flgiht crashes and deaths can be chalked up to lndia's penchant for being cheats and Iiars. Got it. Also stupid white peopIe being so trusting and thinking that aII cuItures are as good as their own...because they're aII equal (i.e. the same). They don't aII have Christendom. They don't aII have the enIightenment. They don't aII have the renaissance. They are not your equaIs.
As a remote Dev that lives in Africa and works for an American company I do agree actually, the big outsourcing firms have a problem, I'm an independent person I just apply independently for remote jobes. I do try to make sure I do thinks right, follow best practices, write things that are maintainable and scalable. But then again I would never work for $9 an hour so it is what it is
Agree. Some of the best work I've done was in collaboration with an on-site Subject Matter Expert (SME). If you look at the companies which are successful at using foreign labor, you'll see close working collaboration and at least one person who knows both the company and the foreign culture.
Basically they did, back to the Board of Directors, they make the choices and the CEO is a puppet winging his little head in a positive manner, it's not surprise they all seem stupid while their CV says another story.
How dare you? Your curelty it's beyond human! [Sarcasm starts] How come are they gonna pay teir 5th mega yate and the 10th mansion with a peasant salary?[Sarcasm ends]
Boeing used to be run by engineers. Now, it's run by financial types whose prioritize profits over everything else, including safety. Back in the 90's I wrote software for air-traffic control systems. Everything was designed, reviewed, and thoroughly tested, and had redundancy built into they system
Oh please stop this accountant bull crap. Dennis Muilenberg (former Boeing CEO, just in case you forgot) is an engineer, and look where Boeing went under his watch. It’s simply greedy a-holes that put greed above anything, not just because someone is a “beancounter” or not.
That is very true. Boeing CEO has been a marketing guy, economist and stuff like that. The times when CEO was a real engineer, are long gone. I made another comment here to tell how MCAS works fine, but the economy-people wanted to get 19000 USD for a software flag that could say "AoA disagree"-alert. Lets see if that comment will be visible, and how it will be at least shadow banned. And yes, there lies the problems of the Boeing Commercial Aircraft Company. The CEO is not an engineer, in a company that is 90% engineering and creating NEW stuff and items. Now Boeing has changed their CEO many times, and still, none of the new ones are engineers. As that is the root cause, including that the current new management board is bunch of children straight out from Harvard (etc), they have zero understanding of anything in airplanes and aviation. The good board members all resigned and stayed in Seattle area, and these new clowns are the 100% culprits of those really bad design principles. For economist, marketing etc person, it must be hard to understand that engineers need to be there, in a huge engineering company, but again, they still haven't tried that. And that is why Boeing will not recover, until they hire from their knowledgeable engineering ranks, and stop murdering whistle blowers.
Since they acquired the DC Douglas folk, they also inherited the toxic management culture which contributed to their current woes. The solution is really simple for Boeing - if you are an engineering / aeronautics company, make sure to put engineers at the helm, not the ones with cushy MBA pencil pushers with their greedy business models.
Companies are turning into financial business instead of serving the purpose of their particular existence. Product quality, customer satisfaction, and worker happiness are all going out the window for "maximize profit".
I remember when Siemens Medical laid off people with decades of experience to contract with an Indian company for dev ops. The new guys took 5 times as long to get things done and often had to reach out to others to figure out how to do their job tasks. Brass did this on purpose to jack up company stock before selling to Cerner to get the price up for stock holders. It's a shame that ethics, quality, and standards are all just things you can cut for capital.
Once you've written specs so detailed that a $9/hr developer can understand, it doesn't take much more time or effort to just code it yourself. I just don't understand this approach.
@@arthurdefreitaseprecht2648 I heard from the management many times that you can teach anyone to write code well, but it has never been true in my experiences. First of all, *most* of low-cost developers do not care much about learning. They think they can just follow instructions without real understanding of stated goals. Moreover, even the minority who care about learning something new often are very slow learners... Take chess as an example. Can you teach anyone to play chess? Sure, but can you teach anyone to play at the GM level, I am not so sure. Even it was possible, can you imagine how much hours of teaching would it require, especially if one is incapable to learn anything on their own?
So based on the difference between what thlse HCL employee resumes claimed vs reality the 737 Max flgiht crashes and deaths can be chalked up to lndia's penchant for being cheats and Iiars. Got it. Also stupid white peopIe being so trusting and thinking that aII cuItures are as good as their own...because they're aII equal (i.e. the same). They don't aII have Christendom. They don't aII have the enIightenment. They don't aII have the renaissance. They are not your equaIs.
A complicated, or large, application, especially if it is "real time", is itself too difficult to code by people whose focus is algorithms. The algorithms require algorithm experts, and the code requires code experts. So, systems engineers write the algorithms, and the test procedures, and the software people write the code.
Cheapskate accountants chopping software engineers is nothing new. Years ago, I worked for a risk management company in the UK and was sent over to Mumbai for a few months to beef up and get the team there up to speed. Whilst I was away, a top-notch fast and accurate software engineer (that I had recruited), worth 4 average engineers was ditched by the accountants because "he was expensive...". Output from the UK dropped noticeably and I quit not so long after, tired with trying to get cheap, inexperienced guys to work accurately. The stories I heard from other departments where the users of the software internally (before selling or upgrading to clients) would berate the engineers because of stupid bugs where the engineers would blithely reply, "Oh, yeah, we know about that, you do x as a work around" was driving us crazy, and I had enough of the fire-fighting and the constant battles with "cheap" engineers to bring them up to spec. The company eventually folded.
Yep. I too have been "downsized" as I was the most expensive. CEO seems to have forgotten that not six weeks before I pulled an all nighter (unpaid) to get enough systems working that we could participate (sucessfully) in an industry wide demo of our tech. Without that, our company would have been a laughingstock. But I was expensive.
So based on the difference between what thlse HCL employee resumes claimed vs reality the 737 Max flgiht crashes and deaths can be chalked up to lndia's penchant for being cheats and Iiars. Got it. Also stupid white peopIe being so trusting and thinking that aII cuItures are as good as their own...because they're aII equal (i.e. the same). They don't aII have Christendom. They don't aII have the enIightenment. They don't aII have the renaissance. They are not your equaIs.
When I heard that Boeing put a single monitor on the outside of the aircraft that if it malfunctioned, would drop the nose of the craft and that pilots would tire within fifteen seconds while trying to regain control, unless they properly reset the system after reading an update to a 500 page manual, I thought... who the hell doesn't have multiple points of redundancy? Who the hell doesn't train pilots on new features? Oh, they fired all of the software engineers whose job it is to think. I could make that determination in my company back when I was getting paid like $70k a year for a $50 million dollar company as a junior software engineer. Boeing, a $50 BILLION dollar company really has NO software engineers that are to that level of competency?
They 737Max was design to compete with the Airbus A320 Neo which uses the new Leap engines which are larger but uses less fuel. Fitting this to the 737 design made the aircraft have a different centre of gravity so MCAS was to make it behave like an old 737. So no retraining or recertification for the pilots were needed since they claimed it worked like the old one
@@tibbydudeza Yep, that was one of the big things. They deliberately didn't train the pilots! Since doing so would have made the 737 Max cost more to operate.
This is not the true. They did safety evaluation of the mcas with some requirements and they end up that risk is minimal so mcas may not have redundancy. And that was all fine. Then someone changed requirements to be more aggressive but there was no second evaluation that could end up with system redesign. And this was not a fault of 10$ / hour devs.
Exactly. If they'd had more experienced internal engineers, they'd have realised the flaws and raised hell about them. This is also what management wanted though. If you read between the lines, you can tell that "engineer says no" was seen as A PROBLEM at Boeing, not a strength. All management could see was "this guy who makes WAY less than me is causing me problems", and their solution was "hire a cheap contractor who WILL DO WHAT I DAMN WELL SAY". The basic problem is that management were making engineering decisions that they weren't qualified to make. They thought they had every right to make those decisions because... aren't they in charge? They didn't realise that reality gets in the way. IMO, this was all 100% a management problem. It's 100% about managers being UNQUALIFIED to manage the things they're managing. There should be no such thing as a manager, at every level, right up to the CEO, who doesn't have at least a decade of actual engineering experience (NOT in management) before being promoted to manage even one person. The "non-technical manager" is the modern scourge of all technical companies, and the source of an overwhelming number of problems.
Dude, working for defense and common fortune 500 companies, a software engineer who tries to discuss tasks with managers or innovate is now called "insubordinate" and "not fit for development teams". All management wants now is go go go, must make the sprint points and deadlines!!! No thinking, just code!
I agree with your assertion that chasing low cost/lower skill engineering talent can often be more costly in the long run, but I believe it is incorrect to blame software engineers for the issues with the Max. The 737 is a design from the 1960's. The original airframe was not designed for the large diameter, high bypass turbofan engines used by the Max to improve fuel efficiency. To provide adequate ground clearance, these bigger engines had to be installed further forward and upward on the wing than the engines on prior 737 models. This moved the center of gravity of the airplane, which in turn changed its performance characteristics at high angles of attack. Boeing needed the airplane to fly just like previous models of 737s so their airline customers could avoid costly aircrew retraining. The MCAS system was added to help pilots recover from nose-high attitudes and make the airplane appear to fly just like earlier 737 models. Unfortunately, the MCAS system was only fed by a single angle of attack sensor. If that sensor malfunctioned, MCAS would aggressively push the nose down with more force than the pilots could overcome. MCAS was a band aid created to squeeze more life out of an outdated airframe. It enabled Boeing to offer a product to compete with the Airbus neo series without the massive expense of developing an entirely new airframe. The entire idea was flawed. The decision to control MCAS from a single, non-redundant sensor was crazy, and the failure to inform or provide training about it to pilots was inexcusable. These actions caused the Max crashes, not the low cost software engineers. The MCAS software performed as designed. It was just a horrible design.
Agree, fundamentally a complete failure of systems engineering, most likely on-shore. No one there put together the big picture in understanding of MCAS operation, failure modes, and safety analysis. It doesn’t help though when the business does so many things to harm the engineering culture and technical capabilities.
From what I gather, there was nothing wrong with the software, the problem that a single sensor was used, which was Boeing's decision. I work in the oil & gas industry and all our safety systems use 3 sensors and 2 out of 3 voting on what signal to use. Additionally, measuring devices are able to indicate when they have a fault. Again, Boeing's choice not to do this. Boeing made the AOA disagree alert an optional feature and it looks like they never considered what would happen if there is a fault on the sensor, which by all accounts is quite regular.
I did some work for a Boeing division (off and on) over the years. When all the stuff going on with the Max happened and the MCAS was identified as the main culprit I hear from a lot of engineers that the real problem was as you suggested with the placement of the engines causing a "hack" to be implemented with the MCAS. Boeing got caught with their pants down with the 737 vs. Airbus A320 Neo. They couldn't take the time to redesign the airframe and landing gear area to gain more clearance for the bigger fuel efficient engines. Basically they would have to go through flight testing and certification again, which would have costed them 5-8 years of work, to be able to compete with the Neo. It would have seriously crippled them in the market as the 737 was their most popular plane.
Yes, But, it was not the retraining but the very costly type specification which prevents changing major parts - like the distance from the wings to the ground of this type. A new type specification costs billions.
@@smmasongt Now you're just blaming the engineers 😆 Their concern is not cutting corners with bad design to compete financially with Airbus, that's on the executives. My guess is most of the engineers looking at the plans thought "WTF are you doing?!" In fact, many of them complained to management and were ignored or told it would be fine. I mean, technically it would work, it was just precarious.
One more trend in Indian consulting firms is that they would have one experienced dev and rest junior devs. Where you would end up with just garbage code due to tight deadlines, experienced devs burning out and quitting, also junior devs don’t get mentored and grow.
Sorry, but it wasn't a software problem. It was a system engineering problem. A sensor with SIL4 (Safety Integrity Level) responsibility (highest safety level) was electrically connected like a sensor without safety responsibility. This was not a software error but a system error.
My understanding is that the aircraft had 2 AoA sensors, but they made the decision to only take input from ONE in the MCAS. It IS a system error but potentially one that could have been spotted and flagged had the devs had more domain knowledge.
This sort of poor management is a problem at Boeing, and MANY other companies and organizations. The senior management no longer has any real world engineering, production, leadership, and basic business skills or experience. They listen to marketing/sales first, cost accountants next, and finally quarterly stock market returns. Senior managements are the winners of decades of internal power politics, in some companies a low grade war. This is all a long way from building great airplanes.
Ugh! This is depressing. I'm already ousted because of my age and it just frustrates me to see it the whole industry trending downward. I wasted money getting a degree.
Indeed the reason of outsourcing now is very simple. CEO and C-level cut cost, get bonuses and leave with gold parachute. Company can die after after that, but it is no interesting for C-level anymore, they don't try to build something, just max money for minimum time. And even after failure in one company, they will be willingly hired in the next one.
Couldn't agree more with this. Also telling that there have been multiple studies that have failed to find a statistically significant relationship between CEOs who move between companies and the performance of those companies, yet the companies feel they have to pay ridiculous money to get the "Best" CEO.
I am a retired software engineer who worked in software for my entire adult working life. I was appalled when I heard about the design defects of the MCAS software in the 737 MAX, and how the people at the top of both Boeing and the FAA seemed oblivious to the problem. The Crux of the problem was that the software would ignore incompatible readings from the AoA sensors, and make automatic trim adjustments using only one (possibly defective) sensor. That is an egregious design error. The only reasonable thing to do in response to disagreement of the AoA sensors is to alert the pilots and refrain from. aking any automatic flight control adjjstments depending on those sensors. Outsourcing software explains it. You have the programming being done by people who are not specialists in Boeing flight control software, and you may have poor communication between the Boeing engineers who (presumably) know how things should work, and the people doing the programming, who may know very little about airplanes.
As far as I know the MAX crashes caused by a faulty system had nothing to do with poor coding. It was the design of the whole system that was faulty from the beginning, and true cost cutting that led to that system being concieved and implemented was the idea that it would enable pilots of the existing -800 series to just switch over to the MAX without having to do any additional type rating.
I worked with the HCL guys in India while in the finance sector, back in the early-ish days of HCL. They were very sharp guys all, even the cheap ones. The issues that did come up, stemned from poor communication from our end as both sides had major technical, social, cultural blind spots towards another, many of them unmentionable. Looking back it was shocking the result was as good as it was despite those issues.
One thing to add, the actual fresh graduate engineers working on this don't even make 9$ an hour. Their salary is 400k-500k INR a year, that is 4,700$ a year.
This is a little misleading. The code didn't have any problems , it was doing exactly what it was meant to do: take control of the plane when a single sensor failed, and tricked the avionics into activating MCAS. This was designed by the engineers at a higher level, point the nose down over and over if one sensor output data that said the plane's nose was pointed up.
They didn't outsource the MCAS, that was all Boeing genius. I'm unclear exactly what they did outsource to India, but they did outsource some 737 Max code.
True that. The MCAS software checks the ONE active AoA sensor (Angle of Attack, eq how high the nose is compared to the airflow around it). If the AoA gets too funky, the MCAS will start pushing the nose down by trimming the horizontal stabilizer to push the nose down. Everything is fine and dandy, as long as the ONE SELECTED AoA sensor is working well. Those senors are of course outside, they have air/sand/garbage flying on them all the time, so they get quite easily broken. Now the broken AoA sensor gives out some funky AoA readings. There is the problem. There is another AoA sensor on the other side of the nose/fuselage. That ought to be ok, but Boeing Chicago management clows (all economics people, kids, straight from universities, knowing nothing at all), decided that checking on AoA sensors, is not needed, and it is a "optional advanced AoA software package, for 19000 USD per plane". That software would then tell the the pilots (and MCAS) that "the left AoA and right AoA are saying very different results", eg AoA DISAGREE. Only Delta ever bought that software, but other airline pilots (not Lion/Ethipioan) also knew what is a runaway-trim and how to fix it. Those two airlines did not know and did not care. Pilots had to select left or right AoA sensor to be used, but as that optional software was not installed, they could not have "Auto, using both". Now when the active AoA sensor fails, the MCAS can freak out, and start helping the pilots by pushing the nose down. Again: Stop the runaway trim by pressing Auto-Trim: CUTOUT right next to the flap lever) AND/OR Stop the running trim wheel by hand (can be done easily). Continue the flight to the destination as normal, with of course now manual trimming. Easy. But again, not easy when pilots are such bad bus drivers, from the lowest tree branch, and Boeing board thinks that all of this is "act of god, nobody knows why the MCAS freaks out...." . Plus of course a lot of corruption, blackmail and threats, and murdering of the whistle blowers, which we are all sure will produce fine Boeing aircrafts in the future..... sick and sad.
$9 per hour for a software developer is bad enough. But there is a bigger question: what about testing the software? Did Boeing test this software to see if it complied with the safety assessments or FAA regulations? Or did they recycle the tests from older designs that weren't as software dependent?
The bigger thing that a lot of airplane safety guys were screaming about right after the investigation came out was lack of redundancy. The software depended on a single sensor for its input data, and when that sensor failed, the software that relied on it went haywire. As you will hear time and time again in _any_ class related to industrial safety, the minimum is two sources for safety (when a system is allowed to fail completely) and three sources for fault tolerance (when it is not allowed to fail completely, usually for safety reasons). The software actually performed perfectly -- assuming the input data wasn't wrong and that the pilot would never need to disengage it quickly and intuitively (the process to disengage it was quite involved and rarely mentioned in training material, if at all).
There’s no magic in safety assessments or FAA regulations. They’re not sufficient to make a good design. If you have engineers who don’t know what they’re doing it’s just a worthless stack of paper.
@@mage3690 that was the first and main issue one of my neighbours mentioned as soon as details about the crashes had been made public. He was an engineer with Airbus. He said that such a design just wouldn't have received approval by European authorities which is why Airbus wouldn't even think of doing something like that. Failed approval procedures are just way too expensive.
Great video. Outsourcing isn't just a recent trend. It's been going on for decades in tech. I first noticed it in the 90's. It was and is getting more prevalent. After that I never worked for a company that didn't depend on cheap contractors. It was always a mess trying to get good or even correct code - cheap labor is cheap labor. Combine that with time lag, language issues and lack of product knowledge or ownership and you get what you get. :)
I have worked for an outsourcing company. I fondly recall them designing and building an overengineered system to achieve something that could have been done by a few tweaks of existing code. A common theme with these companies was a lack of understanding regarding testing.
I spent 25 years in airline ops for various airlines. Over the years technology has pretty much taken away the common sense of engineers and other professionals. To give some perspective. I was involved with weight and balance since the early 80s. We were taught the theory as well as how to correct issues if the computer system crashed (very often in those days). We were also taught on how to use various airlines computer systems to perform our duties. Those days, we had to learn the computer entries and type them in. These days, it is just point, click, drag, drop. or even tell the system to "auto load". Sure, there are error checks, but if you do not have the basic knowledge of the principles, you are pretty much screwed if the error persists or if the system goes down. I use to work with avionics tech folks as well. These days, there is no thaught process involved. You just tell the "AI" what you need and poof - out comes aa sh*t load of spagetti code. Then you outsource because of "costs". Penny wise pound foolish. I digress.
The C-Suites are universally staffed by managers that rarely understand the tech involved. In nearly everything. They are focused on the quarterly results, their personal stock options, and their personal advancement path. Boeing's previous merger caused many experience Boeing engineers to leave the company. The new management structure gave the world the 737-MAX. Oddly enough, one does wonder what those stock options are worth today.
When big trees fall, there is more light for saplings to grow. And big trees fall for a multitude of reasons acting in concert. Boeing looks a bit creaky at the moment and the winds of change are blowing hard.
Problem, Boeing is "too big to fail." It's more like this mutant tree grew to giant size and gobbled up or killed most of the forest. Yet instead of splitting it up, we're actually allowing it to continue to subsume more trees. Alternate example, we built a neighborhood around the monster trees. The only space for saplings to take root is the empty space left by houses the tree destroys when it falls down. Turns out gutting anti-trust, and allowing monopolies to flourish for 30+ years has consequences. Who would have guessed?
This is probably why so many large IT projects go wrong. It is not that programmers in 3rd world programmers are not smart, it is that they are often managed by non-technical “managers” who are thousands of miles away while also not scoping out the project correctly. Failure is inevitable.
From the little I learned about the system, the main issues had nothing whatsoever to do with the code. They were design problems that were based on decisions almost certainly NOT made by programmers or 'overseas' personnel. The two main issues I am aware of were that they deliberately tried to hide the system from oversight and made some decisions for that reason and there were insufficient safeguards if a sensor was giving incorrect readings. The aircraft has two sensors and even I, as someone who has not worked on aircraft software (although I have built a customer reservation system for a chartered aircraft), I know that if you have two sensors then you should read both sensors and if there is a discrepancy you should do something about it whether that be figuring out which is correct and relying on that or informing the pilots that there is a problem etc. But instead they simply use only one sensor and ignore the other making the whole redundancy aspect totally useless. That was NOT a programmers decision. That was a management decision.
Issue is: without the experience and maturity you DO get from well paid, market rate software developers who DO see the total design even though they are on code, you wont have people who state a simple and obvious intelligent point about redundancy, design and integration of the code theyvre written. MOST good software devs will absolutely just tell you when the glaring implementation fault is there. And yeah just make sure you get decent people working on an integrated solution in this scenario, its peoples lives. Boeing should be ashamed.
@@johns5558 In my own job I often point out glaring design flaws or things that should be done, but the reality is that in corporates that doesn't always result in action. The fault lies squarely with management in this case. If I was the programmer, I would have definitely suggested improvements to the system, but if management said no, just stick to the design or we get someone else then I would have stuck to the design and got paid. Outsourcing does have its problems as the employees feel less attached to the company as a whole and are less likely to 'do the right thing'. But lets be clear that Boeing outsources almost everything (and has suffered as a result) and that has nothing to do with whether or not the outsourced company was in India, Italy (recent case) or the US. I once worked for Vodacom (a telco in South Africa) and two thirds of the people at work were 'contractors' including myself. It worked, but it had its problems because there was no job security. Outsourcing has its pros and cons. A contractor is under higher pressure to impress than an employee, but they are less invested in the long term health of the company. I remember a recent case that was reported on where a bank fired a large number of employees because they had installed software that emulates moving the mouse and keyboard. They did this because management was monitoring them and evaluating them with software that monitors mouse and keyboard activity. The employees were fired when the right move would have been to fire the management. Management is hard and evaluating employee performance is hard and when you get it wrong you get the wrong results. Boeing values cost savings over safety and that is what they get.
This is a trend that started decades ago and made famous by Jack Welch as the CEO of GE. When an organization gets hyper-focused on the numbers it starts to get hollowed out with cost-efficiencies. Give it long enough and like GE, it loses its intrinsic value and becomes a hollow shell with significantly reduced value.
India consulting companies have some very good engineers, however not many. They tend to give you the good engineers at the start of contract then swap them for the lower quality engineers after a couple of months. I used a company that over 2000 developers, one of my team would find the error for them, tell them the change the 1 PM, 1 manager and 3 engineers would make the change. When I found out that my engineer was basically doing all the work, they where gone. We weren't outsourcing because of cost, just lack of specialist skills (particular telecoms protocol) their good engineers where probably one of the best in the world for that skill. however they were in project for 3 months then gone, to another project.
Whether it’s India or the US, it’s all the same. There are a lot of bad engineers. If you want to be successful you have to be able to identify the good ones and then retain them. Normally with good work and good pay.
One issue with outsourcing to these contracting companies is that they benefit from delays and rework so there is no incentive to be efficient or provide a quality product first go around.
MCAS was a fairly robust system done a while ago for the military. Digital augmentation of airframe characteristics isn't a new thing in military aviation with fly-by-wire inherently unstable fighters, planes having asymmetric load-outs when they fire externally mounted missiles or receive battle damage etc. The issue was implementing it on the MAX while making the angle of attack sensor non-redundant.
Right, but if you disconnect the software developers from the whole implementation then there's the issue. that they are paid $9.00 an hour only tells you that there's so little experience that not only would they not know about redundancy in the total system but at $9.00 they wont speak up about it either. There are few good ( / well paid ) software developers who won't at some point tell you if there is a fundamental design flaw even though they arent the architect or designer. And they will know. This is the issue and this is Boeings penny pinching fault.
@@johns5558That’s not how software development for safety-certified systems (which includes avionics) works. The coders are the least important part of the process, and they have no influence whatsoever on the design of the software. That is done by different software engineers whose job it is to write specifications.
@@tookitogo , John, you're both almost there, but are missing the particular MCAS context. The *entire point* for that system was to be non-flight-critical & non-redundant, as per Boeing's own requirements - not $9/hr programmers' inexperience or indifference or the savings to be had in hiring them. We're talking about many orders of magnitude more money, elsewhere in the business, as the driver. Please allow me to explain. Let me back-track a little. The 737 sits quite low to the ground, which had played a big role in its success over the decades, as that not only reduced the landing gear weight & complexity & allowed it to be tucked inside the wing instead of the body, it significantly reduced the size and complexity of all the requited airport machinery for service / loading & unloading / de-icing etc. Personnel, procedures, didn't need to be heights-rated. A devastating competitive advantage. As material science and fabrication advances allowed greater engine fuel economy through larger turboprop bypass ratios thanks to giant front fan blades becoming viable, the engine diameters naturally grew. Eventually to a point when compelling new engines would fit under Airbus A380's wing but not under the 737's wing. Boeing engineers pulled a design rabbit out of a hat, flattening engine shroud on the bottom, shortening its pylon and putting the fattest part of the engine way in front of the wing (taking up the wing thickness' worth of space). With all these measures combined, the new engines fit on the old airframe, which ( *this bit is key* ) qualified the 737MAX, as far as the FAA et al were concerned, for the simplified deployment, without extensive pilots' retraining incl simulator hours on what woukd otherwise have been a whole new type. This new type retraining requirement is worth many millions *per airline* in lost profits and risks - that manufacturers typically pony up. It was *key* that the 737MAX not require that retraining, because the A320NEO didn't, b/c even if the Boeing had paid up for the sims etc it'd still take pilots offline, giving airbus oprators a market share in over their brethen running boeings. With me so far? Good. There was only one small issue with Boeing's engineering miracle: the altered engine placement, in certain very small areas of a flight envelope, made the plane suddenly pull up. Boeing has had just the solution, digital flight control augmentation - a little autopilot of sorts that specializes in patching up / normalizing such small irregularities, making the plane behave seamlessly from a pilot's perspective. Many advanced mil airframes (fighters designed for an inherently unstable flight to maximize maneuverability) aren't even flyable by hand without such a system. The only pickle was, while just about everything of the sort is doubly or triply redundant on a military plane, adding a redundant system on a commercial jet automatically flags it as potentially flight safety critical in the FAA review (large chunks of which were delegated by the FAA to Boeing to self-certify). That would in turn require pilot training, defeating the purpose of the entire effort & putting Boeing back to square one. Boeing's plan - as per its lesdership and bean counters NOT programmers - was to sail under the radar of this review, by specifically NOT having MCAS redundant. The rest is history: the gamble didn't pan out, and whistleblowers stripped any shread of plausible deniability, which is how I'm able to explain to you all this without any rational fear.
The problem isn't really with "cheap software engineers" but with the project management at the other end. In our department we had a similar issue: digitalization projects in development hell for years on end. They eventually realized that it was a structural problem and hired a few software engineers to fix that mess. It was painful but now there are no projects in limbo any longer.
From a software point of view this video is spot on, though I happen to personally know that Boeing has far far far far more troubles than software. If it’s Boeing, I’m not going. MCAS has a shortage of air speed sensors such that even if the software was written by a genius it couldn’t safely do its job.
I interviewed a lady that worked in a company that outsourced software development to an Indian company. She told me it took more time to explain what was to be done than do it herself. However, management thought otherwise. I personally dealt with three distinct cases where the outsourced Indian/Pakistani developers did not deliver: a restaurant application, a car spare part application and a ERP system. Spaghetti code full of bugs. Do I imply Indians are not capable? Certainly not. It's just that those that are good developers are not cheaper.
I have worked in developement process of safety critical software components and I too noticed that management is blindly relying on the arbitrary safety standards and documentation which are all completely pointless when carried out by bad programmers
Blaming to outsourced partner is easy, but as a aircraft giant, I'm sure they won't have outsourced their critical components for $9 / hour, it is more than just cheap labour
Don’t be too shocked by $9 outsourcing because it will soon go down to $0 because we won’t need as many programmers. When product developers, like us in Silicon Valley, look at corporate IT (which is what you are referring to) we see low skilled, crusted but highly egoistic people who should be quickly replaced. Their business users hate them, think they are impediments to quick growth and are the biggest bottlenecks for any corporate change. So we’ll get rid of them with generative AI. So don’t be too shocked and I would suggest that this be the topic of your next video.
[upd] nvm, author talks about that but 2 minutes later 3:40 here is exactly part of the article which says that Boeing outsourced "flight display"(which is visualization system) but not MCAS(which is to blame in 2 plane crashes)
I am not sure why you would do something like this. This should be done by local in-house developers. I guess when you make accountants and MBAs head of engineering companies. Even with non-engineering software the quality is so low and has to go through so many cycles of correction and bug fix.
Actually, I do not believe that $9/hour engineers were working on the core of the boeing system. It is not how it works in this sector. Maybe there were working on Boeing webpage but not on the core system ;).
It's first a communication problem. A lot of companies do outsource. It's not really cheaper as you still need project management, testers, developers, etc. on both sides. Although you have someone else work on the software you are still responsible for software quality, operations and integration (i.e. testing). There is nothing new with this. It's an option for several decades now. The costs shift within the balance sheet. It can cause friction due to contractual issues (e.g. implementing requirements as formulated, ignoring errors in requirements or vague requirements). You earn more for fixing error caused by your customer instead of correcting issues earlier.
That pause around 1:52 where you hear crickets is both brilliant composition and devastatingly depressing. I think it is worth noting that the reason that the 737 Max needed MCAS was because Boeing retrofitted the 737 design rather than designing a new aircraft from scratch.
One additional issue is that when outsourcing work literally half-way around the world (e.g. USA to India) makes it very slow and awkward to communicate. Mid-day in one location is midnight in the other and with weekends being skewed you end up with an effective three day work week.
I work on ed tech software and i sometimes wake at night worrying about bugs or some other thing I might have overlooked. If I worked on planes, I would just live in a constant state of worry and testing
While I agree that outsourcing is generally bad, having seen dozens of failed outsourced projects. I think the reference to Intel outsourcing to TSMC is a different case. This is fabrication of processor chips, not software.
I have experience in similar safety oriented industry and there are some glaring points to consider: 1) There are some stringent coding standards (coding standards) which should be maintained. Doing so ensures easy debugging and less possibility of errors. 2) The Verification and Validation team should be able easily pick up flaws after thorough testing using simulators and checking project data/parameters. Seems like coding standards were ignored and poor validation was done. This means that the problem is systemic and completely overlooks/skips all systems of checks and balances.
I'm a controls Software Engineer and was born and raised in SA. I have immigrated to get a decent rate. Man, I do miss SA a lot!! Oddly enough, my situation reminds me of that Mexican fisherman Parable. Sidenote. Great video. Keep up the good content.
Part of the problem is the shift towards a "designed for obsolescence" culture. This trend is so pervasive that it has begun to take root in industries where a more traditional permanently employed loyalist workforce is the more appropriate approach. If Boeing made smart phones, then having a more outsourced approach would probably be the right choice, but they are not in that sector, and they would have to find ways of staying in business while employing a more traditional workforce that stays with the company until retirement, and transfers nuanced intricacies of institutional knowledge down the hierarchy to the junior employees. There are tradeoffs; You could fly for cheaper or you could fly safer.
It's called a tombstone reaction. When people die or the project eats dirt, something starts happening. But not until. Been hearing this topic over & over for many years. Your particular rant is a good one though. Cheers.
I agree there were rookie mistakes made with Boeing's MCAS. Boeing fired many of its veteran flight control engineers a year earlier, saying that their flight controls were already proven. The problems was MCAS was new. Boeing decided to go with one MCAS sensor. One way the sensor can fail is via a bird strike (at least one of the Boeing crashes, is tied to a bird strike on that sensor). An experienced engineer, in the U.S. Knowing full well they can be sued for any failure or omission during the design process, would have pointed out the necessity to install a redundant sensor. And, to test the system against a variety of mechanical failures. And do their best to detect those kinds of failure also in the software (which really would require some push back from the engineers). But, a 3rd party company, with inexperienced engineers, it might never occur to them, this is a bad design (one MCAS sensor). And, if an engineer did realize this, would they be stifled by inexperienced managers that they report this problem to? The retail cost of the 2nd sensor was only about 120k installed (that's what Boeing charged). The actual device was a tiny fraction of that cost. So there is blame to be had by Boeing sales and design at the U.S. level for not thinking about this. But then, if they just fired all of their experienced U.S. engineers, the U.S. sales and design team might have only the bean counters left, who are trying to make sales, anyway possible. Boeing sacked their U.S. engineers, they outsourced the job to a 3rd party, that then went with an inexperienced remote company. That remote company will never pay the real cost for this disaster (which is billions), no one in the remote team can be jailed for gross negligence, which this was. In short, they outsourced because "No Accountability" is way cheaper than "Responsibility". And this isn't a communication problem, because there wasn't anyone at Boeing HQ failing to communicate the requirements. It was a failure of Boeing management to recognize the necessity to keep experienced flight control engineers on payroll for a few more years. Also, there is responsibility at Rockwell and the 3rd party company responsible for developing the software. But, you'll never get them to admit (except in court, which will never happen with the Indian engineers). That they either should have realized the implications of having just one sensor and/or software too inferior to sense it was a mechanical anomaly (which is possible to do, although difficult, and requires testing and more sophisticated software). A few thousand dollars saved, and priceless lives lost. And Boeing deserves it's un Air Worthy reputation, until they fix the organizational problem from the top down.
As a junior, I also thank you for the average wages stats. I’ve been working as a sort of emergency patcher to fix stuff done by outsourced contractants, and always been compared to their prices then told that I’m a junior and can’t ask as much as $40 per hour. Just that my implementations work almost always at first try. I’m sort of angry now lol
I have worked with some great software engineers in India ... but the vast majority of them end up in the US. The best engineers don't stay with contracting companies. This is also a symptom of an even greater issue that is not specific to India or contracting companies. For years, we've pushed people to enter this field with little regard to aptitude. I taught myself to write my first program on my TRS-80 in 1982 using the manual that came with the computer and books from the public library. Started with BASIC (of course), moved on to 6809 assembly and Lattic C on a PC XT in high school. I'm now a senior director of software architecture ... and still write software daily. Why? It's just something (frankly the only thing) I seem to have a natural aptitude for as well as a genuine interest. The same could be said for most any technical field. For some reason, we've gone so far over the top in encouraging everyone to "learn to code". CODING is one thing ... being a software ENGINEER is quite another. I can (and have) installed my own crown molding - but I am certainly NOT a finish carpenter.
Not true, MCAS software did not fail, the active AoA sensor failed to show correct data, and thus MCAS software thought that "ooo, we are going belly first nose up, lets trim the nose down".. The horizonal trim is automatic, but also visible with big wheels in flight deck. When a runaway trim happens, the trim wheel can be stopped by disabling Auto-Trim AND/OR stopping the trim wheel by hand. Easy. The auto-trim-cutout button is right after the flap levers, middle console. Really, easy, been there since 1960s original 37 designs, and still at the same place. So when the incompetent pilots are more focused on being a glorified bus driver, they do not understand even the basics. Yes, Indonesian Lion and Ethiopian are not at all competent, and they have never been safety focused, believe you me. The MCAS operated just fine, but Boeing new management board at Chicago is the real and only culprit. They had zero experienced people from the Seattle board, and Boeing owners (Wall Street and there a certain group...) just said "we can get a much better management board, directly from universities (Harward/MIT), and that is it. Those very new board people knew nothing about aviation, airplanes, how to design, sell, use, operate, own or anything. The new and completely incompetent Boeing Chicago board just tried to see 37M features as a "DLC", since that is what those kids learnt at their schools, how to make games with paid add-ons. This meant that one of the TWO AoA-sensors on the 37M, could break, and no information would be not told to the pilots. That option for the software to say "Aoa Disagree" was "additional paid service for 19000 USD/plane". That software is there, but for that software to be allowed to say that the one of the AoA-sensors are giving freaky results, you had to pay for it. In all other planes the "AoA Disagee" alert is built in, eg when the left AoA produces different values from right AoA. So, in the basic model, with the Boeing Chicago board idiocy, did not tell anybody that AoA sensor was busted. It did not tell that even to the MCAS software, as that would be 19000 USD optional product. Nobody bought that software, except Delta for their future planes. Manual AoA selector is there, and any normal pilot can see the Runaway-Auto-Trim, as it keeps a lot of noise and is visible spinning. So, just stopping the nose-down trim run even by hand, and disabling the auto-trim completely, the plane is allowed to fly to the destination normally. But again, that means the pilots know at least the very basics of aviation, a thing that Lion very secondary airliner and Ethiopian young and minimally qualified 37 pilots just did not want to be bothered with. The horrible worstness of the pilots were just underlined in the fact that they have runaway-nose-down-trim, and those clows try to compensate it by pulling their stick. The whole horizonal stabilizer still moves to trim, and these clows are using their tiny elevators to combat thw whole stabilizer. Well done boys, and all of that when they see and hear the trim wheel spinning nose down, on and on and on and on. Yupp, that is not how airplanes are flown, and that is why these deadly cases happened to those airlines.
The MCAS software was poorly designed, and contained the means to crash the plane. This has been thoroughly explained in the aftermath of the disasters, except for one thing. I have never understood how it came about that Boeing would give the computer priority over the pilots. It was contrary to previous Boeing design philosophy. This video may help clarify what went wrong.
02:18. Oh yes. I was raised a software engineer. Am now a projects manager. Management doesn't understand software. Full stop. Make box do thing must be easy right?
$9/hr or monthly, $9 x 8 hr x 21 days = $1512 = ₹127k (as salaries are generally paid monthly in India) is actually decent for a software engineer in India. That would likely put someone in the top 10% income earners of the country. The problem is that they outsourced to another company, HCL which has no context of Boeing's internal code or standards. HCL (and other outsourcing companies) pay like $2 to $5/hr for fresh graduates. Often, freshers join these companies for some industrial experience in their CV and try to switch to other (often product based) companies. If Boeing wanted to save costs they could've directly employed decent Indian software engineers at like $10/hr and had them trained by experienced Boeing engineers. But, Boeing is a cheapskate and I remember they were offering around ₹800k-₹900k per year or equivalent to $5/hr for fresh Mechanical/Aerospace graduates in India.
Not a good start that you get basic terminology wrong: an *airline* is a company that operates aircraft to carry out flights. Boeing is not an airline, it is an *aircraft manufacturer.* Second, the article (from Bloomberg, a company with a poor track record in tech reporting) talks about software for the flight displays, but the flight displays aren’t what caused the MAX crashes. Third, the widely reported software “responsible” for the crashes, MCAS, was “responsible” in the sense that it carried out the commands, and that it did not have an adequate mechanism to back off when overridden by a pilot. However, the root cause was that the MCAS system, by default, had only one sensor providing it input, flagrantly violating the principle of multiple redundancy that is expected in the aviation industry. That was a hardware limitation, presumably imposed by accountants. (Multiple sensors were available as an optional upgrade.)
Just some Indian context to keep in mind : India is also the country that runs ISRO, with its cheap, well planned and successful Moon and Mars missions, as well as highly successful commercial space launch industry. And ISRO is cheap, and a lot of ISRO engineers get paid at par with 10USD per hour, which still comes out to a lot of money here (= 10 x 84 x 22 x 8 ~= 150K INR / month) In my mind, outsourcing to a company without aerospace engineering expertise seems to be the actual issue. Practically half a dozen aerospace giants "outsource" to India, but they do so in the form of running their own branches in India. These engineers are paid anywhere between 200k to 1000k per month, which is 2500 - 12000 USD per month. They're the cream of the aerospace industry. HCL Technologies is not one of these. It is a body shop. I felt the need to say this only because a good friend works in a leading American aerospace company in Bengaluru and he is getting awarded patents and certificates in flight safety and aerospace communications systems. I would also like to add that cheap Indian developers are pathetic. I suffer them in office every day, being an Indian living in India.
I was at Rolls Royce when they were trialling outsourcing like this. Fortunately it wasn’t critical software at that point, and I hope from what I saw from this supplier it never was. I got fed up with them implementing a feature and taking weeks for something that was really simple. I ended up writing it over night and upsetting everyone in the process. I felt bad for their devs though, they were hours away from home and working insane hours a day
The even bigger problem than the software was tying it to a single sensor. The MCAS being poorly programmed is a thing that happens. Proper testing can sort that out. Tying the MCAS to just one sensor that can fail or be disconnected is a major £ūçkup in the design phase. On top of that, not releasing any technical data to the airlines that bought the aircraft and INSISTING there is absolutely nothing different with the 737 MAX, or the way it operates, when compared to the older 737s...well, that's literally criminal deception and endangerment of the public. MCAS could take over the pitch authority of the aircraft and pilots needes to be informed of it and trained how to troubleshoot it in emergency situations. Again, the biggest problem is tying this system that can take over flying the aircraft to just a single sensor, especially since the 737 MAX comes equipped with TWO angle of attack sensors. This is yet another example of professional business managers making engineering decisions, which always leads to disaster. Consider Fukushima Daichi! The management decided to stick the backup electrical supply into the basement of the nuclear power plant, since a major tsunami that could overtop the seawall and flood the basement, creating the conditions for it to blow up was considered unlikely and the managers could not justify spending the extra money to secure the emergency power supply on an event they estimated was very rare. Fukushima Daino, the twin nuclear power plant to Fukushima Daichi was just a few dozen miles down the coast from it and got hit by the same earthquake and tsunami as Fukushima Daichi. The difference was that the engineers in Daino managed to convince the management to put the emergency power supply on a nearby hill, where it was not flooded by the tsunami and it allowed them to do a cold shutdown of the reactors, just as intended. That's what nuclear engineers get for doing the job correctly - nobody ever has need to talk about them. That's why no one ever head of Fukushima Daino, even as Daichi's reactors were exploding just miles away from it.
Dee you forgot 1# important note!!! they sold the....Angle of Attack Software... as an upgrade those had previously flown that type of Plane could typically handle the engine change. But thanks for the video!!
Boeing's management are to blame. They sold the planes with a single angle-of-attack sensor. Boeing charges $80,000 for the second AoA sensor. Without the second sensor the software can not detect the fault. The REAL fault is the engines are too large and had to be mounted forward on the wing, ahead of the center of gravity. Under acceleration this creates torque around the CG and forces the nose up. Without that they would not need the MCAS to begin with.
The word “accident” is reserved for meteorite strikes. The rest are “crashes”. They have non-random root causes. Please. Just like devs use the word “defect” instead of “bug”.
For the most part low cost first wave outsourcing devs are absolutely comparable to their first world counterparts. That is a very though pill to swallow for many I guess.
My dad was an Intel casualty. Monday was his last day after 25 years. They're floundering. The only thing holding them together is the US government because it's the only fab manufacturer left in the US. He LITERALLY told me to get an AMD CPU for my gaming rig build 😭 These corporations don't think long term. A lot of Indian developers on H-1b visas are leaving to go back home. They've truly pooped the bed on this issue. For myself, I am a hobbyist but would love to work as a dev full time eventually. I'm planning to go to Europe...and I'm from California. I grew UP in Silicon Valley more or less and I'm leaving. 😂 It's already biting them in the ass.
When big companies failed to innovate, they did the next best thing -- cut cost to the cheapest buyers -- so execs can grab whatever values before it sinks. And right on about international devs. The problem is not intl devs per se, there are a lot great ones, but the problem is the cheapest they could get for.
Thanks a lot for this! In Norwegian, we have this saying "Å spare seg til fant", which translates to something like "Saving to become a bum". I think this sums up this whole mess ;)
I am from India and everything said in this video is fully correct. Its unfortunate that it has come to this. Its all about money now and less about quality.
Outsourcing is only part of the issue when the main issue when Boeing though that redundancy wasn't needed and used the same software that was validated with the use of redundant instruments. Not defending a billion dollars company cheaping out on software development and outsourcing in order to give better payout to C-suite executive. but I do feel the issue of removing redundancy is being shadowed by the outsourcing ( where it should have magnified it).
Indian here alot of these consultation firm developers didn't even do cs most are electrical, civil or mechanical grads there's some chemical engineering ones aswell And fyi senior and even junior dev jobs for serious devs are actually really scarce rn most ppl are gettin 1/4 the package that is if they get an offer at all compared to just a few years ago
An aerospace engineer with coding experience will beat a CS graduate with jack of all experience, any time of the day ! Today's Indian graduates are not interested in writing quality code, they just want nice salary to live a luxurious life. This is why majority of Indian youtube videos are about salary and not skill.
nice video.. This has been issue for a long time (at least 20 years). I think that the biggest issue in these companies that make big products is that even tho they have millions of lines of code in their system they still think that they are automation company or something like that and they do not understand the value good software. It looks nice on Quarterly report to have lower costs, but nobody knows how much more that costs in long term. More experienced sw developers change role and get more money when they manage these outsourced developers and do less coding. So these things just pile up.
I agree that outsourcing software to the cheapest source will lead to bad software, but the 737 MAX disasters were not caused by faulty software. In both crashes the MCAS software received incorrect data from malfunctioning angle of attack sensors. Another problem is that pilots were not trained how to recover from a malfunctioning AOA/MCAS system. The root cause of the 737 MAX problems was probably Boeing's attempt to avoid requiring pilots to be retrained on the 737 MAX which had different engines than other 737 variants.
I dont think your assessment is sound. Historically in Boeing and more broadly for effective design implementation, where you have integrated and mature teams, and specifically software developers they will simply know and tell you about redundancy as it is par-for-the-course in IT. Boeings fault here is in the disconnection of functions and downgrading of quality. The quality of the software development team INCLUDES the ability to intelligently see the whole system. Its never not the case even if you arent the designer and its not your domain. Thats what you are actually paying for! And that IS what they got. They walked the intelligent people out the door and people died... holy mother!
Outsourcing for cheap labor joyfully rings the CEOs' and others' ears. Unfortunately, it is a regular practice for large corporations. I would not doubt that Boeing used this cheap labor for its spacecraft and mission to the International Space Station. Now, it is stuck up there until 2/25 unless EM rescues it.
I believe it’s not about whether the cost is $9 or $900 or geographical location etc but rather people (that mid and upper management). In my opinion, hiring non-technical individuals in a tech company is a mistake. I understand this view may seem extreme to some, but to truly grasp what I mean, you need to have spent at least a decade working hands-on in the tech industry. And by real work, I mean contributing directly to projects, not simply managing inflated data points and adding extra burdens to technical teams.Once again, the way this issue is being discussed in the video is not how a technical person would present it. Watching this video again reminds me being in a technical meeting, where non tech management spends 50 minutes talking nonsense in a 60-minute call, focusing more on surface-level details than addressing the core problem.
Hold on though: I have worked with a few of those kinds of people and ime they fall in 2 camps: 1) the ones who care: these go out of their way to understand the context and 2) those working to spec: these don’t care and don’t want to know the context. They are there to write code and that’s that. I witnessed a team of 5, 2 of whom were with us in the office and 3 were back in India. Of the 2, it was a 50/50 split but the “team leader” who was with us in the office would want detailed specs to even get started. That’s how a fair few of them actually make money. The actual failure is totally on Boeing here because once they realised what was happening they should have reneged and cut ties.
Analysts, programmers and software validation need people with an understanding of the product and its function to safely update existing software. Contractors can never do the job. Software almost always gets fixed afterward in house.
I enjoyed this video and will watch for others from you. But I'm surprized you didn't also mention the space capsule "Boeing Starliner" with its failure to reach the Int'l Space Station in Dec 2019, and with the helium leakage problem which prevented the 2-astronaut crew from returning with the capsule last month. You could also have mentioned that Boeing's merger with McDonald Douglas in 1997 has been blamed by some for a change in corporate culture which shifted the emphasis away from engineering excellence to cost-cutting measures.
Those big engines are a big mismatch to start with. The whole sensors/software/fly by wire problem has some issues that have come up in the past. Sometimes the software actually helps the situation, some times it totally wrecks things. You should never have a situation where it is fundamentally impossible to control a crashing plane though.
The hourly rates at 5:25. For what kind of SWE are the for. Those numbers make no sense without more context. Maybe for PHP or low end web dev. But those numbers are way too low.
$9/hr is still a lot… you can get senior developers(5-7 yrs exp) lined up for these price in India. Anyone watching from India, its ~1.33 L per month [$1 = Rs84 at the time of calculation ]
I hate outsourcing of any kind but having said that there is only 1 thing that can automatically control the pitch of an airplane and that is the trim, you learn this from your first flight in a cessna 152. in the 2 737 max crashes the aircraft kept pitching down un-commanded. Meanwhile a couple of big wheels the size of a dinner plates were spinning away right next to the throttle levers. these dinner plates are the trim wheels. All the pilots had to do was turn off the automatic trim, it should not have been hard to diagnose since the ONLY thing on the entire aircraft that can cause an un-commanded pitch down is the automatic trim.. Boeing screwed up big time, and outsourced developers is a horrible idea, but make no mistake those pilots killed all those people no one else.
All big companies outsource a part of the coding work, because there are just not enough engineers in the local market. And in almost all of the cases the outsourcing works well.
I worked for a $2 billion company in the US. It was outsourcing software development to an Indian company. I designed a new command for managing a hardware device. The essence of my design was to gather information about configuration from a database. It was generic - it wasn't dependent on the hardware model. The team in India wanted to hard-code for each hardware model. More money to make if a new model were introduced. My design was eventually used, after significant pushback. The software was still in use 20 years later.
I should point out that it isn't just outsourcing to other countries. Outsourcing to big domestic consulting firms regularly results in catastrophes.
I will give consulting firms one, small, bone. They're often brought in by companies which have some horribly outdated process or another and asked to deal with a problem for the least amount of money. When often the problem is no one wants to actually fix the real process problem. As an internal engineer, you're used to the problems and know what the real requirements are. Meanwhile, the consulting firm is given a set of crappy requirements that are very specific and are not forward thinking at all.
Plus, sub-contractors are used to working with insane clients. For example, on one job the customer demanded we use their ticketing system. To create a ticket for a feature or defect, I had to fill out a PDF and E-mail it to someone. That person would then click the "new" button I wasn't allowed to and manually assign the issue a number. The ticketing system assigned a number and prominently displayed it, but they did not like it, so used their own number in all the paperwork.
@@arthurmoore9488 I've experienced all of that, and what the OP said also.
I'd forgotten the "our ticket ids don't map to your ticket ids" experience.
The Indians are good people but they have a big problem with business ethics. In my case, it happened the same thing
So based on the difference between what thlse HCL employee resumes claimed vs reality the 737 Max flgiht crashes and deaths can be chalked up to lndia's penchant for being cheats and Iiars. Got it.
Also stupid white peopIe being so trusting and thinking that aII cuItures are as good as their own...because they're aII equal (i.e. the same).
They don't aII have Christendom.
They don't aII have the enIightenment.
They don't aII have the renaissance.
They are not your equaIs.
As a remote Dev that lives in Africa and works for an American company I do agree actually, the big outsourcing firms have a problem, I'm an independent person I just apply independently for remote jobes. I do try to make sure I do thinks right, follow best practices, write things that are maintainable and scalable. But then again I would never work for $9 an hour so it is what it is
With over 30 yrs in IT, not having a thorough understanding of a business and its processes, was the biggest impediment to successful outsourcing.
Agree. Some of the best work I've done was in collaboration with an on-site Subject Matter Expert (SME). If you look at the companies which are successful at using foreign labor, you'll see close working collaboration and at least one person who knows both the company and the foreign culture.
That's the actual issue here, not the cost which might be a considerable wage in some economies.
I found that it cost more time to write a spec for outsourcing than to implement in house.
Maybe they should outsource the Director, VP, and CEO tasks.
Many of which could be replaced very easily by a room full of chatbots.
Possible, eventually we'll have 1 person billion dollar companies
Basically they did, back to the Board of Directors, they make the choices and the CEO is a puppet winging his little head in a positive manner, it's not surprise they all seem stupid while their CV says another story.
How do you outsource saying word salads in long meetings 😂
How dare you? Your curelty it's beyond human! [Sarcasm starts] How come are they gonna pay teir 5th mega yate and the 10th mansion with a peasant salary?[Sarcasm ends]
Boeing used to be run by engineers. Now, it's run by financial types whose prioritize profits over everything else, including safety. Back in the 90's I wrote software for air-traffic control systems. Everything was designed, reviewed, and thoroughly tested, and had redundancy built into they system
Oh please stop this accountant bull crap. Dennis Muilenberg (former Boeing CEO, just in case you forgot) is an engineer, and look where Boeing went under his watch. It’s simply greedy a-holes that put greed above anything, not just because someone is a “beancounter” or not.
That is very true. Boeing CEO has been a marketing guy, economist and stuff like that. The times when CEO was a real engineer, are long gone. I made another comment here to tell how MCAS works fine, but the economy-people wanted to get 19000 USD for a software flag that could say "AoA disagree"-alert. Lets see if that comment will be visible, and how it will be at least shadow banned.
And yes, there lies the problems of the Boeing Commercial Aircraft Company. The CEO is not an engineer, in a company that is 90% engineering and creating NEW stuff and items.
Now Boeing has changed their CEO many times, and still, none of the new ones are engineers. As that is the root cause, including that the current new management board is bunch of children straight out from Harvard (etc), they have zero understanding of anything in airplanes and aviation. The good board members all resigned and stayed in Seattle area, and these new clowns are the 100% culprits of those really bad design principles.
For economist, marketing etc person, it must be hard to understand that engineers need to be there, in a huge engineering company, but again, they still haven't tried that. And that is why Boeing will not recover, until they hire from their knowledgeable engineering ranks, and stop murdering whistle blowers.
Bean counters, lawyers, and bad leadership went to the forefront when they brought in McDonnel Diuglas.
Since they acquired the DC Douglas folk, they also inherited the toxic management culture which contributed to their current woes. The solution is really simple for Boeing - if you are an engineering / aeronautics company, make sure to put engineers at the helm, not the ones with cushy MBA pencil pushers with their greedy business models.
Companies are turning into financial business instead of serving the purpose of their particular existence. Product quality, customer satisfaction, and worker happiness are all going out the window for "maximize profit".
just wait until they start "outsourcing" to ChatGPT. That cuts costs even more, right?
According to my AI software resourcing app, ChatGPT would be a good choice as a software developer.
@@GurungyNoHamuster LOL because of course it would.
...and is likely to rise once OpenAI's VCs start asking for money.
There's no such thing as coding without AI anymore. Hasn't been for a while.
@@jtjames79 ya you can't keep up unless your using it too. also its using you to learn to code better it won't be long till it outshines us imo.
I remember when Siemens Medical laid off people with decades of experience to contract with an Indian company for dev ops.
The new guys took 5 times as long to get things done and often had to reach out to others to figure out how to do their job tasks.
Brass did this on purpose to jack up company stock before selling to Cerner to get the price up for stock holders.
It's a shame that ethics, quality, and standards are all just things you can cut for capital.
Once you've written specs so detailed that a $9/hr developer can understand, it doesn't take much more time or effort to just code it yourself. I just don't understand this approach.
Definitely, this is what a sane person sees, unfortunately not what the executives see 😢
@@arthurdefreitaseprecht2648 I heard from the management many times that you can teach anyone to write code well, but it has never been true in my experiences. First of all, *most* of low-cost developers do not care much about learning. They think they can just follow instructions without real understanding of stated goals. Moreover, even the minority who care about learning something new often are very slow learners...
Take chess as an example. Can you teach anyone to play chess? Sure, but can you teach anyone to play at the GM level, I am not so sure. Even it was possible, can you imagine how much hours of teaching would it require, especially if one is incapable to learn anything on their own?
So based on the difference between what thlse HCL employee resumes claimed vs reality the 737 Max flgiht crashes and deaths can be chalked up to lndia's penchant for being cheats and Iiars. Got it.
Also stupid white peopIe being so trusting and thinking that aII cuItures are as good as their own...because they're aII equal (i.e. the same).
They don't aII have Christendom.
They don't aII have the enIightenment.
They don't aII have the renaissance.
They are not your equaIs.
A complicated, or large, application, especially if it is "real time", is itself too difficult to code by people whose focus is algorithms.
The algorithms require algorithm experts, and the code requires code experts. So, systems engineers write the algorithms, and the test procedures, and the software people write the code.
It's management-think.
Cheapskate accountants chopping software engineers is nothing new. Years ago, I worked for a risk management company in the UK and was sent over to Mumbai for a few months to beef up and get the team there up to speed. Whilst I was away, a top-notch fast and accurate software engineer (that I had recruited), worth 4 average engineers was ditched by the accountants because "he was expensive...". Output from the UK dropped noticeably and I quit not so long after, tired with trying to get cheap, inexperienced guys to work accurately. The stories I heard from other departments where the users of the software internally (before selling or upgrading to clients) would berate the engineers because of stupid bugs where the engineers would blithely reply, "Oh, yeah, we know about that, you do x as a work around" was driving us crazy, and I had enough of the fire-fighting and the constant battles with "cheap" engineers to bring them up to spec. The company eventually folded.
Yep. I too have been "downsized" as I was the most expensive.
CEO seems to have forgotten that not six weeks before I pulled an all nighter (unpaid) to get enough systems working that we could participate (sucessfully) in an industry wide demo of our tech. Without that, our company would have been a laughingstock.
But I was expensive.
So based on the difference between what thlse HCL employee resumes claimed vs reality the 737 Max flgiht crashes and deaths can be chalked up to lndia's penchant for being cheats and Iiars. Got it.
Also stupid white peopIe being so trusting and thinking that aII cuItures are as good as their own...because they're aII equal (i.e. the same).
They don't aII have Christendom.
They don't aII have the enIightenment.
They don't aII have the renaissance.
They are not your equaIs.
@@erictheis6093 I wonder who else is expensive and worthless in the company... Hmm..
When I heard that Boeing put a single monitor on the outside of the aircraft that if it malfunctioned, would drop the nose of the craft and that pilots would tire within fifteen seconds while trying to regain control, unless they properly reset the system after reading an update to a 500 page manual, I thought... who the hell doesn't have multiple points of redundancy? Who the hell doesn't train pilots on new features? Oh, they fired all of the software engineers whose job it is to think. I could make that determination in my company back when I was getting paid like $70k a year for a $50 million dollar company as a junior software engineer. Boeing, a $50 BILLION dollar company really has NO software engineers that are to that level of competency?
They 737Max was design to compete with the Airbus A320 Neo which uses the new Leap engines which are larger but uses less fuel.
Fitting this to the 737 design made the aircraft have a different centre of gravity so MCAS was to make it behave like an old 737.
So no retraining or recertification for the pilots were needed since they claimed it worked like the old one
@@tibbydudeza Yep, that was one of the big things. They deliberately didn't train the pilots! Since doing so would have made the 737 Max cost more to operate.
This is not the true. They did safety evaluation of the mcas with some requirements and they end up that risk is minimal so mcas may not have redundancy. And that was all fine. Then someone changed requirements to be more aggressive but there was no second evaluation that could end up with system redesign. And this was not a fault of 10$ / hour devs.
Exactly. If they'd had more experienced internal engineers, they'd have realised the flaws and raised hell about them.
This is also what management wanted though. If you read between the lines, you can tell that "engineer says no" was seen as A PROBLEM at Boeing, not a strength. All management could see was "this guy who makes WAY less than me is causing me problems", and their solution was "hire a cheap contractor who WILL DO WHAT I DAMN WELL SAY".
The basic problem is that management were making engineering decisions that they weren't qualified to make. They thought they had every right to make those decisions because... aren't they in charge? They didn't realise that reality gets in the way.
IMO, this was all 100% a management problem. It's 100% about managers being UNQUALIFIED to manage the things they're managing. There should be no such thing as a manager, at every level, right up to the CEO, who doesn't have at least a decade of actual engineering experience (NOT in management) before being promoted to manage even one person. The "non-technical manager" is the modern scourge of all technical companies, and the source of an overwhelming number of problems.
Dude, working for defense and common fortune 500 companies, a software engineer who tries to discuss tasks with managers or innovate is now called "insubordinate" and "not fit for development teams". All management wants now is go go go, must make the sprint points and deadlines!!! No thinking, just code!
I agree with your assertion that chasing low cost/lower skill engineering talent can often be more costly in the long run, but I believe it is incorrect to blame software engineers for the issues with the Max. The 737 is a design from the 1960's. The original airframe was not designed for the large diameter, high bypass turbofan engines used by the Max to improve fuel efficiency. To provide adequate ground clearance, these bigger engines had to be installed further forward and upward on the wing than the engines on prior 737 models. This moved the center of gravity of the airplane, which in turn changed its performance characteristics at high angles of attack. Boeing needed the airplane to fly just like previous models of 737s so their airline customers could avoid costly aircrew retraining. The MCAS system was added to help pilots recover from nose-high attitudes and make the airplane appear to fly just like earlier 737 models. Unfortunately, the MCAS system was only fed by a single angle of attack sensor. If that sensor malfunctioned, MCAS would aggressively push the nose down with more force than the pilots could overcome. MCAS was a band aid created to squeeze more life out of an outdated airframe. It enabled Boeing to offer a product to compete with the Airbus neo series without the massive expense of developing an entirely new airframe. The entire idea was flawed. The decision to control MCAS from a single, non-redundant sensor was crazy, and the failure to inform or provide training about it to pilots was inexcusable. These actions caused the Max crashes, not the low cost software engineers. The MCAS software performed as designed. It was just a horrible design.
Agree, fundamentally a complete failure of systems engineering, most likely on-shore. No one there put together the big picture in understanding of MCAS operation, failure modes, and safety analysis. It doesn’t help though when the business does so many things to harm the engineering culture and technical capabilities.
From what I gather, there was nothing wrong with the software, the problem that a single sensor was used, which was Boeing's decision. I work in the oil & gas industry and all our safety systems use 3 sensors and 2 out of 3 voting on what signal to use. Additionally, measuring devices are able to indicate when they have a fault. Again, Boeing's choice not to do this. Boeing made the AOA disagree alert an optional feature and it looks like they never considered what would happen if there is a fault on the sensor, which by all accounts is quite regular.
I did some work for a Boeing division (off and on) over the years. When all the stuff going on with the Max happened and the MCAS was identified as the main culprit I hear from a lot of engineers that the real problem was as you suggested with the placement of the engines causing a "hack" to be implemented with the MCAS.
Boeing got caught with their pants down with the 737 vs. Airbus A320 Neo. They couldn't take the time to redesign the airframe and landing gear area to gain more clearance for the bigger fuel efficient engines. Basically they would have to go through flight testing and certification again, which would have costed them 5-8 years of work, to be able to compete with the Neo. It would have seriously crippled them in the market as the 737 was their most popular plane.
Yes, But, it was not the retraining but the very costly type specification which prevents changing major parts - like the distance from the wings to the ground of this type. A new type specification costs billions.
@@smmasongt Now you're just blaming the engineers 😆 Their concern is not cutting corners with bad design to compete financially with Airbus, that's on the executives. My guess is most of the engineers looking at the plans thought "WTF are you doing?!" In fact, many of them complained to management and were ignored or told it would be fine. I mean, technically it would work, it was just precarious.
One more trend in Indian consulting firms is that they would have one experienced dev and rest junior devs. Where you would end up with just garbage code due to tight deadlines, experienced devs burning out and quitting, also junior devs don’t get mentored and grow.
Sorry, but it wasn't a software problem. It was a system engineering problem.
A sensor with SIL4 (Safety Integrity Level) responsibility (highest safety level) was electrically connected like a sensor without safety responsibility.
This was not a software error but a system error.
My understanding is that the aircraft had 2 AoA sensors, but they made the decision to only take input from ONE in the MCAS. It IS a system error but potentially one that could have been spotted and flagged had the devs had more domain knowledge.
Upvoting this for correction
This sort of poor management is a problem at Boeing, and MANY other companies and organizations. The senior management no longer has any real world engineering, production, leadership, and basic business skills or experience. They listen to marketing/sales first, cost accountants next, and finally quarterly stock market returns. Senior managements are the winners of decades of internal power politics, in some companies a low grade war. This is all a long way from building great airplanes.
single-handedly one of the best videos on this subject. Liked and Subscribed.
Ugh! This is depressing. I'm already ousted because of my age and it just frustrates me to see it the whole industry trending downward. I wasted money getting a degree.
Indeed the reason of outsourcing now is very simple. CEO and C-level cut cost, get bonuses and leave with gold parachute. Company can die after after that, but it is no interesting for C-level anymore, they don't try to build something, just max money for minimum time. And even after failure in one company, they will be willingly hired in the next one.
Couldn't agree more with this. Also telling that there have been multiple studies that have failed to find a statistically significant relationship between CEOs who move between companies and the performance of those companies, yet the companies feel they have to pay ridiculous money to get the "Best" CEO.
I am a retired software engineer who worked in software for my entire adult working life. I was appalled when I heard about the design defects of the MCAS software in the 737 MAX, and how the people at the top of both Boeing and the FAA seemed oblivious to the problem. The Crux of the problem was that the software would ignore incompatible readings from the AoA sensors, and make automatic trim adjustments using only one (possibly defective) sensor. That is an egregious design error. The only reasonable thing to do in response to disagreement of the AoA sensors is to alert the pilots and refrain from. aking any automatic flight control adjjstments depending on those sensors.
Outsourcing software explains it. You have the programming being done by people who are not specialists in Boeing flight control software, and you may have poor communication between the Boeing engineers who (presumably) know how things should work, and the people doing the programming, who may know very little about airplanes.
As far as I know the MAX crashes caused by a faulty system had nothing to do with poor coding. It was the design of the whole system that was faulty from the beginning, and true cost cutting that led to that system being concieved and implemented was the idea that it would enable pilots of the existing -800 series to just switch over to the MAX without having to do any additional type rating.
I worked with the HCL guys in India while in the finance sector, back in the early-ish days of HCL. They were very sharp guys all, even the cheap ones. The issues that did come up, stemned from poor communication from our end as both sides had major technical, social, cultural blind spots towards another, many of them unmentionable. Looking back it was shocking the result was as good as it was despite those issues.
One thing to add, the actual fresh graduate engineers working on this don't even make 9$ an hour. Their salary is 400k-500k INR a year, that is 4,700$ a year.
This is a little misleading. The code didn't have any problems , it was doing exactly what it was meant to do: take control of the plane when a single sensor failed, and tricked the avionics into activating MCAS. This was designed by the engineers at a higher level, point the nose down over and over if one sensor output data that said the plane's nose was pointed up.
Yeah, that's my understanding as well. SW was doing exactly what it was designed to do.
Yes, architecs ...
They didn't outsource the MCAS, that was all Boeing genius. I'm unclear exactly what they did outsource to India, but they did outsource some 737 Max code.
True that. The MCAS software checks the ONE active AoA sensor (Angle of Attack, eq how high the nose is compared to the airflow around it). If the AoA gets too funky, the MCAS will start pushing the nose down by trimming the horizontal stabilizer to push the nose down.
Everything is fine and dandy, as long as the ONE SELECTED AoA sensor is working well. Those senors are of course outside, they have air/sand/garbage flying on them all the time, so they get quite easily broken. Now the broken AoA sensor gives out some funky AoA readings. There is the problem.
There is another AoA sensor on the other side of the nose/fuselage. That ought to be ok, but Boeing Chicago management clows (all economics people, kids, straight from universities, knowing nothing at all), decided that checking on AoA sensors, is not needed, and it is a "optional advanced AoA software package, for 19000 USD per plane". That software would then tell the the pilots (and MCAS) that "the left AoA and right AoA are saying very different results", eg AoA DISAGREE.
Only Delta ever bought that software, but other airline pilots (not Lion/Ethipioan) also knew what is a runaway-trim and how to fix it. Those two airlines did not know and did not care.
Pilots had to select left or right AoA sensor to be used, but as that optional software was not installed, they could not have "Auto, using both". Now when the active AoA sensor fails, the MCAS can freak out, and start helping the pilots by pushing the nose down.
Again: Stop the runaway trim by pressing Auto-Trim: CUTOUT right next to the flap lever) AND/OR Stop the running trim wheel by hand (can be done easily). Continue the flight to the destination as normal, with of course now manual trimming. Easy. But again, not easy when pilots are such bad bus drivers, from the lowest tree branch, and Boeing board thinks that all of this is "act of god, nobody knows why the MCAS freaks out...." . Plus of course a lot of corruption, blackmail and threats, and murdering of the whistle blowers, which we are all sure will produce fine Boeing aircrafts in the future..... sick and sad.
$9 per hour for a software developer is bad enough. But there is a bigger question: what about testing the software? Did Boeing test this software to see if it complied with the safety assessments or FAA regulations? Or did they recycle the tests from older designs that weren't as software dependent?
The bigger thing that a lot of airplane safety guys were screaming about right after the investigation came out was lack of redundancy. The software depended on a single sensor for its input data, and when that sensor failed, the software that relied on it went haywire. As you will hear time and time again in _any_ class related to industrial safety, the minimum is two sources for safety (when a system is allowed to fail completely) and three sources for fault tolerance (when it is not allowed to fail completely, usually for safety reasons). The software actually performed perfectly -- assuming the input data wasn't wrong and that the pilot would never need to disengage it quickly and intuitively (the process to disengage it was quite involved and rarely mentioned in training material, if at all).
There’s no magic in safety assessments or FAA regulations. They’re not sufficient to make a good design. If you have engineers who don’t know what they’re doing it’s just a worthless stack of paper.
@@mage3690 that was the first and main issue one of my neighbours mentioned as soon as details about the crashes had been made public. He was an engineer with Airbus. He said that such a design just wouldn't have received approval by European authorities which is why Airbus wouldn't even think of doing something like that. Failed approval procedures are just way too expensive.
Boeing basically runs FAA now. FAA regs are basically "what does Boeing say?"
Great video. Outsourcing isn't just a recent trend. It's been going on for decades in tech. I first noticed it in the 90's. It was and is getting more prevalent. After that I never worked for a company that didn't depend on cheap contractors. It was always a mess trying to get good or even correct code - cheap labor is cheap labor. Combine that with time lag, language issues and lack of product knowledge or ownership and you get what you get. :)
I have worked for an outsourcing company. I fondly recall them designing and building an overengineered system to achieve something that could have been done by a few tweaks of existing code.
A common theme with these companies was a lack of understanding regarding testing.
I spent 25 years in airline ops for various airlines. Over the years technology has pretty much taken away the common sense of engineers and other professionals. To give some perspective. I was involved with weight and balance since the early 80s. We were taught the theory as well as how to correct issues if the computer system crashed (very often in those days). We were also taught on how to use various airlines computer systems to perform our duties. Those days, we had to learn the computer entries and type them in. These days, it is just point, click, drag, drop. or even tell the system to "auto load". Sure, there are error checks, but if you do not have the basic knowledge of the principles, you are pretty much screwed if the error persists or if the system goes down.
I use to work with avionics tech folks as well. These days, there is no thaught process involved. You just tell the "AI" what you need and poof - out comes aa sh*t load of spagetti code.
Then you outsource because of "costs". Penny wise pound foolish.
I digress.
The C-Suites are universally staffed by managers that rarely understand the tech involved. In nearly everything. They are focused on the quarterly results, their personal stock options, and their personal advancement path.
Boeing's previous merger caused many experience Boeing engineers to leave the company. The new management structure gave the world the 737-MAX. Oddly enough, one does wonder what those stock options are worth today.
When big trees fall, there is more light for saplings to grow. And big trees fall for a multitude of reasons acting in concert. Boeing looks a bit creaky at the moment and the winds of change are blowing hard.
Problem, Boeing is "too big to fail." It's more like this mutant tree grew to giant size and gobbled up or killed most of the forest. Yet instead of splitting it up, we're actually allowing it to continue to subsume more trees.
Alternate example, we built a neighborhood around the monster trees. The only space for saplings to take root is the empty space left by houses the tree destroys when it falls down.
Turns out gutting anti-trust, and allowing monopolies to flourish for 30+ years has consequences. Who would have guessed?
This is probably why so many large IT projects go wrong. It is not that programmers in 3rd world programmers are not smart, it is that they are often managed by non-technical “managers” who are thousands of miles away while also not scoping out the project correctly. Failure is inevitable.
From the little I learned about the system, the main issues had nothing whatsoever to do with the code. They were design problems that were based on decisions almost certainly NOT made by programmers or 'overseas' personnel. The two main issues I am aware of were that they deliberately tried to hide the system from oversight and made some decisions for that reason and there were insufficient safeguards if a sensor was giving incorrect readings. The aircraft has two sensors and even I, as someone who has not worked on aircraft software (although I have built a customer reservation system for a chartered aircraft), I know that if you have two sensors then you should read both sensors and if there is a discrepancy you should do something about it whether that be figuring out which is correct and relying on that or informing the pilots that there is a problem etc. But instead they simply use only one sensor and ignore the other making the whole redundancy aspect totally useless. That was NOT a programmers decision. That was a management decision.
Issue is: without the experience and maturity you DO get from well paid, market rate software developers who DO see the total design even though they are on code, you wont have people who state a simple and obvious intelligent point about redundancy, design and integration of the code theyvre written. MOST good software devs will absolutely just tell you when the glaring implementation fault is there. And yeah just make sure you get decent people working on an integrated solution in this scenario, its peoples lives. Boeing should be ashamed.
@@johns5558 In my own job I often point out glaring design flaws or things that should be done, but the reality is that in corporates that doesn't always result in action. The fault lies squarely with management in this case. If I was the programmer, I would have definitely suggested improvements to the system, but if management said no, just stick to the design or we get someone else then I would have stuck to the design and got paid. Outsourcing does have its problems as the employees feel less attached to the company as a whole and are less likely to 'do the right thing'. But lets be clear that Boeing outsources almost everything (and has suffered as a result) and that has nothing to do with whether or not the outsourced company was in India, Italy (recent case) or the US. I once worked for Vodacom (a telco in South Africa) and two thirds of the people at work were 'contractors' including myself. It worked, but it had its problems because there was no job security. Outsourcing has its pros and cons. A contractor is under higher pressure to impress than an employee, but they are less invested in the long term health of the company.
I remember a recent case that was reported on where a bank fired a large number of employees because they had installed software that emulates moving the mouse and keyboard. They did this because management was monitoring them and evaluating them with software that monitors mouse and keyboard activity. The employees were fired when the right move would have been to fire the management. Management is hard and evaluating employee performance is hard and when you get it wrong you get the wrong results. Boeing values cost savings over safety and that is what they get.
This is a trend that started decades ago and made famous by Jack Welch as the CEO of GE. When an organization gets hyper-focused on the numbers it starts to get hollowed out with cost-efficiencies. Give it long enough and like GE, it loses its intrinsic value and becomes a hollow shell with significantly reduced value.
India consulting companies have some very good engineers, however not many. They tend to give you the good engineers at the start of contract then swap them for the lower quality engineers after a couple of months.
I used a company that over 2000 developers, one of my team would find the error for them, tell them the change the 1 PM, 1 manager and 3 engineers would make the change. When I found out that my engineer was basically doing all the work, they where gone.
We weren't outsourcing because of cost, just lack of specialist skills (particular telecoms protocol) their good engineers where probably one of the best in the world for that skill. however they were in project for 3 months then gone, to another project.
Experienced that also. The person I was left with (overseas) didn't understand what a function call was.
Whether it’s India or the US, it’s all the same. There are a lot of bad engineers. If you want to be successful you have to be able to identify the good ones and then retain them. Normally with good work and good pay.
It’s not really “airline software”, but rather aerospace software.
Much more critical.
There is a reason studies showed psychopathic tendencies among executives.
One issue with outsourcing to these contracting companies is that they benefit from delays and rework so there is no incentive to be efficient or provide a quality product first go around.
This is the same across industry, way too many middle-managers and the 'MBA types'.
MCAS was a fairly robust system done a while ago for the military. Digital augmentation of airframe characteristics isn't a new thing in military aviation with fly-by-wire inherently unstable fighters, planes having asymmetric load-outs when they fire externally mounted missiles or receive battle damage etc.
The issue was implementing it on the MAX while making the angle of attack sensor non-redundant.
Right, but if you disconnect the software developers from the whole implementation then there's the issue.
that they are paid $9.00 an hour only tells you that there's so little experience that not only would they not know about redundancy in the total system but at $9.00 they wont speak up about it either. There are few good ( / well paid ) software developers who won't at some point tell you if there is a fundamental design flaw even though they arent the architect or designer. And they will know.
This is the issue and this is Boeings penny pinching fault.
@@johns5558That’s not how software development for safety-certified systems (which includes avionics) works. The coders are the least important part of the process, and they have no influence whatsoever on the design of the software. That is done by different software engineers whose job it is to write specifications.
@@tookitogo , John, you're both almost there, but are missing the particular MCAS context. The *entire point* for that system was to be non-flight-critical & non-redundant, as per Boeing's own requirements - not $9/hr programmers' inexperience or indifference or the savings to be had in hiring them. We're talking about many orders of magnitude more money, elsewhere in the business, as the driver. Please allow me to explain.
Let me back-track a little. The 737 sits quite low to the ground, which had played a big role in its success over the decades, as that not only reduced the landing gear weight & complexity & allowed it to be tucked inside the wing instead of the body, it significantly reduced the size and complexity of all the requited airport machinery for service / loading & unloading / de-icing etc. Personnel, procedures, didn't need to be heights-rated. A devastating competitive advantage.
As material science and fabrication advances allowed greater engine fuel economy through larger turboprop bypass ratios thanks to giant front fan blades becoming viable, the engine diameters naturally grew. Eventually to a point when compelling new engines would fit under Airbus A380's wing but not under the 737's wing. Boeing engineers pulled a design rabbit out of a hat, flattening engine shroud on the bottom, shortening its pylon and putting the fattest part of the engine way in front of the wing (taking up the wing thickness' worth of space). With all these measures combined, the new engines fit on the old airframe, which ( *this bit is key* ) qualified the 737MAX, as far as the FAA et al were concerned, for the simplified deployment, without extensive pilots' retraining incl simulator hours on what woukd otherwise have been a whole new type. This new type retraining requirement is worth many millions *per airline* in lost profits and risks - that manufacturers typically pony up. It was *key* that the 737MAX not require that retraining, because the A320NEO didn't, b/c even if the Boeing had paid up for the sims etc it'd still take pilots offline, giving airbus oprators a market share in over their brethen running boeings.
With me so far? Good. There was only one small issue with Boeing's engineering miracle: the altered engine placement, in certain very small areas of a flight envelope, made the plane suddenly pull up. Boeing has had just the solution, digital flight control augmentation - a little autopilot of sorts that specializes in patching up / normalizing such small irregularities, making the plane behave seamlessly from a pilot's perspective. Many advanced mil airframes (fighters designed for an inherently unstable flight to maximize maneuverability) aren't even flyable by hand without such a system.
The only pickle was, while just about everything of the sort is doubly or triply redundant on a military plane, adding a redundant system on a commercial jet automatically flags it as potentially flight safety critical in the FAA review (large chunks of which were delegated by the FAA to Boeing to self-certify). That would in turn require pilot training, defeating the purpose of the entire effort & putting Boeing back to square one. Boeing's plan - as per its lesdership and bean counters NOT programmers - was to sail under the radar of this review, by specifically NOT having MCAS redundant. The rest is history: the gamble didn't pan out, and whistleblowers stripped any shread of plausible deniability, which is how I'm able to explain to you all this without any rational fear.
The problem isn't really with "cheap software engineers" but with the project management at the other end.
In our department we had a similar issue:
digitalization projects in development hell for years on end.
They eventually realized that it was a structural problem and hired a few software engineers to fix that mess. It was painful but now there are no projects in limbo any longer.
From a software point of view this video is spot on, though I happen to personally know that Boeing has far far far far more troubles than software. If it’s Boeing, I’m not going. MCAS has a shortage of air speed sensors such that even if the software was written by a genius it couldn’t safely do its job.
I interviewed a lady that worked in a company that outsourced software development to an Indian company. She told me it took more time to explain what was to be done than do it herself. However, management thought otherwise. I personally dealt with three distinct cases where the outsourced Indian/Pakistani developers did not deliver: a restaurant application, a car spare part application and a ERP system. Spaghetti code full of bugs. Do I imply Indians are not capable? Certainly not. It's just that those that are good developers are not cheaper.
I have worked in developement process of safety critical software components and I too noticed that management is blindly relying on the arbitrary safety standards and documentation which are all completely pointless when carried out by bad programmers
Blaming to outsourced partner is easy, but as a aircraft giant, I'm sure they won't have outsourced their critical components for $9 / hour, it is more than just cheap labour
Don’t be too shocked by $9 outsourcing because it will soon go down to $0 because we won’t need as many programmers. When product developers, like us in Silicon Valley, look at corporate IT (which is what you are referring to) we see low skilled, crusted but highly egoistic people who should be quickly replaced. Their business users hate them, think they are impediments to quick growth and are the biggest bottlenecks for any corporate change. So we’ll get rid of them with generative AI. So don’t be too shocked and I would suggest that this be the topic of your next video.
[upd] nvm, author talks about that but 2 minutes later
3:40 here is exactly part of the article which says that Boeing outsourced "flight display"(which is visualization system) but not MCAS(which is to blame in 2 plane crashes)
She addresses this afterwards, I think you did not watch the whole video.
@@arthurdefreitaseprecht2648 you're right. I did not. I'll make extra effort and will watch this ranting till its end
I am not sure why you would do something like this. This should be done by local in-house developers. I guess when you make accountants and MBAs head of engineering companies. Even with non-engineering software the quality is so low and has to go through so many cycles of correction and bug fix.
Actually, I do not believe that $9/hour engineers were working on the core of the boeing system. It is not how it works in this sector. Maybe there were working on Boeing webpage but not on the core system ;).
It's first a communication problem. A lot of companies do outsource. It's not really cheaper as you still need project management, testers, developers, etc. on both sides. Although you have someone else work on the software you are still responsible for software quality, operations and integration (i.e. testing). There is nothing new with this. It's an option for several decades now. The costs shift within the balance sheet. It can cause friction due to contractual issues (e.g. implementing requirements as formulated, ignoring errors in requirements or vague requirements). You earn more for fixing error caused by your customer instead of correcting issues earlier.
That pause around 1:52 where you hear crickets is both brilliant composition and devastatingly depressing. I think it is worth noting that the reason that the 737 Max needed MCAS was because Boeing retrofitted the 737 design rather than designing a new aircraft from scratch.
One additional issue is that when outsourcing work literally half-way around the world (e.g. USA to India) makes it very slow and awkward to communicate. Mid-day in one location is midnight in the other and with weekends being skewed you end up with an effective three day work week.
I work on ed tech software and i sometimes wake at night worrying about bugs or some other thing I might have overlooked. If I worked on planes, I would just live in a constant state of worry and testing
it's not about the money it's about the quality..
To be fair, the report didn’t say which software supplier was blamed
While I agree that outsourcing is generally bad, having seen dozens of failed outsourced projects. I think the reference to Intel outsourcing to TSMC is a different case. This is fabrication of processor chips, not software.
Perhaps companies should start recognising that "There are no cheap options".
I have experience in similar safety oriented industry and there are some glaring points to consider:
1) There are some stringent coding standards (coding standards) which should be maintained. Doing so ensures easy debugging and less possibility of errors.
2) The Verification and Validation team should be able easily pick up flaws after thorough testing using simulators and checking project data/parameters.
Seems like coding standards were ignored and poor validation was done. This means that the problem is systemic and completely overlooks/skips all systems of checks and balances.
I'm a controls Software Engineer and was born and raised in SA. I have immigrated to get a decent rate. Man, I do miss SA a lot!!
Oddly enough, my situation reminds me of that Mexican fisherman Parable.
Sidenote. Great video. Keep up the good content.
Minimize spending, minimize quality. You can pick two: fast, cheap, or good. Expecting 3 guarantees failure.
Part of the problem is the shift towards a "designed for obsolescence" culture. This trend is so pervasive that it has begun to take root in industries where a more traditional permanently employed loyalist workforce is the more appropriate approach. If Boeing made smart phones, then having a more outsourced approach would probably be the right choice, but they are not in that sector, and they would have to find ways of staying in business while employing a more traditional workforce that stays with the company until retirement, and transfers nuanced intricacies of institutional knowledge down the hierarchy to the junior employees. There are tradeoffs; You could fly for cheaper or you could fly safer.
It's called a tombstone reaction. When people die or the project eats dirt, something starts happening.
But not until.
Been hearing this topic over & over for many years. Your particular rant is a good one though. Cheers.
I agree there were rookie mistakes made with Boeing's MCAS. Boeing fired many of its veteran flight control engineers a year earlier, saying that their flight controls were already proven. The problems was MCAS was new.
Boeing decided to go with one MCAS sensor. One way the sensor can fail is via a bird strike (at least one of the Boeing crashes, is tied to a bird strike on that sensor).
An experienced engineer, in the U.S. Knowing full well they can be sued for any failure or omission during the design process, would have pointed out the necessity to install a redundant sensor. And, to test the system against a variety of mechanical failures. And do their best to detect those kinds of failure also in the software (which really would require some push back from the engineers).
But, a 3rd party company, with inexperienced engineers, it might never occur to them, this is a bad design (one MCAS sensor). And, if an engineer did realize this, would they be stifled by inexperienced managers that they report this problem to?
The retail cost of the 2nd sensor was only about 120k installed (that's what Boeing charged). The actual device was a tiny fraction of that cost.
So there is blame to be had by Boeing sales and design at the U.S. level for not thinking about this. But then, if they just fired all of their experienced U.S. engineers, the U.S. sales and design team might have only the bean counters left, who are trying to make sales, anyway possible.
Boeing sacked their U.S. engineers, they outsourced the job to a 3rd party, that then went with an inexperienced remote company. That remote company will never pay the real cost for this disaster (which is billions), no one in the remote team can be jailed for gross negligence, which this was.
In short, they outsourced because "No Accountability" is way cheaper than "Responsibility".
And this isn't a communication problem, because there wasn't anyone at Boeing HQ failing to communicate the requirements. It was a failure of Boeing management to recognize the necessity to keep experienced flight control engineers on payroll for a few more years. Also, there is responsibility at Rockwell and the 3rd party company responsible for developing the software. But, you'll never get them to admit (except in court, which will never happen with the Indian engineers). That they either should have realized the implications of having just one sensor and/or software too inferior to sense it was a mechanical anomaly (which is possible to do, although difficult, and requires testing and more sophisticated software).
A few thousand dollars saved, and priceless lives lost.
And Boeing deserves it's un Air Worthy reputation, until they fix the organizational problem from the top down.
As a junior, I also thank you for the average wages stats. I’ve been working as a sort of emergency patcher to fix stuff done by outsourced contractants, and always been compared to their prices then told that I’m a junior and can’t ask as much as $40 per hour. Just that my implementations work almost always at first try. I’m sort of angry now lol
I have worked with some great software engineers in India ... but the vast majority of them end up in the US. The best engineers don't stay with contracting companies. This is also a symptom of an even greater issue that is not specific to India or contracting companies. For years, we've pushed people to enter this field with little regard to aptitude. I taught myself to write my first program on my TRS-80 in 1982 using the manual that came with the computer and books from the public library. Started with BASIC (of course), moved on to 6809 assembly and Lattic C on a PC XT in high school. I'm now a senior director of software architecture ... and still write software daily. Why? It's just something (frankly the only thing) I seem to have a natural aptitude for as well as a genuine interest. The same could be said for most any technical field. For some reason, we've gone so far over the top in encouraging everyone to "learn to code". CODING is one thing ... being a software ENGINEER is quite another. I can (and have) installed my own crown molding - but I am certainly NOT a finish carpenter.
Not true,
MCAS software did not fail, the active AoA sensor failed to show correct data, and thus MCAS software thought that "ooo, we are going belly first nose up, lets trim the nose down"..
The horizonal trim is automatic, but also visible with big wheels in flight deck. When a runaway trim happens, the trim wheel can be stopped by disabling Auto-Trim AND/OR stopping the trim wheel by hand. Easy.
The auto-trim-cutout button is right after the flap levers, middle console. Really, easy, been there since 1960s original 37 designs, and still at the same place. So when the incompetent pilots are more focused on being a glorified bus driver, they do not understand even the basics. Yes, Indonesian Lion and Ethiopian are not at all competent, and they have never been safety focused, believe you me.
The MCAS operated just fine, but Boeing new management board at Chicago is the real and only culprit. They had zero experienced people from the Seattle board, and Boeing owners (Wall Street and there a certain group...) just said "we can get a much better management board, directly from universities (Harward/MIT), and that is it. Those very new board people knew nothing about aviation, airplanes, how to design, sell, use, operate, own or anything.
The new and completely incompetent Boeing Chicago board just tried to see 37M features as a "DLC", since that is what those kids learnt at their schools, how to make games with paid add-ons. This meant that one of the TWO AoA-sensors on the 37M, could break, and no information would be not told to the pilots. That option for the software to say "Aoa Disagree" was "additional paid service for 19000 USD/plane". That software is there, but for that software to be allowed to say that the one of the AoA-sensors are giving freaky results, you had to pay for it. In all other planes the "AoA Disagee" alert is built in, eg when the left AoA produces different values from right AoA.
So, in the basic model, with the Boeing Chicago board idiocy, did not tell anybody that AoA sensor was busted. It did not tell that even to the MCAS software, as that would be 19000 USD optional product. Nobody bought that software, except Delta for their future planes.
Manual AoA selector is there, and any normal pilot can see the Runaway-Auto-Trim, as it keeps a lot of noise and is visible spinning. So, just stopping the nose-down trim run even by hand, and disabling the auto-trim completely, the plane is allowed to fly to the destination normally. But again, that means the pilots know at least the very basics of aviation, a thing that Lion very secondary airliner and Ethiopian young and minimally qualified 37 pilots just did not want to be bothered with.
The horrible worstness of the pilots were just underlined in the fact that they have runaway-nose-down-trim, and those clows try to compensate it by pulling their stick. The whole horizonal stabilizer still moves to trim, and these clows are using their tiny elevators to combat thw whole stabilizer. Well done boys, and all of that when they see and hear the trim wheel spinning nose down, on and on and on and on. Yupp, that is not how airplanes are flown, and that is why these deadly cases happened to those airlines.
The MCAS software was poorly designed, and contained the means to crash the plane. This has been thoroughly explained in the aftermath of the disasters, except for one thing. I have never understood how it came about that Boeing would give the computer priority over the pilots. It was contrary to previous Boeing design philosophy. This video may help clarify what went wrong.
02:18. Oh yes. I was raised a software engineer. Am now a projects manager. Management doesn't understand software. Full stop. Make box do thing must be easy right?
$9/hr or monthly, $9 x 8 hr x 21 days = $1512 = ₹127k (as salaries are generally paid monthly in India) is actually decent for a software engineer in India. That would likely put someone in the top 10% income earners of the country.
The problem is that they outsourced to another company, HCL which has no context of Boeing's internal code or standards. HCL (and other outsourcing companies) pay like $2 to $5/hr for fresh graduates. Often, freshers join these companies for some industrial experience in their CV and try to switch to other (often product based) companies.
If Boeing wanted to save costs they could've directly employed decent Indian software engineers at like $10/hr and had them trained by experienced Boeing engineers. But, Boeing is a cheapskate and I remember they were offering around ₹800k-₹900k per year or equivalent to $5/hr for fresh Mechanical/Aerospace graduates in India.
They have completely lost their minds.
When you get more money, but you kill or harming people's health, you're committing a crime.
That's oblivious.
$9 is double the minimum wage in Poland, you can get a good software engineer here for that price. The problem is who they chose to hire.
Not a good start that you get basic terminology wrong: an *airline* is a company that operates aircraft to carry out flights. Boeing is not an airline, it is an *aircraft manufacturer.*
Second, the article (from Bloomberg, a company with a poor track record in tech reporting) talks about software for the flight displays, but the flight displays aren’t what caused the MAX crashes.
Third, the widely reported software “responsible” for the crashes, MCAS, was “responsible” in the sense that it carried out the commands, and that it did not have an adequate mechanism to back off when overridden by a pilot. However, the root cause was that the MCAS system, by default, had only one sensor providing it input, flagrantly violating the principle of multiple redundancy that is expected in the aviation industry. That was a hardware limitation, presumably imposed by accountants. (Multiple sensors were available as an optional upgrade.)
Just some Indian context to keep in mind :
India is also the country that runs ISRO, with its cheap, well planned and successful Moon and Mars missions, as well as highly successful commercial space launch industry. And ISRO is cheap, and a lot of ISRO engineers get paid at par with 10USD per hour, which still comes out to a lot of money here (= 10 x 84 x 22 x 8 ~= 150K INR / month)
In my mind, outsourcing to a company without aerospace engineering expertise seems to be the actual issue.
Practically half a dozen aerospace giants "outsource" to India, but they do so in the form of running their own branches in India.
These engineers are paid anywhere between 200k to 1000k per month, which is 2500 - 12000 USD per month.
They're the cream of the aerospace industry.
HCL Technologies is not one of these. It is a body shop.
I felt the need to say this only because a good friend works in a leading American aerospace company in Bengaluru and he is getting awarded patents and certificates in flight safety and aerospace communications systems.
I would also like to add that cheap Indian developers are pathetic. I suffer them in office every day, being an Indian living in India.
I was at Rolls Royce when they were trialling outsourcing like this. Fortunately it wasn’t critical software at that point, and I hope from what I saw from this supplier it never was. I got fed up with them implementing a feature and taking weeks for something that was really simple. I ended up writing it over night and upsetting everyone in the process.
I felt bad for their devs though, they were hours away from home and working insane hours a day
The even bigger problem than the software was tying it to a single sensor.
The MCAS being poorly programmed is a thing that happens. Proper testing can sort that out.
Tying the MCAS to just one sensor that can fail or be disconnected is a major £ūçkup in the design phase.
On top of that, not releasing any technical data to the airlines that bought the aircraft and INSISTING there is absolutely nothing different with the 737 MAX, or the way it operates, when compared to the older 737s...well, that's literally criminal deception and endangerment of the public. MCAS could take over the pitch authority of the aircraft and pilots needes to be informed of it and trained how to troubleshoot it in emergency situations.
Again, the biggest problem is tying this system that can take over flying the aircraft to just a single sensor, especially since the 737 MAX comes equipped with TWO angle of attack sensors.
This is yet another example of professional business managers making engineering decisions, which always leads to disaster.
Consider Fukushima Daichi!
The management decided to stick the backup electrical supply into the basement of the nuclear power plant, since a major tsunami that could overtop the seawall and flood the basement, creating the conditions for it to blow up was considered unlikely and the managers could not justify spending the extra money to secure the emergency power supply on an event they estimated was very rare.
Fukushima Daino, the twin nuclear power plant to Fukushima Daichi was just a few dozen miles down the coast from it and got hit by the same earthquake and tsunami as Fukushima Daichi. The difference was that the engineers in Daino managed to convince the management to put the emergency power supply on a nearby hill, where it was not flooded by the tsunami and it allowed them to do a cold shutdown of the reactors, just as intended. That's what nuclear engineers get for doing the job correctly - nobody ever has need to talk about them. That's why no one ever head of Fukushima Daino, even as Daichi's reactors were exploding just miles away from it.
Dee you forgot 1# important note!!! they sold the....Angle of Attack Software... as an upgrade those had previously flown that type of Plane could typically handle the engine change.
But thanks for the video!!
What is the cost of human life. Apparently $9/hr if you ask Boeing.
that is the equation
That's too much, the figure includes commission of the HCL as well, the end engineer gets only $2-3/hr.
Boeing's management are to blame. They sold the planes with a single angle-of-attack sensor. Boeing charges $80,000 for the second AoA sensor. Without the second sensor the software can not detect the fault. The REAL fault is the engines are too large and had to be mounted forward on the wing, ahead of the center of gravity. Under acceleration this creates torque around the CG and forces the nose up. Without that they would not need the MCAS to begin with.
The word “accident” is reserved for meteorite strikes. The rest are “crashes”. They have non-random root causes. Please. Just like devs use the word “defect” instead of “bug”.
For the most part low cost first wave outsourcing devs are absolutely comparable to their first world counterparts. That is a very though pill to swallow for many I guess.
My dad was an Intel casualty. Monday was his last day after 25 years. They're floundering. The only thing holding them together is the US government because it's the only fab manufacturer left in the US. He LITERALLY told me to get an AMD CPU for my gaming rig build 😭
These corporations don't think long term. A lot of Indian developers on H-1b visas are leaving to go back home. They've truly pooped the bed on this issue. For myself, I am a hobbyist but would love to work as a dev full time eventually. I'm planning to go to Europe...and I'm from California. I grew UP in Silicon Valley more or less and I'm leaving. 😂 It's already biting them in the ass.
When big companies failed to innovate, they did the next best thing -- cut cost to the cheapest buyers -- so execs can grab whatever values before it sinks. And right on about international devs. The problem is not intl devs per se, there are a lot great ones, but the problem is the cheapest they could get for.
The Netflix Documentary is called
Downfall: The Case Against Boeing
Thanks a lot for this! In Norwegian, we have this saying "Å spare seg til fant", which translates to something like "Saving to become a bum". I think this sums up this whole mess ;)
I am from India and everything said in this video is fully correct. Its unfortunate that it has come to this. Its all about money now and less about quality.
When I think about the people I had to work with in Europe, I wonder how much better a project is that fails for €80/hour🤔
Outsourcing is only part of the issue when the main issue when Boeing though that redundancy wasn't needed and used the same software that was validated with the use of redundant instruments.
Not defending a billion dollars company cheaping out on software development and outsourcing in order to give better payout to C-suite executive. but I do feel the issue of removing redundancy is being shadowed by the outsourcing ( where it should have magnified it).
Indian here alot of these consultation firm developers didn't even do cs most are electrical, civil or mechanical grads there's some chemical engineering ones aswell
And fyi senior and even junior dev jobs for serious devs are actually really scarce rn most ppl are gettin 1/4 the package that is if they get an offer at all compared to just a few years ago
An aerospace engineer with coding experience will beat a CS graduate with jack of all experience, any time of the day !
Today's Indian graduates are not interested in writing quality code, they just want nice salary to live a luxurious life. This is why majority of Indian youtube videos are about salary and not skill.
"Boeing must be run as a business by business people to be profitable.", they said.
We gave seen the result.
nice video.. This has been issue for a long time (at least 20 years). I think that the biggest issue in these companies that make big products is that even tho they have millions of lines of code in their system they still think that they are automation company or something like that and they do not understand the value good software. It looks nice on Quarterly report to have lower costs, but nobody knows how much more that costs in long term. More experienced sw developers change role and get more money when they manage these outsourced developers and do less coding. So these things just pile up.
I agree that outsourcing software to the cheapest source will lead to bad software, but the 737 MAX disasters were not caused by faulty software. In both crashes the MCAS software received incorrect data from malfunctioning angle of attack sensors. Another problem is that pilots were not trained how to recover from a malfunctioning AOA/MCAS system.
The root cause of the 737 MAX problems was probably Boeing's attempt to avoid requiring pilots to be retrained on the 737 MAX which had different engines than other 737 variants.
I dont think your assessment is sound. Historically in Boeing and more broadly for effective design implementation, where you have integrated and mature teams, and specifically software developers they will simply know and tell you about redundancy as it is par-for-the-course in IT. Boeings fault here is in the disconnection of functions and downgrading of quality. The quality of the software development team INCLUDES the ability to intelligently see the whole system. Its never not the case even if you arent the designer and its not your domain. Thats what you are actually paying for! And that IS what they got. They walked the intelligent people out the door and people died... holy mother!
The code that is an airliner needs to be safe code. Which means it needs to be high quality among other things.
Outsourcing for cheap labor joyfully rings the CEOs' and others' ears. Unfortunately, it is a regular practice for large corporations. I would not doubt that Boeing used this cheap labor for its spacecraft and mission to the International Space Station. Now, it is stuck up there until 2/25 unless EM rescues it.
I believe it’s not about whether the cost is $9 or $900 or geographical location etc but rather people (that mid and upper management). In my opinion, hiring non-technical individuals in a tech company is a mistake. I understand this view may seem extreme to some, but to truly grasp what I mean, you need to have spent at least a decade working hands-on in the tech industry. And by real work, I mean contributing directly to projects, not simply managing inflated data points and adding extra burdens to technical teams.Once again, the way this issue is being discussed in the video is not how a technical person would present it. Watching this video again reminds me being in a technical meeting, where non tech management spends 50 minutes talking nonsense in a 60-minute call, focusing more on surface-level details than addressing the core problem.
Hold on though: I have worked with a few of those kinds of people and ime they fall in 2 camps: 1) the ones who care: these go out of their way to understand the context and 2) those working to spec: these don’t care and don’t want to know the context. They are there to write code and that’s that. I witnessed a team of 5, 2 of whom were with us in the office and 3 were back in India. Of the 2, it was a 50/50 split but the “team leader” who was with us in the office would want detailed specs to even get started. That’s how a fair few of them actually make money. The actual failure is totally on Boeing here because once they realised what was happening they should have reneged and cut ties.
Analysts, programmers and software validation need people with an understanding of the product and its function to safely update existing software. Contractors can never do the job. Software almost always gets fixed afterward in house.
4:02 Cheap engineers are too expensive. The errors, the rework, ugh.
HCL costs 1/3, but they put 4x people on it and take 2x too long.
'if we do not cut costs, we can not pay our directors...'
I enjoyed this video and will watch for others from you. But I'm surprized you didn't also mention the space capsule "Boeing Starliner" with its failure to reach the Int'l Space Station in Dec 2019, and with the helium leakage problem which prevented the 2-astronaut crew from returning with the capsule last month. You could also have mentioned that Boeing's merger with McDonald Douglas in 1997 has been blamed by some for a change in corporate culture which shifted the emphasis away from engineering excellence to cost-cutting measures.
Those big engines are a big mismatch to start with. The whole sensors/software/fly by wire problem has some issues that have come up in the past. Sometimes the software actually helps the situation, some times it totally wrecks things. You should never have a situation where it is fundamentally impossible to control a crashing plane though.
The hourly rates at 5:25. For what kind of SWE are the for. Those numbers make no sense without more context.
Maybe for PHP or low end web dev. But those numbers are way too low.
Most of the coding done in those forsaken third world countries is putrid and is a reminder why we should be cautious as to what is outsourced.
$9/hr is still a lot… you can get senior developers(5-7 yrs exp) lined up for these price in India.
Anyone watching from India, its ~1.33 L per month [$1 = Rs84 at the time of calculation ]
I hate outsourcing of any kind but having said that there is only 1 thing that can automatically control the pitch of an airplane and that is the trim, you learn this from your first flight in a cessna 152. in the 2 737 max crashes the aircraft kept pitching down un-commanded. Meanwhile a couple of big wheels the size of a dinner plates were spinning away right next to the throttle levers. these dinner plates are the trim wheels. All the pilots had to do was turn off the automatic trim, it should not have been hard to diagnose since the ONLY thing on the entire aircraft that can cause an un-commanded pitch down is the automatic trim.. Boeing screwed up big time, and outsourced developers is a horrible idea, but make no mistake those pilots killed all those people no one else.
All big companies outsource a part of the coding work, because there are just not enough engineers in the local market. And in almost all of the cases the outsourcing works well.