A Cultural Laziness?

  • 2
  • Idea
  • Updated 2 months ago
From a recent IEEE article comes this interesting observation:

I believe the relative ease—not to mention the lack of tangible cost—of software updates has created a cultural laziness within the software engineering community. Moreover, because more and more of the hardware that we create is monitored and controlled by software, that cultural laziness is now creeping into hardware engineering—like building airliners. Less thought is now given to getting a design correct and simple up front because it’s so easy to fix what you didn’t get right later.

Every time a software update gets pushed to my Tesla, to the Garmin flight computers in my Cessna, to my Nest thermostat, and to the TVs in my house, I’m reminded that none of those things were complete when they left the factory—because their builders realized they didn’t have to be complete. The job could be done at any time in the future with a software update.

https://spectrum.ieee.org/aerospace/aviation/how-the-boeing-737-max-disaster-looks-to-a-software-dev...


Photo of Dave - W6OVP

Dave - W6OVP

  • 304 Posts
  • 96 Reply Likes

Posted 2 months ago

  • 2
Photo of Doug - K3TZR

Doug - K3TZR

  • 115 Posts
  • 15 Reply Likes
Dave,

As a retired software developer I'm very aware that more and more of life is controlled by software. Unfortunately, some of what is software controlled is also life critical (e.g. airplanes). To design a bridge you need an Engineering degree, a PE certification, etc. To design software, there's no similar set of requirements. In addition, the "triple all your calculations and nothing will ever break" thinking of civil engineering has no counterpart in software engineering. One errant line of code can "crash" the entire system.

It's a serious problem deserving of immediate attention. As the article points out, money may be in the way at times.
Photo of Reggie

Reggie

  • 91 Posts
  • 8 Reply Likes
Just because there are certifications and standards do not mean civil engineers do not make mistakes with their bridge designs.  Last year, a newly built bridge collapsed in Miami, FL.  Medical doctors are educated for years, pass multiple certifications, and do years of residency, but they are not infallible.  They kill people every day.  Software engineering is no different.  Software engineers can run simulations, they can do alpha testing, and beta testing, but there are always bugs because no complex software program is perfect.  Look at Obamacare website.  They spend nearly $2 billions dollars creating that website, and it had major issues.  Sometimes money is not the issue.
Photo of Doug - K3TZR

Doug - K3TZR

  • 115 Posts
  • 15 Reply Likes
Reggie, I think you are correct; standards don't stop these types of failures. Nothing will ever be perfect but the software development world is the "wild west" in many businesses. In my experience, hardware companies are the worst, their view is that software is an unfortunate need to be dealt with as expeditiously as possible.

As we are in the process of inserting software into every aspect of life I'm hoping that the 737 fiasco will be a wakeup call that some safer, more thoughtful approach is needed.
Photo of James Del Principe

James Del Principe

  • 304 Posts
  • 45 Reply Likes
Dave, based on your examples, it would be easy to agree with you and in many respects I do. That said, however, it is tempting to confuse an upgrade or added feature, feeling that it should have been provided in the first release. The problem is the nature of software. It can always be improved with features added. Then a developer must make a decision. When should a product or SW release be rolled out?   Should it have every possible feature?  In that case it would never see a customer.  Remember that perfection is the enemy of the good.  So I really don't disagree with you but considerations must be taken.     Best 73, Jim
Photo of James Charlton

James Charlton

  • 41 Posts
  • 7 Reply Likes
A fundamental law of coding:
Users = beta testers

or, let the customers find the bugs, they paid for that right  ;-)
Jim Charlton  AD0AB
Photo of James Del Principe

James Del Principe

  • 301 Posts
  • 44 Reply Likes
James, I just had to chuckle. In my past life, I spent 43 years in medical x-ray for a manufacturer.  We did alpha testing in house and beta testing for selected customers (volunteers).....still had issues over time with weird combinations of usage or even certain dates....a September 1 fault and a leap year fault......  customers were NOT amused.
Photo of James Charlton

James Charlton

  • 41 Posts
  • 7 Reply Likes
Hi James,
Thank you.  Another fundamental rule in the software business is:  "Never buy version 1.0."

In the interest of full disclosure, I've been both a software writer and tester, but that doesn't stop me from making fun of my fellow code-monkeys.

I believe there is some theory that says no program can never be proven to be bug-free.  As you point out, there are so many combinations of inputs etc that it is almost impossible to test them all.  

One method of testing was to set up a test bed and start the new code and measure how long it took for a bug to reveal itself.  At first, that Time to Failure was measured in seconds.  But as fixes were implemented it stretched out to hours and days.  So, the question wasn't is the software bug free, but how long do we test before it can be released?

That was a management, not a technical, decision.

I'm not trying to justify sloppy or buggy code.  Now that I'm retired and out of the business of writing and testing it I say that all code should be signed by the author so we can find them and take away their keyboards!  Better yet, their phone numbers should be published so we can call them at night and weekends then their stuff doesn't work!

That was a lot of work, now it's time for my nap.
73,
  Jim Charlton  AD0AB
Photo of Peter K1PGV

Peter K1PGV, Elmer

  • 543 Posts
  • 321 Reply Likes
The author is right... software these days is sloppy. But he ascribes the cause to”laziness” which is not anywhere close to correct, in my observation.

The abysmal state of software quality,, in my opinion, primarily has to do with three things:

1) A general acceptance, in the software dev and user communities, of bugs. People have been conditioned to expect that software doesn’t work correctly all the time. The make allowances for, and accept, that “well, THAT feature is obviously broken.” A lot of this inadvertent training comes from the web, where web based apps as frequently buggy in one or more browsers.

2) Slavish adherence to agile processes, which are often poorly understood and very badly managed. My antipathy toward agile for the type of software that I write professionally (operating systems) is well documented. For projects where agile is used, it is most often run by people who are clueless as to how it’s supposed to really work, and mostly serves as a DISincentive to carefully design, test, and perfect your code. If I had a dollar for every time I’ve head some idiot say “Just get something working and we’ll fix it in the next sprint” I’d own a PGXL today. Shipping software that’s not fully cooked is the definition of agile. When you couple this with the new trend of flighting releases and “testing in production” you get a whole culture based being proud to ship software that doesn’t really work.

3) A lack of value, or even understanding, of software architecture and design. I could write a whole article on this. It dovetails with the introduction of agile, also. With the push to “just get something working” nobody values, and few people understand, defining an overall architecture for a system. There’s little architecture done these days. Design is not architecture.

So... it’s not laziness. It’s worse than laziness. It’s failure of the underlying systems of how most software is written. It’s conflating “I through some crap together and it seems to work pretty well” with “I designed and implemented a solution.”

Peter
K1PGV
Photo of KC0EM

KC0EM

  • 85 Posts
  • 9 Reply Likes

Perhaps, I am a bit too old to add to this discussion, but what I see as part of the problem is the lack of connection between the person doing the programming and the end user of that program. My first job was working in a postdoctoral research laboratory. There was a mixture of EEs, physicists, and other scientific professionals. One of the major problems was handling all of the data being generated in the experiments. I remember seeing high-speed printers turning out foot tall piles of paper that were nothing more than numbers. Some members of a team would want to look for patterns in the printouts. Some members would want to test theoretical equations against the printouts and so on. If you wrote a program to test a theoretical equation against the data set and it didn't work, you had to look that researcher in the eye. It was tough enough to see the disappointment when a theoretical equation didn't work out that you didn't want your work to make even a bigger mess of things. A programmer today never gets to see the end-users disappointment when their programming efforts fail or go awry.


Footnote: I was working there in the 1970s. My biggest hit in programming was one that allowed the user to change the strategies in card counting to maximize their odds playing blackjack. I was surprised by how many scientists are gamblers.

VC
KCØEM
Photo of James Del Principe

James Del Principe

  • 301 Posts
  • 44 Reply Likes
I think you are on to something here. Too much of a disconnect between developer and end user. BTW, you are never too old to contribute. In my case, I am old on the outside but young on the inside.
Photo of k0eoo

k0eoo

  • 612 Posts
  • 86 Reply Likes
I disagree Dave, Every product I ever developed and shipped was working perfect, as per the original design.  Unfortunately, after my many customers got a hold of my products and ran them in ways I never anticipated, problems cropped up that needed to be fixed, some by hardware but mostly and hopefully by software...  As a designer/developer, fixing problems via software/firmware is so much cheaper/faster and better for the customer than via hardware, consequently, all my designs started going in that direction.  Big S little H....  YMMV

Dennis, k0eoo
Photo of Steve Sampson

Steve Sampson

  • 4 Posts
  • 4 Reply Likes
People - and their desire to make things work properly - hasn't changed.  Systems today are so very much more complicated than any historic levels (and I've run a number of developmental programs), the opportunities to make - or miss - mistakes pushes human ability.  Complexity and perfection don't play well together.   
Photo of Craig Williams

Craig Williams

  • 201 Posts
  • 64 Reply Likes
Resistance is futile. 
Photo of James Del Principe

James Del Principe

  • 304 Posts
  • 45 Reply Likes
but......capacitance is reactive.
Photo of Pat - WH6HI

Pat - WH6HI

  • 354 Posts
  • 66 Reply Likes
A lot that goes on in product design is driven by the marketing side of the business. I think that this was something that will be shown in the Boeing incidents. The desire to compete with the competition on price some times trumps the engineering side. In this compromising safety, with unintended or on educated sales force actions. On the engineering side trying to get the software to account for all conditions no matter how trivial has consequences. Evaluation is difficult and time consuming and can be costly the more complex the engineering. Is.
Photo of Erika - KØDD

Erika - KØDD

  • 4 Posts
  • 4 Reply Likes
I'm with Dennis K0EOO on software design verses end users quandary.

The 737 Max Series Aircraft are NOT Tesla Autos, Nest thermostats, Garmin or Flex Radios.  The problem here is a CURRENT design of 737's the 700, 800 and 900's were confronted with AIRBUS' competing product that was smoking the Boeing in fuel economy.  When you're burning thousands of pounds in fuel during a flight, Better Gas Mileage is Important at keeping an airline's cost per seat mile as low as possible.

Well a NEW engine, an even higher bypass turbo fan was developed and to bolt a pair of those units on to the 700-800-900 airframes Boeing needed to extend the landing gear, and shove the engines closer to the wing and further forward.

The end result is an aircraft that additional thrust causes the NOSE of the aircraft to rise increasing the angle of attack...  They had to implement this MCAS system to allow the aircraft to FEEL LIKE the previous versions of the aircraft so pilots could move between them without feeling a change.

The airlines just need to implement differences training between the New Generation 700 etc and the Max...  The system allows the Airframe to handle LIKE the previous airplanes.  Here's the wording.

****The Maneuvering Characteristics Augmentation System (MCAS) is a flight control law managed by the flight control computer (FCC) and introduced on the 737 MAX to help it handle like a 737 Next Generation (NG), particularly at slow speeds and high angles of attack (AOA).****

The problem is the dual Angle of Attack Sensors were optional even with the MCAS (they made them mandatory now... but the pilots still need training if an AOA sensor fails and how to diagnose that.) Unfortunately there were a couple aircraft that fell out of the sky due to the pilots not receiving enough simulator and diagnostics training along the way and had no clue what was happening.

I watched an in cockpit simulation of the second crash with discussion while it was happening and I was unable to follow why the A/C pitched sharply down and everybody died...

I stepped out of an airline pilot's seat in 1993 and admit being rusty however I'd never felt like I was out of control for no reason in perfect flying conditions before, I was able to follow their flight right up to the point the pilots started freaking out.  Of course I couldn't feel the plane fighting at the controls. DUH?

This IS nor WAS ever a software issue...  This in both cases had been an AOA Sensor failure issue.  Even on the twin sensor systems, the pilots NEED to be trained at How to diagnose which sensor has failed and which is correct for them to properly fly the plane.  Do some of that Pilot S*** Mav...  I do believe these pilots didn't get enough training to safely fly these birds.

These ARE NOT software controlled airplanes...  So the problem isn't bad software the problem is/was inadequate training.  Boeing is NOW adding lines of code to anticipate undertrained pilots???

What about SDR?  Advances and revisions in SDR software if an issue was found and a fix was designed for that issue that's one thing... Adding additional features that's different, Repairing an issue with the original hardware?  Doubtful.

Even with an SDR radio $$$$ must be spent in the original design to build components in the Signal Path like LO's, and A/D conversion devices at the absolute leading edge of technology...  and then designing software that anticipates every possible feature for each of those devices to please the customers  When You're talking new technology it would take Johnny Carson as Carnac the Magnificent  to foresee every possible vision the customer might come up with or future need so the processor can deliver at the design onset.

Be pleased they keep updating our software....  My windows 10 Pro x64 still sees a blue screen every once in a while...  Microsoft KNOWS there's thousands of potential issues running around in these computers...  NOBODY is smart enough to design a software O/S that complex with the first revision.

Sincerely,

Erika KØDD
Photo of Pat - WH6HI

Pat - WH6HI

  • 348 Posts
  • 65 Reply Likes
I think in this case the 737 MAX was designed to have the complete package installed. Eliminating part of the package had unintended results. Yes training is a factor, and not doing proper training exasperated the problem. This is why Boeing is installing the complete package with sw updates and requiring more training to insure that there is no repeat.
(Edited)
Photo of Mark  K1LSB

Mark K1LSB

  • 83 Posts
  • 27 Reply Likes
"I think in this case the 737 MAX was designed to have the complete package installed. Eliminating part of the package had unintended results."

Pat,

What part of the "package" didn't Boeing install?
Photo of Jay / NO5J

Jay / NO5J

  • 1655 Posts
  • 254 Reply Likes
Mark 

https://en.wikipedia.org/wiki/Boeing_737_MAX_groundings#Maneuvering_Characteristics_Augmentation_System_(MCAS) 

https://en.wikipedia.org/wiki/Maneuvering_Characteristics_Augmentation_System

It's been reported that Boeing was selling an economy 737 MAX package, to some of their customers that didn't include redundant angle of attack sensors, which are included in the regular more expensive packages.

    SDRgadgets

#FlexRadio IRC Chat

   73, Jay - NO5J

Photo of Mark  K1LSB

Mark K1LSB

  • 83 Posts
  • 27 Reply Likes
Thank you, Jay.  Now that you mention it, to Pat's credit I do remember now reading somewhere last week about a less-than-complete package that Boeing offered as a cost-saving option.

According to the article in the IEEE link in the OP, there were AOA sensors on both side of the aircraft but the two flight control computers were each wired only to the AOA sensor on their respective side of the aircraft.

Thanks again,

Mark  K1LSB
Photo of Erika - KØDD

Erika - KØDD

  • 4 Posts
  • 4 Reply Likes
Mark I believe we're talking about the optional DUAL Angle of Attack Sensors vs a Single AOA Option for Less $$$  plus the little indicators lights on the displays stating the two sensors are NO LONGER IN AGREEMENT and HELLO its time to figure which one is really working...

I believe any additional code they write will be an attempt to make the Troubleshooting and Notification Process easier for ALL PILOTS not just the Smart Ones with advanced diagnostic skills or the proper Simulator Training..

No Matter what, this last crew correctly disabled the MCAS system briefly I bet in a guess or by golly move...  Maybe hoping the aircraft attitude would magically self correct for some reason I suspect. (It's not going to.  They needed to retrim the bird themselves) 

Unfortunately when the A/C trim didn't magically self correct they flipped the thing back on and the Pitch Trim did a runaway nose down situation, resulting in a Big Hole in the Desert with innocent victims burning and tossed haphazardly around the Crater.  What Can I do at this point other than shake my head.

In the meantime Boeing isn't shipping 737 Airplanes and are hemorrhaging dollar Signs by the billions.

One thing I DO want to state...  Between Airbus and Boeing they've had each a share of fatal crashes on the 737 and Airbus Equivellent Unit...  737's had worn out trim screw bushings in the tails and resulting crashes, and Airbus had the verticals just plain old breaking off and crashing.  This size aircraft is not only the most common, but are flown with the most takeoff and landing cycles... Have a nice day.

Erika DD

Photo of Jay / NO5J

Jay / NO5J

  • 1655 Posts
  • 254 Reply Likes
Mark 

This economy package also eliminated, an optional pilot training program designed to train 737 MAX pilots about handling situations where the MCAS system malfunctioned. 

     SDRgadgets

#FlexRadio IRC Chat

   73, Jay - NO5J