Some SD-CARD Observations associated with V3.2.39 / V3.3.29

Dwayne_AB6A

1. The migrations from Delkin 256MB SLC (60,000 cycles) card to a Delkin 4GB MLC (3,000 cycles) card as specific in Delkin documentation would mean the 4GB card actually provides a little less wear level endurance.
2. The change to EXT2 to EXT3 sounds attractive since it is now a journaled file system and can protect against crash consistency... Based on the various reports from users and Flex comments, wear leveling appears to be the issue and not crash consistency. Due to the need to maintain a "journal" in EXT3, write IO actually increases 2x since you now need to write the journal together with the data block. This would increase SD-CARD wear.
3. The industry recognized that current file systems don't address the needs of flash devices and have addressed this with a focus on a new file system type: Flash Friendly File System (F2FS).
4. Unrelated, numerous comments from Flex saying the downgrade form V3.3.29 to V3.2.39 retains the file system "benefits" introduced in V3.3.29 are incorrect. The partition size changes are retained, but the root partition for the Angstrom OS is reverted back from ext3 to ext2.
5. How others have mitigated this issue is to minimize file system writes with the addition of a small RAM disk and to move move hot IO to utilize the RAM disk.

If Flex was addressing SD-CARD crash consistency, inadvertently, Flex has increased wear up to 2x as a byproduct. Likewise, if Flex was addressing SD-CARD wear leveling, Flex has increased the wear 2x and it is not mitigated by using a larger capacity SD-CARD since the card chosen has significantly less wear level protection than the card it replaced.

To recover V3.3.29/ext3 4GB MLC to the same wear level protection of the V3.2.39/ext2 256MB SLC, an 8GB Delkin MLC card would be required, or go back to a larger capacity SLC cards with the 60K cycle capability.

Eric-KE5DTO

(1) You are correct that the larger card is rated for less cycles (even assuming we only use 256MB as this is a 16x multiplier on the available cycles -- still 48k vs 60k cycles). However, this is not something we are concerned about.

Allow me to explain: When we qualified these SD cards, we profiled our usage of the card on the radio. Typical operation barely shows up at all when measuring card cycles used compared to a single software update. This is because we made efforts to only hit the card in order to write to the database and we cache any updates there to only happen a few times a minute. This is to ensure that if you lose power or have some other issue that requires a reboot that you pick up at least close to where you left off.

So a software update exercises the card in the same way that many years of typical operation does. Knowing this allowed us to setup a brutal test that requires any SD card that we qualify to endure at least 3000 software update cycles before it is considered. This takes over a week to complete uninterrupted.

While it is anecdotal, it is worth mentioning that the same SD cards that I used to qualify that were put through this test are the ones that I use in my radios on my desk every day when testing and developing software.

We also sent a number of cards to the manufacturer in the first round of the issue before we identified the CPU errata issue. They confirmed that the failed cards from the field still had at least 95% cycles left after several years of use.

All of this says that wear leveling is not the issue and THUS, we aren't super concerned about giving up some of the rated cycles. Keep in mind that this is assuming that SLC cycles and MLC cycles can be compared directly.

(2) You are again correct. There is certainly some overhead for the journaling in ext3. However, because of what I described in (1), this is a really good tradeoff to ensure data integrity, which is our primary concern. This should give us more confidence when we do end up writing to the SD card, even though that is pretty sparse in typical operation.

(3) I had to google F2FS as I wasn't familiar with this (I'm not a linux or file system guru by any means). Thanks for mentioning this. I'm sure some of our team would be more familiar with this and they may want to comment.

(4) You are correct here. This is my fault as I gave bad data to Michael on this subject. This is why they keep me out of linux land. ;) When downgrading to a version prior to SmartSDR v3.3, the file system does indeed revert to using the less robust ext2 instead of ext3 that is used in SmartSDR v3.3+.

(5) We do actually use a RAM disk to help minimize writes to the card. This does actually help us minimize the card write impact during an update. It would otherwise be around twice as many cycles. We don't keep the database on the RAM disk though as this would mean losing recent settings if you lost power.

Thanks for all of the thoughtful comments and excellent questions. We appreciate it.

Mike-VA3MW

@Eric-KE5DTO Thanks Eric for taking the time to provide such a detailed response.

Dwayne_AB6A

Thanks Eric,

While I appreciate the position Flex has taken on crash protection vs wear leveling, many comments from Flex detail "irreparable damage" to SD-Cards both in reference to previous software updates (the journey to V3.2.39) together with a large number of failures associated with this update.

1. What does Flex mean by "Irreparable" damage and why so many failures since there is a huge difference between "Irreparable damage" and a corrupted file system?
2. Flex discussions regarding the recent failures talk about "sanity checks". It seems that attempts are made to resize partitions and update the file system from ext2 to ext3 on the second partition fail leaving the partitions corrupted. Since Flex has stated wear leveling is not the issue, why not simply improve the update process to copy key files (id etc) to the first exfat partition and delete/format/reload the two Linux partitions.
3. In the context of crash protection, would it not be in Flex's best interest to focus on resolving OS crashes in the first place? It could do with some patching right?

Al_NN4ZZ

Dwayne,

Great information and a lot more about the SD card operation than I was aware of. What do you think about the change from EXT2 to EXT3 and the other impacts from changing to the 4GB card for those of us on V2.x?

It is not clear whether Flex plans to make those changes for the V2.x users. If they migrate to the 4GB card for V2.x radios but don’t change from EXT2 to EXT3, will the failure rate go up?

It sounds like the most reliable solution for the V2.x users would be to adopt both changes, does that sound right? Maybe that is the plan but I know Flex typically does not disclose planned changes until they are ready.

Thanks, Al / NN4ZZ

WX7Y

Just my thought's, With Version 2 being locked on any new features, Why does it need to be updated to EXT3.

Version 3 is NOT "new feature locked" so FLEX has already said that for the future updates and features they need to update to EXT3 to accomplish what the future may bring.

FLEX knows exactly how many Version 2 SDCards have failed and how many Version 3 SDCards in service so evidently in comparison the Version 2 hasn't lost near as many SDCards as Version 3 no mater what the Age and usage of the SDCards have on them.

Not sure why people can't understand this simple thing.

BE CAREFUL what you wish for, you might just get it along with all the Bugs and issues that comes with it like we have had with version 3.

No matter what FLEX does people grip at them for it, no wonder we fill we are at the bottom of the totem pole with updates and software progress, it's our own doing for not knowing were the software is headed.

good luck

Al_NN4ZZ

Bret wrote: Just my thought's, With Version 2 being locked on any new features, Why does it need to be updated to EXT3. 

Hi Bret,

Perhaps incorrectly, but I got the impression that the change to EXT3 was being done to reduce the number of SD card failures. So it is a maintenance fix rather than a new feature and those types of updates are still available to V2.x.

BTW, although I don’t want or need any of the features in V3.x, I would be happy to upgrade to V3 once the bugs are fixed and they add the Profile Autosave On/Off switch option. Like a number of others I’m still on V2.4.10 to avoid the profile autosave change.

73, Al / NN4ZZ

Al_NN4ZZ

Bret wrote: No matter what FLEX does people grip at them for it, no wonder we fill we are at the bottom of the totem pole with updates and software progress, it's our own doing for not knowing were the software is headed.

Hi Bret,

About NOT knowing where the software is headed. Back a number of years Flex did provide road maps but no longer provides guidance on the future plans. We only learn about what is being delivered in an update when it happens. I understand why but it makes it hard to know or guess the where the software is headed.

73, Al / NN4ZZ

WX7Y

Yep exactly I've been running Flex radios since 2008, with the GREAT PowerSDR software which is leaps and bounds beyond SmartSDR in some area's like EYE CANDY other then the ability to have Multi and Remote USER capability.

We used to get updates some times 2 or 3 times a day to play with and give our feed back with PowerSDR, then people started complaining and that was end of it.

NOT TO POINT ANY Fingers but People just don't have a clue what ramifications of OUR negativity does to widdle away at FLEX wanting to do anything for us, for instance looking at the leaps and bounds that PowerSDR improvements have came being partly Public domain software, look at Thetis-X64 for Ethernet SDR's and KE9NS for Firewire Radio's.

I never hear or see people complaining about that Software and it's development but positive responsiveness giving the Software writers incentive to WANT TO DO MORE and keep their Software Free.

SmartSDR still has a huge lead in the market place for Remote and Multiuser functionality for now so my point is find the GOOD things that they are doing and the Support we DO get from them and tell them once in a while.

I for one give FLEX a huge THANK YOU for Doing the work on EXT3 SD Card with OUT crashing EVERY radio out there and NOT charging us a penny for OUT OF WARENTEE Radio SD Cards like mine.

If you have ever tried to modify a Operating partition with out killing it on a running Partition I take my hat off to them, and glad to see they are getting ready for the NEXT big thing (I HOPE) that requires a EXT3 file system at least it's a step in the right direction.

I lost SD Card's in my 6300 and 6700 radios but worked on my 6600M so been there and done that and a BIG THANKS again FLEX Radio for getting me back up and running in a matter of Days.

I'm sure people will flame me but what ever turns their crank, Help make it better or help make it worse the decision is up to us all.

EA7JQA

Man, that the radio is left hanging due to a software update recommended by the brand itself... There can be no other answer, other than the one it has had from Flex... The equipment and software are not exactly cheap... It is their obligation if they want us to continue trusting them.

I only hope that what happened is to make a leap in terms of the development of Smart SDR, because the truth is that for me the update has not meant anything new that I observe, it is even more so that the version for Mac has not been updated.

If it would be convenient to consider a software update, since I think that there is free software that is being developed such as Power sdr used for the old versions of flex that improve many aspects that smartsdr does not have, not to mention the difference between the version for windows and the one developed for Mac...

For me what happened would have no other explanation than the possible arrival of an update to a possible version 4 of SmartSdr, otherwise what happened with the cards is not really explained.....

Another thing I don't understand is why this thread is not at the top of the comments made in the community...

Al_NN4ZZ

Ea7jqa wrote “Another thing I don't understand is why this thread is not at the top of the comments made in the community...”

I asked the same question…see this post and maybe some feedback.

https://community.flexradio.com/discussion/8026488/community-posts-and-listing-by-date

https://community.flexradio.com/discussion/8026488/community-posts-and-listing-by-date#latest

Dwayne_AB6A

Just an update. Looks like Flex Radio moved from the Delkin cards to the SanDisk Industrial card... 8GB as well. You are welcome Flex Radio.

Regards Dwayne AB6A