> repurposes the high-speed commercial processor inside the Ingenuity Helicopter Base Station ... This radiation can induce “transient bit flips” in memory the software needs to function, and the team developed an effective software-based approach to radiation hardening
Anyone know what they're doing?
It sounds like a single processor, so no best-of-three or similar systems. Software solution only, possibly running on the same bitflipping CPU that's being checked. I'd be very interested to know how it's solved.
This is interesting and I googled a bunch of this.
The helicopter base station is an off the shelf snapdragon 801 cpu with 4 cores. By comparision the CPU in the rover is a BAE RAD750 processor at 200hz, essentially a radiation hardened 1998 G3 imac chip. They run calculations on each snapdragon core and if there is a consensus take the value. I don't know how their voting system works. If core 1 and 2 agree and 3 and 4 don't, does it just agree with core 1 and 2's consensus if 3 and 4 don't match? Or does it was 3 minimum and re-runs the calculation.
It also performs aggressive memory scrubbing. This sounds like software ECC by performing various memory checksums by comparing values to the flash drive's files. So if script.sh on disk doesn't match script.sh in memory, they know the memory is bad. I guess they are assuming flash will always be correct, which is interesting, but in space maybe its just a 'pick a lesser evil' thing because flash can also be corrupted by radiation. They store critical files with triple redundancy on the flash, so the flash sounds pretty rugged here. There is also a secondary radiation hardened flash storage 'golden image' the unprotected snapdragon's running flash can be compared to.
Then the software identifies damaged memory addresses and blocks them. One example is the software found 25 corrupted bits in its 1gb memory and has marked them as unusable. I'm guessing over time more and more bits will be unusable as radiation permanently destroys those transistors. So its a bit of a race against the clock. Better to use the base station until it can't be used anymore.
Ingenuity does have an RT ProASIC3 (an older rad hard FPGA) on board so presumably that could be what's handling the voting. Since the Snapdragon was originally used for navigation and operates much, they could be voting at a higher level rather than actual CPU output. When it was still a helicopter, the various sensors and actuators all connect to the FPGA so the voting could be based on commands to the actuators (turn left, reduce power, etc.)
I asked Gemini the parent question and got a very comprehensive answer. I didn't want to cut/paste it here because the villagers have torches. Your summation is very good and took some effort, thanks!
> After running thousands of simulations, the team raised the rover’s tilt limit to 24 degrees and reduced the clearance required under the rover’s belly pan to 20 centimeters (about 7.9 inches). The team also updated the rover’s braking software to avoid rock hazards, nearly doubled steer motor current limits, updated flight software with “dynamic slip tables” to model expected slip, and used cross-slope driving to better anchor wheels on exposed bedrock.
I wonder why they didn't model and use these parameters in the original design (I'm not at all suggesting a mistake on their part; I wonder what their thinking has been):
Were Martian environmental parameters less well-known (e.g., how gravity, air resistance / wind, and scree dynamics interact)? Bot performance in Martian environment (e.g., motor power output) less well-known? Were they being understandably conservative with their irreplaceable bot?
> Were Martian environmental parameters less well-known (e.g., how gravity, air resistance / wind, and scree dynamics interact)? Bot performance in Martian environment (e.g., motor power output) less well-known? Were they being understandably conservative with their irreplaceable bot?
I haven't worked on the Mars rover programs, but I have worked on a NASA moon rover. Based on that experience I would hazard a guess that it's all three of these things.
Environment: We actually have quite a bit more experience roving on Mars so I expect the environmental uncertainty is a bit attenuated for the Mars missions, but one thing I was really surprised by during my time on the moon rover mission was just how little confidence our scientists had about the physical conditions at the specific landing site we are targeting (which is quite far from the Apollo missions for example)
Performance: Building accurate hardware tests on Earth for operations on another planetary body is incredibly hard. Of course NASA engineers are smart and can do this with fairly high confidence, but it's far from perfect. For the moon's 1/6th gravity, we couldn't just build a copy of the moon rover and test it on Earth because the suspension would collapse, so we had a whole separate copy of the moon rover mobility and suspension that was built to operate in Earth gravity yet mimic the performance the actual rover would have on the moon. That unit in and of itself is a feat of engineering that is definitely not immune to mistakes.
More data can support changing performance limits. They also may have been more conservative in the past when the rover was new because they wanted to ensure it met its mission requirements. Since the rover has long past its original mission duration (1 year, it's been 5), they can relax a bit more and take bigger risks to achieve bigger goals.
On one hand, you have to care for the hand that feeds you. On the other, I'd love for NASA to basically ignore the orange convicted felon and see what happens.
> repurposes the high-speed commercial processor inside the Ingenuity Helicopter Base Station ... This radiation can induce “transient bit flips” in memory the software needs to function, and the team developed an effective software-based approach to radiation hardening
Anyone know what they're doing?
It sounds like a single processor, so no best-of-three or similar systems. Software solution only, possibly running on the same bitflipping CPU that's being checked. I'd be very interested to know how it's solved.
This is interesting and I googled a bunch of this.
The helicopter base station is an off the shelf snapdragon 801 cpu with 4 cores. By comparision the CPU in the rover is a BAE RAD750 processor at 200hz, essentially a radiation hardened 1998 G3 imac chip. They run calculations on each snapdragon core and if there is a consensus take the value. I don't know how their voting system works. If core 1 and 2 agree and 3 and 4 don't, does it just agree with core 1 and 2's consensus if 3 and 4 don't match? Or does it was 3 minimum and re-runs the calculation.
It also performs aggressive memory scrubbing. This sounds like software ECC by performing various memory checksums by comparing values to the flash drive's files. So if script.sh on disk doesn't match script.sh in memory, they know the memory is bad. I guess they are assuming flash will always be correct, which is interesting, but in space maybe its just a 'pick a lesser evil' thing because flash can also be corrupted by radiation. They store critical files with triple redundancy on the flash, so the flash sounds pretty rugged here. There is also a secondary radiation hardened flash storage 'golden image' the unprotected snapdragon's running flash can be compared to.
Then the software identifies damaged memory addresses and blocks them. One example is the software found 25 corrupted bits in its 1gb memory and has marked them as unusable. I'm guessing over time more and more bits will be unusable as radiation permanently destroys those transistors. So its a bit of a race against the clock. Better to use the base station until it can't be used anymore.
Ingenuity does have an RT ProASIC3 (an older rad hard FPGA) on board so presumably that could be what's handling the voting. Since the Snapdragon was originally used for navigation and operates much, they could be voting at a higher level rather than actual CPU output. When it was still a helicopter, the various sensors and actuators all connect to the FPGA so the voting could be based on commands to the actuators (turn left, reduce power, etc.)
I asked Gemini the parent question and got a very comprehensive answer. I didn't want to cut/paste it here because the villagers have torches. Your summation is very good and took some effort, thanks!
> After running thousands of simulations, the team raised the rover’s tilt limit to 24 degrees and reduced the clearance required under the rover’s belly pan to 20 centimeters (about 7.9 inches). The team also updated the rover’s braking software to avoid rock hazards, nearly doubled steer motor current limits, updated flight software with “dynamic slip tables” to model expected slip, and used cross-slope driving to better anchor wheels on exposed bedrock.
I wonder why they didn't model and use these parameters in the original design (I'm not at all suggesting a mistake on their part; I wonder what their thinking has been):
Were Martian environmental parameters less well-known (e.g., how gravity, air resistance / wind, and scree dynamics interact)? Bot performance in Martian environment (e.g., motor power output) less well-known? Were they being understandably conservative with their irreplaceable bot?
> Were Martian environmental parameters less well-known (e.g., how gravity, air resistance / wind, and scree dynamics interact)? Bot performance in Martian environment (e.g., motor power output) less well-known? Were they being understandably conservative with their irreplaceable bot?
I haven't worked on the Mars rover programs, but I have worked on a NASA moon rover. Based on that experience I would hazard a guess that it's all three of these things.
Environment: We actually have quite a bit more experience roving on Mars so I expect the environmental uncertainty is a bit attenuated for the Mars missions, but one thing I was really surprised by during my time on the moon rover mission was just how little confidence our scientists had about the physical conditions at the specific landing site we are targeting (which is quite far from the Apollo missions for example)
Performance: Building accurate hardware tests on Earth for operations on another planetary body is incredibly hard. Of course NASA engineers are smart and can do this with fairly high confidence, but it's far from perfect. For the moon's 1/6th gravity, we couldn't just build a copy of the moon rover and test it on Earth because the suspension would collapse, so we had a whole separate copy of the moon rover mobility and suspension that was built to operate in Earth gravity yet mimic the performance the actual rover would have on the moon. That unit in and of itself is a feat of engineering that is definitely not immune to mistakes.
Thank you!
I assume everything was very conservative and safe back during initial design.
Now they have a ton a data and a long unimaginably successful mission they can take a few more risks.
More data can support changing performance limits. They also may have been more conservative in the past when the rover was new because they wanted to ensure it met its mission requirements. Since the rover has long past its original mission duration (1 year, it's been 5), they can relax a bit more and take bigger risks to achieve bigger goals.
>generative SI
Eyeroll. Terrible that NASA must kowtow to the orange moron.
The S stands for slopcore
On one hand, you have to care for the hand that feeds you. On the other, I'd love for NASA to basically ignore the orange convicted felon and see what happens.