Hi Ian, here a few tips you can try out. 9:14 disconnect the external interface on the backside, a faulty component there can be the issue. 10:45 the SPI header is usually (as you probably know) connected 1 to 1 to the 8 pin IC, where the first 2 pins of the header are not used. If this is the case, then you can do a complete dump of that 8pin IC , wipe the IC and flash the dumpfile back. This dump is usually double in size compared to the firmware file, otherwise you could update the firmware via this option. Regarding to this, extracting the contents of this 8pin IC and replace this IC is also an option. Another trick that others also suggested, is to use cold spray or heat to isolate a component that is "living on the edge of working", a little bit of cold or heat can cause the component to stop work properly. i don't think it's a pure overheating issue, because the crashes are randomly, useally a overheating issue occurs after a specific time, but if you have Thermal imaging camera, you can use that to rule this out. And last, disconnect the powersupply and feed the required voltage rails from a external powersupply.
I had a similar problem with a board that would fail after 5-10 minutes for no apparent reason. Turned out to be a bad crystal. When I probed it, the signals were quite weak compared to a working board. Replaced it and it ran solidly for a 12 hour burnin and no problems since.
I think I would use a FLIR or other to see if anything is getting a bit too warm. It seems as though something is on the edge. If you find something questionable, you could put a heatsink on it to see if that's all that is needed. Also, since the USB demonstrates a repeatable issue, I would disable it and see if the unit still crashes. Plugging in the stick draws power ... which could be leading to something getting grounded or voltage pulled down. I would also check to see what the USB power voltage is. If it's a bit low (e.g.~4.7, 4.8 ...), that could be a problem when plugging in the stick. I would monitor USB voltage before adding the stick, and after to see what happens. I would also disconnect the output pcb and any other non-essentials to get as bare as possible ... then if that works, add back until the problem returns. Good Luck! Cheers.
Another option is to use a different USB drive, or a USB test load that doesn't connect to data lines. If it doesn't crash anymore, then it's not the 5V supply. There's always the chance of issues with the signal lines that aren't due to the actual USB connection, but just noise or injection of noise as well!
Sounds like a PSU issue in the 5v rail. Specifically since plugging a USB drive triggers the issue consistently. You could try connecting a small load to the USB port to see if that triggers the issue.
Heh. I was surprised to see you use un-fluxed solder wick... As for idea: Check your oscillators or crystals. I've had an old battery charger that I repaired (video must be somewhere on the channel, not sure if it's still public). The thing did the same. Intermittent lock up after some time, but different amount of time each run. I figured at the time it was because of the environment it was used in, getting tossed in the back of a van by the outdoor service technician... Same idea could apply here, with manufacturing inconsistency in the crystal, combined with shipping/handling. Very hard to spot problem in my case though, as the capacitance introduced by the scope probe made it seem functional when probing, but leaving it alone for some time revealed it. Now that I think of it, it wouldn't explain the issue popping up when plugging in the USB drive, but hey...
I would look to see if I can influence the fault by putting a small load on the 5V (just the USB stick power lines and not the data lines initially) prior to selecting a firmware update. Also look for power supply noise (not just single spikes / dips). As others have commented, a can of freezer spray can also reveal intermittent components and it would be interesting to see whether the micro oscillator is still running when the lockup occurs.
That’s a nice problem 🤪. I’ve seen Jerry Walker fixing a Siglent digital power supply with similar random froze up. He add some capacitor on power rails due spurious that were triggering the rails watchdogs. In your case, plugging the stick and loading +5v, fails immediately. It might be a clue. At 15:47 your 5v is already 4.7v and a small deviation can stop the instrument to run without triggering the scope. Very interesting fail, I hope to see a winning continuation. 👍
yeah update firmware 1st, va hardware flashing method... then 2 check datasheets for things like cpld and mcu, see where to hook up oscilloscope, if there is any detectable power glitches that is then disrupting or crashing the mcu execution but really its like the other guy already said: the 5v usb power rail must hold the clue, to finding if it is in fact power glitching on that specific rail, or on some co-driven adjacent power rail, that is near to those 5v usb circuit. would seem to be the sorts of places to speculatively hook up your oscilloscope channels onto. and trigger so on, the fact that its reliably reproducable issue, always by inserting or removing a usb key is great ! although admittedly annoying to have to keep rebooting the thing. ok then best wishes & am certain you will be finding it, because you always do! some great problem solving fault finding skills. always learning from this channel
Can you plug in the USB drive before you power on the unit? Wondering if it hangs the unit or you can proceed with a firmware update? Also wondering if you can determine what line is the hardware IRQ line to the microcontroller. Check and see if it always asserted when it hangs. If it is traceback to the device triggering it. It may be one of the 74 series chips you pointed out.
try to heat up or cool down some ICs on the board. It could be a thermal issue somewhere. As some others decided, load the 5V supply and try to replicate the issue when powering the USB. The unit can freeze due to a 5V issue, or communication.
Cold spray parts one at a time, to see if the freezing makes it better or worse? The reset circuitry of the MCU? No floating pins due to bad solder of pull-up or filter caps? May create interrupt on change that are not serviced in the firmware? It could be so many other things, but this is what I would try next. Good luck.
Maybe is also thermal related? Maybe there are some drift on the MCU crystal oscillator? You could check that. The only logical thing to consider is that when you insert the USB key you are loading some supply line but you excluded this with your scope measure. But are we sure the MCU do not use other voltages generated like by an LDO so we are not seen a voltage drop on the main 3V3 line?
If possible disconnect the measure parts from ucontroller, and disconnet the LCD and see how it behave. -The AC filter board and secondary DC-DC psu is seem close to the data lines the data may gets corrupted by EMP. - the grounding points on the screen case, or shield on a flat cable, sometimes it charged up slowly and the controller or lvds lines had a esd strike from that issue. - the oscillator output can contain the second harmonics of the main frequency, it can cause lost or jittering clocks. Or simply the txco overdrives the clock input, or have current modulated grounding. -See the USB interface ESD protection diodes, or metal shild. The usb to board cable seem not shielded, it can cause problem when the protocol is work only on a shielded cable.
4:00, there is a slight graphical glitch on the screen in the lower right corner. Where it says "Clear", the pixels look misaligned. It is subtle but it's there.
I have no knowledge about the controller in your device, but some have a voltage supervisor build in witch can be activated via software or programming of a fuse. Those controllers therefore have a special supply voltage input for analog purposes, mostly used for the A/D converters but also used for the voltage supervisor. The power connection to this pin mostly is done via an inductor and a capacitor to ground for noise minimization. The internal resistance of this inductor reduces the voltage on this pin, also affecting the voltage supervisor. While software controlled usage of the A/D converters this voltage drops even more and may come below the voltage level of the supervisor. So if the supervisor level is set to high, with this scenario it can come to randomly software holds or restarts, depending on controller settings.
I've had systems lock up when the 5v was too low or getting dragged down, only needed to get down to 4.85 ish to be an issue. Maybe checking at 4.7v is too much?
I would double check the USB port for any rubbish or shorts. The previous buyer might have broken something, could also e.g. not completed a firmware update. and make it corrupt. I would check the osccilator circuit and check the reset line with a trigger. Also not wait but just put a usb stick in to see if it crashes. Capacitors around the CPU? Hard one this little meter.
Since inserting a USB stick can cause it to crash, I'd be looking at how that's connected. You insert the USB stick and it gets power, then it should try to read it. But why would that cause it to crash? Somehow I think that's telling though.
I would focus on the reproducable lockup too. Ian, remove that ribbon cable from the USB and see if the crash still happens. If it does, check the voltage on the datalines, they normally clamped down to 3.3v with 3.6V zeners.
Some good attempts 👍 Personally I'd just leave it as is if you plan to keep it. I rarely use an LCR meter for more than a few minutes at a time. Even if you use it for longer to match components or something, it really isn't bad having to reset it every 30 minutes. Maybe it will degrade in the future and then it'll be easier to narrow down the issue. Cheers, Jake
About screen is missing model number. Corrupt nvram/flash? Maybe why the update crashes as the model number is missing and causing problems. Doesn't help with the random hang though. Just a thought.
Check the core voltage of that MCU, plus also check the voltage, using another probe, to check the return current path from the MCU to the power supply. Easy to have a poor connection in a through plated hole, that is high resistance, or goes intermittently open. So one probe on the MCU itself power pin, likely a 1V8 or so rail, and anoyther on MCU ground, both referenced to the ground connection of the 7805 regulator. Then possibly a third on the 5V supply to the USB port as well, and a fourth on the output of that clock oscillator. Set the triggers on each of the 3 to 200mV below standard voltage, and then trigger off one at a time, and not off the clock, so that when one triggers you can see clock still present at trigger, or the clock has gone. 4 channel scope, use all 4 channels.
Most oscilloscopes has a trigger function on window instead of edge, allows you to set a say +/- 0.3 V window around 3.3 V and trigger if signal "goes out the window"
Can you put a logic analyzer on the address/data lines? Are you hitting a watchdog timer? Can you program the firmware directly or do you need to go through the UI to do it? ? If you have to use the UI to program it, you could get someone with newer firmware to just read their chip directly and you could write that image onto your chip. Can you reset it to factory defaults It's like it's trying to access a block of data, but the address line is not getting set or is stuck low. Have you double-checked the filename (sometimes they are case sensitive). Maybe that lockup is just a different bug entirely. Or caused by the device trying to reach a memory address but it's address bit is stuck low. Is there any serial output you could trace?
There could be a problem with the voltage supervisor, if there is any. You could feed one power line after the other with an external supply. For example starting with the 5V rail: set the external powersupply to apx. 5.2 V and watch the behavior.
My experience as firmware developer about update is that problem is in size and formatiing of USB drive. It freezes because of some int32 variable in file system stack. Check if drive needs to be fat16 formatted or something similar. As firmware is from 2017 I am sure that at that time noone wrote firmware that is compatible with drives >4GB ( probably they have used some open source library). Maybe crashes will diasppear after update. Check if there is some uart tx line . Most of the time there is crash handler that prints on serial even if logging is not enabled in firmware.
Check to see if lockup is related to temperature. Check clock stability. Is there an external watchdog? What's the Addr/Data bus doing during a lockup? Any external interrupts causing problems? I am doubting its a watchdog issue as, if it had one then the unit would just reset on its own.. but maybe its faulty.... Random failures could be firmware but on embedded systems, the firmware is reacting to external signals. In the spirit of GIGO (garbage in garbage out) an unexpected external signal (spike or state change) can cause firmware to do weird things if it hits an "edge case" condition. Best to poke around some more, these things look expensive.. I looked on Amazon and aother model from the same manufacturer was $2000CAD new. Cheers,
I noticed both a crystal as well as a crystal oscillator close to the NXP MCU. It won't break the bank replacing those. I have seen crystal oscillators 'loosing it', while working fine again after a power cycle. That said, the firmware upgrade systematically locking up is really weird. It would point imo to a corrupted firmware ROM or something to that effect. 🤔
And what about the clock ckt of the uprocessor, Ian? I can see a quartz close to it, apart from the square one (it looks more like an oscillator). It seems to be a faulty connection or a clock signal stopped, IMHO. Also try with freezer and hot air soldering machine, and observe the results... Just a couple of ideas. BTW, it's always nice to see you here again...but finishing the equipment, not leaving "ill"...
It looks like it crashes everytime you connect a USB stick to it. Did you tried to load that +5V line through the USB port? And does this cause a crash? If not, then try to connect something different than a USB flash drive, e.g. mouse or something similar. That thing is definitelly not enumerated as a USB stick, so it shouldn't trigger too much of internal code related to the usb, which could cause a different behavior. This is just to explore how much is that USB issue related to the firmware. Furthermore, where the firmware is permanently stored? That MCU contains 512kB of Flash, but this doesn't mean this Flash is used to store complete firmware. There is some SPI flash chip. And what about the front panel? Did you tried inserting completely unconnected plug into that USB socket?
that 5V regulator is screaming at me. .. i would apply a known 3.3v and 5v and isolate that PS board. else, check the oscillator. really interesting issue. keep me posted.
My guess is courrupted firmware. The LPC2478F doesn't appear to have a boot loader in ROM! It appears from a quick look at the data sheet that in system programming programming is via UART0. If the initial programming was defective or it has been bricked by a failed software update or possibly the flash memory has lost data due to a semiconductor fault then I guess there is no easy recovery without the code and access to UART0.
Software issue on the main microcontroller perhaps a bad dump or is there a firmware update or downgrade perhaps had a usb A port is that for firmware update or just a 5v port
Nice bug :D I'm wondering why you are always waiting when you are able to simulate the problem by plugin USB stick. You can even log the power line behavior or any other line by plugin USB. You are lucky to have this possibility. Check if there is no intermittent problem with pullups (on flash or any other). Hot / cold spray could help. X-tal could be also a culprit.
Seems I was right :D I didn't have X-tal issue so often but often enough to point out to that :D And it is one of those head scratching issue which could take a weeks to identify.
Hi. Maybe someone plugged a USB killer or who knows what in the USB port, and caused some damage to the data lines, maybe scope them for abnormal logic levels.
always use a really small (256Mb) usb stick, formatted FAT32 to update the firmwares. it has a firmware made in 2017, it might not recognize bigger usb sticks.
Could the flash chip itself be bad ? But I suppose if it was it wouldn't boot now and then. Also can't believe you didn't flood board with expensive flux when cleaning the sdram pads🤔. I know not much of a helpful comment but I enjoy your videos. Thank you very much. Hope you get to bottom of this one. I am in the market for an LCR meter. Might look into this one.
Did you watch and understand the video? The RAM has been replaced and not fixed the problem. He can't upgrade because it crashes when a USB Flash drive (with new firmware) is inserted.
@@Brian_Of_Melbourne I commented before I watched the complete video, obviously yes something more complicated. Perhaps related to the usb controller chip?
Wrong type of electrons, you are using northern supplied ones. Maybe sending it down south might be the answer. I hear that the northern ones are prone to sticking in component leads due to their cold and clammy nature. 🙂 I could charge a battery up with some good southern one and post it up if that helps. ;-)
@@IanScottJohnston Make sure you filter out the damp ones and those that have too many wee small dram's 🙂 looks to be a nice bit of kit once you get it working. But then it does seem to work as long as I do before needing a wee cup of tea. maybe unplugging it and swapping the socket for a kettle we would just never notice it.
Defective decoupling I'd say, just trips over too much draw. Could be the display itself if it has any smarts in it. Plugging the USB in takes the micro over the edge.
If you are planning to use it 'as is' I would suggest an external RESET button, connected to the microprocessor reset line. That should save you power-off power-ons.
CPU collapsing, possibly under run-of-the-mill-orders to execute and definitely when loading a USB stick, any common devices used. The MCU is marginal is my best guess after the PSU seemed OK.
Hi Ian, I am really disappointed. (Not really, I love all your videos) I was anxiously awaiting another video on the Tektronix 2430, because I have one on the workbench right now and I was hoping to get some insights into your troubleshooting process on this way too complicated machine. Will you make another video and if so when can I expect it? Thanks!
I will make another video, but not sure when......it looks like I need to put in a lot of research into the issue and maybe acquire some spare parts first. Looking for a cheap working unit, or parts.
Clock circuit working properly? Is it overclocking, underclocking or even stalling the CPU or other peripherals? Maybe the keypad (on the front) is sending rogue signals and the software isn't monkey-proof and locks up on bad inputs i.e. jumps into some infinite random loop.
Hi lan, I'm Wendy from MATRIX, I'm glad to see your updated video. I have sent your video to our technician, hoping he can provide some further help.
Wendy, see my Part 2 video, the unit is now working!
@@IanScottJohnston Congratulations!🥳
How does one go about buying stuff that has been returned to you?
Hi Ian, here a few tips you can try out.
9:14 disconnect the external interface on the backside, a faulty component there can be the issue.
10:45 the SPI header is usually (as you probably know) connected 1 to 1 to the 8 pin IC, where the first 2 pins of the header are not used. If this is the case, then you can do a complete dump of that 8pin IC , wipe the IC and flash the dumpfile back. This dump is usually double in size compared to the firmware file, otherwise you could update the firmware via this option. Regarding to this, extracting the contents of this 8pin IC and replace this IC is also an option.
Another trick that others also suggested, is to use cold spray or heat to isolate a component that is "living on the edge of working", a little bit of cold or heat can cause the component to stop work properly. i don't think it's a pure overheating issue, because the crashes are randomly, useally a overheating issue occurs after a specific time,
but if you have Thermal imaging camera, you can use that to rule this out.
And last, disconnect the powersupply and feed the required voltage rails from a external powersupply.
I had a similar problem with a board that would fail after 5-10 minutes for no apparent reason. Turned out to be a bad crystal. When I probed it, the signals were quite weak compared to a working board. Replaced it and it ran solidly for a 12 hour burnin and no problems since.
I think I would use a FLIR or other to see if anything is getting a bit too warm. It seems as though something is on the edge. If you find something questionable, you could put a heatsink on it to see if that's all that is needed.
Also, since the USB demonstrates a repeatable issue, I would disable it and see if the unit still crashes. Plugging in the stick draws power ... which could be leading to something getting grounded or voltage pulled down.
I would also check to see what the USB power voltage is. If it's a bit low (e.g.~4.7, 4.8 ...), that could be a problem when plugging in the stick. I would monitor USB voltage before adding the stick, and after to see what happens.
I would also disconnect the output pcb and any other non-essentials to get as bare as possible ... then if that works, add back until the problem returns.
Good Luck! Cheers.
Another option is to use a different USB drive, or a USB test load that doesn't connect to data lines. If it doesn't crash anymore, then it's not the 5V supply. There's always the chance of issues with the signal lines that aren't due to the actual USB connection, but just noise or injection of noise as well!
Sounds like a PSU issue in the 5v rail. Specifically since plugging a USB drive triggers the issue consistently. You could try connecting a small load to the USB port to see if that triggers the issue.
Heh. I was surprised to see you use un-fluxed solder wick...
As for idea:
Check your oscillators or crystals. I've had an old battery charger that I repaired (video must be somewhere on the channel, not sure if it's still public). The thing did the same. Intermittent lock up after some time, but different amount of time each run.
I figured at the time it was because of the environment it was used in, getting tossed in the back of a van by the outdoor service technician... Same idea could apply here, with manufacturing inconsistency in the crystal, combined with shipping/handling. Very hard to spot problem in my case though, as the capacitance introduced by the scope probe made it seem functional when probing, but leaving it alone for some time revealed it.
Now that I think of it, it wouldn't explain the issue popping up when plugging in the USB drive, but hey...
How are oscillators and crystals checked?
@@jheissjr With an osciloscope. 😊
peace be upon you sir and zamzam water
I would look to see if I can influence the fault by putting a small load on the 5V (just the USB stick power lines and not the data lines initially) prior to selecting a firmware update. Also look for power supply noise (not just single spikes / dips). As others have commented, a can of freezer spray can also reveal intermittent components and it would be interesting to see whether the micro oscillator is still running when the lockup occurs.
That’s a nice problem 🤪. I’ve seen Jerry Walker fixing a Siglent digital power supply with similar random froze up. He add some capacitor on power rails due spurious that were triggering the rails watchdogs. In your case, plugging the stick and loading +5v, fails immediately. It might be a clue. At 15:47 your 5v is already 4.7v and a small deviation can stop the instrument to run without triggering the scope. Very interesting fail, I hope to see a winning continuation. 👍
Where can you purchase matrix equipment that has been returned to them? Do they have a special store?
Did you try probing data lines to check levels? Maybe it is a bad I2C pull-up resistor effecting main comms data lines?
yeah update firmware 1st, va hardware flashing method... then 2 check datasheets for things like cpld and mcu, see where to hook up oscilloscope, if there is any detectable power glitches that is then disrupting or crashing the mcu execution
but really its like the other guy already said: the 5v usb power rail must hold the clue, to finding if it is in fact power glitching on that specific rail, or on some co-driven adjacent power rail, that is near to those 5v usb circuit. would seem to be the sorts of places to speculatively hook up your oscilloscope channels onto. and trigger so on,
the fact that its reliably reproducable issue, always by inserting or removing a usb key is great ! although admittedly annoying to have to keep rebooting the thing. ok then best wishes & am certain you will be finding it, because you always do! some great problem solving fault finding skills. always learning from this channel
Great video Ian =D I am thinking about ROM, maybe try temperature in different areas of the PCB to localise it. Maybe hot air / freezer spray.
Can you plug in the USB drive before you power on the unit? Wondering if it hangs the unit or you can proceed with a firmware update? Also wondering if you can determine what line is the hardware IRQ line to the microcontroller. Check and see if it always asserted when it hangs. If it is traceback to the device triggering it. It may be one of the 74 series chips you pointed out.
what about the crystal oscillator for the CPU maybe it's intermittent.
try to heat up or cool down some ICs on the board. It could be a thermal issue somewhere.
As some others decided, load the 5V supply and try to replicate the issue when powering the USB. The unit can freeze due to a 5V issue, or communication.
Cold spray parts one at a time, to see if the freezing makes it better or worse?
The reset circuitry of the MCU? No floating pins due to bad solder of pull-up or filter caps? May create interrupt on change that are not serviced in the firmware?
It could be so many other things, but this is what I would try next. Good luck.
Maybe is also thermal related? Maybe there are some drift on the MCU crystal oscillator? You could check that. The only logical thing to consider is that when you insert the USB key you are loading some supply line but you excluded this with your scope measure. But are we sure the MCU do not use other voltages generated like by an LDO so we are not seen a voltage drop on the main 3V3 line?
If possible disconnect the measure parts from ucontroller, and disconnet the LCD and see how it behave.
-The AC filter board and secondary DC-DC psu is seem close to the data lines the data may gets corrupted by EMP.
- the grounding points on the screen case, or shield on a flat cable, sometimes it charged up slowly and the controller or lvds lines had a esd strike from that issue.
- the oscillator output can contain the second harmonics of the main frequency, it can cause lost or jittering clocks. Or simply the txco overdrives the clock input, or have current modulated grounding.
-See the USB interface ESD protection diodes, or metal shild. The usb to board cable seem not shielded, it can cause problem when the protocol is work only on a shielded cable.
4:00, there is a slight graphical glitch on the screen in the lower right corner. Where it says "Clear", the pixels look misaligned. It is subtle but it's there.
To completely rule out the power supply try using external supplies. Then go down the USB, firmware and clock rabbit holes if that doesn't work. 😊
I have no knowledge about the controller in your device, but some have a voltage supervisor build in witch can be activated via software or programming of a fuse. Those controllers therefore have a special supply voltage input for analog purposes, mostly used for the A/D converters but also used for the voltage supervisor. The power connection to this pin mostly is done via an inductor and a capacitor to ground for noise minimization. The internal resistance of this inductor reduces the voltage on this pin, also affecting the voltage supervisor. While software controlled usage of the A/D converters this voltage drops even more and may come below the voltage level of the supervisor. So if the supervisor level is set to high, with this scenario it can come to randomly software holds or restarts, depending on controller settings.
I've had systems lock up when the 5v was too low or getting dragged down, only needed to get down to 4.85 ish to be an issue. Maybe checking at 4.7v is too much?
Does the crystal continue oscillating after the crash?
Did you try to power up and do a firmware update with the USB stick already connected before power on?
I would double check the USB port for any rubbish or shorts. The previous buyer might have broken something, could also e.g. not completed a firmware update. and make it corrupt. I would check the osccilator circuit and check the reset line with a trigger. Also not wait but just put a usb stick in to see if it crashes. Capacitors around the CPU? Hard one this little meter.
Since inserting a USB stick can cause it to crash, I'd be looking at how that's connected. You insert the USB stick and it gets power, then it should try to read it. But why would that cause it to crash? Somehow I think that's telling though.
I would focus on the reproducable lockup too. Ian, remove that ribbon cable from the USB and see if the crash still happens. If it does, check the voltage on the datalines, they normally clamped down to 3.3v with 3.6V zeners.
Have you probed the clock and reset on the processor to see if there is a perturbation during the crash event?
Some good attempts 👍 Personally I'd just leave it as is if you plan to keep it. I rarely use an LCR meter for more than a few minutes at a time. Even if you use it for longer to match components or something, it really isn't bad having to reset it every 30 minutes. Maybe it will degrade in the future and then it'll be easier to narrow down the issue.
Cheers,
Jake
About screen is missing model number. Corrupt nvram/flash? Maybe why the update crashes as the model number is missing and causing problems. Doesn't help with the random hang though. Just a thought.
Check the core voltage of that MCU, plus also check the voltage, using another probe, to check the return current path from the MCU to the power supply. Easy to have a poor connection in a through plated hole, that is high resistance, or goes intermittently open. So one probe on the MCU itself power pin, likely a 1V8 or so rail, and anoyther on MCU ground, both referenced to the ground connection of the 7805 regulator. Then possibly a third on the 5V supply to the USB port as well, and a fourth on the output of that clock oscillator. Set the triggers on each of the 3 to 200mV below standard voltage, and then trigger off one at a time, and not off the clock, so that when one triggers you can see clock still present at trigger, or the clock has gone. 4 channel scope, use all 4 channels.
Worth attacking ICs with the hot air or freezer spray to try and trigger?
I hope you will update us with the outcome.
Looks like there was a fan mounted on the side right by the ram chip that you just replaced? Thanks
Most oscilloscopes has a trigger function on window instead of edge, allows you to set a say +/- 0.3 V window around 3.3 V and trigger if signal "goes out the window"
What about checking for excess voltage spikes or events on the power rails?
Maybe there is a weak resistor on the reset-line? Or is something holding the reset line low?
Can you put a logic analyzer on the address/data lines? Are you hitting a watchdog timer? Can you program the firmware directly or do you need to go through the UI to do it? ? If you have to use the UI to program it, you could get someone with newer firmware to just read their chip directly and you could write that image onto your chip. Can you reset it to factory defaults
It's like it's trying to access a block of data, but the address line is not getting set or is stuck low.
Have you double-checked the filename (sometimes they are case sensitive). Maybe that lockup is just a different bug entirely. Or caused by the device trying to reach a memory address but it's address bit is stuck low.
Is there any serial output you could trace?
There could be a problem with the voltage supervisor, if there is any.
You could feed one power line after the other with an external supply. For example starting with the 5V rail: set the external powersupply to apx. 5.2 V and watch the behavior.
My experience as firmware developer about update is that problem is in size and formatiing of USB drive. It freezes because of some int32 variable in file system stack. Check if drive needs to be fat16 formatted or something similar. As firmware is from 2017 I am sure that at that time noone wrote firmware that is compatible with drives >4GB ( probably they have used some open source library). Maybe crashes will diasppear after update. Check if there is some uart tx line . Most of the time there is crash handler that prints on serial even if logging is not enabled in firmware.
Matrix gave me explicit instructions on size, format etc…..but it didn’t help.
Usb power distribution chip, have you scoped it out?
Check to see if lockup is related to temperature. Check clock stability. Is there an external watchdog? What's the Addr/Data bus doing during a lockup? Any external interrupts causing problems?
I am doubting its a watchdog issue as, if it had one then the unit would just reset on its own.. but maybe its faulty....
Random failures could be firmware but on embedded systems, the firmware is reacting to external signals. In the spirit of GIGO (garbage in garbage out) an unexpected external signal (spike or state change) can cause firmware to do weird things if it hits an "edge case" condition.
Best to poke around some more, these things look expensive.. I looked on Amazon and aother model from the same manufacturer was $2000CAD new.
Cheers,
Check the low value resistors networks that usually are terminators and if one of the resistors is out of value can slow rise and fall edge.
I noticed both a crystal as well as a crystal oscillator close to the NXP MCU. It won't break the bank replacing those. I have seen crystal oscillators 'loosing it', while working fine again after a power cycle. That said, the firmware upgrade systematically locking up is really weird. It would point imo to a corrupted firmware ROM or something to that effect. 🤔
Maybe a ceramic capacitor . You can check with a thermal camera.
It’s always a capacitor… lol 😂
And what about the clock ckt of the uprocessor, Ian? I can see a quartz close to it, apart from the square one (it looks more like an oscillator). It seems to be a faulty connection or a clock signal stopped, IMHO. Also try with freezer and hot air soldering machine, and observe the results... Just a couple of ideas. BTW, it's always nice to see you here again...but finishing the equipment, not leaving "ill"...
It looks like it crashes everytime you connect a USB stick to it. Did you tried to load that +5V line through the USB port? And does this cause a crash? If not, then try to connect something different than a USB flash drive, e.g. mouse or something similar. That thing is definitelly not enumerated as a USB stick, so it shouldn't trigger too much of internal code related to the usb, which could cause a different behavior. This is just to explore how much is that USB issue related to the firmware.
Furthermore, where the firmware is permanently stored? That MCU contains 512kB of Flash, but this doesn't mean this Flash is used to store complete firmware. There is some SPI flash chip.
And what about the front panel? Did you tried inserting completely unconnected plug into that USB socket?
Maby add some capacitance close to cpu . I had once stability issue in an bose device fixed by that
that 5V regulator is screaming at me. .. i would apply a known 3.3v and 5v and isolate that PS board. else, check the oscillator.
really interesting issue. keep me posted.
My guess is courrupted firmware. The LPC2478F doesn't appear to have a boot loader in ROM! It appears from a quick look at the data sheet that in system programming programming is via UART0. If the initial programming was defective or it has been bricked by a failed software update or possibly the flash memory has lost data due to a semiconductor fault then I guess there is no easy recovery without the code and access to UART0.
Software issue on the main microcontroller perhaps a bad dump or is there a firmware update or downgrade perhaps had a usb A port is that for firmware update or just a 5v port
Maybe get a terminal going via UART and see if serial data shows anything interesting.
Nice bug :D I'm wondering why you are always waiting when you are able to simulate the problem by plugin USB stick. You can even log the power line behavior or any other line by plugin USB. You are lucky to have this possibility. Check if there is no intermittent problem with pullups (on flash or any other). Hot / cold spray could help. X-tal could be also a culprit.
Seems I was right :D I didn't have X-tal issue so often but often enough to point out to that :D And it is one of those head scratching issue which could take a weeks to identify.
Having dealt with issues like this with computers, I'd look at the CPU or RAM. I'd lean more towards the CPU. Is it possible to get replacements?
Did you watch and understand the video? The RAM has been replaced and not fixed the problem.
@@Brian_Of_Melbourne He removed it, cleaned up the pads and resoldered it. I don't recall him saying that he replaced it with a new chip.
@@GlennNiesen Watch carefully from 16:40.
As the USB stick lets the instrument crash immediately, pls. check the value of the +5V rail with a DMM .. maybe you see an effect.
That keypad though. Entering numbers would end up wrong. Always. But you don't need to do that very often, I guess.
@7:00 makes me think it is a 5V supply rail issue, crashing as it sags under increased load. EDIT, maybe not then.
Maybe a faulty cap causing ripple, but even that isn't that likely.
Hi. Maybe someone plugged a USB killer or who knows what in the USB port, and caused some damage to the data lines, maybe scope them for abnormal logic levels.
I'd focus on what is happening with that USB interface, seems very repeatable. Bad pullups? I'm sure you'll get this fixed Ian! 🙂
I would try several other usb sticks both with and without the fw image and observe behavior.
always use a really small (256Mb) usb stick, formatted FAT32 to update the firmwares. it has a firmware made in 2017, it might not recognize bigger usb sticks.
Could the flash chip itself be bad ? But I suppose if it was it wouldn't boot now and then. Also can't believe you didn't flood board with expensive flux when cleaning the sdram pads🤔. I know not much of a helpful comment but I enjoy your videos. Thank you very much. Hope you get to bottom of this one. I am in the market for an LCR meter. Might look into this one.
Bad ram ic or eeprom, I'd definitely try updating the firmware or even try down grading
Did you watch and understand the video? The RAM has been replaced and not fixed the problem. He can't upgrade because it crashes when a USB Flash drive (with new firmware) is inserted.
@@Brian_Of_Melbourne I commented before I watched the complete video, obviously yes something more complicated. Perhaps related to the usb controller chip?
Hmm? A story without a happy ending 🙂
Wrong type of electrons, you are using northern supplied ones. Maybe sending it down south might be the answer.
I hear that the northern ones are prone to sticking in component leads due to their cold and clammy nature. 🙂
I could charge a battery up with some good southern one and post it up if that helps. ;-)
Maybe it needs an electron exorcism, I’ll buy a job lot of mixed electrons and try that first.
@@IanScottJohnston Make sure you filter out the damp ones and those that have too many wee small dram's 🙂
looks to be a nice bit of kit once you get it working. But then it does seem to work as long as I do before needing a wee cup of tea. maybe unplugging it and swapping the socket for a kettle we would just never notice it.
Defective decoupling I'd say, just trips over too much draw. Could be the display itself if it has any smarts in it. Plugging the USB in takes the micro over the edge.
If you are planning to use it 'as is' I would suggest an external RESET button, connected to the microprocessor reset line. That should save you power-off power-ons.
CPU collapsing, possibly under run-of-the-mill-orders to execute and definitely when loading a USB stick, any common devices used. The MCU is marginal is my best guess after the PSU seemed OK.
Hi Ian, I am really disappointed. (Not really, I love all your videos)
I was anxiously awaiting another video on the Tektronix 2430, because I have one on the workbench right now and I was hoping to get some insights into your troubleshooting process on this way too complicated machine.
Will you make another video and if so when can I expect it? Thanks!
I will make another video, but not sure when......it looks like I need to put in a lot of research into the issue and maybe acquire some spare parts first. Looking for a cheap working unit, or parts.
Oscillator maybe? Secondly, I'd go see why USB is causing crashes
its not made by matrix, its sold under many different names. typical alibaba import product
It looks too hard, ie, too many things it could be, unless you get another and start swapping components.
send it back
Clock circuit working properly? Is it overclocking, underclocking or even stalling the CPU or other peripherals? Maybe the keypad (on the front) is sending rogue signals and the software isn't monkey-proof and locks up on bad inputs i.e. jumps into some infinite random loop.
Hi!