About a week ago, I got a very worrying message from Daniel. He had built 12 of the new Pokitto boards with the RTC (Real-Time Clock). The clock worked but none - not a single one - was able to go into the bootloader mode and connect to the PC via the USB.
This was bad news indeed. We had a working design that suddenly had developed a fatal flaw. With you, dear Pokitto backers & community in mind I thought: “this could hit the shipping time badly”. My second thought was, that unless we find the bug fast, we will remove the RTC and go back to the original design. My apology message to the community was already written!
Throughout my “mini-holiday” we sent messages back and forth with Daniel: “check ISP enable lines”, “check reset signal”, “check VBUS voltages” … all were correct. And yet: no USB connection. Daniel quickly dispatched 4 boards to me.
Now, it is hard to explain how tricky it is to debug something like this. The fault can be in so many places. It can be a hardware fault, it can be a broken chip somewhere, it can be the addition of the real-time clock and its auxillary power supply that changes the behavior of the Power Management Unit (responsible for power saving modes) of the main chip. Worst of all is the fact that the USB routines are in the ROM of the chip: there is no way to step through the code that is burned inside the chip. USB is notoriously difficult to debug: the protocol is quite complex and events happen very fast. You need good tools to be able to follow whats going on.
Yesterday the boards arrived and I quickly assembled 2 Pokittos and began the detective work, not really knowing what to look for.
The whole day I was trying out things, measuring signal voltages, comparing schematics and eliminating possible sources of problems one by one. Daniel had already done these steps, but no harm in double checking.
At 10 in the evening I finally had a breakthrough. I had written a small program that makes the Pokitto to behave like a USB Serial device and connect to the PC as a virtual COM port. First I ran it on the faulty version 4 board. No show. Then I put it on the working v3 Pokitto: wham bang! COM port pops up and “Hello world!” message on the PC. Repeating the test gave same results. Now I knew the USB was the culprit. Not the RTC or any other of the thousands of things.
Because the USBSerial was now a software program on the chip instead of a hidden ROM routine, I was able to step through the USB handshake routine (called the “chirp”) and spot the point of failure. Something was pooping - yes, no kinder word for it - pooping on the USB data lines, messing up the signal.
I wrote about my findings to Daniel. The reply came almost immediately: check the ESD (electrostatic discharge) protection chip on the USB data lines.
And sure enough, a chip and feature that had been on the Pokitto PCB from day 1 of the design had chosen this revision of the board to suddenly rotate 90 degrees on its place, spewing crap on the USB lines. And all the time we were cursing the new RTC feature! Turning the chip around automagically solved all problems. Everything works again.
It was a close call, dear readers. Pokitto almost lost its Real-Time Clock. If the error would not have been found within 2-3 days, we would have gone back to the previous design in order not to screw up the project timeline completely.
But now I have a working v4 Pokitto on my table with Real-Time Clock and we are back in the game!
Mugshot of the culprit:
