Traumwind - Traumwind RSS reader

Imaging old 3.5" floppy disks (DOS 720k, DOS 1.44M, ADFS 320k, ADFS 800k, DFS 100k, DFS 200k) in New Zealand using libdsk/dsktrans

Wow, it's been a while since I last posted. Since then I started working at Google, made a ton of hardware projects for the Acorn series of computers: Electron, BBC Micro, BBC Master, and have enjoyed a year and a half of marriage.

I spent a week in New Zealand visiting family for Christmas. My main project while there was to clean up a whole lot of stuff I left behind when I moved to the USA, which, among a whole lot of physical stuff like university notes, included various hard disks, and several hundred floppy disks in various formats. Most of them were 1.44M DOS disks, but the most interesting ones (because I was hoping to find traces of the earliest programs I'd written, in the 80s) were in various Acorn formats like 320k ADFS and 200k DFS, as well as the newer 800k and 1.6M ADFS.

Luckily it appears that the last desktop computer I bought, an Athlon XP machine from 2005, was capable of reading *all* the formats. As confirmed by Andrew Benham in London, the libdsk project and its dsktrans tool work really well and can autodetect all the formats I was interested in.

Healthy disks could be copied quickly with: dsktrans /dev/fd0 image.dsk

Disks with errors required: dsktrans -retry 10 -stubborn /dev/fd0 image.dsk

I never needed to specify -format; I experimented a bit with some unknown disks, but it looks like the ones that dsktrans couldn't autodetect were unreadable using any of the formats I tried.

This results in .DSK format files, which contain the data from the sectors on the disk, but also quite a bit of metadata about the disk, and can be converted into raw images suitable for reading in emulators by running them through dsktrans again:

For DOS disks, which macOS will happily open if given .dmg extensions: dsktrans image.dsk -otype raw image.dmg

For ADFS disks: dsktrans image.dsk -otype logical image.adf

I assume the ADFS approach will also work for DFS, but either I haven't found an emulator that understands 3.5" DFS disks yet, or my disks were too damaged.

Comment

guid

http://www.myelin.co.nz/post/2018/1/16/#201801161

pubdate

Mon, 16 Jan 2018 20:30:01 GMT

comments

http://www.myelin.co.nz/post/comments?u=2&p=201801161&link=http://www.myelin.co.nz/post/2018/1/16/#201801161

summary

Wow, it's been a while since I last posted. Since then I started working at Google, made a ton of hardware projects for the Acorn series of computers: Electron, BBC Micro, BBC Master, and have enjoyed a year and a half of marriage.

I spent a week in New Zealand visiting family for Christmas. My main project while there was to clean up a whole lot of stuff I left behind when I moved to the USA, which, among a whole lot of physical stuff like university notes, included various hard disks, and several hundred floppy disks in various formats. Most of them were 1.44M DOS disks, but the most interesting ones (because I was hoping to find traces of the earliest programs I'd written, in the 80s) were in various Acorn formats like 320k ADFS and 200k DFS, as well as the newer 800k and 1.6M ADFS.

Luckily it appears that the last desktop computer I bought, an Athlon XP machine from 2005, was capable of reading *all* the formats. As confirmed by Andrew Benham in London, the libdsk project and its dsktrans tool work really well and can autodetect all the formats I was interested in.

Healthy disks could be copied quickly with: dsktrans /dev/fd0 image.dsk

Disks with errors required: dsktrans -retry 10 -stubborn /dev/fd0 image.dsk

I never needed to specify -format; I experimented a bit with some unknown disks, but it looks like the ones that dsktrans couldn't autodetect were unreadable using any of the formats I tried.

This results in .DSK format files, which contain the data from the sectors on the disk, but also quite a bit of metadata about the disk, and can be converted into raw images suitable for reading in emulators by running them through dsktrans again:

For DOS disks, which macOS will happily open if given .dmg extensions: dsktrans image.dsk -otype raw image.dmg

For ADFS disks: dsktrans image.dsk -otype logical image.adf

I assume the ADFS approach will also work for DFS, but either I haven't found an emulator that understands 3.5" DFS disks yet, or my disks were too damaged.

Comment

date_timestamp

1516653001

Things you learn working at a startup

So it's kinda stressful, and I wouldn't recommend spending your whole life like this, but that adage about learning everything all at once when you're working at a tiny startup is kinda true. If I don't write all this down, I'm going to forget it, so here's what I've been up to in the last few weeks:

On September 15th, Mode Media, where I'd worked since 2011 (or 2009 if you include the time at Ning before we got acquired by Mode), went out of business. I was lucky enough to be rescued from the ashes of the company because an investor, Cyndx, wanted to buy Ning and keep it running, and immediately sent six of us ex-Mode employees contracts for the next week, and put in a bid to buy Ning. A week later, a completely unrelated investor, Noosphere Ventures, swept in and bought the company instead, which was exciting to say the least, but seems to be working out well.

This meant that Ning went from being a low priority maintenance mode job, supported by a ton of people at Mode who worked on other things most of the time, to being extremely important, and also super understaffed. We were down to three engineers and no ops, IT, or QA. I had been taking care of most of the backend of the system for a while, but now I was also responsible for ops, DBA, and IT work. So, basically, we went from a 150-person company to a 6-person startup. Luckily our entire engineering team is pretty seasoned (I think that each of us has at least 10-15 years of industry experience), our CEO/GM has decades of experience negotiating with suppliers to keep everything under control, and we got to keep our two most experienced support folks, to keep the lines of communication open with customers.

Notable stuff I've done since then:

- Hunted around to find passwords, SSH keys, and SSL certificates for everything in the system. Edited databases to give myself administrator access where necessary.

- Poked my way through the system until I found the LDAP servers, and brought up an OpenDJ replica on EC2, along with copies of our Confluence and JIRA servers.

- Debugged a ton of MySQL server outages, mainly caused by failing disks or partitions getting full after a replication failure caused binary log purging to stop. Rebuilt several 900GB databases from backups and fixed up replication.

- Moved everything from two AWS accounts onto a third account (for the new company). This meant coding up a hacked up MapReduce-y type set of Python scripts to copy 140 terabytes (400 million files) from an S3 bucket in one AWS account to a bucket in another account in a couple of days, learning enough about CloudFormation to bring up a copy of an old stack in a new account, rebuilding a bunch of Wordpress blogs from backups and disk snapshots, and copying a ton of random data around in S3.

- Learned that granting someone from another account access to your S3 bucket gives them the ability to create files that you have no access to, unless you set a bucket policy to force them to give you full control. Good thing I'd optimized my copy scripts so I could rerun the whole thing in 12 hours...

- Learned some tricks for setting up an AWS VPC, like making sure to turn on DNS hostnames, to avoid getting a warning about not being able to resolve the hostname every time you run sudo, and also some other apps refusing to start up.

More to come, I'm sure... once this is all taken care of, I imagine there'll be way more cloud work, and tasks to bring as much of our infrastructure as possible into 2016, and enable future development. It's definitely keeping me busy, but it's exciting for sure!

Comment

guid

http://www.myelin.co.nz/post/2016/10/19/#201610191

pubdate

Mon, 20 Oct 2016 02:17:48 GMT

comments

http://www.myelin.co.nz/post/comments?u=2&p=201610191&link=http://www.myelin.co.nz/post/2016/10/19/#201610191

summary

So it's kinda stressful, and I wouldn't recommend spending your whole life like this, but that adage about learning everything all at once when you're working at a tiny startup is kinda true. If I don't write all this down, I'm going to forget it, so here's what I've been up to in the last few weeks:

On September 15th, Mode Media, where I'd worked since 2011 (or 2009 if you include the time at Ning before we got acquired by Mode), went out of business. I was lucky enough to be rescued from the ashes of the company because an investor, Cyndx, wanted to buy Ning and keep it running, and immediately sent six of us ex-Mode employees contracts for the next week, and put in a bid to buy Ning. A week later, a completely unrelated investor, Noosphere Ventures, swept in and bought the company instead, which was exciting to say the least, but seems to be working out well.

This meant that Ning went from being a low priority maintenance mode job, supported by a ton of people at Mode who worked on other things most of the time, to being extremely important, and also super understaffed. We were down to three engineers and no ops, IT, or QA. I had been taking care of most of the backend of the system for a while, but now I was also responsible for ops, DBA, and IT work. So, basically, we went from a 150-person company to a 6-person startup. Luckily our entire engineering team is pretty seasoned (I think that each of us has at least 10-15 years of industry experience), our CEO/GM has decades of experience negotiating with suppliers to keep everything under control, and we got to keep our two most experienced support folks, to keep the lines of communication open with customers.

Notable stuff I've done since then:

- Hunted around to find passwords, SSH keys, and SSL certificates for everything in the system. Edited databases to give myself administrator access where necessary.

- Poked my way through the system until I found the LDAP servers, and brought up an OpenDJ replica on EC2, along with copies of our Confluence and JIRA servers.

- Debugged a ton of MySQL server outages, mainly caused by failing disks or partitions getting full after a replication failure caused binary log purging to stop. Rebuilt several 900GB databases from backups and fixed up replication.

- Moved everything from two AWS accounts onto a third account (for the new company). This meant coding up a hacked up MapReduce-y type set of Python scripts to copy 140 terabytes (400 million files) from an S3 bucket in one AWS account to a bucket in another account in a couple of days, learning enough about CloudFormation to bring up a copy of an old stack in a new account, rebuilding a bunch of Wordpress blogs from backups and disk snapshots, and copying a ton of random data around in S3.

- Learned that granting someone from another account access to your S3 bucket gives them the ability to create files that you have no access to, unless you set a bucket policy to force them to give you full control. Good thing I'd optimized my copy scripts so I could rerun the whole thing in 12 hours...

- Learned some tricks for setting up an AWS VPC, like making sure to turn on DNS hostnames, to avoid getting a warning about not being able to resolve the hostname every time you run sudo, and also some other apps refusing to start up.

More to come, I'm sure... once this is all taken care of, I imagine there'll be way more cloud work, and tasks to bring as much of our infrastructure as possible into 2016, and enable future development. It's definitely keeping me busy, but it's exciting for sure!

Comment

date_timestamp

1477275468

Phil's guide to Burning Man, part 1

This year will be Burning Man #201603311" onclick="window.open( this.href, 'comments', 'width=515, height=480, location=0, resizable=1, scrollbars=1, status=0, toolbar=0, directories=0'); return(false);" title="Click here to comment on this post.">Comment

guid: http://www.myelin.co.nz/post/2016/3/31/#201603311
category: Burning Man
pubdate: Mon, 31 Mar 2016 08:50:14 GMT
comments: http://www.myelin.co.nz/post/comments?u=2&p=201603311&link=http://www.myelin.co.nz/post/2016/3/31/#201603311
summary: This year will be Burning Man #201603311" onclick="window.open( this.href, 'comments', 'width=515, height=480, location=0, resizable=1, scrollbars=1, status=0, toolbar=0, directories=0'); return(false);" title="Click here to comment on this post.">Comment
date_timestamp: 1459759814

Cheap Chinese STM32F103 boards

I recently ordered a bunch of STM32F103 boards from China. They have a rectangular footprint like the Teensy or Arduino Micro, and a 48-pin STM32F103C8T6 MCU on board -- a 72MHz Cortex-M3 with 64kB of flash and 20kB of RAM. It's just the single MCU chip on board, with a bootloader, and the SWD pins brought out to a header, which should make it quite flexible; I'll be able to program/debug them with Eclipse and OpenOCD if I want full control, or I can use the STM32duino Arduino port if I just want to push code to them. The best part is that they only cost $2.25 each on Aliexpress, or about $3.50 on eBay , which is about half the cost of the chip alone when bought in the US.

The boards I've ordered are 2.1" long by 0.9" wide, with a rows of 20 pins along each side, and 0.6" from the center of one row to the other. I've made a KiCad part and footprint for them:

Part/library → myelin-kicad-libraries/stm32f103c8t6-module-china.lib

Footprint/module → myelin-kicad.pretty/stm32f103c8t6-module-china.kicad_module

These should be good for controlling LEDs over USB, or for soldering into projects where I feel that a Teensy would be too expensive.

Comment

guid

http://www.myelin.co.nz/post/2016/3/15/#201603151

pubdate

Mon, 15 Mar 2016 23:03:50 GMT

comments

http://www.myelin.co.nz/post/comments?u=2&p=201603151&link=http://www.myelin.co.nz/post/2016/3/15/#201603151

summary

I recently ordered a bunch of STM32F103 boards from China. They have a rectangular footprint like the Teensy or Arduino Micro, and a 48-pin STM32F103C8T6 MCU on board -- a 72MHz Cortex-M3 with 64kB of flash and 20kB of RAM. It's just the single MCU chip on board, with a bootloader, and the SWD pins brought out to a header, which should make it quite flexible; I'll be able to program/debug them with Eclipse and OpenOCD if I want full control, or I can use the STM32duino Arduino port if I just want to push code to them. The best part is that they only cost $2.25 each on Aliexpress, or about $3.50 on eBay , which is about half the cost of the chip alone when bought in the US.

The boards I've ordered are 2.1" long by 0.9" wide, with a rows of 20 pins along each side, and 0.6" from the center of one row to the other. I've made a KiCad part and footprint for them:

Part/library → myelin-kicad-libraries/stm32f103c8t6-module-china.lib

Footprint/module → myelin-kicad.pretty/stm32f103c8t6-module-china.kicad_module

These should be good for controlling LEDs over USB, or for soldering into projects where I feel that a Teensy would be too expensive.

Comment

date_timestamp

1458601430

CMSIS-DAP work: implemented raw JTAG support, and ported the HID firmware to Pro Micro and Teensy 3.2

While I am so happy to have my J-Link back, the couple of weeks without it have been very productive in terms of open source contributions. After finding out that OpenOCD didn't support raw JTAG mode on CMSIS-DAP adapters, I bit that off as a potential project, and eventually got it working, then did some performance tuning, and I'm pretty pleased with it now. With my LPC-Link2, it can program the flash in an ATMEGA32U4 over JTAG at about 1/4-1/3 of the speed of the J-Link (which is kind of a speed demon). I'm going to let it soak on GitHub for a while, then clean it up and submit it to OpenOCD once it's had a bit of scrutiny.

→ Clone myelin/openocd and check out the cmsis-dap-jtag branch to try it out.

Implementing this required getting pretty familiar with the CMSIS-DAP source code and protocol, and at some point I realized that it wouldn't be super hard to port the SWD/JTAG debugger part of the CMSIS-DAP firmware over to any USB capable microcontroller. Full CMSIS-DAP support requires the debugger part, plus a serial port and a mass storage device emulator that's also capable of flashing a .bin file to a chip, but I'm not doing this to provide a USB interface to a custom board (like the ones on the mbed site), so I've skipped those two. I'll probably add in the serial port sometime, because I wrote serial bridge code for teensy-openocd-remote-bitbang already.

This was mainly a matter of getting rid of Keil-specific code, plus a small amount of debugging:

The _OUT functions (e.g. PIN_SWDIO_OUT) take a uint32_t with a boolean value in the LSB, but often junk in the higher bits.
A 32-bit processor is expected, so there was one point where I needed to add (uint32_t) casts to avoid losing the high bytes of a word.

→ So here's CMSIS-DAP firmware for your Pro Micro (ATMEGA32U4) or Teensy 3.2 (MK20DX256) board.

Enjoy!

Comment

guid

http://www.myelin.co.nz/post/2016/3/12/#201603121

pubdate

Mon, 13 Mar 2016 00:36:51 GMT

comments

http://www.myelin.co.nz/post/comments?u=2&p=201603121&link=http://www.myelin.co.nz/post/2016/3/12/#201603121

summary

While I am so happy to have my J-Link back, the couple of weeks without it have been very productive in terms of open source contributions. After finding out that OpenOCD didn't support raw JTAG mode on CMSIS-DAP adapters, I bit that off as a potential project, and eventually got it working, then did some performance tuning, and I'm pretty pleased with it now. With my LPC-Link2, it can program the flash in an ATMEGA32U4 over JTAG at about 1/4-1/3 of the speed of the J-Link (which is kind of a speed demon). I'm going to let it soak on GitHub for a while, then clean it up and submit it to OpenOCD once it's had a bit of scrutiny.

→ Clone myelin/openocd and check out the cmsis-dap-jtag branch to try it out.

Implementing this required getting pretty familiar with the CMSIS-DAP source code and protocol, and at some point I realized that it wouldn't be super hard to port the SWD/JTAG debugger part of the CMSIS-DAP firmware over to any USB capable microcontroller. Full CMSIS-DAP support requires the debugger part, plus a serial port and a mass storage device emulator that's also capable of flashing a .bin file to a chip, but I'm not doing this to provide a USB interface to a custom board (like the ones on the mbed site), so I've skipped those two. I'll probably add in the serial port sometime, because I wrote serial bridge code for teensy-openocd-remote-bitbang already.

This was mainly a matter of getting rid of Keil-specific code, plus a small amount of debugging:

The _OUT functions (e.g. PIN_SWDIO_OUT) take a uint32_t with a boolean value in the LSB, but often junk in the higher bits.
A 32-bit processor is expected, so there was one point where I needed to add (uint32_t) casts to avoid losing the high bytes of a word.

→ So here's CMSIS-DAP firmware for your Pro Micro (ATMEGA32U4) or Teensy 3.2 (MK20DX256) board.

Enjoy!

Comment

date_timestamp

1457915811

Trying out the CNC mill at Noisebridge

I've been doing quite a bit of fabrication work at Noisebridge over the last year or so -- mostly electronics, plus a bit of 3D printing. One bit of equipment I've been interested in for a while is the MaxNC 10 CNC mill, which has been covered by a sheet for a while and appeared out of order, but has recently been unearthed and looks a bit happier. I gave it a try tonight, and it looks like it should be usable!

It's controlled by a computer under the desk running an ancient version of Ubuntu, with a realtime kernel for the control software. Booting it up and double-clicking the maxnc10ol link on the desktop brought up LinuxCNC. Flipping the red switch on the mill brought it to life, and the machine started up properly after hitting F1 then F2.

The axes are set up so that the origin is the bottom left front corner of the thing being milled, and the positive direction is right/back/up in the x/y/z axes:

X axis: + moves the platform left, - moves the platform right
Y axis: + moves the platform towards the user, - moves the platform towards the machine
Z: + moves the bit up, - moves the mill down

When homing axes, you can only move an inch in the 'negative' direction before the software will stop you -- need to rehome the axis (redefine your negative position as 0) then hit F1 twice and F2 again, then you can move it another inch. So it took a few tries to get the machine to move far enough to the left (moving the platform over to the right), and a couple to get the Y axis homed.

Once the spindle looks like it's in a safe place, you can run your program with Machine | Run Program. This will start the spindle and move through the program, then stop wherever it finished. It happily ran through its demo program.

The mill is a little small for the sorts of things I have in mind (large wooden boards cut in interesting shapes, like a lot of art at Burning Man these days) but could work nicely for PCB milling -- someone at Noisebridge has had some luck with this -- if I get some cheap PCB blanks from Aliexpress, and a couple of carbide mill bits. I suspect that the sweet spot here would be for very simple single-sided boards which are SMD-only... for example, one-off custom LED fixtures with a string of addressable LEDs and no onboard controller (or a very simple one that doesn't need any complicated traces or small pads -- so my favourite tiny MCU, the MKE04Z8VTG4, would be out, but the SOIC version would be OK, or maybe the 0.8mm TQFP44 MKE*VLD* chips).

Comment

guid

http://www.myelin.co.nz/post/2016/3/8/#201603081

pubdate

Mon, 09 Mar 2016 03:03:17 GMT

comments

http://www.myelin.co.nz/post/comments?u=2&p=201603081&link=http://www.myelin.co.nz/post/2016/3/8/#201603081

summary

I've been doing quite a bit of fabrication work at Noisebridge over the last year or so -- mostly electronics, plus a bit of 3D printing. One bit of equipment I've been interested in for a while is the MaxNC 10 CNC mill, which has been covered by a sheet for a while and appeared out of order, but has recently been unearthed and looks a bit happier. I gave it a try tonight, and it looks like it should be usable!

It's controlled by a computer under the desk running an ancient version of Ubuntu, with a realtime kernel for the control software. Booting it up and double-clicking the maxnc10ol link on the desktop brought up LinuxCNC. Flipping the red switch on the mill brought it to life, and the machine started up properly after hitting F1 then F2.

The axes are set up so that the origin is the bottom left front corner of the thing being milled, and the positive direction is right/back/up in the x/y/z axes:

X axis: + moves the platform left, - moves the platform right
Y axis: + moves the platform towards the user, - moves the platform towards the machine
Z: + moves the bit up, - moves the mill down

When homing axes, you can only move an inch in the 'negative' direction before the software will stop you -- need to rehome the axis (redefine your negative position as 0) then hit F1 twice and F2 again, then you can move it another inch. So it took a few tries to get the machine to move far enough to the left (moving the platform over to the right), and a couple to get the Y axis homed.

Once the spindle looks like it's in a safe place, you can run your program with Machine | Run Program. This will start the spindle and move through the program, then stop wherever it finished. It happily ran through its demo program.

The mill is a little small for the sorts of things I have in mind (large wooden boards cut in interesting shapes, like a lot of art at Burning Man these days) but could work nicely for PCB milling -- someone at Noisebridge has had some luck with this -- if I get some cheap PCB blanks from Aliexpress, and a couple of carbide mill bits. I suspect that the sweet spot here would be for very simple single-sided boards which are SMD-only... for example, one-off custom LED fixtures with a string of addressable LEDs and no onboard controller (or a very simple one that doesn't need any complicated traces or small pads -- so my favourite tiny MCU, the MKE04Z8VTG4, would be out, but the SOIC version would be OK, or maybe the 0.8mm TQFP44 MKE*VLD* chips).

Comment

date_timestamp

1457924597

Still trying to get JTAG working with the ESP8266

Good news -- my replacement J-Link is in the mail! Reliable hardware on the way.

In the meantime, I'm still working on my CMSIS-DAP adapter and my ESP8266 board. I'd mis-soldered a 10uF capacitor between TCK and ground (instead of 3V3 and ground), which was stopping anything from working, but after hooking my Saleae box up to the board, I debugged that fairly quickly. I fixed some bugs in the code, but OpenOCD still failed to do the initial chain scan, so something's still broken.

To debug this, I changed tack: OpenOCD also has a "remote_bitbang" driver, where it sends a series of ASCII characters to a TCP socket, to drive a very very simple JTAG adapter. This sounds like a recipe for the worst JTAG performance ever, except that the Teensy 3.2's serial-over-USB performance is very very good, and I saw some pulses as short as 2us in the logic capture, so this should be able to get me ~250 kbit/s, which is certainly good enough for now.

→ Here's the teensy-openocd-remote-bitbang code, on GitHub

It took a bit of messing around with socat on OS X to get it to bridge between a TCP socket and a serial port, but I found the magic set of arguments eventually:

socat -d -d -d file:/dev/tty.usbmodem1485121,clocal=1,cs8,nonblock=1,ixoff=0,ixon=0,ispeed=9600,ospeed=9600,raw,echo=0,crtscts=0 tcp-listen:3335,reuseaddr

The corresponding OpenOCD incantation is:

openocd -d -c "reset_config srst_only; interface remote_bitbang; remote_bitbang_port 3335; remote_bitbang_host localhost" -f target/esp8266.cfg

Unfortunately OpenOCD *still* couldn't scan the JTAG chain, and it did get bits out TDO this time, so that hardware is not completely broken, but I still have some debugging to do.

Update: I suspect TDO is in high-impedance mode -- it looks like it's floating around 1.5V. Perhaps the ESP8266 doesn't enable its JTAG pins while in reset? Either that or maybe I need to program the chip with code that enables the JTAG pins.

Update 2: Got it working! It turns out that my understanding of remote_bitbang's reset signals was the wrong way around -- RESET is active-low electrically, but the remote_bitbang protocol spec assumes active high. So "reset=0" means 3v3 on the reset pin, and vice versa. The ESP8266 JTAG pins are high-impedance in reset, which is why I saw TDO floating. This doesn't explain why my CMSIS-DAP code didn't work, but at least now I have a known-good state to compare with.

Comment

guid

http://www.myelin.co.nz/post/2016/2/27/#201602271

pubdate

Mon, 28 Feb 2016 00:37:54 GMT

comments

http://www.myelin.co.nz/post/comments?u=2&p=201602271&link=http://www.myelin.co.nz/post/2016/2/27/#201602271

summary

Good news -- my replacement J-Link is in the mail! Reliable hardware on the way.

In the meantime, I'm still working on my CMSIS-DAP adapter and my ESP8266 board. I'd mis-soldered a 10uF capacitor between TCK and ground (instead of 3V3 and ground), which was stopping anything from working, but after hooking my Saleae box up to the board, I debugged that fairly quickly. I fixed some bugs in the code, but OpenOCD still failed to do the initial chain scan, so something's still broken.

To debug this, I changed tack: OpenOCD also has a "remote_bitbang" driver, where it sends a series of ASCII characters to a TCP socket, to drive a very very simple JTAG adapter. This sounds like a recipe for the worst JTAG performance ever, except that the Teensy 3.2's serial-over-USB performance is very very good, and I saw some pulses as short as 2us in the logic capture, so this should be able to get me ~250 kbit/s, which is certainly good enough for now.

→ Here's the teensy-openocd-remote-bitbang code, on GitHub

It took a bit of messing around with socat on OS X to get it to bridge between a TCP socket and a serial port, but I found the magic set of arguments eventually:

socat -d -d -d file:/dev/tty.usbmodem1485121,clocal=1,cs8,nonblock=1,ixoff=0,ixon=0,ispeed=9600,ospeed=9600,raw,echo=0,crtscts=0 tcp-listen:3335,reuseaddr

The corresponding OpenOCD incantation is:

openocd -d -c "reset_config srst_only; interface remote_bitbang; remote_bitbang_port 3335; remote_bitbang_host localhost" -f target/esp8266.cfg

Unfortunately OpenOCD *still* couldn't scan the JTAG chain, and it did get bits out TDO this time, so that hardware is not completely broken, but I still have some debugging to do.

Update: I suspect TDO is in high-impedance mode -- it looks like it's floating around 1.5V. Perhaps the ESP8266 doesn't enable its JTAG pins while in reset? Either that or maybe I need to program the chip with code that enables the JTAG pins.

Update 2: Got it working! It turns out that my understanding of remote_bitbang's reset signals was the wrong way around -- RESET is active-low electrically, but the remote_bitbang protocol spec assumes active high. So "reset=0" means 3v3 on the reset pin, and vice versa. The ESP8266 JTAG pins are high-impedance in reset, which is why I saw TDO floating. This doesn't explain why my CMSIS-DAP code didn't work, but at least now I have a known-good state to compare with.

Comment

date_timestamp

1456706274

Kinetis E and OpenOCD update, and some Kinetis FTMRH flasher code and JTAG rambling

Following up on my last post, I managed to get OpenOCD flashing a Kinetis MKE02Z64VLD2 chip, to the point that I could hook it up to Kinetis Design Studio and program/debug almost as easily as with my J-Link, albeit without unlimited flash breakpoints, and it would often get confused about where it was on startup, although after hitting F8 everything would come back to normal.

I posted to the OpenOCD-devel mailing list and got a bunch of responses from others who had been working on the same problem, including Ivan Meleca, who had written a flash driver for the KE02/4/6 series but hadn't had time to upload it. I tested this out tonight on my MKE02Z64VLD2 and MKE04Z8VTG4 boards and it works well, so look forward to seeing that in mainline OpenOCD soon.

Ivan's code works, so I don't want to mess with it, but I'm proud of a couple of little tricks I figured out for my own implementation. I wrote a Python script to process the output of arm-none-eabi-gcc -g -O0 -mcpu=cortex-m0plus -mthumb -c -Wa,-adhln, extract out the opcodes, and comment everything up nicely, which lets you build self-contained routines that are easy to download and run on target devices. I'm not sure how everybody else wrote their flasher code, but they all seem to have ended up with slightly differently formatted hex bytes embedded in C files, so maybe by hand? Anyway, my flash algorithms and C-to-assembly-to-C code are up on GitHub as myelin-mcu-flash.

Next, my esp8266-jtag boards came back from OSHPark, and I soldered one up and connected it to my LPC-Link2, only to find that OpenOCD doesn't support JTAG operations on CMSIS-DAP adapters, only SWD. So I started writing some code to get that working. The next day, someone wrote on OpenOCD-devel that they were trying to do the same thing. So hopefully we can combine efforts and sort it out. I've learned a lot about JTAG in the last few days -- in a nutshell it's a very complicated way to reset a chip and read and write two registers, but it turns out that you can do a *lot* with those building blocks :)

Here are some notes from my attempts to get OpenOCD to support the raw JTAG operations in the CMSIS-DAP API, to talk to my ESP8266 board.

To get JTAG to work, you need to be able to clock in arbitrary sequences on the TMS pin, and you need to be able to clock in arbitrary sequences on the TDI pin with TMS low, while capturing output on the TDO pin, and setting TMS high as you clock in the final bit. This makes more sense if you look at the JTAG state diagram. TEST LOGIC RESET resets the test logic. Entering CAPTURE-DR copies data from the device into the DR register, and entering UPDATE-DR copies data from the register to the device. Every entry into SHIFT-DR clocks a bit from TDI into DR (on the rising edge of TCK), and a bit from DR out TDO (on the falling TCK edge). CAPTURE-IR, SHIFT-IR, and UPDATE-IR work similarly. All other states are just intermediaries that make it convenient to navigate around the state machine without needing too long of a sequence on TMS.

The relevant CMSIS-DAP operations, using the defines in the mbed CMSIS-DAP code:

ID_DAP_SWJ_Sequence, which clocks out a sequence on TMS
ID_DAP_JTAG_Sequence, which clocks out sequences on TDI, and optionally sets TMS and captures TDO.

DAP_SWJ_Sequence is already supported, because it's necessary to switch between JTAG and SWD mode. ID_DAP_JTAG_Sequence isn't used for SWD, but we need it for raw JTAG. The packet format seems to be:

byte 0: the number of TDI bit sequences in the packet
byte 1: sequence info for the first sequence
byte 2..length(sequence 1): sequence data for the first sequence
... other sequences follow ...

The max sequence length is 8 bytes (64 bits), and the sequence info byte is: (0x80 if capturing TDO, else 0) | (0x40 if TMS is set, else 0) | (sequence length in bits, with 64 encoded as 0).

This suggests that a 'scan' operation (writing something into the IR or DR register) will require one DAP_SWJ_Sequence command to move the TAP to Shift_IR or Shift_DR and one DAP_JTAG_Sequence command to write the command. We can probably just leave the TAP in the shift state once the command is done, but we could use another DAP_SWJ_Sequence to get back to the idle state if that's not okay.

Looking at the bitbang driver, it looks like the desired end state is given in cmd->cmd.scan->end_state, the input and type (input or output) can be obtained via jtag_build_buffer and jtag_scan_type, and it's an IR scan if cmd->cmd.scan->ir_scan is set. The actual scan function moves to the TAP_DRSHIFT or TAP_IRSHIFT state as required, then clocks the data in/out on TDI/TDO, then executes another state_move to get to the desired end state. So I guess we'll actually require three DAP operations to do this, because of the final requirements for TMS (unless we want to try to pack those into the JTAG_Sequence packet).

11pm update: Implemented! It compiles, but I haven't tested it at all yet, so expect many bugs..

Comment

guid

http://www.myelin.co.nz/post/2016/2/25/#201602251

pubdate

Mon, 26 Feb 2016 03:36:12 GMT

comments

http://www.myelin.co.nz/post/comments?u=2&p=201602251&link=http://www.myelin.co.nz/post/2016/2/25/#201602251

summary

Following up on my last post, I managed to get OpenOCD flashing a Kinetis MKE02Z64VLD2 chip, to the point that I could hook it up to Kinetis Design Studio and program/debug almost as easily as with my J-Link, albeit without unlimited flash breakpoints, and it would often get confused about where it was on startup, although after hitting F8 everything would come back to normal.

I posted to the OpenOCD-devel mailing list and got a bunch of responses from others who had been working on the same problem, including Ivan Meleca, who had written a flash driver for the KE02/4/6 series but hadn't had time to upload it. I tested this out tonight on my MKE02Z64VLD2 and MKE04Z8VTG4 boards and it works well, so look forward to seeing that in mainline OpenOCD soon.

Ivan's code works, so I don't want to mess with it, but I'm proud of a couple of little tricks I figured out for my own implementation. I wrote a Python script to process the output of arm-none-eabi-gcc -g -O0 -mcpu=cortex-m0plus -mthumb -c -Wa,-adhln, extract out the opcodes, and comment everything up nicely, which lets you build self-contained routines that are easy to download and run on target devices. I'm not sure how everybody else wrote their flasher code, but they all seem to have ended up with slightly differently formatted hex bytes embedded in C files, so maybe by hand? Anyway, my flash algorithms and C-to-assembly-to-C code are up on GitHub as myelin-mcu-flash.

Next, my esp8266-jtag boards came back from OSHPark, and I soldered one up and connected it to my LPC-Link2, only to find that OpenOCD doesn't support JTAG operations on CMSIS-DAP adapters, only SWD. So I started writing some code to get that working. The next day, someone wrote on OpenOCD-devel that they were trying to do the same thing. So hopefully we can combine efforts and sort it out. I've learned a lot about JTAG in the last few days -- in a nutshell it's a very complicated way to reset a chip and read and write two registers, but it turns out that you can do a *lot* with those building blocks :)

Here are some notes from my attempts to get OpenOCD to support the raw JTAG operations in the CMSIS-DAP API, to talk to my ESP8266 board.

To get JTAG to work, you need to be able to clock in arbitrary sequences on the TMS pin, and you need to be able to clock in arbitrary sequences on the TDI pin with TMS low, while capturing output on the TDO pin, and setting TMS high as you clock in the final bit. This makes more sense if you look at the JTAG state diagram. TEST LOGIC RESET resets the test logic. Entering CAPTURE-DR copies data from the device into the DR register, and entering UPDATE-DR copies data from the register to the device. Every entry into SHIFT-DR clocks a bit from TDI into DR (on the rising edge of TCK), and a bit from DR out TDO (on the falling TCK edge). CAPTURE-IR, SHIFT-IR, and UPDATE-IR work similarly. All other states are just intermediaries that make it convenient to navigate around the state machine without needing too long of a sequence on TMS.

The relevant CMSIS-DAP operations, using the defines in the mbed CMSIS-DAP code:

ID_DAP_SWJ_Sequence, which clocks out a sequence on TMS
ID_DAP_JTAG_Sequence, which clocks out sequences on TDI, and optionally sets TMS and captures TDO.

DAP_SWJ_Sequence is already supported, because it's necessary to switch between JTAG and SWD mode. ID_DAP_JTAG_Sequence isn't used for SWD, but we need it for raw JTAG. The packet format seems to be:

byte 0: the number of TDI bit sequences in the packet
byte 1: sequence info for the first sequence
byte 2..length(sequence 1): sequence data for the first sequence
... other sequences follow ...

The max sequence length is 8 bytes (64 bits), and the sequence info byte is: (0x80 if capturing TDO, else 0) | (0x40 if TMS is set, else 0) | (sequence length in bits, with 64 encoded as 0).

This suggests that a 'scan' operation (writing something into the IR or DR register) will require one DAP_SWJ_Sequence command to move the TAP to Shift_IR or Shift_DR and one DAP_JTAG_Sequence command to write the command. We can probably just leave the TAP in the shift state once the command is done, but we could use another DAP_SWJ_Sequence to get back to the idle state if that's not okay.

Looking at the bitbang driver, it looks like the desired end state is given in cmd->cmd.scan->end_state, the input and type (input or output) can be obtained via jtag_build_buffer and jtag_scan_type, and it's an IR scan if cmd->cmd.scan->ir_scan is set. The actual scan function moves to the TAP_DRSHIFT or TAP_IRSHIFT state as required, then clocks the data in/out on TDI/TDO, then executes another state_move to get to the desired end state. So I guess we'll actually require three DAP operations to do this, because of the final requirements for TMS (unless we want to try to pack those into the JTAG_Sequence packet).

11pm update: Implemented! It compiles, but I haven't tested it at all yet, so expect many bugs..

Comment

date_timestamp

1456716972

Working toward being able to flash Kinetis E chips with OpenOCD, using USBDM flash routines

My J-Link broke yesterday, so I'm back to using either my FRDM-KE04Z board with USBDM firmware or my LPC-Link2 with CMSIS-DAP firmware to program my Kinetis E devices. USBDM seems to only support Windows and Linux, so I'm trying out the LPC-Link2 with OpenOCD today.

My chip is a Kinetis MKE02Z64VLD2 (64 kB flash, 4 kB RAM, 20 MHz max), which OpenOCD doesn't have a flash algorithm for. With any luck it should be able to debug my chip out of the box, but I'm going to have to steal code from USBDM to get it to download my code.

First things first... Homebrew seems to install OpenOCD 0.8.0 when you run 'brew install open-ocd', so I built and installed 0.10.0-dev from git. Copying /usr/local/share/openocd/script/kl25.cfg as ke02.cfg and running openocd with the following in openocd.cfg seems to get it started:

interface cmsis-dap
adapter_khz 500
transport select swd

source ke02.cfg

Once OpenOCD is running, it listens on ports 3333 (GDB remote protocol) and 4444 (command shell). telnet localhost 4444 gives you something like the J-Link Commander console. 'reset' and 'halt' do the expected things, so it looks like it's handling the SWD protocol just fine.

Now to figure out how to extract the KE02Z64 flashing code from USBDM... here's what I've figured out:

USBDM flashes the KE02 using its Kinetis-FTMRH-flash-program-gcc helper, which is downloaded into the target over SWD, then executed to copy chunks of code from RAM to flash.
The metadata mapping devices to flash routines is in arm_devices.xml.
It looks like flashing happens via the FlashProgrammer_ARM class: loadLargeTargetProgram loads the flash helper into the target's RAM, and executeTargetProgram runs it.
It looks like the target program contains a header, at a known location in its data segment.

So it looks like we have all the pieces of the puzzle... the trick will be to get FlashProgrammer_ARM.cpp wired up to OpenOCD's interface code, so it can write and execute code on the target.

(to be continued)

Comment

guid

http://www.myelin.co.nz/post/2016/2/15/#201602151

pubdate

Mon, 15 Feb 2016 23:30:20 GMT

comments

http://www.myelin.co.nz/post/comments?u=2&p=201602151&link=http://www.myelin.co.nz/post/2016/2/15/#201602151

summary

My J-Link broke yesterday, so I'm back to using either my FRDM-KE04Z board with USBDM firmware or my LPC-Link2 with CMSIS-DAP firmware to program my Kinetis E devices. USBDM seems to only support Windows and Linux, so I'm trying out the LPC-Link2 with OpenOCD today.

My chip is a Kinetis MKE02Z64VLD2 (64 kB flash, 4 kB RAM, 20 MHz max), which OpenOCD doesn't have a flash algorithm for. With any luck it should be able to debug my chip out of the box, but I'm going to have to steal code from USBDM to get it to download my code.

First things first... Homebrew seems to install OpenOCD 0.8.0 when you run 'brew install open-ocd', so I built and installed 0.10.0-dev from git. Copying /usr/local/share/openocd/script/kl25.cfg as ke02.cfg and running openocd with the following in openocd.cfg seems to get it started:

interface cmsis-dap
adapter_khz 500
transport select swd

source ke02.cfg

Once OpenOCD is running, it listens on ports 3333 (GDB remote protocol) and 4444 (command shell). telnet localhost 4444 gives you something like the J-Link Commander console. 'reset' and 'halt' do the expected things, so it looks like it's handling the SWD protocol just fine.

Now to figure out how to extract the KE02Z64 flashing code from USBDM... here's what I've figured out:

USBDM flashes the KE02 using its Kinetis-FTMRH-flash-program-gcc helper, which is downloaded into the target over SWD, then executed to copy chunks of code from RAM to flash.
The metadata mapping devices to flash routines is in arm_devices.xml.
It looks like flashing happens via the FlashProgrammer_ARM class: loadLargeTargetProgram loads the flash helper into the target's RAM, and executeTargetProgram runs it.
It looks like the target program contains a header, at a known location in its data segment.

So it looks like we have all the pieces of the puzzle... the trick will be to get FlashProgrammer_ARM.cpp wired up to OpenOCD's interface code, so it can write and execute code on the target.

(to be continued)

Comment

date_timestamp

1455579020

Using a Teensy as a USB-Serial bridge to program an ESP8266

Hopefully in future I'll be able to program my ESP8266 modules using JTAG, but I have a project from last year which used the ESP8266, didn't bring out the JTAG pins, and tied GPIO15 (TDO) to ground, which means it would require desoldering the ESP-03 module to fix. I programmed it using the ESP8266 Arduino extension and a Sparkfun FTDI Basic Breakout board, which unfortunately doesn't bring out RTS, which meant for a lot of power cycling. To make matters worse, the FT232 seems to crash sometimes when subjected to this sort of thing, losing control of its DTR output and needing power cycling *itself*.

I happen to have a couple of Teensy 3.2 boards here, however, and they're super easy to program up to act as a more flexible serial adapter. Here's the sketch I ended up with, which happily programs my ESP-03 at the max upload speed selectable in the ESP8266 Arduino IDE, 921600 baud, and with any luck will work with the ESP GDB stub:

// Simple USB-Serial bridge for programming ESP8266 modules

// new connector:
// GND (black) RXD (grey)  GPIO0 (brown)
// 5V (n/c)    TXD (white) RTS (purple)

// RX2 = pin 9, white wire, connects to ESP8266 TX [saleae ch0]
// TX2 = pin 10, grey wire, connects to ESP8266 RX [saleae ch1]
#201602131" onclick="window.open( this.href, 'comments', 'width=515, height=480, location=0, resizable=1, scrollbars=1, status=0, toolbar=0, directories=0'); return(false);" title="Click here to comment on this post.">Comment

guid
http://www.myelin.co.nz/post/2016/2/13/#201602131
category
ESP8266Electronics
pubdate
Mon, 14 Feb 2016 00:20:18 GMT
comments
http://www.myelin.co.nz/post/comments?u=2&p=201602131&link=http://www.myelin.co.nz/post/2016/2/13/#201602131
summary

Hopefully in future I'll be able to program my ESP8266 modules using JTAG, but I have a project from last year which used the ESP8266, didn't bring out the JTAG pins, and tied GPIO15 (TDO) to ground, which means it would require desoldering the ESP-03 module to fix.  I programmed it using the ESP8266 Arduino extension and a Sparkfun FTDI Basic Breakout board, which unfortunately doesn't bring out RTS, which meant for a lot of power cycling.  To make matters worse, the FT232 seems to crash sometimes when subjected to this sort of thing, losing control of its DTR output and needing power cycling *itself*.


I happen to have a couple of Teensy 3.2 boards here, however, and they're super easy to program up to act as a more flexible serial adapter.  Here's the sketch I ended up with, which happily programs my ESP-03 at the max upload speed selectable in the ESP8266 Arduino IDE, 921600 baud, and with any luck will work with the ESP GDB stub:


// Simple USB-Serial bridge for programming ESP8266 modules

// new connector:
// GND (black) RXD (grey)  GPIO0 (brown)
// 5V (n/c)    TXD (white) RTS (purple)

// RX2 = pin 9, white wire, connects to ESP8266 TX [saleae ch0]
// TX2 = pin 10, grey wire, connects to ESP8266 RX [saleae ch1]
#201602131" onclick="window.open( this.href, 'comments', 'width=515, height=480, location=0, resizable=1, scrollbars=1, status=0, toolbar=0, directories=0'); return(false);" title="Click here to comment on this post.">Comment
date_timestamp

1455495618