Weekly report - Week 11

Pencil down is today, so it's probably my last weekly report...

Status:

• Improved the documentation on my wiki page, including a paragraph about a potential problem with bandwidth allocation.
• Got my FPGA4U to work.
• No success at getting the TI MSP430 Launchpad to work...
• Attempted to add support for high-bandwidth isochronous endpoints, but it hit yet another bug in the MUSB driver, happening even with DMA disabled...

Plans:

• I may be able to fix the MSP430 problem...

Risks:

• ?

Weekly report - Week 10

Status:

• Dynamic sizing of MUSB FIFOs: there is a new module parameter, fifo_config, that allows to set the FIFO configuration for each endpoint. An example would be: ep1in:1024,ep3out:32,ep4in:512. In that case, EP1 IN will have 1024 bytes of buffer, EP3 OUT 32 bytes, etc... Note that FIFO sizes must be powers of 2.
• Modified the sniff script to take that into account: it reads the device descriptor, and chooses the closest bigger or equal power of 2 size for the endpoint FIFO.
• Improved the instructions on the wiki, taking into account feedback from Frans.

Plans:

• I just bought a USB 2.0 webcam, that needs high-bandwith isochronous endpoints. Getting that to work will require modifications to the MUSB driver

Risks:

• Lack of time... There is only a week left until the suggested pencil down date...
• Adding high-bandwidth endpoints support to MUSB may be tricky.

Weekly report - Week 9

As expected, it wasn't a very productive week...

Status:

• Tested my MUSB modification when double-buffering is enabled. It seems to cause a lot of packet losses with isochronous transfers, so I'm wondering if my patch is breaking something.
• Added recipes for libpcap 1.1.1 and tcpdump 4.1.1 in OpenEmbedded. Commits: libpcap, and tcpdump.
• These versions are able to capture USB traffic in a format that is compatible with wireshark.
• Added an all-in-one capture script: sniff. See the script here, and updated instructions.

Plans:

• Investigate the MSP430 Launchpad problem.
• Choose the FIFO size for each of the endpoints from the device descriptors: right now, I have 2 extra FIFO modes, 6, and 7, used respectively for the webcam, and the headset. When that is done, we will have some fully automatic sniffing solution.
• MUSB related: try to find out what is going wrong when double-buffering is enabled, and experiment with high-bandwidth transfers.

Risks:

• Lack of time... This week will be the last week I will be able to work full-time on this project.

Weekly report - Week 8

Status:

• Managed to fix the bug with short ISO packets in the MUSB driver (see Friday's post).
• Retested the proxy driver with the devices I have. I did some more thorough Bluetooth testing, including file transfers (table here).

Plans:

• Test with a few more devices, before leaving Switzerland.
• The MSP430 Launchpad doesn't work with the proxy (it seems a bit fragile, even when connected directly to my PC, I suspect a driver problem): investigate this.
• It would be good to add support for high-bandwidth endpoints in the MUSB driver, and test it with the proxy (I don't own any device requiring this at the moment).
• Test packet capture with libpcap on the BeagleBoard (to get wireshark-compatible capture files).
  

Risks:

• I'll be on a plane for most of Wednesday and Thursday (and jet-lagged after that...), so don't expect too much out of this week...

MUSB isochronous transfers fixed (hopefully)

So, it took a while, but I think I managed to fix the MUSB bug, the relevant commit is here.

The MUSB DMA supports 2 different transfer modes: mode-0 transfers packets one at a time, requiring CPU intervention to reload the next packet, while mode-1 is able to transfer multiple packets, before triggering an interrupt.

For every mode-0 transfer, or for the last mode-1 packet, the packet in the FIFO must be manually flushed, that is, MUSB_TXCSR_TXPKTRDY must be set. This was somehow done in musb_g_tx (musb_gadget.c:513), but incorrectly. Using the procedure described on the wiki, with a packet size of 4 (parameter -x 4), I got the following usbmon output:

S Zi:2:120:1 -115:1024:0 1 -18:0:512 512 <
C Zi:2:120:1 0:1024:5584:0 1 0:0:512 512 = de010203 de050607 de090a0b de0d0e0f de111213 de151617 de19
1a1b de1d1e1f
S Zi:2:120:1 -115:1024:5584 1 -18:0:512 512 <
C Zi:2:120:1 0:1024:6608:0 1 0:0:4 4 = df010203
S Zi:2:120:1 -115:1024:6608 1 -18:0:512 512 <
C Zi:2:120:1 0:1024:7632:0 1 0:0:4 4 = df010203
S Zi:2:120:1 -115:1024:7632 1 -18:0:512 512 <
C Zi:2:120:1 0:1024:464:0 1 0:0:4 4 = df010203
S Zi:2:120:1 -115:1024:464 1 -18:0:512 512 <
...

The first packet is 512 bytes, which is normal, and then same data was repeated again and again: df is supposed to be incremented, that is, the next packets should contain e0010203, e1010203. I tried commenting out

if (csr & MUSB_TXCSR_TXPKTRDY)
  return;

as recommend by Ajay, with the same result...

I then realized that the following lines (which, BTW, do not necessarily send a zero packet, they simply ask the MUSB controller to fetch the next packet from the FIFO):

DBG(4, "sending zero pkt\n");
musb_writew(epio, MUSB_TXCSR, MUSB_TXCSR_MODE
  | MUSB_TXCSR_TXPKTRDY);

were clearing all flags from MUSB_TXCSR (including things like MUSB_TXCSR_ISO or >MUSB_TXCSR_DMAEN). So I replaced these with:

DBG(4, "sending zero pkt\n");
musb_writew(epio, MUSB_TXCSR, csr
  | MUSB_TXCSR_TXPKTRDY);

where csr is the current value of MUSB_TXCSR. I'm not sure if some other flags must be cleared, but at least, it works for my test case...

And the output looks better:

S Zi:2:125:1 -115:1024:0 1 -18:0:512 512 <
C Zi:2:125:1 0:1024:2608:0 1 0:0:512 512 = de010203 de050607 de090a0b de0d0e0f de111213 de151617 de191a1b de1d1e1f
S Zi:2:125:1 -115:1024:2608 1 -18:0:512 512 <
C Zi:2:125:1 0:1024:3632:0 1 0:0:12 12 = df010203 e0010203 e1010203
S Zi:2:125:1 -115:1024:3632 1 -18:0:512 512 <
C Zi:2:125:1 0:1024:4656:0 1 0:0:12 12 = e2010203 e3010203 e4010203
S Zi:2:125:1 -115:1024:4656 1 -18:0:512 512 <
C Zi:2:125:1 0:1024:5680:0 1 0:0:4 4 = e5010203
S Zi:2:125:1 -115:1024:5680 1 -18:0:512 512 <
C Zi:2:125:1 0:1024:6704:0 1 0:0:4 4 = e6010203
...

Except that 3 requests were merged into one packet (twice), which is not supposed to happen, I suppose: If the gadget driver sends a short request to the gadget controller, it probably means that it wants a short packet to be transmitted.

In any case, a bigger problem appeared, if the driver sent packets of 5 bytes, instead of 4 bytes:

S Zi:2:009:1 -115:8:0 1 -18:0:512 512 <
C Zi:2:009:1 0:8:4568:0 1 0:0:512 512 = de010203 de050607 de090a0b de0d0e0f de111213 de151617 de191a1b de1d1e1f
S Zi:2:009:1 -115:8:4568 1 -18:0:512 512 <
C Zi:2:009:1 0:8:4576:0 1 0:0:15 15 = df010203 e0010203 e1010203 de0d0e
S Zi:2:009:1 -115:8:4576 1 -18:0:512 512 <
C Zi:2:009:1 0:8:4584:0 1 0:0:10 10 = e1e20203 e3010203 e301
S Zi:2:009:1 -115:8:4584 1 -18:0:512 512 <
C Zi:2:009:1 0:8:4592:0 1 0:0:15 15 = e4010203 e5010203 e6010203 e60102
S Zi:2:009:1 -115:8:4592 1 -18:0:512 512 <
C Zi:2:009:1 0:8:4600:0 1 0:0:5 5 = e7010203 e7
S Zi:2:009:1 -115:8:4600 1 -18:0:512 512 <
C Zi:2:009:1 0:8:4608:0 1 0:0:5 5 = e8010203 e8
...

The second packet data is badly corrupted, it should be df010203 04e00102 0304e101 020304. It seems like the DMA engine is not able to properly write unaligned data to the FIFO.

After seeing that problem (added to the fact that requests should not be merged together in the first place), I realized that MUSB_TXCSR_TXPKTRDY was probably not set at the right place in the code: it should be set right after the DMA finishes copying the data to the FIFO.

This happens in musbhsdma.c, function dma_controller_irq. And the code was already there, it's just that, for some reasons, it was only enabled for the host mode, and not the peripheral mode (if (devctl & MUSB_DEVCTL_HM), line 533). I enabled that code path, with a fix for DMA mode 0: it is useless to clear MUSB_TXCSR_DMAMODE in that case, and MUSB_TXCSR_DMAENAB may need to be cleared before clearing MUSB_TXCSR_DMAMODE, but the flag needs to be set again, otherwise this confuses the driver...

Then, I disabled the code setting MUSB_TXCSR_TXPKTRDY in musb_g_tx, and replaced it with another piece of code that forces the packet to be sent, by setting MUSB_TXCSR_FLUSHFIFO, and, it seems to work!

S Zi:2:016:1 -115:8:0 1 -18:0:512 512 <
C Zi:2:016:1 0:8:1328:0 1 0:0:512 512 = de010203 de050607 de090a0b de0d0e0f de111213 de151617 de191a1b de1d1e1f
S Zi:2:016:1 -115:8:1328 1 -18:0:512 512 <
C Zi:2:016:1 0:8:1336:0 1 0:0:5 5 = df010203 df
S Zi:2:016:1 -115:8:1336 1 -18:0:512 512 <
C Zi:2:016:1 0:8:1344:0 1 0:0:5 5 = e0010203 e0
S Zi:2:016:1 -115:8:1344 1 -18:0:512 512 <
C Zi:2:016:1 0:8:1352:0 1 0:0:5 5 = e1010203 e1
...

Long story short, the webcam works, even with DMA enabled... And g_ether still works (i.e., I didn't break everything else)...

The latest code is available here: the kernel is now based on a 2.6.34 tree.

Weekly report - Week 7

Status:

• Switched to a more recent kernel, based on 2.6.34, to make sure I have the latest bug fixes, as recommended by Ajay.
  
• I shifted focus towards debugging the MUSB driver, and particularly the code path used when DMA is used. It could be possible to debug the MUSB driver using my proxy driver, connecting the webcam, but the transfer rate and packet sizes are rather unpredictable, making it hard to reproduce problems.
  
• My first idea was to program a Cypress FX2 to stream isochronous packets, connect the FX2 to the BeagleBoard, with my proxy driver. I tried to find some ready-made test firmwares, without much success: none of these tested isochronous transfers, and most of them were written for the Cypress EZ-USB chip (i.e., a (incompatible) predecessor to the FX2).
  
• Since the problem lies in the MUSB driver, I don't really need to use my proxy driver. I found a testing gadgetfs driver, which supports isochronous transfers. I could then hack the driver to force it to send short packets, triggering the MUSB bug (some instructions can be found here).

Plans:

• Continue debugging the MUSB driver with short isochronous packets. If I cannot find a solution after a few days, I'll report the problem to the linux-usb list.
  
• High-throughput endpoints are not supported in the MUSB driver, that's something I could try to implement.

Risks:

• Debugging the MUSB driver is a bit difficult: since isochronous are delay sensitive, displaying some debugging messages causes packet losses.

Weekly report - Week 6

Status:

• Mostly worked on the webcam, I got isochronous transfers to work, after identifying a bug in the MUSB driver (related to the DMA controller). See last Friday's post.

Plans:

• Attempt to fix the MUSB driver with isochronous endpoints, disabling the DMA works, but hurts the performance...
  
• Program a Cypress FX2 with a high-speed, high-bandwidth endpoint (the MUSB does not seems to handle that, so it would be good to fix it as well). Maybe the FX2 can be used for bulk bandwidth tests too.

Risks:

• Lack of available documentation about MUSB.

Isochronous transfers - Webcam

These past 2 weeks, I focused on getting a USB webcam to work, and particularly the 2 isochronous endpoints, used for audio and video coming from the webcam. This is a full speed device, and its descriptor is below (I trimmed it a bit):

Bus 002 Device 061: ID 046d:08da Logitech, Inc. QuickCam Messanger
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0         8
  idVendor           0x046d Logitech, Inc.
  idProduct          0x08da QuickCam Messanger
  bcdDevice            1.00
  iManufacturer           0 
  iProduct                0 
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength          336
    bNumInterfaces          3
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xa0
      (Bus Powered)
      Remote Wakeup
    MaxPower              100mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass    255 Vendor Specific Subclass
      bInterfaceProtocol    255 Vendor Specific Protocol
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            1
          Transfer Type            Isochronous
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0000  1x 0 bytes
        bInterval               1
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval              10
    Interface Descriptor: ...
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       6
      bNumEndpoints           2
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass    255 Vendor Specific Subclass
      bInterfaceProtocol    255 Vendor Specific Protocol
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            1
          Transfer Type            Isochronous
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0300  1x 768 bytes
        bInterval               1
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval              10
    Interface Descriptor: ...
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      ...
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       0
      bNumEndpoints           0
      bInterfaceClass         1 Audio
      bInterfaceSubClass      2 Streaming
      bInterfaceProtocol      0 
      iInterface              0 
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       1
      bNumEndpoints           1
      bInterfaceClass         1 Audio
      bInterfaceSubClass      2 Streaming
      bInterfaceProtocol      0 
      iInterface              0 
      AudioStreaming Interface Descriptor: ....
      AudioStreaming Interface Descriptor: ....
      Endpoint Descriptor:
        bLength                 9
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            1
          Transfer Type            Isochronous
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0010  1x 16 bytes
        bInterval               1
        bRefresh                0
        bSynchAddress           0
        AudioControl Endpoint Descriptor: ....
Device Status:     0x0000
  (Bus Powered)

Audio comes through EP3, and video through EP1.

The first thing to note is that isochronous endpoints are not allowed to be part of the default configuration for a device, so, to support these transfers, the SET_INTERFACE command has to be supported. For example, when a video player is launched, the alternate setting 6 is selected for interface 0, allowing packets of up to 768 bytes on EP1 IN.

When I tried with audio playback, URBs could not be submitted to the device, with error -28, i.e. ENOSPC, which has something to do with a problem in USB bandwidth allocation. The kernel configuration CONFIG_USB_EHCI_TT_NEWSCHED is meant to help with that kind of problems, and it does, no more -28 errors, and audio worked. I realised much later that the same problem happens when I connect the webcam to my PC, through the same USB hub I normally use with the BeagleBoard, so this is probably a hub problem. The webcam works fine with another USB hub on my PC, but that USB hub refuses to work with the BeagleBoard, so I need to find a better supported USB hub.

The next problem was about the packet size used for EP1, 768 bytes is more than the FIFO size (512 bytes) in the default endpoint configurations (fifo_mode=5, see here, __initdata mode_5_cfg). I simply created a new configuration, with an EP1 FIFO size of 1024 bytes (fifo_mode=6).

And then, nothing. When I started a video player, some packets were received properly from the webcam. The proxy driver would then submit a request to the MUSB controller, and, then, the request completion handler was only called for the first request, and never for any subsequent packet. After a few hours of debugging, I found that the MUSB driver sends a zero-length packet after sending a packet smaller than the maximum size (see musb_gadget.c, around line 520). This may make sense for bulk transfers (and I have doubts...), but not for isochronous transfers. Furthermore, sending that packet seemed to stall the endpoint, and no further requests would be processed. I tried a quick hack (around line 503), and avoid sending a zero-length packet, but transfers were still not working.

After seeing the line testing #ifdef CONFIG_USB_INVENTRA_DMA around there, I started wondering about that DMA engine, and disabled it (use_dma=0 parameter to musb_hdrc), and the webcam video started to work.

Now, the question is, why did it work with audio, but not video? The answer is quite simple, audio packets are always filled: Each packet contains exactly 16 bytes, as can be seen in the following usbmon line:

ffff88006e576100 2368995185 C Zi:6:075:3 0:1:831804651:0 1 0:0:16 16 =
6aff81ff 65ff6fff 7fff6fff 9dff86ff

(The interesting bit is 0:0:16, the 3rd number indicates that the packet contains 16 bytes. Other packets are similar, just with different payload and timestamps.)

However, for the webcam, the packet size is variable:

ffff880080052000 2351992862 C Zi:6:075:1 0:1:831787618:0 32 0:0:250
0:768:292 0:1536:338 0:2304:362 0:3072:254 6807 = ...

Here, the first packet contains 250 bytes, the second one 292, then 338, etc., triggering the MUSB/DMA bug.

Finally, you can try the latest version by using the instructions found here, and the branch stable-20100702 (here).

Weekly report - Week 5

Status:

• Managed to fix the throughput issue with my USB flash device. I now get 19.11 MB/sec, close to the maximum throughput when the USB device is directly connected to the host.
  In my previous version, I used only one urb/request pair per endpoint: when data came from the device (through an urb), it was forwarded to the host (as a request), and the driver would only resubmit the urb when the data was properly sent to the host.
  


• I fixed this by having 8 urb/requests submitted to the USB controller driver, to make sure that there is always some buffer available to write incoming data to. This alone improved the performance to 12.79 MB/sec.
• The second thing I did is to have a larger buffer per urb/request: instead of allocating 512 bytes (i.e. wMaxPacketSize) per urb/request, I allocated 4096 bytes. I guess that this reduces the number of interrupts (and the number of times my handlers are called), giving an extra performance gain. This should also be transparent to the host/device, as transfers larger than wMaxPacketSize are split into smaller packets of size wMaxPacketSize (except the last one). Since our buffer size is a multiple of wMaxPacketSize, this should not cause problems.
  


• SET_INTERFACE command works (this is a prerequisite for isochronous transfers). This was a bit tricky, as disabling endpoints cannot be done in interrupt context, so I submit a work_struct to do that in process context.
• I started working on isochronous transfers. I got clean audio coming from the microphone integrated in a webcam. The video still does not work, but I am working on it.
  

Plans:

• Fix isochronous transfers, as the urb buffer is filled in a special way in these cases (I'll write about that next week), my code works by chance with the audio from the webcam.
• Code cleanup, again. Some parts are unstable again, causing oopses and warning, after some of my recent updates.
  
• Maybe, collect statistics on urb/request usage, to see how many urb/requests are needed for maximum performance.
  

Risks:

• From what I see in MUSB, the driver does not support high-bandwidth endpoints (i.e. isochronous transfers with maxPacketSize > 1024 bytes), that are used by High-Speed device. The driver may need to be updated.
• Bandwidth allocation: with this proxy model, there is nothing much with can do if bandwidth allocation fails on the BeagleBoard. If the host believes enough bandwidth is available, it will use high-throughput interfaces, and there is nothing that can be done on the BeagleBoard to prevent that, in case there is not enough bandwidth available on the EHCI controller of the BeagleBoard. This causes problems with the video on the USB webcam, but the configuration option CONFIG_USB_EHCI_TT_NEWSCHED seems to help, though.

Weekly report - Week 4

Status:

• Did some code cleanup, hopefully the code should not crash anymore, and be free of memory leaks. Instructions on how to use my code can be found here, and you should use this branch of my git tree.
• Documented a bit more the data flow, in the code.
• Tested more systematically with a few different devices. A table of those devices can be found on the wiki page.

Plans:

• Improve performance with the USB flash device (current speed: 3.77MB/sec, whereas we can get as high as 19.76 MB/sec without proxy).
• Isochronous transfers (see last week).
  

Risks:

• Risks from last week are still around.
  

Weekly report - Week 3

Status:

• I now have a more general proxy driver, that should work with any device, as long as it does not use any isochronous endpoints, and does not require any change in configuration or interface (i.e., only one SET_CONFIGURATION control request is used, and no SET_INTERFACE).
• Managed to get a DVB-T tuner to work properly (mine only uses bulk transfers). There is no noticeable difference between connecting the tuner directly to a PC, or through the BeagleBoard, even though only a single transfer is buffered in the proxy driver.
  

Plans:

• Code cleanup. There are frequent kernel oops (notably upon module unloading), a few memory leaks I know about, and the code is lacking error checking in many places.
• Some documentation about the data flow in the code (a transfer goes through a few different callbacks, on the gadget and device sides, and it is a bit hard to follow what is happening by looking at the code).
• Isochronous transfers seems to work differently from control/interrupt/bulk transfers. I need to look at that, and implement them (I have a USB webcam and a headset, which would make good test cases). Since isochronous endpoints cannot be part of the default interface of a device, I need to implement SET_INTERFACE as well.
  

Risks:

• Endpoints maximum packet size. With the current FIFO configuration, the MUSB block probably only supports a maximum of 512 bytes per transfer. More would be needed for the webcam to work (768 at max, the standard allowing up to 3072 in High-Speed mode).
• I tried the g_audio gadget, which uses isochronous transfers, but experienced some corruption (the music plays, but with noticeable corruption), I do not know if it is due to the MUSB, or to the g_audio gadget.
• Bandwidth allocation. I picked some functions from the USB core to initialize the USB device and its endpoints correctly, but I may have left some parts aside, one of them being the function that allocates USB bandwidth.

Bidirectional communication with an Arduino

I now have transparent communication with an Arduino (programming the device work, as well as serial communication with the loaded program). You can find the proxy driver code here, or fetch the whole branch here.

Basically, the Arduino appears as a USB->serial converter to the host, with 2 bulk endpoints, with directions IN and OUT (for both directions of the serial transfers):

Bus 007 Device 120: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0         8
  idVendor           0x0403 Future Technology Devices International, Ltd
  idProduct          0x6001 FT232 USB-Serial (UART) IC
  bcdDevice            6.00
  iManufacturer           1 FTDI
  iProduct                2 FT232R USB UART
  iSerial                 3 A800evGn
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           32
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0xa0
      (Bus Powered)
      Remote Wakeup
    MaxPower               90mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass    255 Vendor Specific Subclass
      bInterfaceProtocol    255 Vendor Specific Protocol
      iInterface              2 FT232R USB UART
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
Device Status:     0x0000
  (Bus Powered)

The first endpoint (EP 1 IN), for communication from the Arduino to the host, worked almost immediately: apart from the transfer type, this is the same as for HID devices.

Getting the second endpoint (EP 2 OUT) to be configured properly on the gadget side was a little tricky. In fact, the infrastucture provided in drivers/usb/gadget/epautoconf.c, namely the function usb_ep_autoconfig, does not allow to choose an endpoint address: This function looks at the list of endpoints provided by the gadget controller driver, and simply selects the first endpoint that supports the request transfer type, direction and maximum packet size (here, Bulk OUT, 64 bytes). In our case, the function selects EP 1 OUT, while the host still thinks transfers should be done on EP 2 OUT.

To get it to work, I had to rewrite usb_ep_autoconfig (see find_gadget_endpoint), and I was a little horrified to see the way gadget controllers endpoints are advertised: Each endpoint structure has a char* name field, which tells the endpoint address, direction, and possibly the supported transfer types. To find EP 2 OUT, the function has to go through all the endpoints, until it finds one whose name is ep2out. A bit kludgy, but well, it works, at least for the MUSB controller...

Data transfers are handled differently, whether they happens on a IN, or OUT, endpoint. For IN endpoints, we follow the following procedure:
1. Initialization (see bridge_endpoint): We submit an URB (USB request block) to the device, asking for data.
2. Device callback (see device_epin_irq): We got some data. Copy it to a request (the gadget equivalent of an URB), and submit it.
3. Gadget callback (see gadget_epin_complete): The request was submitted correctly. Resubmit the URB, so we can receive the next packet.

OUT endpoints are handled similarly: Upon initialization, a request is submitted (bridge_endpoint), the gadget callback (gadget_epout_complete) submits an URB. When the URB completes (device_epout_irq), the request is resubmitted.

This method, that is, with only one URB/request "flying" at the time, seems to work well for low throughput applications, but buffering may be required for higher throughput.

Weekly report - Week 2

Status:

• Modified the proxy driver to be more generic: descriptors requests are forwarded to the device, and the descriptors are parsed (and modified if required) on the fly: but still, only interrupt transfers from the device to the host on endpoint 1 are supported.
• Ported my set of patches to the kernel 2.6.35-rc1 (i.e., the latest vanilla tree): a few functions got renamed, but nothing major. I compiled and ran that kernel on the BeagleBoard. I had to tweak some files to get the MUSB block to accept being run in peripheral-only mode (commit), but the MUSB block seems buggy (even with the Ethernet gadget), so I gave up on that. From a discussion on #beagle, it seems like it would be better to use the linux-omap git tree (but I don't know which branch), and apply "some patches" from Angstrom.
• Started working with a new device: the FTDI USB->Serial converter found, for example, on Arduino boards. Device->PC communication uses endpoint 1, in bulk mode, and worked with little modifications of the existing code.
  

Plans:

• Get PC->Device communication to work with the Arduino, which uses endpoint 2 in bulk mode (to get at least one OUT endpoint working).
• From there, dynamically connect all endpoints advertised in the configuration.
  

Risks/problems:

• So far I've been targeting kernel 2.6.32 (from angstrom-linux/beagleboardXM), but at some point I would like to target the latest kernel, so I would need to get the MUSB block to work properly with a recent kernel.
• I have encountered strange problems with a keyboard: pressing num/caps lock only takes effect at the next key stroke, I need to investigate this.
  

Towards a more general solution

I spent the last few days generalizing my proxy driver, in such a way that most HID devices (mice, keyboards), should now be working. Actually, any device that uses only endpoint 1 in interrupt mode and IN direction should be supported by the current implementation.

There are no static descriptors hard-coded in the driver anymore, so GET_DESCRIPTOR requests are intercepted, and the configuration descriptors are parsed and cached (using functions found in the kernel). Also, the descriptors are modified on the fly, to compensate for the fact that the communication between the PC and the BeagleBoard happens in Full Speed, while the BeagleBoard-device communication uses Low Speed.

Two fields are dynamically modified:

• bMaxPacketSize0 in the device descriptor. It seems like the MUSB driver only support 64 bytes as maximum packet size (which is what is mandated by the High Speed mode), while the mouse uses 8 bytes (the only possible value for Low Speed devices). In theory, Full Speed devices can advertise a maximum packet size between 8 and 64 bytes, but the MUSB driver does not seem to support that.
• bInterval in endpoints descriptors. As mentionned before, the meaning of that field varies with the device speed.
  

To get the proxy driver to work, you need to follow these steps:

• Clone my kernel git tree. Use the backup-20100602 branch.
• Compile the kernel with the default beagleboard configuration (see here). You just need to add CONFIG_USB_G_PROXY=m. I also disabled MUSB in host and OTG mode, as well as USB suspend, but this may not be necessary.
• Install the new kernel on the board.
• Clone the helper scripts git tree, and copy the content of the arm directory to the BeagleBoard (you need to modify load/setup scripts if you do not have have a copy of musb_hdrc.ko and g_proxy.ko in the same directory).
• Run ./setup on the BeagleBoard, this will unload the g_ether gadget driver.
• Plug your USB hub + a HID mouse/keyboard to the BeagleBoard (this can be done earlier).
• Plug your PC to the BeagleBoard USB slave port (this can be done earlier as well).
• Run ./unbind: This will unbind the device from the normal Linux driver.
• Run ./load: this will (re)load the g_proxy driver.
• Use the mouse/keyboard, it should work.
  

The code is still full of TODO/FIXME, and is rather likely to cause kernel panics.

Weekly report - Week 1

Status:

• Managed to build Angstrom's console-image, as well as the kernel (outside OpenEmbedded), both work well on the BeagleBoard.
• Wrote a prototype kernel proxy driver, and managed to get USB data to be forwarded properly between a USB mouse and a Linux PC, with the BeagleBoard in between (git tree here, but no really usable yet).
• Modified the MUSB controller driver to disable high-speed USB if the gadget asks for full-speed (commit here).
  

Plans:

• Get the proxy driver more generic, and get completely rid of the static descriptor copy.
• Dynamically connect endpoints, according to the endpoint descriptor.
  

Risks:

• Differences between low-speed, full-speed, high-speed, for example in the meaning of some descriptor fields (e.g., bInterval for endpoints): since the BeagleBoard MUSB cannot operate in low-speed, this would require some "rewriting" of the descriptors to get the device to work.
• Also, my Linux PC asks for the Device_Qualifier descriptor (a high-speed only descriptor) when connected to the BB, but not when connected directly to the mouse, even though the device is advertised and detected as full-speed.

A first device works: USB mouse

Today I managed to get the following to work: I connect a USB mouse to the host port of my BeagleBoard (via a hub, since the BeagleBoard does not support low-speed USB), and the slave USB port of my BeagleBoard to my PC. Then, I can use that mouse to control the cursor of my PC.

For that, I have a "proxy" kernel driver on the BeagleBoard, that is registered both as a USB device driver (on the EHCI side), and as a USB gadget driver (on the MUSB/OTG side).

Most USB drivers register at the interface level (that is, as a struct usb_driver), so control only a part of a device (one device could have multiple interfaces, for multiple functions). That is not what we want, so, instead, the proxy driver registers as a struct usb_device_driver, that claims the device at a lower level. Normally, there is only one of such low-level drivers, found in drivers/usb/core/generic.c. This generic driver takes care of all USB devices attached to the system, and looks for interface-level drivers for each of the interfaces of these devices.

Since the generic driver claims all new devices (and never releases them), the trick is to unbind the device from the generic driver, so that it can be claimed by our proxy driver. This is done using a command like:

echo 1-2 > /sys/bus/usb/drivers/usb/unbind

Once "unbound", our driver is able to claim it.

On the gadget side, there is nothing special: the driver is registered as a struct usb_gadget_driver. Then, we do everything we can to make the PC believe that it is connected directly to a USB mouse. For that purpose, we reply to GET_DESCRIPTOR control requests from the PC with the following device descriptor:

Bus 001 Device 005: ID 046d:c00e Logitech, Inc. M-BJ58/M-BJ69 Optical Wheel Mouse
Device Descriptor:
bLength                18
bDescriptorType         1
bcdUSB               2.00
bDeviceClass            0 (Defined at Interface level)
bDeviceSubClass         0
bDeviceProtocol         0
bMaxPacketSize0         8
idVendor           0x046d Logitech, Inc.
idProduct          0xc00e M-BJ58/M-BJ69 Optical Wheel Mouse
bcdDevice           11.10
iManufacturer           1 Logitech
iProduct                2 USB-PS/2 Optical Mouse
iSerial                 0
bNumConfigurations      1
Configuration Descriptor:
  bLength                 9
  bDescriptorType         2
  wTotalLength           34
  bNumInterfaces          1
  bConfigurationValue     1
  iConfiguration          0
  bmAttributes         0xa0
    (Bus Powered)
    Remote Wakeup
  MaxPower               98mA
  Interface Descriptor:
    bLength                 9
    bDescriptorType         4
    bInterfaceNumber        0
    bAlternateSetting       0
    bNumEndpoints           1
    bInterfaceClass         3 Human Interface Device
    bInterfaceSubClass      1 Boot Interface Subclass
    bInterfaceProtocol      2 Mouse
    iInterface              0
      HID Device Descriptor:
        bLength                 9
        bDescriptorType        33
        bcdHID               1.10
        bCountryCode            0 Not supported
        bNumDescriptors         1
        bDescriptorType        34 Report
        wDescriptorLength      52
       Report Descriptors:
         ** UNAVAILABLE **
    Endpoint Descriptor:
      bLength                 7
      bDescriptorType         5
      bEndpointAddress     0x81  EP 1 IN
      bmAttributes            3
        Transfer Type            Interrupt
        Synch Type               None
        Usage Type               Data
      wMaxPacketSize     0x0004  1x 4 bytes
      bInterval              10
Device Status:     0x0000
(Bus Powered)

Currently, I have a copy of the descriptor statically defined in the driver, but of course, in the future, this descriptor will have to be read from the device itself.

Then, whenever the drivers receives a control request it does not know how to answer (for example, a class-specific HID control request), it forwards the request to the device, waits for the reply, then writes back the reply to the PC.

With that, the mouse is fully recognized by the PC, but nothing happens when you move it, because mouse events use interrupt transfers on endpoint 1. That can be fixed by listening for interrupt transfers from the device, and forwarding those to the PC.

At that point, the cursor moved, but with some huge latency: clicking was fast, but there was a delay of a few seconds when moving the mouse around... This was due to the following: the mouse is a low-speed USB device, and its endpoint descriptor asks the host to poll it for events every 10 ms (bInterval = 10). However, the MUSB device controller on the BeagleBoard appears as a high-speed device (there is probably a way to force the controller in full-speed mode, but the current driver does not seem to support that), and the definition of the polling interval, for a high-speed device, is 0.125 ms * 2^(bInterval-1). By setting bInterval to 10, the BeagleBoard device controller only got polled every 64 ms, so, when I moved the mouse, this created a lot of events, which got buffered somewhere in the BeagleBoard, until they could be released to the PC.

I fixed the problem by modifying the descriptor, and putting bInterval = 7, that is a poll every 8 ms.

I can write up some instructions about how to get this setup to work, but I guess this is not so useful if you do not have the same mouse as I have, or a similar enough mouse at least...

All the kernel modifications have been pushed in my git repository. The proxy driver code, in particular, can be found here, but the code is still far from clean and full of TODO/FIXME tags.

Received my BeagleBoard!

This morning I received a package from Digikey (thanks to Cathy!), with a BeagleBoard, a 5V power adapter, and a IDC10-DB9 cable.

I connected the BeagleBoard to my PC, using the ICD10-DB9 cable, followed by a USB->RS232 adapter. Then I powered it on, some leds blinked, and, nothing on the terminal... I then tried connecting it to another PC, with a serial port, nothing again.

I started wondering if I needed a null-modem cable, so I checked the pinout on the ICD10-DB9 to check if it wasn't one already. And I realised that the pinout was completely non-sensical (ground connected to RX, ring signal connected to TX)... Since the colors on the RS232 wires are not standardized, it looks like the people who built the cable did not realize that the colors changed with their latest supply of cables...

It seems like the problem affect quite a few people who ordered the cable recently, so here is how to fix it:

• Take out all the wires from the ICD10 connector (easy to do with a needle).
• The red and green wires should be together on one pin. Cut the green one.
• Then, connect the red wire (TX, pin 3) on pin 2 of the BeagleBoard (RX).
• The brown one (RX, pin 2), on pin 3 of the BeagleBoard (TX).
• The yellow one (GND, pin 5), on pin 5 of the BeagleBoard.
• Leave the other wires dangling, you don't need them (beware of shorts though, I taped them to avoid problems).
  

A picture of what it looks like in the end:

Note that this is only for this batch of cables, and it's always better to check the wiring with a ohmmeter, if you have one (the RS232/DB9 pinout can be found here).

Weekly report: Week 0

First, I should receive my BeagleBoard tomorrow, so I can get up to speed on the project.

Status:

• Tried to build Angstrom (the latest git version), but ran into disk space problems (20+ gigs do not seem to be enough). Since I only need to do kernel development, I can also use demo images, and pre-built toolchains.
• Downloaded the pre-built toolchain (angstrom-2010.4-test-20100421-x86_64-linux-armv7a-linux-gnueabi-toolchain-qte-4.6.2.tar.bz2).
• Built a Beagleboard kernel based on angstrom-linux/beagleboardXM, using that toolchain.
• Formatted a 1GiB SD-card according to the guide, and copied the content from the demo image to it. Since my SD-card is only 1GiB, and the image is 2GiB, I "dd"-ed the first FAT partition (to make sure that files are in the right order), but copied all the files on the ext3 partition.
  

Plans:

• Try to finish building Angstrom (I freed some disk space, and maybe I messed up somewhere).
• Test the SD-card I formatted.
• Replace the kernel on the SD-card with the one I built.
• Use the newest root filesystem (if I manage to finish building it).
• Test the USB gadget and EHCI controllers with a few different drivers/devices.
• Start working on the proxy driver (i.e., the project itself).
  

Risks:

• Unable to build Angstrom: nevermind, I can use demo images (at least for now).
• ?
  

Device emulation

So, as it seems that hub emulation cannot be done, here is how device emulation would work.

We need a kernel driver that lies between the slave (gadget, MUSB) port and the master (EHCI) port. This "proxy" driver will be registered both as a USB device driver, and as a USB gadget driver. It will then "simply" forward USB packets back and forth between the 2 interfaces.

First, the sniffed device is connected to the master port of the BeagleBoard. The Linux kernel assign an address to it, reads its descriptors, select a configuration, and tries to find a driver for the new device. Reading the descriptors and assigning the address are probably fine (it needs to be done anyway, and it should be a rather standard operation), but we probably do not want to set a configuration, and clearly do not want any driver to initialize the device.

Then, we want to bind the device to our proxy driver (this can be done using echo [device] > /sys/bus/usb/drivers/xxx/[un]bind ).
The proxy driver will then copy the descriptors, and activate the slave port. The host PC enumerates the new device, sets an address to it, and obtains the copied descriptors (and hopefully believes that it is connected to the actual device).

Some additional control transfers (i.e. endpoint 0) may happen, such as configuration selection. These need to be forwarded to the real device.

To be able to actually use the device, other endpoints (in addition to the mandatory endpoint 0) need to be configured in the gadget driver, and registered on the device side, so that the data can be forwarded. The required endpoints can be obtained from the device descriptor (below, the (very simple) descriptor for a mouse):

Bus 006 Device 003: ID 046d:c018 Logitech, Inc. Optical Wheel Mouse
Device Descriptor:
bLength                18
bDescriptorType         1
bcdUSB               2.00
bDeviceClass            0 (Defined at Interface level)
bDeviceSubClass         0
bDeviceProtocol         0
bMaxPacketSize0         8
idVendor           0x046d Logitech, Inc.
idProduct          0xc018 Optical Wheel Mouse
bcdDevice           43.01
iManufacturer           1 Logitech
iProduct                2 USB Optical Mouse
iSerial                 0
bNumConfigurations      1
Configuration Descriptor:
bLength                 9
bDescriptorType         2
wTotalLength           34
bNumInterfaces          1
bConfigurationValue     1
iConfiguration          0
bmAttributes         0xa0
 (Bus Powered)
 Remote Wakeup
MaxPower              100mA
Interface Descriptor:
 bLength                 9
 bDescriptorType         4
 bInterfaceNumber        0
 bAlternateSetting       0
 bNumEndpoints           1
 bInterfaceClass         3 Human Interface Device
 bInterfaceSubClass      1 Boot Interface Subclass
 bInterfaceProtocol      2 Mouse
 iInterface              0
   HID Device Descriptor:
     bLength                 9
     bDescriptorType        33
     bcdHID               1.11
     bCountryCode            0 Not supported
     bNumDescriptors         1
     bDescriptorType        34 Report
     wDescriptorLength      52
    Report Descriptors:
      ** UNAVAILABLE **
 Endpoint Descriptor:
   bLength                 7
   bDescriptorType         5
   bEndpointAddress     0x81  EP 1 IN
   bmAttributes            3
     Transfer Type            Interrupt
     Synch Type               None
     Usage Type               Data
   wMaxPacketSize     0x0005  1x 5 bytes
   bInterval              10
Device Status:     0x0000
(Bus Powered)

From that descriptor, we can see that we need to configure the gadget driver to handle outgoing interrupt transfers on endpoint 1, and that we need to listen for interrupt transfers on endpoint 1, coming from the device.

My plan for the following weeks:

• Manually copy the descriptor of a simple device (a mouse/keyboard would do)
• Handle control transfers forwarding
• Connect one endpoint on the slave and master sides (EP 1 IN, interrupt for mouse/keyboard)
• Forward interrupt transfers

Some other thoughts:

• Mice and keyboards are usually Low Speed devices, which is not supported by many gadget controllers (including MUSB), does it hurt if we advertise them as Full Speed devices on the gadget-host link?
• If a device has multiple configurations, an endpoint may have different types and/or directions depending on the configuration, we need to be able to handle that (and intercept SET_CONFIGURATION requests).
• Some devices change descriptor (and disconnect/reconnect) after a firmware load (e.g., FX2-based devices), we need to find a way to handle that, and reattach to the "new" device automatically.

Device vs hub emulation

There are 2 main alternatives to implement the USB sniffer:

• Device emulation: Emulate the sniffed device itself: That is, pretend that we are the device itself, and forward packets back and forth between the device and the host.
• Hub emulation: Emulate a USB hub. That is, act as if we were a USB hub, with several ports. One (or more) of these ports would be connected to the device to be analysed. Other ports would still be available for other USB gadgets (e.g., Ethernet, serial gadgets).
  

For the hub emulation, I should first try to explain how a USB hub works (or at least, how I understand it works, skipping details):
An USB hub is a simple repeater: that is, each packet coming from the host is broadcasted to all the devices connected to it. An USB token packet contains, among other things, an address field. As each device is assigned an unique address, it can safely ignore transfers that are not meant for it. Packets coming from the device are not broadcasted (it would not make any sense), and are simply forwarded back to the host.

When a new device is connected to a hub, the hub informs the host. The new device, by default, takes address 0, and the host can then initiate basic communication with it using that address. One of the first thing the host does it to send a SET_ADDRESS command to the device, the new device will then answer on that new address (and not on address 0 anymore).

Now, the question is, can we implement this on the BeagleBoard, with the MUSB controller? For this, the MUSB controller needs to be able to listen on all possible USB addresses (similar to a promiscuous mode on an Ethernet controller). Depending on the address, we could then forward the packet to the sniffed device, or on any other virtual gadget device (since we have more processing power than a normal USB hub, we can be more clever and avoid broadcasting the data to every device).

From what I see in the code of the MUSB controller driver, there seems to be only one register containing the USB address to listen on. This register (F_ADDR) is updated at line 706, after a SET_ADDRESS command is received from the host:

 if (musb->set_address) {
  musb->set_address = false;
  musb_writeb(mbase, MUSB_FADDR, musb->address);
 }

Therefore, it seems like the MUSB controller is only able to listen on a single address, so it looks like it may not be possible to implement a USB hub.

Project Goal

The goal of this Google Summer of Code 2010 project is to have some hardware USB sniffer on the Beagle Board. The host PC is connected on the slave USB port of the Beagle Board, and the device to be analysed on the host port of the Beagle Board.

The goal is to have a completely transparent solution to forward USB packets from the host PC to the device (and back), while logging the data: that is, neither the host PC nor the device must be able to realise that there is a sniffer in the middle, intercepting packets.

Software solutions to capture packets already exist (e.g., USB Snoopy on Windows, USB monitor on Linux), but a hardware solution would have the advantage of:

• Not requiring any software modification
• Supporting proprietary OSes, where you may not have access to the software
• Allowing debugging of a new USB stack: If, let's say, you are developing a new USB stack for your preferred operating system, any software logging solution would obviously be as reliable as the stack you are currently developing, hence the usefulness of a hardware solution.
• Depending on the capabilities of the OTG and EHCI chips, being able to capture USB frames at a lower level, and maybe monitor for transmission errors for example.