#5 ✓resolved
Matt Colyer

Integrate usbmuxd

Reported by Matt Colyer | March 24th, 2009 @ 03:26 AM | in 1.0 Release

Comments and changes to this ticket

  • Matt Colyer

    Matt Colyer March 24th, 2009 @ 03:26 AM

    • Title changed from “Integrate usbmuxd ” to “Integrate usbmuxd (when it'”
  • Matt Colyer

    Matt Colyer March 24th, 2009 @ 03:28 AM

    • Title changed from “Integrate usbmuxd (when it'” to “Integrate usbmuxd (when it's ready)”

    See Nikias' email:

    git clone http://pims.selfip.net/git/libip...

    and of course usbmuxd is here:

    git clone http://pims.selfip.net/git/usbmuxd

    Until now, the libiphone fork opens the first device reported by usbmuxd. It works quite well for now. Simultaneous connections to the same device do work, e.g. an ifuse mount and running iphoneclient or additional ssh connections over the cable using iproxy/usbmux-proxy. There are still bugs in it so don't expect it to work every time.

  • Paul Sladen

    Paul Sladen March 25th, 2009 @ 02:05 AM

    • Tag set to “usbmuxd”
  • Nikias Bassen

    Nikias Bassen April 15th, 2009 @ 09:07 PM

    STATUS UPDATE

    The current usbmuxd implementation is quite stable, and it is also when running multiple connections (e.g. multiple ifuse mounts and/or iphoneclient sessions working at the same time). However I think a bit more testing and feedback would be good to round things up.

    My suggestion for libiphone integration is to put the code in a subdirectory 'usbmuxd' and adopt the configure-script to have an option '--enable-usbmuxd' which defines 'ENABLE_USBMUXD'. The libiphone code is then modified like this:

    
#ifdef ENABLE_USBMUXD
// usbmuxd specific code
#else
// current libiphone code
#endif

    Basically, the changes only affect iphone.{c|h} and usbmux.{c|h}. However, the makefiles (or makefile creation) need some work, as the usbmuxd version additionally has to be linked against libusbmuxd.so. And the udev rules for the current libiphone conflict with the udev rules for usbmuxd. make install should copy only one of these files depending on --enable-usbmuxd has been configured or not.

    I think I'll make a patch for the code itself (when the latest tickets have been resolved), but I need some help with the autoconf/automake stuff, and the usbmuxd code also does not use automake yet.

  • Jonathan Beck

    Jonathan Beck April 16th, 2009 @ 07:33 AM

    Since we want to include it in the end, I'd say we should integrate usbmuxd without all the #ifdef stuff, and fix bugs when they come.

    Then there is no complication with two automake config. It also has the advantage to enforce a lot more testing on this part.

    Also since this change is quite big, maybe a merge with your branch is better, so we can keep the log.

  • Martin S.

    Martin S. April 16th, 2009 @ 11:58 AM

    No #ifdef stuff since there is not much need to keep the "old usbmux way" in once usbmuxd is integrated and working.

  • Martin S.

    Martin S. April 24th, 2009 @ 12:00 PM

    Looked roughly over the usbmuxd implementation (great work Nikias), some comments though to get a merge going:

    • The code style needs adjustment so it follows libiphone code style (mainly indentation like using a mix of spaces/tabs)
    • There is no need for iphone* naming, straight usbmux* should be fine
    • Things like "iphone_get_specific_device" totally belong into usbmuxd code only
    • Needs to get rid of iphone.[ch] and any mix between libiphone/usbmux code
    • Merge of the libiphone udev rules into usbmuxd's (to change config for all known devices)
    • iphone_device_t belongs into libiphone only, usbmux should not use it
    • helpers like usbmuxd_send/usbmuxd_recv could abstract some more code in consumers
    • libusbmuxd should provide: a) device discovery/list devices b) connect/disconnect to a device c) send/recv to a device
    • needs autofoo :)

    Related for libiphone:

    • libiphone needs to use libusbmuxd for a) discovering devices b) connecting to devices c) probably could use helpers like usbmuxd_send/usbmuxd_recv to communicate
    • the handle structs need some cleanup to rip out/adjust to usbmux stuff
    • in the end libiphone should have no libusb dependency, only usbmuxd
    • libiphone should have no udev rule, usbmuxd does usb stuff and should therefore use it

    Basically I'd propose to alter usbmuxd into a state first where it can be compiled separately, then we'd brew a branch for libiphone to remove all usbmux related cruft and make use of libusbmuxd for communication.

    In the end we'd finish by patching all consumers (iFuse, dev/ tools) to the new libiphone API, since device discovery/communication will change to select by uuid if I see it correctly.

  • Nikias Bassen

    Nikias Bassen April 24th, 2009 @ 02:38 PM

    Well, this is a plan. Thanks Martin for writing this together, I think everything you wrote is exactly what we need to do.

  • Martin S.

    Martin S. May 2nd, 2009 @ 11:43 AM

    Below patch adds autotools to usbmuxd and moves files into appropriate subdirectories.

    Would it be feasible to add "usbmuxd_recv()" and "usbmuxd_send()" to libusbmuxd which could abstract the socket stuff away for consumers (like libiphone) which rather want to move buffers around?

    The idea is that instead of libiphone having to introduce it's own socket "layer" (like iphone_mux_send), it could instead directly use usbmuxd_send() everywhere.

  • Nikias Bassen

    Nikias Bassen May 5th, 2009 @ 01:12 PM

    Thanks for the patch Martin, I just pushed the changes. Concerning usbmuxd_send/usbmuxd_recv: It's not so easy, as you need to specify where to send to/receive from. Every connection (lockdown, AFC, ssh, ...) gets a socket file descriptor which is returned by usbmuxd_connect(). My libiphone-usbmuxd uses send/recv with the fd of the connection to send/receive on. It's implemented inside usbmux.c/h, as well as the management of the connections (storing the socket fd, etc.)...

  • Martin S.

    Martin S. May 5th, 2009 @ 02:21 PM

    One more thing (patch attached if needed), the Makefile should not be tracked since it is generated, somehow it ended up in the sources now. :)

    Regarding, usbmuxd_send/usbmuxd_recv, I think something like this could work:

    • new: Store result of usbmuxd_connect() in related client struct (lockdown, afc, ...) when creating a new client
    • send/recv: client has own send/recv functions which use usbmuxd_send/usbmuxd_recv and pass socket from the client struct as first argument
    • delete: When freeing a client, call usbmud_disconnect() with the socket

    Kind of like "make the returned socket" a "handle" to work on.

    Main benefit of this from what I see is that one does not have to reimplement some of the usbmuxd functionality (connection management for instance) while using libusbmuxd with libiphone.

  • Nikias Bassen

    Nikias Bassen May 5th, 2009 @ 05:42 PM

    I removed 'Makefile' didn't notice it was still in ;)

    So what you describe is basically already done in my libiphone-usbmuxd implementation, inside the usbmux functions, but kind of hidden. Each of libiphone's clients has a iphone_umux_client_t field, and iphone_umux_client_int has a field sfd that is used in send/recv.

    I think this should be unwrapped a little bit. The usbmux part of libiphone is not really required anymore. So each client could just call usbmuxd_connect() in its *_new_client function, and store the file descriptor in the client struct. Then, instead of calling iphone_mux_send/iphone_mux_recv, usbmuxd_send/usbmuxd_recv are called with the corresponding file descriptor from the client struct.

    This also would let us remove usbmux.c/h from libiphone.

  • Martin S.

    Martin S. May 8th, 2009 @ 04:04 PM

    • Tag changed from “usbmuxd” to “udev, usbmux, usbmuxd”
    • State changed from “new” to “open”
    • Milestone set to “1.0 Release”

    You got my idea, had in mind exactly what you describe Nikias! I think such a change would clean up/simplify code aswell.

    Setting this for 1.0 release milestone since using usbmuxd will definitely cause API changes for libiphone.

    In order to not break API too much, it's probably better to patch libiphone for usbmuxd before making a 1.0 release.

  • Nikias Bassen

    Nikias Bassen May 9th, 2009 @ 11:22 AM

    I totally agree. I'll see if I can make it work like this and then we should integrate it as soon as possible.

  • Nikias Bassen

    Nikias Bassen May 18th, 2009 @ 10:36 PM

    Here's a patch for current libiphone to make use of usbmuxd. It removes usbmux.c and usbmux.h and does not link against libusb anymore.
    There's also an API change as the *new_client functions had a source port parameter which is not required anymore as usbmuxd handles this.
    I also removed the udev directory as usbmuxd has it's own udev rules.

    Please test happily and report.

    Remember to get the latest usbmuxd code from git clone http://pims.selfip.net/git/usbmuxd

    libusbmuxd now provides disconnect and send/recv functions as discussed previously.

  • Nikias Bassen

    Nikias Bassen May 18th, 2009 @ 10:38 PM

    And here's a patch to make ifuse work with the above patch. It also closes the lockdown connection after AFC has been started, as this connection is not needed afterwards (and in fact causes and endless read loop when the iphone is plugged out and after that an umount is performed). BTW, the iPhone closes the lockdown connection itself after some seconds of inactivity.

    What is still missing is the device selection, which should be done by the uuid of the device (using the newly introduced function iphone_get_device_by_uuid in the previous patch).

  • Matt Colyer

    Matt Colyer May 19th, 2009 @ 02:03 AM

    Two things.

    • ./autogen.sh fails on a fresh checkout of libusbmux (needs an m4 directory).
    • Also there wasn't a pkg-config in libusbmux which the above patch depends on.

    I'm going to push the patches anyway to get the integration rolling.

  • Nikias Bassen

    Nikias Bassen May 19th, 2009 @ 05:35 AM

    Fixed. Forget to git push and the m4 issue has been resolved, too.

  • Nikias Bassen

    Nikias Bassen May 19th, 2009 @ 11:20 AM

    Hi again. I had to comment out the device selection by usb bus and device number in iphoneinfo and iphonesyslog due to the API change introduced by the previous patch.
    The attached patch fixes that by allowing to select a specific device with -u or --uuid.

    I also included a tool called iphone_id that allows to list all attached devices (iphone_id -l) or retrieve the device's name (iphone_id DEVICE_UUID). This was intended to be used for automounting later, to give the mount point a fancy name like "Steve's iPhone".

    NOTE: the patch from ticket #39 has to be applied before applying this one.

  • Nikias Bassen

    Nikias Bassen May 19th, 2009 @ 12:04 PM

    Forgot to mention something: the udev rules file from usbmuxd is launching usbmuxd automatically when a device is plugged in. So if your system is using udev and you did make install, it should work right away.

  • Matt Colyer

    Matt Colyer May 19th, 2009 @ 02:39 PM

    Okay I added in that patch as well (and closed #39). The updated usbmuxd repository resolved my issues.

    I had to make a minor change to the python wrapper. Can someone check that my change is correct?

  • Nikias Bassen

    Nikias Bassen May 19th, 2009 @ 02:49 PM

    The change is correct! I forgot to commit it, just noticed this either.

    BTW, I have just updated usbmuxd so multiple devices are handled correctly by udev. usbmuxd was terminated when one device was unplugged even if another one is still attached to the computer.

  • Matt Colyer

    Matt Colyer May 20th, 2009 @ 03:17 PM

    So I now have it running but it can't seem to find my iPhone's UUID (and none of the rest of the stuff seems to work without it). It looks like it is timing out on main.c:691.

    This is the output from usbmuxd:

    usbmuxd: starting
usbmuxd: Successfully dropped privileges
usbmuxd: waiting for connection
usbmuxd: new client connected (fd=5)
usbmuxd: usbmuxd_client_init_thread[424bc950]: started (fd=5)
usbmuxd: usbmuxd_client_init_thread[424bc950]: Got scan packet!
usbmuxd: usbmuxd_send_result: tag=2 result=0
usbmuxd: usbmuxd_client_init_thread[424bc950]: usb init
usbmuxd: usbmuxd_client_init_thread[424bc950]: usb find busses
usbmuxd: usbmuxd_client_init_thread[424bc950]: usb find devices
usbmuxd: usbmuxd_client_init_thread[424bc950]: Looking for attached devices...
usbmuxd: usbmuxd_client_init_thread[424bc950]: Found device on bus 0, id 95
usbmuxd: usbmuxd_client_init_thread[424bc950]: Waiting for connect request
recv_buf_timeout: fd=5 recv returned 0
usbmuxd: usbmuxd_client_init_thread[424bc950]: Did not receive any connect request.
usbmuxd: usbmuxd_client_init_thread[424bc950]: terminating
usbmuxd: usbmuxd_client_init_thread[424bc950]: terminated
usbmuxd: reclaimed client thread (fd=5)

    Any ideas?

  • Matt Colyer

    Matt Colyer May 20th, 2009 @ 03:47 PM

    Found it, the reason it fails is that you need root privileges to access the usb device on my system. However usbmuxd is dropping the permissions before it attempts to access the device.

    How should we go about fixing this?

  • Nikias Bassen

    Nikias Bassen May 21st, 2009 @ 08:31 PM

    I added a parameter -p to prevent usbmuxd from dropping the privileges. This is a workaround, if you want to use this for your system you need to add -p to the udev rules file yourself.

    Matt, what linux system are you using? On my ubuntu this is working flawlessly.

    Does iphone_id -l work for you with dropped privileges?

    Hm, it just came to my mind, perhaps it would be useful to let the user specify the user to use when dropping the privileges (instead of nobody).

  • Matt Colyer

    Matt Colyer May 22nd, 2009 @ 03:07 AM

    I am running stock Ubuntu 8.10 as far as I know. Which device file is the one that libusb uses again? (I am pretty sure it's owned to root on my system).

  • Nikias Bassen

    Nikias Bassen May 23rd, 2009 @ 11:24 AM

    Hi,
    I refined the debugging output of usbmuxd a bit, including debugging output from libusb. This way we should find out what's wrong.

    Could you retry with the current code and run:

    sudo usbmuxd -f -v -v -v

    then try:

    iphone_id -l

    and then:

    iphone_id $YOUR_UUID
    even if the previous one seems to fail.

    You can send me the debugging output via email if it gets too much...

  • Martin S.

    Martin S. May 23rd, 2009 @ 05:44 PM

    Superb work Nikias. Below is a patch to fix the open bits of the udev rules. I also created initial packages for usbmuxd in my repository.

    I also noticed that usbmuxd fails for me due to issues on enumerating usb devices. This was fixed in libiphone to work correctly once but that change was not backported to usbmuxd. I'll see to cook up a patch for that, too.

    The main issue was that the busnum/location variables apparently are not always correct and almost all libusb tools use the dirname/filename strings to look for a bus:dev combination.

    Also main.c:787 uses "usbmux_get_specific_device(0, c_req->device_id, &phone)", you always pass 0 as busnum... which can't be correct.

  • Martin S.

    Martin S. May 23rd, 2009 @ 07:02 PM

    Use the patch below, last one had a minor typo.

  • Nikias Bassen

    Nikias Bassen May 23rd, 2009 @ 07:29 PM

    Thanks for the patch. I was about to push the changes, but the symlink creation does not work on my system, it looks like this:

    $ ls -l /dev/usbmux/
insgesamt 0
lrwxrwxrwx 1 root root 21 2009-05-23 21:17 in -> ../../usbdev3.40_ep85
lrwxrwxrwx 1 root root 21 2009-05-23 21:17 out -> ../../usbdev3.40_ep04

    The "serial" part is missing as a subdirectory; cat /sys/bus/usb/devices/3-2-or-whatever/serial prints my device serial number correctly so I don't know what's wrong...

    BTW, I worked out the usb device enumeration to use bus and device number, so it should work now. Matt, this might fix your problem too.

  • Martin S.

    Martin S. May 23rd, 2009 @ 09:34 PM

    Had also cooked up something for the device enumeration, just ignore it as you seem to have the same done already.

    I need to pass "-p" to not drop privileges in order for usbmuxd to work for me regardless of this change, too.

    The udev serial and thus the directory is correctly set for me.

    My versions:

    # rpm -q udev libusb-devel kernel-pae
udev-128-9.7.1
libusb-devel-0.1.12-136.10
kernel-pae-2.6.27.7-9.1

    Perhaps you could try if "$attr{serial}" works better for you.
    Also verify that output of "udevadm info -a -p $(udevinfo -q path -n /dev/usbdev*_ep85)" lists you the parent device chain and also has the "serial" attribute. It is supposed to lookup the parent device chain for a property which does not exist for the current device.

  • Matt Colyer

    Matt Colyer May 23rd, 2009 @ 11:34 PM

    http://gist.github.com/116875

    The output of iphone_id -l is this:
    handle=524299 product_id=1290 uuid=

  • Nikias Bassen

    Nikias Bassen May 24th, 2009 @ 11:47 AM

    This is really strange. The device files that are opened are in /dev/bus/usb/BUSNUM/DEVNUM. On my system they all have access mode rw-rw-r-- and are owned by 'root', group 'root'. I have no clue why it works with dropped privileges...

    Assuming that more than just your systems will be affected, I have committed a change that removes the '-p' parameter and adds a '-d' parameter. The privileges are not dropped unless '-d' is passed to usbmuxd.

    I also committed the udev rules patch by Martin. However my udev (Version 141-1) on Ubuntu Jaunty seems not to be able to read parent attributes, even the man page says so (regardless if I use $attr{serial} or %s{serial}). I haven't found the reason for that yet.
    The udevadm info query show me that the "grandfather" of the 'usb_endpoint/usbdev*_ep85' is the first to have the serial attribute.

  • Martin S.

    Martin S. May 24th, 2009 @ 01:21 PM

    @Matt: I get this output if usbmuxd drops privileges.

    @Nikias: Try to check the output of "udevadm test $(udevadm info -q path -n /dev/usbdev*_ep85)". It should show exactly what udev is doing when processing the rules (without actually doing anything).

    It seems libusb sometime looked for usb devices at /proc/bus/usb while libusb-0.1.12+ is looking at /dev/bus/usb which somehow introduced permission issues.

    Also when running usbmuxd + iproxy 5000 22 + ssh root@localhost -p 5000, then attempting to use "iphone_id " fails with "encountered USB read error: -104" on read and write. Once I close ssh, iphone_id works again.

    iphonesyslog shows me this which probably is related:

    Sun May 24 14:58:41 unknown com.apple.mobile.lockdown[21] <Notice>: Could not receive size of message
Sun May 24 14:58:41 unknown kernel[0] <Debug>: Error: void AppleUSBDeviceMux::handleMuxTCPInput(mbuf*) something is wrong, sending RST and closing 0xc6864f40
Sun May 24 14:58:41 unknown lockdownd[21] <Error>: (0x81fa00) handle_connection: Could not receive USB message #1 from unknown. Killing connection
Sun May 24 14:58:41 unknown kernel[0] <Debug>: void AppleUSBDeviceMux::handleMuxTCPInput(mbuf*) th.th_flags = 0x1, not TH_ACK(0x10)
  • Nikias Bassen

    Nikias Bassen May 24th, 2009 @ 04:29 PM

    Take a look:

    https://bugs.launchpad.net/ubuntu/+source/udev/+bug/348513

    A quick look into the sources reveals that version 141 and later definitely DOES NOT walk the path up looking for attributes as the documentation says. It just looks at the parent device, but in this case it has to look at the parent of the parent ;) This explains why it does not work.

    I made the rule working now by using 'usb_id' included in udev.

    Regarding the concurrency problem: Seems as if some packets could not be sent. I can use ifuse, iproxy+ssh, and iphone_id together without problems.

    Try if running usbmuxd -f -v -v -v reveals some details about what's going on? You can send me the log or post it somewhere because I think it'll be long.

  • Matt Colyer

    Matt Colyer May 25th, 2009 @ 07:10 PM

    @nikias: So it turns out if you have virtualbox installed those devices get owned to the vboxusers group. Now that I have added myself to that group it shouldn't be an issue. Maybe it's something we should document?

    I have also had trouble with multiple simulatenous connections with usbmuxd. (ifuse and calendar syncing)

  • Nikias Bassen

    Nikias Bassen May 25th, 2009 @ 10:19 PM

    @Matt: putting yourself into vboxusers will not change anything as the permission drop is using user 'nobody' and not your user account.

    I'd like some debugging output to see what's going on with simultaneous connections. If you could please run "sudo usbmuxd -f -v -v -v -v -v" and then do something that does not work for you, so mount via ifuse and then run one of the tools (e.g. iphone_id your_UUID) once. usbmuxd will produce a lot of debugging output which you send to me.

    Another thing to try is this: http://pims.selfip.net/cgi-bin/gitweb.cgi?p=libiphone-usbmuxd;a=sna...

    It's a snapshot of my libiphone-usbmuxd before I removed the usbmux.c and usbmux.h files. It was a big change so perhaps there's something that got messed up.

  • Martin S.

    Martin S. May 26th, 2009 @ 01:18 PM

    A level 5 usbmuxd debug log here: http://gist.github.com/118058

    Whatever combinations of clients I try, I occasionally get that error. At the same time the iphone's syslog shows me what I posted in the comment above which makes me believe something wrong is sent to the device.

    Once I got some more time I can attempt to debug it myself.

  • Nikias Bassen

    Nikias Bassen May 27th, 2009 @ 11:56 PM

    According to the log file, a second client program never connects.
    The line

    usbmuxd: new client connected (fd=5)
    Is from the first client performing usbmuxd_scan().

    And this:

    usbmuxd: new client connected (fd=6)
    is from the first client performing usbmuxd_connect().

    When a second client connects, there will be another two "new client connected" lines. Perhaps there's something wrong with the socket file... I have no idea to be honest.

  • Martin S.

    Martin S. May 28th, 2009 @ 08:55 AM

    I have updated the log at http://gist.github.com/118058 to include the full communication process.

    What I did in to get that log was basically:

    • Start usbmuxd -f -v -v -v -v -v
    • Start iproxy 5000 22
    • Connect with ssh to localhost:5000 and login to the device
    • Use ./dev/iphone_id to get the name which failed with ERROR: Can't connect to device
    • Then quit the ssh session
    • Terminated iproxy
    • Terminated usbmuxd

    Again, once I got some time (moving towns now) I can attempt to debug it myself.

  • Paul Sladen

    Paul Sladen May 30th, 2009 @ 12:18 PM

    • Title changed from “Integrate usbmuxd (when it's ready)” to “udev matching”

    Martin, would you be able to investigate matching based on:

    #ATTRS{bInterfaceClass}=="ff", ATTRS{bInterfaceSubClass}=="fe", ATTRS{bInterfaceProtocol}=="02", SYMLINK+="usbmux/interface"

    This is what we /really/ want to be matching based on and it would be good if it could at least stay commented in the udev file in the meantime.

  • Martin S.

    Martin S. May 30th, 2009 @ 01:00 PM

    I did before I updated the current rules, the problem with this is:

    • It trys to match an "invalid" vendor interface
    • udev does not treat an usb interface as a device (rather as a device container), does not create one and there is neither something to symlink nor give a name to (Since actually using the NAME="usbmuxintf" rule should create "/dev/usbmuxintf"). If you use the udevadm test thingy you can see how udev carefully skips this.
  • Paul Sladen

    Paul Sladen May 30th, 2009 @ 01:32 PM

    Martin: nod.... maybe it needs fixing upstream in udev for it to be possible.

    Martin: Looking through that last log, there is an enourmous of communication with to port 62078 (lockdownd) before any communication/attempt to port 22 (ssh).

    Any idea where that's coming from if iproxy is the first client started?

  • Martin S.

    Martin S. June 7th, 2009 @ 01:39 PM

    • Title changed from “udev matching” to “Integrate usbmuxd”
    • State changed from “open” to “resolved”

    Closing this ticket as the initial goal "Integrate usbmuxd" has basically been reached. :)

    I'll try to move the two other issues discussed here (failed multiplexing and udev rules) into new tickets.

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป

A project around supporting the iPhone in Linux.

See http://libimobiledevice.org

People watching this ticket

Attachments

Tags

Referenced by

Pages