Friday, May 27, 2011

I'm slowly learning to hate Eclipse less

I'm doing more development on my Mac, which means more Eclipse. Of course I hate it. I've been using Visual Studio since 1998, and Turbo C++ before that, and compared to those IDEs Eclipse feels like... well, it feels like Linux. There's no better way to put it. It feels like a hodgepodge of great ideas put together by a large number of incredibly smart people with almost no coordination whatsoever.

Fortunately I work with a large number of people who are both smarter and more patient than myself, so I was able to get help. Fred Sauer, better known as one of the minds behind Angry Birds Chrome, sat down and shared some tips that ended up making a huge difference to me. Here's a selection of the tips that helped me the most:

  1. Eclipse "workspaces" are pretty much just like Visual Studio solutions. They seem different and weird at first, because (a) they're directories, not files, and (b) Eclipse makes you choose one at startup. But end of the day, they're just containers for projects.
  2. "Close Unrelated Projects" is the magic command to unclutter your workspace--it closes up every project that doesn't live in the dependency chain of the project you're currently working on. Nice!
  3. Command+3 (Ctrl+3 on Windows) is your friend. It brings up a quick search of all menu items, keyboard commands, and shortcuts. This is one Eclipse feature I really wish Visual Studio had--I use it all the time and it makes learning the IDE far easier. Unless, of course, you don't know what the command you want is named....
  4. The command I want most but couldn't figure out the name for: Backward History. It's the rough equivalent of Ctrl+Tab in Visual Studio. There's also a Forward History.
  5. Eclipse builds all the time by default. You can turn this off in the Project menu, at which point Ctrl+B does what F7 does in Visual Studio.
Thanks to Fred's tips--especially the one about Command+3--I've been able to use Eclipse for a week or two now without wanting to chew off my own arm and use it to club people to death. So that's an improvement. I won't say I love Eclipse--I doubt I ever will--but I'm coming to terms with it. If I find any more useful tips I'll post them here.

Wednesday, May 18, 2011

NDK debugging without root access

Recently I made a comment to the effect that while ndk-gdb enables debugging on unrooted devices, Nvidia's Eclipse plugin and WinGDB's Visual Studio plugin don't. This means that if you want easy, IDE-based device debugging today, you need to start by running "adb root". Maybe that doesn't seem like a problem if you've already rooted your phone or if most of your debugging is done on an emulator. But if you're trying to develop on your personal phone, and it's a subsidized phone from a carrier that locks out root access, and you're trying to debug something the emulator doesn't support (cough, OpenGL 2, cough) this is kind of a big deal. Even if you've got a rooted device, why run as root if you don't have to?

This isn't meant as a criticism of either NVidia or the WinGDB folks. WinGDB has explicitly said they're only supporting emulators right now, and NVidia is targeting its own devkits that have AFAIK always shipped rooted. I highly doubt that they tried and failed to debug without root, I just don't think that they've had any real reason to try. But I was surprised to see just how small of a change needs to be made in order to enable debugging on unrooted devices.

The short answer to the question "how can I debug a process if I'm not root?" is simple: the debugger needs to run under the same account as the process it's debugging, and it can't do things (e.g. create sockets) which might be forbidden to that account. The long answer is more involved, because running under the same account isn't quite as simple as it sounds. To understand why this is tricky, let's first revisit how accounts and privileges work on Android.

When it comes to restricting privilege, desktop OSes have nothing on Android. Windows, OSX, and Linux all have the concept of root and non-root accounts, but they apply these concepts mostly to users (although both consumer OSes have followed *nix's lead in allowing sudo-style temporary privilege escalation, which while sometimes annoying, at least prevents everyone from running as root all the time). In general, every process the user runs has access to the same files and settings as any other process, give or take. Which means that "OpenOffice.exe" and "MalwareRiddenToolbarInstalledWithSomePornMyRoommateDownloaded.exe" have approximately equal ability to read, write and delete your stuff.

Android takes a different tack, one that's more common in mobile systems: it creates a different account for each app. This has two immediate advantages. First, every app gets its own "home directory" that other apps can't access, so it's harder for programs to screw with each other. Second, it makes it easy for the system to customize a set of privileges for each application. Some apps need to access the Internet, write to your contacts list, and send SMS texts; others don't.

But this compartmentalized security model makes life hard for gdbserver. While the underlying Linux permission model for debugging is pretty reasonable--as a normal user, you're allowed to debug any app that's running under the same account--that model assumes that accounts are tied to users or roles, not individual apps. What works on the desktop fails on Android because by default the app and the debugger will run under separate identities.

As an example, here's me trying to run the command line that WinGDB issued for my most recent debugging session. I'm running it against an unrooted Xperia Play. (I extracted the command line using ProcMon .)


>adb shell /data/data/com.example.testwingdb/lib/gdbserver :1001 --attach 1221
Cannot attach to process 1221: Operation not permitted (1)


No dice. The process I want to debug is running under the identity that's been assigned to com.example.testwingdb. But gdbserver is running under a different account--in this case, the default shell identity.

So what to do? Well, it turns out that there is a simple way out of this mess. Android ships with a utility called run-as. The run-as command takes a package name and a command line, then turns around and executes that command line under the security identity of the package you named. It's just like sudo, except run-as lets you specify which identity you want to run under while sudo always uses root. [Edited to add: Some Android device builds have issues with run-as. See my comment (comment #3 below this post).]

Here's me running the unix "id" command first without, then with "run-as". (For this example and the ones that follow, I'm using a package called com.example.testwingdb. If you're playing along at home, substitute your own package name.)


>adb shell id
uid=2000(shell) gid=2000(shell) groups=1003(graphics),1004(input),1007(log),1009(mount),1011(adb),1015(sdcard_rw),3001(net_bt_admin),3002(net_bt),3003(inet)


>adb shell run-as com.example.testwingdb id
uid=10117(app_117) gid=10117(app_117) groups=1003(graphics),1004(input),1007(log
),1009(mount),1011(adb),1015(sdcard_rw),3001(net_bt_admin),3002(net_bt),3003(inet)



So we use run-as and all is dandy, right? Not quite yet. The last part of the trick is that the executable we want to run with run-as, in this case gdbserver, needs to be in a place where our app uid has permission to execute. Sharp-eyed readers may have noted that my instance of gdbserver lives in /data/data/com.example.testwingdb/lib. Fortunately for the lazier programmers among us, it's not there by accident. It got there because ndk-build automatically puts it there when it makes a debug build. It puts it there because that's where it needs to be if you want to run it under com.example.testwingdb's uid.

With this in mind, we can make a very small tweak to the command line:


>adb shell run-as com.example.testwingdb /data/data/com.example.testwingdb/lib/gdbserver :1001 --attach 1221
Attached; pid = 1221


Woohoo!! Gdbserver is launched and has successfully attached to my process. All is perfect.... well, until this happens:



Can't bind address: Permission denied.
Exiting

Here we see the second consequence of the Android security model: apps have fine grained permissions. In this case, my app never asked for Internet permissions, so it's unable to open a socket--and because gdbserver is running under my app's uid, it can't open sockets either.

There's two ways to solve this. We could just modify our app manifest to request Internet permission. But that would suck: we don't need that permission for anything else, so we'd be making a significant change to our app's capabilities just to make it debuggable. A better solution is to do what ndk-gdb does: create a named pipe that gdbserver can use instead of a socket. Communication over named pipes doesn't require special permission as long as the pipe itself is accessible to the app, and adb includes the "forward" command that magically turns a device-side named pipe into a host-side socket:


>adb forward tcp:5039 localfilesystem:/data/data/com.example.testwingdb/debug-pipe

To use the named pipe, we launch gdbserver like this:


>adb shell run-as com.example.testwingdb /data/da
ta/com.example.testwingdb/lib/gdbserver +debug-pipe --attach 1221
Attached; pid = 1221
Listening on sockaddr socket debug-socket



And BAM. We can now connect up our favorite gdb client to port 5039 on the host, and it will communicate with the device-side instance of gdbserver over the named pipe /data/data/com.example.testwingdb/debug-pipe.

As far as I can tell, that's all it takes to enable rootless debugging. Let's review:

  1. Launch gdbserver under the uid of the process to be debugged, using run-as.
  2. Tell gdbserver to use a named pipe instead of a socket to communicate with the host.
  3. Use adb forward to forward the device-side named pipe to a host-side tcp socket.
Hope that helps!

Monday, May 16, 2011

Android talks from GDC...

...are now available online, mostly. And mostly in new improved form. We were able to get almost all of the talks accepted to Google I/O this year, and those sessions were taped. So check these out:


  • My talk, Bringing C and C++ Games to Android, which I got to late and with an incredibly hoarse voice. Aaargh. Dan Galpin was my co-presenter and he covered for me on the first couple of slides. Thanks Dan! This talk is a good intro to the NDK, and also contains an overview of common pitfalls that we've seen in more than one game.
  • Dan's other talk, Evading Pirates and Stopping Vampires, covers nifty license server tricks for protecting your IP. This is especially important for games that host additional content (as in, the stuff that didn't fit in your apk) on the web. Nonpaying customers can suck your bandwidth bill and kill your profits if you don't take some precautions, but the Android license server can help protect you from these vampires as well as from more traditional pirates.
  • Nico Weber's 3D Graphics on Android: Lessons Learned from Google Body. Nico ported Google Body to Android, and he pulls no punches when he talks about the issues he ran into. A must-see for anyone who's tackling serious OpenGL programming on the Android platform.
  • And the great Chris Pruett, no longer a Googler but still passionate about Android gaming, reprised his excellent presentation Building Aggressively Compatible Android Games. Check out Chris's new venture, Robot Invader. He's already got a better blog than mine, so go over there and read his stuff now.
Sorry to the audio fans out there--the Android audio team was busy writing code and couldn't come to I/O to present the OpenSL talk that they did at GDC. Bummer! We'll try and get Glenn and Jean-Michel out to another conference or Dev Day soon so we can get their talk on the Web. 

Victory!

I finally got the latest version of WinGDB For Android to really truly work. Best news: it's faster than the Eclipse debugger. A lot faster. So far it feels about as fast as the Xbox 360 debugger--not as fast as local, but very usable. So, about as good as you can expect from what is, after all, a remote debugger operating over a serial link.

Just a few small snags in what is, after all, still a beta product:

  • Your device has to be rooted. I think this is because nobody's quite figured out that you need to do some special things to run gdbserver as non-root. The ndk-gdb shell script does it right; WinGDB and the NVidia Eclipse plugin don't.
  • You have to run Visual Studio as administrator. I found this really surprising, since it's been possible for years to build a VS plugin that didn't require admin privileges to run. Even a debugger replacement
  • I couldn't get WinGDB's "Deploy" option to work. Even when it said it worked, it turned out to have not in fact succeeded. Their website says this is due to an adb bug. I'm skeptical; I was able to "ant install" from my project directory quite reliably. On the other hand, I was just using ant to install to the default device, and they're trying to make sure the deployment goes to a specific device (the one you said you wanted to debug on), so maybe they're hitting an edge case that makes adb fall over.
  • Dependency checking seems to be nonexistent--doing anything seems to trigger a build of the world. And I mean anything, including pressing the "Debug" toolbar button. This is pretty annoying even when "the world" is a sample project with one source file in it. On an actual application, it will be crippling. If they fix nothing else, this has to be remedied.
  • Intellisense is broken. This sort of doesn't surprise me; the VS plugin mechanism makes it kind of hard to replace the native debugger and build system without replacing absolutely fucking everything. I was kind of hoping Visual Assist might step in and lend a hand, but no dice. From Visual Studio's point of view, you're not in C++ anymore, you're in some alien project that it has no hope of understanding. The good news is that the WinGDB folks know this and claim to have a fix in the works. I'll be really interested to see what they come up with.
  • And a smattering of small UI bugs that you'd expect from a beta build.
All in all, not bad. The inability to do a minimal build has me worried, but I'm going to do a little poking around and see if it's something I'm doing wrong. If I can get it fixed, this might turn into my main debugging tool.