I learned something today

Darkness on the Internet

2012-01-18T07:29:00.000-08:00

Today is Stop SOPA day, with sites like Wikipedia going dark in protest. Google isn't going dark, but it's got a doodle about it, so you know it's got to be at least as important as, say, Doraemon.

Please take a moment to sign a petition or contact your congressperson about SOPA and PIPA. This is a complicated issue and there are two sides to every argument, but as an internet user I feel SOPA is a horrible idea.

To understand why SOPA is a bad idea, consider this analogy. Let's say I call you up on your T-Mobile phone, and leave you a voicemail. (I'll use voicemail as an example because it's stored on T-Mobile's servers.) As part of my voicemail message, I play you a bit off my new Lady Gaga CD. (I have to use Lady Gaga as an example, because I work for Google. Long story.) Let's assume for the sake of argument that this is illegal. Who is at fault?

Am I at fault for playing the song? To the extent that playing it was illegal, I'd say yes. Absolutely, I'm at fault. Punish me.

Are you at fault for listening? You know what, I'll bite. Let's punish you, too--you should have hung up as soon as you realized there was unauthorized Gaga on the line.

Is T-Mobile at fault for storing my voicemeil? Think for a minute about what that means. How exactly was T-Mobile supposed to prevent this infringement? There's only one way: eavesdropping on every single voicemail that is ever left by anyone that uses their system.

Quite aside from the privacy concerns, this is a logistical nightmare. They'd have to have an army of people listening in on every conversation that anyone ever made, ready to flip a kill switch the moment they heard "Poker Face." And that's an easy example. What if the infringement involved someone less famous, like Elvis Hitler or Toad the Wet Sprocket? What if I am Elvis Hitler, and I'm playing you a track from my long-awaited followup to "Disgraceland?" There's going to be a lot of judgment calls, and a lot of eavesdropping.

Now let's talk about punishment. In this analogy, SOPA would not only require T-Mobile to do the eavesdropping and make the judgment calls. It would also allow any entertainment company to shut off T-Mobile completely just on the allegation that infringement occurred. Until it proved itself innocent, T-Mobile would be unable to connect to the U.S. telephone system or use U.S. airwaves. (In this analogy I'm granting the government miraculous powers over the electromagnetic spectrum--which, if you think about it, is not much more far-fetched than the miraculous powers it would need to have in order to actually enforce SOPA the way Congress apparently thinks it can.)

Think about that for a minute: an entire company put on hold on the basis of an allegation by some media multinational. Millions of employees out of work. Tens of millions of customers who can no longer communicate. All in the name of stopping infringment.

Replace "T-Mobile" with "YouTube," "Facebook," or "Google" and that's SOPA.

Listen, I don't want to come across as a big leftie here, but here in America we don't shut companies down even when we have solid proof that they are killing people. Now we're talking about shutting down Internet businesses not because they did something wrong, and not even because we have proof of wrongdoing, but because some entertainment company alleged that that one of that company's customers did something that might be wrong.

And don't even get me started on the actual enforcement clauses, which would empower our government to set up its own Great Firewall. I don't want to sound like Glenn Beck, but folks, this is how liberty ends. Ask a Chinese friend what the Great Firewall is for. Unless they're really dialed into politics, chances are they'll tell you it's to keep out porn and terrorists. We in the West think we'd never stifle political dissent, but we'll do it in a heartbeat if instead of calling it "dissent" you call it "terrorism" or "indecency" or "infringement."

Whatever you think about copyright law, SOPA is not the right way to enforce it. Even if the entertainment industry is right (newsflash: they're not) and a lack of SOPA would cause the entertainment industry to wither up and die, would we really censor the entire Internet in order to keep those Jerry Bruckheimer movies coming every summer? Ben Franklin derided those who would trade liberty for safety. What would he think of those who ask us to trade essential liberties for continued access to mindless entertainment?

So please, contact your congressperson. Wikipedia has a search tool to help you. Don't copy a form letter, write something short and succint and from the heart. Even if it's just a few words, write something. If you're an avid Internet user, SOPA is literally an attack on your way of life. Help put a stop to it.

My first "official" Android Blog post...

2011-11-28T13:38:00.001-08:00

...went live last week here! It's about my own personal pet peeve, games that play audio when you're not playing them. One of my achievements this year has been to improve the technical standards for featuring on the Android Market; at this point, it's almost impossible to get a new title featured if it has this bug. Which means, of course, that part of my job is to send off emails to promising game developers, asking if they can fix this sort of issue so that we can feature their game. I'm hoping that publishing this sort of blog post will help.

Stay tuned for more; in return for being allowed to title my post "Part I of a series," I had to commit to at least three more episodes. :-)

Bug in LockFreePipe

2011-09-19T12:13:00.000-07:00

Dmitry Vyukov, one of my fellow engineers at Google, pointed out a bug in the implementation of LockFreePipe that you can find here and there on the internet, for example here. That reminded me that Bruce Dawson (who coauthored the original with me) had discovered this bug a while ago and tried to explain it to me, but at the time we didn't have the code in front of us and I didn't completely grok what he was trying to tell me.

Anyway, this particular bug is simple: in Read(), after the call to memcpy(), there should be a barrier before the read pointer is updated. Otherwise the read pointer update can become visible before the memcpy is finished, giving the writer an opportunity to overwrite the memory before it's read.

        unsigned long cbTailBytes = min( bytesLeft, c_cbBufferSize - actualReadOffset );
#ifdef _XBOX_VER
        XMemCpy( pbDest, m_pbBuffer + actualReadOffset, cbTailBytes );
#else
        memcpy( pbDest, m_pbBuffer + actualReadOffset, cbTailBytes );
#endif
        bytesLeft -= cbTailBytes;

        if( bytesLeft )
        {
#ifdef _XBOX_VER
            XMemCpy( pbDest + cbTailBytes, m_pbBuffer, bytesLeft );
#else
            memcpy( pbDest + cbTailBytes, m_pbBuffer, bytesLeft );
#endif
        }

// OOPS: no barrier here. These operations could be reordered

// by the compiler or by the CPU. Should have lwsync or _ReadWriteBarrier here.
        readOffset += cbDest;
        m_readOffset = readOffset;

Dmitry also pointed out that a store-release is meaningless without a load-acquire, so there should also be a barrier in Write() after m_readOffset is read:

   bool __forceinline          Write( const void* pvSrc, unsigned long cbSrc )
    {
        // Reading the read offset here has the same caveats as reading
        // the write offset had in the Read() function above. 
        DWORD readOffset = m_readOffset;

// OOPS: need lwsync or _ReadWriteBarrier here if we want it to work
        DWORD writeOffset = m_writeOffset;

Needless to say, I'm humiliated :-) and curious to know why this worked so well in the past. My first guess was that MSVC did not reorder because the read pointer was marked volatile. Of course, that raises the question of whether any of our barriers were necessary, if volatile is so magic. Well, yes, they were; MSVC volatile semantics at the time prevented compiler reordering, but didn't insert any barriers at the CPU level. The write could still have become visible any time before the read was issued. So why didn't it?

I'd say we got lucky. Just because the CPU can reorder stores before reads, doesn't mean that it does. There's a good reason for the PowerPC to delay stores--it's something that can usually be deferred until later, because in general nobody's really waiting for a store to complete. (If the data is getting reused, it's probably in one of the PPC's copious number of registers. And if it's not... see "load-hit-store.") There's seldom a good reason for a read to get delayed, because the read is very frequently a dependency of the next instruction in the queue. I'm speaking very generally and in a fashion that involves waving my hands quite a lot, but the point is that the window for failure on this is pretty small. Not, however, small enough to be nonexistent, which is what you need if you're doing lock-free work. Bummer.

The sad thing is that I really like this class. It's small and lightweight, doesn't do any allocation, and is basically perfect for things like passing data in and out of audio routines where blocking would cause glitches. I originally wrote it to dump sample data out of custom DSP routines on the Xbox 360, and it worked awesomely well for that purpose. But after talking to Dmitry, I realize that it really only worked as well as it did because of quirks in the 360's CPU architecture. It's not a generic piece of code.

In any case, if you're using this class, please fix it by inserting memory barriers as described in the code snippet above. And stay tuned--there may be other bugs that I haven't found yet.

My current frustration

2011-09-10T17:35:00.001-07:00

Checking In...

2011-09-06T14:00:00.000-07:00

I haven't been good about posting to this blog lately, probably because I originally started it to talk about the Android NDK, and I haven't been doing much of that lately. I've been jumping around to different parts of Android, trying to learn more about the entire system. But I've made what Tim Bray says is a fundamental blogging mistake, which is to only write about what you think your audience wants to hear. Oops. I guess I should really start writing about what I've been doing, whether or not it's NDK-related (or even Android related).

What I've mostly been working on, besides the inevitable "partner support" email queue, is a Google Tasks app. Yeah, I know, not groundbreaking. But all of the apps I've found so far do it wrong. They don't authenticate with my Google account token--instead, they ask me for my actual Google credentials, which I'm not going to give them. And they don't sync correctly. I want a Tasks app that's as solid as the Gmail app, and since I have the code to the Gmail app, I think I ought to be able to do it. I've got most of the account management stuff worked out, and I have access to the GTasks API (which is, by the way, kind of incomplete), and now I'm writing a ContentProvider so that I can hook it to a SyncAdapter and make background sync work. That's taken me down a path of inventing a lightweight ORM so that I can write a ContentProvider without having to do a whole lot of redundant keyboarding. (I almost said "typing," but that's an overloaded term in this context.)

So that's probably what I'll be writing about for the next little while: my quixotic attempt to do a simple Android app The Right Way. I want it to be a shining example in every possible way. I've already found a few places where that's going to be impossible, because no Right Way exists. But that's a subject for another post...

Ndk-gdb and service processes don't (currently) mix

2011-08-08T15:00:00.000-07:00

Got a bug from one of our partners complaining that they had two almost identical Android projects, but one of them was debuggable and the other one wasn't. After checking all the usual culprits, we finally narrowed it down to a single line in the manifest of the undebuggable project (names have been changed, obviously):

It was, of course, very tempting to blame the SomeMiddleware company and its purportedly Useful service, but it wasn't their fault at all. Instead, it turned out to be a very small error in one of the ndk's awk scripts. It turns out that the script that finds the pid of the process you're supposed to be debugging (extract-pid.awk) was, for various reasons, doing a substring match on the process name. This almost always works, because there's almost always only one process running under your package name at any one time. But it doesn't always work.

In this case we were bitten by a quirk of the Android service architecture. The manifest is allowed to specify a process name under which to run the service by using the android:process attribute. Normally this just specifies the exact name of the process that the service should run under. But there's a special case: if the process name so specified begins with a colon, then the service process is considered "private" and its name is concatenated with that of the main process. So instead of running as "com.somemiddleware.useful.process" the service runs as "com.mycompany.mygame:com.somemiddleware.useful.process."

See the problem? Now we have two processes that match our package id: our main game process that runs as "com.mycompany.mygame" and the middleware service that runs as "com.mycompany.mygame:com.somemiddleware.useful.process" -- and guess which one gets matched by the ndk-gdb awk script?

The fix to extract-pid.awk is simple, and I've already submitted a patch, so hopefully that'll get accepted and shipped in the next NDK. Meanwhile, if you happen to run into this bug, you can either patch the script yourself, or you can just eliminate the colon from the beginning of your service process name while you're debugging.

If you're interested, here's a gnu diff version of the patch:

25,27d24

< # NOTE: For some reason, simply using $9 == PACKAGE does not work

< # with this script, so use pattern matching instead.

< #

31a29

> RS = "\r\n"

40c38

< gsub("\\.","\\.",PACKAGE)

---

> #gsub("\\.","\\.",PACKAGE)

46c44

< $9 ~ PACKAGE {

---

> $9 == PACKAGE {

Yes, I am now on Google+

2011-07-12T11:33:00.000-07:00

Of course.

Anyone want to do some Android ports?

2011-07-08T10:38:00.000-07:00

A friend of mine is looking for a team to port a couple of titles to Android. Email me if you're interested: i lewis at google dot com.

I'm slowly learning to hate Eclipse less

2011-05-27T08:36:00.000-07:00

I'm doing more development on my Mac, which means more Eclipse. Of course I hate it. I've been using Visual Studio since 1998, and Turbo C++ before that, and compared to those IDEs Eclipse feels like... well, it feels like Linux. There's no better way to put it. It feels like a hodgepodge of great ideas put together by a large number of incredibly smart people with almost no coordination whatsoever.

Fortunately I work with a large number of people who are both smarter and more patient than myself, so I was able to get help. Fred Sauer, better known as one of the minds behind Angry Birds Chrome, sat down and shared some tips that ended up making a huge difference to me. Here's a selection of the tips that helped me the most:

Eclipse "workspaces" are pretty much just like Visual Studio solutions. They seem different and weird at first, because (a) they're directories, not files, and (b) Eclipse makes you choose one at startup. But end of the day, they're just containers for projects.
"Close Unrelated Projects" is the magic command to unclutter your workspace--it closes up every project that doesn't live in the dependency chain of the project you're currently working on. Nice!
Command+3 (Ctrl+3 on Windows) is your friend. It brings up a quick search of all menu items, keyboard commands, and shortcuts. This is one Eclipse feature I really wish Visual Studio had--I use it all the time and it makes learning the IDE far easier. Unless, of course, you don't know what the command you want is named....
The command I want most but couldn't figure out the name for: Backward History. It's the rough equivalent of Ctrl+Tab in Visual Studio. There's also a Forward History.
Eclipse builds all the time by default. You can turn this off in the Project menu, at which point Ctrl+B does what F7 does in Visual Studio.

Thanks to Fred's tips--especially the one about Command+3--I've been able to use Eclipse for a week or two now without wanting to chew off my own arm and use it to club people to death. So that's an improvement. I won't say I love Eclipse--I doubt I ever will--but I'm coming to terms with it. If I find any more useful tips I'll post them here.

NDK debugging without root access

2011-05-18T22:03:00.000-07:00

Recently I made a comment to the effect that while ndk-gdb enables debugging on unrooted devices, Nvidia's Eclipse plugin and WinGDB's Visual Studio plugin don't. This means that if you want easy, IDE-based device debugging today, you need to start by running "adb root". Maybe that doesn't seem like a problem if you've already rooted your phone or if most of your debugging is done on an emulator. But if you're trying to develop on your personal phone, and it's a subsidized phone from a carrier that locks out root access, and you're trying to debug something the emulator doesn't support (cough, OpenGL 2, cough) this is kind of a big deal. Even if you've got a rooted device, why run as root if you don't have to?

This isn't meant as a criticism of either NVidia or the WinGDB folks. WinGDB has explicitly said they're only supporting emulators right now, and NVidia is targeting its own devkits that have AFAIK always shipped rooted. I highly doubt that they tried and failed to debug without root, I just don't think that they've had any real reason to try. But I was surprised to see just how small of a change needs to be made in order to enable debugging on unrooted devices.

The short answer to the question "how can I debug a process if I'm not root?" is simple: the debugger needs to run under the same account as the process it's debugging, and it can't do things (e.g. create sockets) which might be forbidden to that account. The long answer is more involved, because running under the same account isn't quite as simple as it sounds. To understand why this is tricky, let's first revisit how accounts and privileges work on Android.

When it comes to restricting privilege, desktop OSes have nothing on Android. Windows, OSX, and Linux all have the concept of root and non-root accounts, but they apply these concepts mostly to users (although both consumer OSes have followed *nix's lead in allowing sudo-style temporary privilege escalation, which while sometimes annoying, at least prevents everyone from running as root all the time). In general, every process the user runs has access to the same files and settings as any other process, give or take. Which means that "OpenOffice.exe" and "MalwareRiddenToolbarInstalledWithSomePornMyRoommateDownloaded.exe" have approximately equal ability to read, write and delete your stuff.

Android takes a different tack, one that's more common in mobile systems: it creates a different account for each app. This has two immediate advantages. First, every app gets its own "home directory" that other apps can't access, so it's harder for programs to screw with each other. Second, it makes it easy for the system to customize a set of privileges for each application. Some apps need to access the Internet, write to your contacts list, and send SMS texts; others don't.

But this compartmentalized security model makes life hard for gdbserver. While the underlying Linux permission model for debugging is pretty reasonable--as a normal user, you're allowed to debug any app that's running under the same account--that model assumes that accounts are tied to users or roles, not individual apps. What works on the desktop fails on Android because by default the app and the debugger will run under separate identities.

As an example, here's me trying to run the command line that WinGDB issued for my most recent debugging session. I'm running it against an unrooted Xperia Play. (I extracted the command line using ProcMon .)

>adb shell /data/data/com.example.testwingdb/lib/gdbserver :1001 --attach 1221
Cannot attach to process 1221: Operation not permitted (1)

No dice. The process I want to debug is running under the identity that's been assigned to com.example.testwingdb. But gdbserver is running under a different account--in this case, the default shell identity.

So what to do? Well, it turns out that there is a simple way out of this mess. Android ships with a utility called run-as. The run-as command takes a package name and a command line, then turns around and executes that command line under the security identity of the package you named. It's just like sudo, except run-as lets you specify which identity you want to run under while sudo always uses root. [Edited to add: Some Android device builds have issues with run-as. See my comment (comment #3 below this post).]

Here's me running the unix "id" command first without, then with "run-as". (For this example and the ones that follow, I'm using a package called com.example.testwingdb. If you're playing along at home, substitute your own package name.)

>adb shell id
uid=2000(shell) gid=2000(shell) groups=1003(graphics),1004(input),1007(log),1009(mount),1011(adb),1015(sdcard_rw),3001(net_bt_admin),3002(net_bt),3003(inet)

>adb shell run-as com.example.testwingdb id
uid=10117(app_117) gid=10117(app_117) groups=1003(graphics),1004(input),1007(log
),1009(mount),1011(adb),1015(sdcard_rw),3001(net_bt_admin),3002(net_bt),3003(inet)

So we use run-as and all is dandy, right? Not quite yet. The last part of the trick is that the executable we want to run with run-as, in this case gdbserver, needs to be in a place where our app uid has permission to execute. Sharp-eyed readers may have noted that my instance of gdbserver lives in /data/data/com.example.testwingdb/lib. Fortunately for the lazier programmers among us, it's not there by accident. It got there because ndk-build automatically puts it there when it makes a debug build. It puts it there because that's where it needs to be if you want to run it under com.example.testwingdb's uid.

With this in mind, we can make a very small tweak to the command line:

>adb shell run-as com.example.testwingdb /data/data/com.example.testwingdb/lib/gdbserver :1001 --attach 1221
Attached; pid = 1221

Woohoo!! Gdbserver is launched and has successfully attached to my process. All is perfect.... well, until this happens:

Can't bind address: Permission denied.
Exiting

Here we see the second consequence of the Android security model: apps have fine grained permissions. In this case, my app never asked for Internet permissions, so it's unable to open a socket--and because gdbserver is running under my app's uid, it can't open sockets either.

There's two ways to solve this. We could just modify our app manifest to request Internet permission. But that would suck: we don't need that permission for anything else, so we'd be making a significant change to our app's capabilities just to make it debuggable. A better solution is to do what ndk-gdb does: create a named pipe that gdbserver can use instead of a socket. Communication over named pipes doesn't require special permission as long as the pipe itself is accessible to the app, and adb includes the "forward" command that magically turns a device-side named pipe into a host-side socket:

>adb forward tcp:5039 localfilesystem:/data/data/com.example.testwingdb/debug-pipe

To use the named pipe, we launch gdbserver like this:

>adb shell run-as com.example.testwingdb /data/da
ta/com.example.testwingdb/lib/gdbserver +debug-pipe --attach 1221
Attached; pid = 1221
Listening on sockaddr socket debug-socket

And BAM. We can now connect up our favorite gdb client to port 5039 on the host, and it will communicate with the device-side instance of gdbserver over the named pipe /data/data/com.example.testwingdb/debug-pipe.

As far as I can tell, that's all it takes to enable rootless debugging. Let's review:

Launch gdbserver under the uid of the process to be debugged, using run-as.
Tell gdbserver to use a named pipe instead of a socket to communicate with the host.
Use adb forward to forward the device-side named pipe to a host-side tcp socket.

Hope that helps!

Android talks from GDC...

2011-05-16T22:46:00.000-07:00

...are now available online, mostly. And mostly in new improved form. We were able to get almost all of the talks accepted to Google I/O this year, and those sessions were taped. So check these out:

My talk, Bringing C and C++ Games to Android, which I got to late and with an incredibly hoarse voice. Aaargh. Dan Galpin was my co-presenter and he covered for me on the first couple of slides. Thanks Dan! This talk is a good intro to the NDK, and also contains an overview of common pitfalls that we've seen in more than one game.
Dan's other talk, Evading Pirates and Stopping Vampires, covers nifty license server tricks for protecting your IP. This is especially important for games that host additional content (as in, the stuff that didn't fit in your apk) on the web. Nonpaying customers can suck your bandwidth bill and kill your profits if you don't take some precautions, but the Android license server can help protect you from these vampires as well as from more traditional pirates.
Nico Weber's 3D Graphics on Android: Lessons Learned from Google Body. Nico ported Google Body to Android, and he pulls no punches when he talks about the issues he ran into. A must-see for anyone who's tackling serious OpenGL programming on the Android platform.
And the great Chris Pruett, no longer a Googler but still passionate about Android gaming, reprised his excellent presentation Building Aggressively Compatible Android Games. Check out Chris's new venture, Robot Invader. He's already got a better blog than mine, so go over there and read his stuff now.

Sorry to the audio fans out there--the Android audio team was busy writing code and couldn't come to I/O to present the OpenSL talk that they did at GDC. Bummer! We'll try and get Glenn and Jean-Michel out to another conference or Dev Day soon so we can get their talk on the Web.

Victory!

2011-05-16T22:09:00.000-07:00

I finally got the latest version of WinGDB For Android to really truly work. Best news: it's faster than the Eclipse debugger. A lot faster. So far it feels about as fast as the Xbox 360 debugger--not as fast as local, but very usable. So, about as good as you can expect from what is, after all, a remote debugger operating over a serial link.

Just a few small snags in what is, after all, still a beta product:

Your device has to be rooted. I think this is because nobody's quite figured out that you need to do some special things to run gdbserver as non-root. The ndk-gdb shell script does it right; WinGDB and the NVidia Eclipse plugin don't.
You have to run Visual Studio as administrator. I found this really surprising, since it's been possible for years to build a VS plugin that didn't require admin privileges to run. Even a debugger replacement.
I couldn't get WinGDB's "Deploy" option to work. Even when it said it worked, it turned out to have not in fact succeeded. Their website says this is due to an adb bug. I'm skeptical; I was able to "ant install" from my project directory quite reliably. On the other hand, I was just using ant to install to the default device, and they're trying to make sure the deployment goes to a specific device (the one you said you wanted to debug on), so maybe they're hitting an edge case that makes adb fall over.
Dependency checking seems to be nonexistent--doing anything seems to trigger a build of the world. And I mean anything, including pressing the "Debug" toolbar button. This is pretty annoying even when "the world" is a sample project with one source file in it. On an actual application, it will be crippling. If they fix nothing else, this has to be remedied.
Intellisense is broken. This sort of doesn't surprise me; the VS plugin mechanism makes it kind of hard to replace the native debugger and build system without replacing absolutely fucking everything. I was kind of hoping Visual Assist might step in and lend a hand, but no dice. From Visual Studio's point of view, you're not in C++ anymore, you're in some alien project that it has no hope of understanding. The good news is that the WinGDB folks know this and claim to have a fix in the works. I'll be really interested to see what they come up with.
And a smattering of small UI bugs that you'd expect from a beta build.

All in all, not bad. The inability to do a minimal build has me worried, but I'm going to do a little poking around and see if it's something I'm doing wrong. If I can get it fixed, this might turn into my main debugging tool.

Big life changes!

2011-04-27T12:04:00.000-07:00

It's been a crazy few weeks for me. First and most importantly, our daughter Amanda Evelyn (Chinese name: Yilan) Ni-Lewis was born on April 9! Mother and baby are doing fine, although at 8lbs 8oz little Amanda was quite a burden on my not-so-large wife.

Second, on the day before Amanda was born, I officially transferred to Android Developer Relations. I'd started out in Chrome and spent a few months covering the gap left by Mark DeLoura's departure as Game Developer Advocate lead, but I've been wanting to get back into a more technical and product-focused role. When my friend Chris Pruett quit Android to go chase his dreams, an opportunity opened up and I took it. So I'll be working much more closely with Android game developers now. I'm still helping to coordinate Google-wide game developer relations activities, but that'll be more of a 20% project for me from now on.

I'm also taking the plunge and trying to learn to love Eclipse. I'm not sure how that's going to go. I'll keep you posted. :-)

GDC slides, how do they work?

2011-03-28T13:34:00.000-07:00

[update: (versions of) (most of) these talks are now online!]

tomaswoj writes, "I would love to see the presentation you gave on NDK/C++ in Android on GDC2011. You've mentioned that it will be up for d/l at some point of time. GDC already shares lots of this year presentations in their 'free' section, but yours is not among them. Does it mean, it will be in the pay area only? Or just overlooked? "

So, here's the deal. Back in a more innocent time of my life (late February), I was under the impression that it would be fairly easy to get slides posted online. I thought it would be about as easy as getting them OK'd for GDC in the first place--a little tedious, but not that hard. Hah. I was wrong.

First and foremost, game developer relations deals with multiple product teams, each of which has their own PR org. We have to make sure that each team is ok with what we're saying about their product. And in the case of posting slides online, we need to be sure that nothing can be taken out of context, because we don't have the luxury of explaining them or doing Q&A. So Chrome, App Engine, YouTube, and Android all need to be cool with not just the slides, but with all of the many uses and misuses they might suffer as they get passed around on the web.

Most of the product teams have signed off or requested small changes. But Android is different, and this is why the answer to @tomaswoj is not going to make him (or me) happy. See, Android is not a Google product in the way that Chrome or Gmail is a Google product. Android is a product of the Open Handset Alliance, which is an industry consortium involving many, many companies--carriers, hardware manufacturers, and so on. It's true that Google has an important role in the Android ecosystem, but it's not a lone wolf. The challenges of balancing the needs of all of the OHA's members make Android PR extra careful.

All of which boils down to one sad fact: Android's policy, which I learned after the fact, is to release slides only if they are accompanied by video. And we did not have budget for video at Moscone. (It's a union shop--we couldn't just handycam it ourselves.)

So, what's going to happen to those slides? Well, there is one possibility that I'm exploring. The developer relations team does have a budget for video production at Google I/O, which is happening in May. It turns out that many of the presentations my team is giving at I/O are very similar to the ones we gave at GDC. (One might even say we were recycling the material. I would respond, waste not, want not.) So there's a reasonably good chance that we can post the material online after I/O.

Does this suck? Yes. I learned a sad lesson this month. I'm very sorry that others--our excellent speakers and the developers who need this material--have to suffer for my sins. We'll do what we can to make the material available in some way, as soon as we can. But I can't make any promises about the NDK talk or any other Android material.

Handedness! Argh!

2011-03-24T10:18:00.000-07:00

This week I started hacking on Unity, because all the cool kids are doing it. I wanted to see what all the fuss was about. So far I'm pretty impressed. I like the component-oriented architecture, which is something I've been lazily experimenting with since I left Offset Software eighteen months ago. One of the frustrating things about Offset (and many other engines) is the monolithic entity model: every entity has physics, audio, etc., whether it's needed or not. In Unity every entity starts out empty and you add components as necessary. I don't know much about the underlying code in Unity, so I don't know how far down the component-oriented thing goes, but it seems pretty promising so far.

I also got a copy of Maya, because I was sick of trying to learn Blender and we ended up with an extra copy at work. W00t! So now I can kick it like a real artist. (If only I were a real artist. So far I've been able to create a bad copy of the ship from Elite, and that's about it.) And best of all, Unity imports Maya files directly. No exporter FTW!!!.....

...and that's where the problems began. Because Maya is all about right-handed coordinates, and Unity is resolutely left-handed. And neither product seems to offer any feature that would fix this. Maya has a left-handed tangent space option, but nothing for actual world space. I can flip my models 180° in Maya (this works because they're symmetrical around Z), but that doesn't actually change the vertices in Unity--it just adds a 180° rotation to the model's transform. Which is useless, because that's the world transform--there's no "import space to model space" node that I can find. Useful things like the Unity follow-cam use that transform to figure out which way the model is pointed. This led to hilarity as I tried multiple ways of rotating the model in Maya, only to find that no matter how I rotated it, and no matter what orientation the model appeared to have in my Unity scene, the follow-cam always floated around to the exact same spot right in front of my little spaceship. (In case you're new to this, let me emphasize that the follow-cam is supposed to always be in back of what it's following.)

I guess the right thing to do is probably to parent the model under some other game object, and let the parent-to-child transform take care of the handedness switch. (I still need to figure out how to load a Unity transform with a matrix that inverts Z, because doing a 180 around Y isn't going to cut it forever.) But it still chafes me that neither Maya nor Unity has a simple way to invert Z. This seems like the kind of problem that would crop up fairly frequently.

Maybe someone reading this knows how to fix the handedness mismatch between Maya and Unity in a clean way?

So close but yet so far...

2011-03-19T19:02:00.000-07:00

So WinGDB has a beta up of their "mobile edition," which is light years beyond the hacktastic preview I wrote about a couple of months ago. I installed it today and wow. This is what Android hacking on VS was meant to be. You can create projects via the normal new project dialog, you can start debugging with the normal play button, and all of the random settings and junk that we had to deal with before are gone.

What's really scary-cool is that they're actually working within the NDK make system. The project file is not a .vcproj or an MSBuild file, it is Android.mk. You add a file to the project via Visual Studio, and that's where it lands. That means they have completely rewritten the storage part of the Visual Studio project system, which is kind of a big deal. I'm not sure if I agree 100% with the design (I'm actually kind of a fan of MSBuild, at least for Visual Studio stuff)*, but I have to respect that kind of work.

Only problem is, right now it doesn't work! Not on my computer, anyway. It does what looks like a full and successful build, but then tells me that the build failed and refuses to launch the debugger. I turned diagnostics up to 11, but I can't find anything that even looks like a failure in my build log. Building from the command line via ndk-build works fine. But no luck getting Visual Studio to act like all is well. I'll ping the WinGDB guys, of course. I'd be interested to know if anyone reading this can get this to work. I must be doing something wrong, but I've tried this with both the NDK samples and a scratch project, and I'm not having any luck.

Well, it's probably for the best. I'd promised myself this would be the weekend that I finally learned how to use Unity. So I'll get onto that. Leave a comment if you get the WinGDB beta working better than I did...

Weekend Blogging: a recipe hack

2011-03-19T18:22:00.000-07:00

Three weeks ago I made a soup with Lagunitas GnarlyWine that was so astonishingly not vile that I wanted to share the recipe. Of course, I promptly forgot it--probably because the GnarlyWine was accompanied by a half liter of Hop Stoopid, which did not go into the soup. (Neither did half the GnarlyWine, come to think of it.) So I have no recipe to remember that weekend by. In fact, I have very little memory of that weekend in general, except for a recollection of Hop Stoopid being one of the most awesome beers I've ever had.

So this weekend I'm writing down what I remember of my most recent experiment. Alcohol was not involved--quite the opposite, in fact. I wanted to make Beef Burgundy in our new automatic pressure cooker. But, the wife is pregnant, and pouring an entire bottle of wine into her beef stew sounded like a bad idea. Even if most of the alcohol did cook out, she'd kill me--she's very, very careful of our baby's health.

So how to spice beef stew without wine? I stumbled upon what turned out to be an amazing mix: onion powder, balsamic vinegar, white truffle oil, and just a bit of cheap-ass Costco brown gravy mix. Oh, and some McCormick garlic Season-All, probably because I use so much of it that nothing tastes right without it. Here's the recipe I ended up with (all measurements are approximate):

16 oz ribeye steak, cubed
1 lb bag frozen peas & carrots
3 red potatos, also cubed
1 tbsp 10 year old balsamic vinegar
1 tbsp white truffle oil
1/2 tbsp onion powder
1/2 tbsp brown gravy mix
1 tsp garlic Season-All
1 tsp basil
1 tsp oregano
1 tbsp extra-virgin olive oil

Combine in pressure cooker, pressure cook for 30 minutes (the auto cooker's "soup" setting) and stir vigorously to break up the potatoes a bit so they thicken the stew. Mwah!

The pressure cooker really is amazing. The meat was so tender. Well, ok, it was USDA choice ribeye, so no real surprise there. But everything was better than it had any right to be after half an hour of cooking. It tasted like it had been slow cooking all day. All the juices and flavors were mixed--even the potatoes were flavorful all the way through. I didn't even want the thing, it was my wife's purchase, but now I'm totally sold.

Oh man, you know what would go awesome with this? A bottle of burgundy. Good thing I'm not pregnant... :-)

In case you're wondering...

2011-03-12T11:02:00.000-08:00

It's been a while since my last update, I know. I caught some kind of flu at GDC that's left me tired and weak. Didn't help that, due to missing our anniversary on Feb. 27, I'd promised my wife to spend Monday on whatever she wanted. That turned out to be moving lots of heavy furniture around to get ready for our baby and for her mom to come stay with us. So on top of the flu I threw my back out.

Anyway, I'm still sick and doing more than answering email is pretty much beyond me at this point. But I do have a beta of WinGDB 2 and I'll hopefully get a chance to try it out before it actually gets released. So stay tuned :-)

Another Open Letter

2011-03-02T10:24:00.000-08:00

An Open Letter to whomever stole my Motorola Xoom tablet at GDC.

Dear Sir or Madam:
You stole my stuff. You suck.

Unless, of course, you use that Motorola Xoom to create a fantastic app for the Android Market. In which case, good on you and I'm glad you're a little lightfingered.

:-)

Ian

Letter to a Bike Thief

2011-03-01T20:48:00.000-08:00

An open letter to whomever stole my bike from the rack at the Moscone Center last night.

Dear Sir or Madam,

First of all, ha ha ha ha ha ha ha. I hope you enjoy your new bike. Especially the way the chain falls off when you downshift. Or the way the right-hand shifter sticks and has to be yanked back with your thumb. Enjoy the slow leaks in both of the tires. And especially enjoy the way you'll have to pump up those leaky tires with two separate bike pumps, because one of the tires has a normal red-blooded American valve, and the other one has one of those elitist European things with the long stem and the little screw gadget on the end.

Second, let me just be the first to salute what I can only assume to be your giant, brass-cored, steel-belted, chrome-plated set of cojones. That bike was locked up right on Market Street in full view of thousands of convention-goers. Admittedly your average Game Developer Conference attendee isn't the most physically imposing specimen, but still. There's an awful lot of them, and they're going back and forth in that area all day and all night because they're diehard party animals. So bravo, sir or madam, you have got to be swinging a massive pair.

Lastly, I want you to understand just what this means to my life. You didn't just steal a reasonably nice but sensibly priced road bike from me. You took away my mode of transportation. That means I have to buy another bike. And that's five hundred to a thousand dollars that could have gone to pay for so many other things.

For example, will I now be able to convince my wife to let me buy a new Xbox 360, the new black one with that fancy new camera controller thingy, even though I already have like five 360s sitting in the garage waiting to get their red rings of death fixed? No. And I was this close to talking her into it, too, man. But now that money is bike money. So I will not be playing Dance Central using my body as a controller this month. There will be no evacuating the dance floor for me. I'm going to have to get by with a regular controller and a stupid white Xbox just like I did last month.

And it's not just the Xbox, either. A bike costs a lot of money. Will I be buying a stack of new games or a bunch of e-books this month? No, that's bike money now. Can I get my 1995 Acura Integra detailed and waxed 'til it shines like the top of the Chrysler building, like I was planning? Gonna have to wait until the bike's paid off. Will I be sipping Champagne and dining on the finest cuisine San Francisco has to offer? Well, yes, because I plan to expense that. But those new shoes that Baby needs? Bike.

Okay, thinking about it, my life is pretty good. You probably need the bike more than I do. After all, I still do have one Xbox that hasn't red-ringed yet, and I've always got my 1995 Acura Integra (last model year before they nerfed the engine control computer!) to fall back on.

So enjoy the bike. You're the one that has to live with being a lousy thief.

GDC: Coming up for air

2011-03-01T20:44:00.000-08:00

After 2.5 days of GDC I'm finally coming up for air a bit. It's going really well. Our Google sessions played to a nearly full house, and we've been turning people away from the Android sessions all day long.

I spoke last night at the Social and Online summit about lessons we learned from Google TV, and again today at the Android dev day on the Android NDK. What do those topics have to do with each other? Nothing. What do I know about either of them? Well... let's just say I had a lot of studying to do.

I actually co-presented both talks. The main speaker for the Google TV talk was Andrés Ferraté, who has given me two gifts in life: one, a much better understanding of Google TV, and two, the knowledge of how to type an 'é' on the Mac. (Everyone who knows Andrés learns how to type é at some point. It's kind of a thing with him.) Andrés put together the main slides and I stepped in to play the role of "guy who knows console gaming" to talk about how Google TV is not a console. It was a fun talk to give, very fluffy but entertaining. For me, at least.

The talk today was much more technical. I played the role of "guy who will explain the basics of the NDK," which was right in line with the work I've done so far. I talked about the ins and outs of getting started and using the NDK on Windows, and introduced things that I've been talking about on this blog, like WinGDB and vs-android. Then Dan stepped up to the plate and laid down his unique brand of magic, otherwise known as "knowing pretty much everything there is to know about NDK programming." Go teamwork!

So dev days are over, and I'm relieved. There were some gaffes but overall I'm pleased. Thanks much to everyone who attended!

One week to go...

2011-02-21T18:08:00.000-08:00

I learned something today: putting on a show, even just a little corner of a show like Google Developer Days at GDC, is really hard. Combine that with the fact that I somehow volunteered to give two talks, and write a bunch of content for our new website, and I'm just about worn out.

I've definitely learned my lesson about the talks: next time I'll either volunteer to organize things or to develop original material, but not both. It's not that either one is difficult, or that I didn't have time for both; it's that the two activities are very different and trying to juggle both was mentally exhausting. Not to mention that it put me in the position of being both boss and worker--I was developing my own material (and blowing deadlines, natch) as I was also trying to crack the whip on other developers and try and get them to meet the same deadlines I was having trouble with. I've gotten better at the management gig over the last twenty years, but it's still really hard to push people to do something I'd have a hard time doing myself.

But my slides are done (well, almost--a little extra polish never hurts), and rehearsals are almost over, and in a week we're going to put on a great show. And then I am going to go hide in a corner and write code for a couple of months. :-)

GDC

2011-02-13T18:14:00.000-08:00

If you've been following my twitter feed--or any of my coworkers' feeds--you've probably heard that we launched a GDC microsite a few days ago. This was my first experience launching a site at Google, and overall it wasn't bad. We got Ryan Thomas, who designed the Google TV site, to do the layout for us. Ryan is actually working on something a lot bigger for us, which we haven't yet announced, but he agreed to take a little time out of his busy schedule and put this together for us.

If you haven't already, this would be a good time to check out google.com/gdc2011. Especially check out the speaker bios--it turns out we have more game developers working for Google than you might have thought! We'll probably be adding some latecomers, too, as some of the talks will have more than one speaker. For instance, Dave Springer is probably going to show up. Who's Dave Springer? Oh, he just founded a little company called Alias. You might have heard of it.

Hope to see you all at GDC. Come check out the Google developer days--I can promise you it will be well worth your time.

In which I access a file

2011-01-26T20:14:00.000-08:00

Hola amigos! Sorry it's been so long since I rapped at ya, but shit's been going down, you know? Shit, in this case, being Google's GDC presentations--two dev days and a couple of sponsored sessions, twelve talks total. We finally have a schedule and with luck we might even be able to post it soon, woohoo!! Now if I could just get my slides done....

But anyway... I got a little more time to work on Android NDK programming last night. I was planning to play around with VBOs a bit, but I got a little sidetracked. See, I wanted to load up my shaders from a file, like a human being, not put them in constant strings like some damn dirty ape. And files are kind of weird on Android, because everything goes in the apk. Getting stuff into the apk is super easy--just toss it in the "assets" directory, easy as pie--but getting stuff out is another story.

OK, first I have to thank Jeebus that I'm using Gingerbread and NDKr5, because I don't actually need to play games in Java to get access to the apk. If you want to know how blessed you are to be living in 2011 instead of 2010, just check out the old instructions for apk access (h/t to Dan Galpin):

Use the asset manager code to get an AssetFileDescriptor:

AssetFileDescriptor afd =

context.getAssets().openFd("ToothyDroid.png");

Read the offset and length from the AssetFileDescriptor:

long offset = afd.getStartOffset();

long length = afd.getDeclaredLength();

afd.close();

Get the APK file name:

context.getApplicationInfo().sourceDir;

Open the APK file at the specified offset and read the specified length. [Code omitted from original, because... why is that, Dan? Because it's a pain to write? ;-)]

Yeah. Weird.

Anyway, with r5 things get a lot simpler, because there's now an AssetManager library that lets you use the asset manager directly. If you're using NativeActivity and the native app glue, you even get your application's asset manager passed down from Java for free as part of the application class. So you end up with code that looks a lot saner:

  AAsset* pAsset = AAssetManager_open(ctx->app->activity->assetManager, "readme.txt", AASSET_MODE_UNKNOWN);

  const int count = 80;
  char buf[count + 1] = {0};

  while (AAsset_read(pAsset, buf, count) > 0) {
  LOGI(buf);
  }
  AAsset_close(pAsset);

That's better, right? It at least feels something like a normal set of filesystem calls. Rock on!!

Of course, it isn't a set of normal filesystem calls... which immediately makes me look ahead to the time when I'm going to want to access other filesystems (the SD card, for instance), and probably want to do so without having to plumb two different sets of file APIs all through my code. And while I'm at it, I probably want to abstract whether I'm even using a filesystem, or whether my files are in memory, or compressed, or streamed, or packed together in one huge bundle. I guess what I'm saying is, I've written enough data access code in my life to know that I'm going to want an abstraction layer at some point.

And so I pull out my trusty stream implementation... and run smack dab into the NDK build system. Ah, but that is definitely a post for another day. And probably at least a slide or two in my GDC talk.

Interview with TechWeb

2011-01-18T13:29:00.000-08:00

UBM Game Network newsletter did an interview with me as part of their pre-GDC coverage. You can read the interview here. This was the first interview I've done over email. I'm pretty pleased with how it turned out--a lot of what I wrote got cut, but I expected that, and what's left is pretty much my own words.

I'm really excited about what we've got planned for GDC this year. We'll be doing two full days of presentations on Monday and Tuesday, and we'll have a booth on the show floor.

I'm in charge of the presentation content, so I'm doing a lot of running around trying to get everyone to finish their slides on time. Of course people will be fixing things all the way up to the day of the show, but we have to at least pretend to have deadlines. :-) The content is shaping up pretty well. We've got some really in-depth technical talks on Android and Chrome both. Most of the really deep tech stuff is going to be presented by the engineers that actually implemented the systems we're talking about. Should be really good.

If you want to hear more about GDC, or about game development at Google in general, you might want to follow @ilewis_goog on Twitter. I generally don't post more than a couple of times a week.

Android Debugging, Visual Studio style!

2011-01-11T14:38:00.000-08:00

UPDATE: More recent beta versions of wingdb make these instructions obsolete. Try it here.

The high point of my week, courtesy of the good people at www.wingdb.com... (click to enlarge)

Full Android NDK debugging in Visual Studio. This is using version 1.8, which is kind of tricky to set up--mad props to the WinGDB support folks who walked me through it. Version 2.0 is coming soon, though, and according to WinGDB, they will have full support for Android NDK.

For the curious, here's how I got this working. If you want to try this, do so at your own risk and please don't go bugging the WinGDB folks about it; they're busy enough working on version 2.0 and they don't need anything slowing them down! For this example, I'm using the project I described in an earlier post.

First, install all of the normal prerequisites--Cygwin, Android SDK, and Android NDK. I'm using the latest Android stuff, which as of now is SDK tools r8 and NDK r5. Install Visual Studio and WinGDB as well. Run "android.bat" and make sure everything is up to date. You might also want to set the environment variables ANDROID_SDK and ANDROID_NDK to point to the root of the SDK and NDK, respectively.
Since the current WinGDB build was based on the previous SDK, you'll need to copy adb.exe back to where it used to be in the old SDK: copy %ANDROID_SDK%\platform-tools\adb.exe %ANDROID_SDK%\tools
Create an NDK project (or use one of the samples). Build the project using ndk-build. Then, run ndk-gdb --start. This will set up the files that gdb needs to see in order for WinGDB to attach. As soon as the gdb prompt appears, type quit to dismiss it. Elsewhere in these instructions I'll assume that the environment variable PROJECT_ROOT points to the root directory of your project (where AndroidManifest.xml lives).
Regedit time! You need to add an entry for "AndroidSDKPath" to WinGDB/Preferences/General. The exact location of this key will vary depending on your OS. I have 64-bit Windows7 so mine shows up under HKLM/SOFTWARE/Wow6432Node. YMMV. In any case, the value for AndroidSDKPath should be the full path to wherever you installed the Android SDK, like so:
In Visual Studio, open the WinGDB preferences dialog (choose WinGDB|Preferences... from the main menu). Set the following options:

Default debugger path: %ANDROID_NDK%\toolchains\arm-eabi-4.4.0\prebuilt\windows\bin\arm-eabi-gdb.exe
Use Cygwin mode for local sessions: yes
Cygwin installation (root) path: wherever you installed this (default is c:\cygwin)

Dismiss the preferences dialog.
From the Visual Studio main menu, select WinGDB|Attach to process...
Set the following options (you should only need to do this once):

Target type: Embedded Linux System (gdbserver)
Executable path: <path to your project>\obj\local\armeabi\app_process
Target login: <your device id>:adb (or emulator:adb if you're using the emulator)

Press the "Advanced Settings..." button. In the Advanced Settings dialog, set the following options:

Environment tab

Debugger path: %ANDROID_NDK%\toolchains\arm-eabi-4.4.0\prebuilt\windows\bin\arm-eabi-gdb.exe

Directories tab

Additional source directories: %ANDROID_NDK%\platforms\android-9\arch-arm
Shared library directories: %ANDROID_NDK%\platforms\android-9\arch-arm\usr\lib;%PROJECT_ROOT%\obj\local\armeabi

Target tab

Sysroot on host: /cygdrive/c/sdk/android-ndk-r5/platforms/android-9/arch-arm (notice that this has to be in Cygwin path format, so I can't use %ANDROID_NDK%. In this case my ndk is installed under c:\sdk\android-ndk-r5, which in Cygwin turns into /cygdrive/c/sdk/android-ndk-r5. You'll need to change that to whatever your local NDK path is in Cygwinese.)

Server tab

Launch server from sysroot: no
Path to gdbserver: /system/bin/gdbserver
Server port: an unused IP port (I used 1100)
Server port is forwarded: yes
Forwarded server port: another unused port (in my case, 11000).

Check "Save these settings as project defaults" unless you enjoy typing them in over and over again. (Yeah, I learned that the hard way.)
Dismiss the Advanced Settings dialog.
Back in the Attach to remote process dialog, check "Show processes from all users."
If you don't see any processes, click Refresh. You should see a long list of processes that are running on your device or emulator. User processes are listed by package name, so search for your package name and select it.
Click the Attach button and wait a bit
You'll see a message box warning you that there are no debug symbols in the executable. Dismiss it. It's true that the executable on the device doesn't have symbols that WinGDb can use; that's why you set up the shared libraries directory back in step 9.2.
Set a breakpoint and enjoy!

To debug again, just repeat from step 12. You'll only need to fiddle with the settings again if you switch to a new project, in which case you'll probably need to adjust the paths that are relative to your project root.

As far as I can tell, WinGDB works pretty well. I was able to set breakpoints, step through code, and use the watch window. There are limitations, of course; I was only able to attach to processes, not launch them, and the "autos" window didn't work, for instance. But just having the ability to debug in my favorite environment... that made my week.

Just a reminder, what I've described in this post is not a supported use of WinGDB 1.8! Don't go emailing WinGDB asking them to fix things they don't officially support; that's just not cool. (If you do have problems, though, leave a comment here--I can't promise to solve it, but I or some other reader may be able to.)

Good luck and let me know if you find this useful.

Adventures on Android NDK, Part II: Some useful tools

2011-01-09T18:17:00.000-08:00

Finally got some more time to play around with the NDK. And yes, I'm still stubbornly in Visual Studio. Here's what I'm using to make my life easier:

Visual Assist (of course!)
NShader for GLSL syntax highlighting
GnuWin32 so I can take advantage of tools like "which" and "sed" without having to go full Cygwin. (I never go full Cygwin if I can help it.)
WinGDB which almost, any day now, will support native Android debugging.... I'm counting the days!

I also upgraded to VS2010. Jury's still out on whether that's an improvement. They moved some stuff around and made it purple.

Quick Tip: Per-Project Environment Scripts

2011-01-06T19:28:00.000-08:00

Setting up build scripts for my Android exploration reminded me of something that I learned a while back on the Xbox 360 XDK: scripts that set up a project environment for you. I don't know if this is common knowledge generally--it's a pretty simple idea--but I don't see a lot of game developers doing it, so it seems like it's worth sharing.

The idea behind per-project environment scripts is to create a set of environment variables that describe all of the paths and settings and such for one project. Why would you want to do this? Well, for a simple project it's probably not important. For a project the size of the XDK, though, it was a lifesaver. The determining factor isn't really project size, though. It's the number of permutations you have to deal with. On the XDK, I usually had to deal with four or five completely different build environments:

My top-of-tree development branch
The current release candidate branch
The last released branch
A retail install of the XDK toolchain
A build of DirectX, where we shared some code from the XDK (and thus had to sanity-check every change we made to make sure we hadn't broken DX).
A (possibly modified) copy of the XDK that the game I was currently debugging had built against
All of the above, replicated on my laptop and on my machine at home.

To that set of permutations, add all of the different per-user and machine-specific settings you have in a large engineering team. It gets out of hand pretty quickly.

IDEs like Visual Studio have some features to deal with build and environment permutations, but in my experience they're fragile and hard to discover. In any case, the solution I'm going to talk about here works fine with Visual Studio, as long as you launch devenv.exe from a command line after running the per-project script (or, more likely, invoke devenv.exe from the script itself).

The solution that the XDK used, which I've rewritten and refined a bit to suit my own needs, was really elegant. At the root of each project tree is a shell script named something short and easy like "go.cmd." The script starts* with the most important magic scripting incantation you will ever need to know: the shell script command that remembers the path to the script itself.

SET PROJECT_ROOT=%~dp0

or, in the much less cryptic bash lingo:

export PROJECT_ROOT=`dirname $0`

Either command sets PROJECT_ROOT to the directory where the script lives. It doesn't matter where you called the script from, or where the current directory is--PROJECT_ROOT is dependent only on the location of the script. This is what makes the magic happen. It lets you have multiple copies of the same script on your local filesystem, each of which does a subtly different job. Each copy of the script sets up essentially the same environment, but it sets it up relative to its own project root.

After that first magic line, the script settles down to setting all of the variables that are dependent on PROJECT_ROOT, like INCLUDE and BINDIR and OUTDIR (or whatever you call the place where object files and executables land in your project). Pretty mundane stuff. This is also a pretty good place to set up values that are shared between all of the solutions in your project--you can use IDE features like Visual Studio solution variables for this, too, but this way seems simpler and easier to me. YMMV.

After the project setup there's one more bit of cleverness:

call %PROJECT_ROOT%\Users\%USERNAME%\setup_environment.cmd

The script calls a per-user override file. This is a script that lives in the user's directory under the project root, and calling it gives the user the chance to add their own specific environment variables or what have you. (Notice, by the way, that it has a longer and more descriptive name than "go.cmd." That's because nobody calls this directly, so its name can be as unwieldy as you want.)

Wait, back up a minute: Per-user directories? Yeah, here's the deal with that: Somewhere under the project root, in source control, you make a directory called "users." In that directory there's a directory for each user that works on the project. This might seem a little weird if you haven't worked on a project that's structured like this before, but there are two great reasons for doing this:

It gives everyone a place where they can store version-controlled junk that nobody else wants to see. For instance, I used it to store test data for features that weren't ready yet. Since everyone unmaps all of the users/ directory except for their own subdirectory, it gives you a way to keep things in the repository without pissing off all of your teammates.
It lets you add the per-user configuration scripts that we just talked about. And that lets each user set up their own, possibly temporary, environment: special #defines, filesystem mappings, command aliases, or what have you.

This idea isn't limited to just users, either. On a couple of projects I worked on I've also called a per-computer script, so that I could have different configurations on my office, laptop, and home machine. On one project I also built in a per-branch config, which seemed helpful at the time. A little crazy, yes, but so what? As long as you check to see whether a script exists before you call it, you can loop in any number of per-whatever config scripts without actually requiring everyone to write the things.

The last thing you do, if you're so inclined, is to launch whatever IDE you're working with. I don't, personally--I think every project should build from the command line, whether or not there's also an IDE involved. But that's just me.

And that's it. Very simple idea, you can set it up in an hour, but it's very powerful. It makes it trivial to manage multiple projects, and multiple versions of each project. And it eliminates the need for every new team member to go and set up their machine environment, and their Visual Studio include paths, and everything else that new team members usually need to set up. Instead, the only thing that a new team member needs to configure is their source control client, because everything else is checked in.**

*Well, after the normal stuff like "echo off" and "pushd" and such.

**Including, if you're really good at this stuff, your entire toolchain. All builds should be 100% reproducible from source.

Adventures on Android NDK, Part I

2011-01-03T14:32:00.000-08:00

Over Christmas I brought home my shiny new Nexus S and downloaded the Gingerbread SDK. I'm pretty stoked about Gingerbread, since a lot of the changes between Froyo and Gingerbread seem really helpful for game developers. For instance, you can now write a game entirely in C++, without writing anything in Java.* And there's OpenSL ES, so finally some real support for sound.

So, here's my story: former Xbox 360 developer, longtime DirectX user, Visual Studio pro, C++ devotee and C# aficianado... now trying my hand at Android development for the first time. I'll try and share what I've learned so far. This might be old hat to Android pros, but it's new to me, so maybe it'll come in handy for other developers with my kind of experience.

I'm going to start this in Visual Studio. Android really wants you to use Eclipse. I'm totally going to do that someday, because I feel like I should know how to use something other than Visual Studio to write code. But not today. So I set up a makefile project and hooked it up to some code I copied out of the Android NDKr5 NativeActivity sample.

Cygwin Woes
First unpleasant realization: Cygwin is absolutely required if you want to build an NDK project on Windows. At least, if you want to build an NDK project that works the way it should. Most of the NDK make system's complications seem to stem from its support for backwards compatibility with different versions of the Android OS. Hmmm. I'm only interested in Gingerbread; do I want to break down the makefile and reconstitute it as an MSBuild project? Well... yes, yes I do. But not today. So, I installed Cygwin.

Next bit of unpleasantness came when I hooked up the ndk-build script to my Visual Studio build action. It took forever to start up. Well, maybe five or six seconds. It felt like forever. Yeah, I've worked on projects that took more than an hour to build, but this is just a couple hundred lines of C code--what gives. It turns out that newer versions of Cygwin start bash-completion on login, and that's slow. Not sure what it's doing (scanning my hard drive for things to complete?), but I'm pretty sure I don't need it. So out it goes.

Visual Studio
I added the NDK include directories to my project file under "additional includes." I told Visual Studio to ignore the standard includes because I don't want Windows stuff cropping up in my Intellisense. And yes, Intellisense works! Nice job, Visual Assist. :-)

To build, I wrote a couple of scripts to launch ndk-build and ant. The first is a DOS batch script to set up the paths and call Cygwin; the second runs everything inside of a Cygwin bash shell.

The scripts are pretty simple, although there were a couple of tricky bits that I might post about later. Most of my trouble at was with the regular expression syntax in sed.** It's just slightly different than in perl, which is different from Visual Studio's system (which is, inexplicably, different from the .Net regexp library... and so on). Ah, regexp, you never cease to surprise me. Anyway, after figuring out that in sed you escape the parens that surround a group, rather than *not* escaping them the way you do in perl, I was on my way.

And... build works! The only thing that still sucks is that linker errors from the ndk toolchain don't contain the word "error," so Visual Studio doesn't list them in the error log. Well, that and the fact that I can't yet launch from Visual Studio. But that's a topic for another day.

* Although, now that there's a concurrent garbage collector, Java might not be that bad. But when it comes to managed languages, I'm a C# guy. So anything that gets me out of having to learn another language is good, at least for now.
** I usually use perl for this, for no really good reason other than I always have a Perl distro on my machine. Sed is actually a much better choice, because it operates on stdin as a stream, rather than buffering it up the way Perl does. For a task like munging compiler output, sed is totally the way to go.

I learned something today

2011-01-01T13:39:00.000-08:00

Remember the old seasons of South Park, back when Kenny was still getting killed every episode? Back then, a lot of the episodes had sort of an "afterschool special" ending to them. Kyle or Stan would turn to the camera and say, "You know, I learned something today..." And then he'd explain whatever the moral of the episode was. Usually something sick and twisted, but that's what you expect from South Park, right?

I feel like that line--"I learned something today"--pretty much sums up my whole career right now. My job (which I'll post more about later) involves a lot of learning, digesting, and regurgitating different technologies. I go down a lot of dead ends that don't turn into anything, but I still learn something from them. A lot of what I learn isn't groundbreaking, and sometimes it's not even useful, but if I write it down then maybe it'll help keep others from making the same dumb mistakes I did.

So, welcome to my blog. Hope you learn something. :-)