Wednesday, January 26, 2011

In which I access a file

Hola amigos! Sorry it's been so long since I rapped at ya, but shit's been going down, you know? Shit, in this case, being Google's GDC presentations--two dev days and a couple of sponsored sessions, twelve talks total. We finally have a schedule and with luck we might even be able to post it soon, woohoo!! Now if I could just get my slides done....

But anyway... I got a little more time to work on Android NDK programming last night. I was planning to play around with VBOs a bit, but I got a little sidetracked. See, I wanted to load up my shaders from a file, like a human being, not put them in constant strings like some damn dirty ape. And files are kind of weird on Android, because everything goes in the apk. Getting stuff into the apk is super easy--just toss it in the "assets" directory, easy as pie--but getting stuff out is another story.

OK, first I have to thank Jeebus that I'm using Gingerbread and NDKr5, because I don't actually need to play games in Java to get access to the apk. If you want to know how blessed you are to be living in 2011 instead of 2010, just check out the old instructions for apk access (h/t to Dan Galpin):

Use the asset manager code to get an AssetFileDescriptor:
  AssetFileDescriptor afd = 
Read the offset and length from the AssetFileDescriptor:
  long offset = afd.getStartOffset();
  long length = afd.getDeclaredLength();
Get the APK file name:
Open the APK file at the specified offset and read the specified length. [Code omitted from original, because... why is that, Dan? Because it's a pain to write? ;-)]

Yeah. Weird.

Anyway, with r5 things get a lot simpler, because there's now an AssetManager library that lets you use the asset manager directly. If you're using NativeActivity and the native app glue, you even get your application's asset manager passed down from Java for free as part of the application class. So you end up with code that looks a lot saner:

  AAsset* pAsset = AAssetManager_open(ctx->app->activity->assetManager, "readme.txt", AASSET_MODE_UNKNOWN);
  const int count = 80;
  char buf[count + 1] = {0};

  while (AAsset_read(pAsset, buf, count) > 0) {

That's better, right? It at least feels something like a normal set of filesystem calls. Rock on!!

Of course, it isn't a set of normal filesystem calls... which immediately makes me look ahead to the time when I'm going to want to access other filesystems (the SD card, for instance), and probably want to do so without having to plumb two different sets of file APIs all through my code. And while I'm at it, I probably want to abstract whether I'm even using a filesystem, or whether my files are in memory, or compressed, or streamed, or packed together in one huge bundle. I guess what I'm saying is, I've written enough data access code in my life to know that I'm going to want an abstraction layer at some point.

And so I pull out my trusty stream implementation... and run smack dab into the NDK build system. Ah, but that is definitely a post for another day. And probably at least a slide or two in my GDC talk.

    Tuesday, January 18, 2011

    Interview with TechWeb

    UBM Game Network newsletter did an interview with me as part of their pre-GDC coverage. You can read the interview here. This was the first interview I've done over email. I'm pretty pleased with how it turned out--a lot of what I wrote got cut, but I expected that, and what's left is pretty much my own words. 

    I'm really excited about what we've got planned for GDC this year. We'll be doing two full days of presentations on Monday and Tuesday, and we'll have a booth on the show floor. 

    I'm in charge of the presentation content, so I'm doing a lot of running around trying to get everyone to finish their slides on time. Of course people will be fixing things all the way up to the day of the show, but we have to at least pretend to have deadlines. :-) The content is shaping up pretty well. We've got some really in-depth technical talks on Android and Chrome both. Most of the really deep tech stuff is going to be presented by the engineers that actually implemented the systems we're talking about. Should be really good.

    If you want to hear more about GDC, or about game development at Google in general, you might want to follow @ilewis_goog on Twitter. I generally don't post more than a couple of times a week.

    Tuesday, January 11, 2011

    Android Debugging, Visual Studio style!

    UPDATE: More recent beta versions of wingdb make these instructions obsolete. Try it here.

    The high point of my week, courtesy of the good people at (click to enlarge)

    Full Android NDK debugging in Visual Studio. This is using version 1.8, which is kind of tricky to set up--mad props to the WinGDB support folks who walked me through it. Version 2.0 is coming soon, though, and according to WinGDB, they will have full support for Android NDK.

    For the curious, here's how I got this working.  If you want to try this, do so at your own risk and please don't go bugging the WinGDB folks about it; they're busy enough working on version 2.0 and they don't need anything slowing them down! For this example, I'm using the project I described in an earlier post.
    1. First, install all of the normal prerequisites--Cygwin, Android SDK, and Android NDK. I'm using the latest Android stuff, which as of now is SDK tools r8 and NDK r5. Install Visual Studio and WinGDB as well. Run "android.bat" and make sure everything is up to date. You might also want to set the environment variables ANDROID_SDK and ANDROID_NDK to point to the root of the SDK and NDK, respectively.
    2. Since the current WinGDB build was based on the previous SDK, you'll need to copy adb.exe back to where it used to be in the old SDK: copy %ANDROID_SDK%\platform-tools\adb.exe %ANDROID_SDK%\tools
    3. Create an NDK project (or use one of the samples). Build the project using ndk-build. Then, run ndk-gdb --start. This will set up the files that gdb needs to see in order for WinGDB to attach. As soon as the gdb prompt appears, type quit to dismiss it. Elsewhere in these instructions I'll assume that the environment variable PROJECT_ROOT points to the root directory of your project (where AndroidManifest.xml lives).
    4. Regedit time! You need to add an entry for "AndroidSDKPath" to WinGDB/Preferences/General. The exact location of this key will vary depending on your OS. I have 64-bit Windows7 so mine shows up under HKLM/SOFTWARE/Wow6432Node. YMMV. In any case, the value for AndroidSDKPath should be the full path to wherever you installed the Android SDK, like so:
    5. In Visual Studio, open the WinGDB preferences dialog (choose WinGDB|Preferences... from the main menu). Set the following options:
      • Default debugger path: %ANDROID_NDK%\toolchains\arm-eabi-4.4.0\prebuilt\windows\bin\arm-eabi-gdb.exe
      • Use Cygwin mode for local sessions: yes
      • Cygwin installation (root) path: wherever you installed this (default is c:\cygwin)
    6. Dismiss the preferences dialog. 
    7. From the Visual Studio main menu, select WinGDB|Attach to process...
    8. Set the following options (you should only need to do this once):
      • Target type: Embedded Linux System (gdbserver)
      • Executable path: <path to your project>\obj\local\armeabi\app_process
      • Target login: <your device id>:adb (or emulator:adb if you're using the emulator)
    9. Press the "Advanced Settings..." button. In the Advanced Settings dialog, set the following options:
      1. Environment tab
        • Debugger path: %ANDROID_NDK%\toolchains\arm-eabi-4.4.0\prebuilt\windows\bin\arm-eabi-gdb.exe
      2. Directories tab
        • Additional source directories: %ANDROID_NDK%\platforms\android-9\arch-arm
        • Shared library directories: %ANDROID_NDK%\platforms\android-9\arch-arm\usr\lib;%PROJECT_ROOT%\obj\local\armeabi
      3. Target tab
        • Sysroot on host: /cygdrive/c/sdk/android-ndk-r5/platforms/android-9/arch-arm (notice that this has to be in Cygwin path format, so I can't use %ANDROID_NDK%. In this case my ndk is installed under c:\sdk\android-ndk-r5, which in Cygwin turns into /cygdrive/c/sdk/android-ndk-r5. You'll need to change that to whatever your local NDK path is in Cygwinese.)
      4. Server tab
        • Launch server from sysroot: no
        • Path to gdbserver: /system/bin/gdbserver
        • Server port: an unused IP port (I used 1100)
        • Server port is forwarded: yes
        • Forwarded server port: another unused port (in my case, 11000).
    10. Check "Save these settings as project defaults" unless you enjoy typing them in over and over again. (Yeah, I learned that the hard way.)
    11. Dismiss the Advanced Settings dialog.
    12. Back in the Attach to remote process dialog, check "Show processes from all users."
    13. If you don't see any processes, click Refresh. You should see a long list of processes that are running on your device or emulator. User processes are listed by package name, so search for your package name and select it.

    14. Click the Attach button and wait a bit
    15. You'll see a message box warning you that there are no debug symbols in the executable. Dismiss it. It's true that the executable on the device doesn't have symbols that WinGDb can use; that's why you set up the shared libraries directory back in step 9.2.
    16. Set a breakpoint and enjoy!
    To debug again, just repeat from step 12. You'll only need to fiddle with the settings again if you switch to a new project, in which case you'll probably need to adjust the paths that are relative to your project root.

    As far as I can tell, WinGDB works pretty well. I was able to set breakpoints, step through code, and use the watch window. There are limitations, of course; I was only able to attach to processes, not launch them, and the "autos" window didn't work, for instance. But just having the ability to debug in my favorite environment... that made my week.

    Just a reminder, what I've described in this post is not a supported use of WinGDB 1.8! Don't go emailing WinGDB asking them to fix things they don't officially support; that's just not cool. (If you do have problems, though, leave a comment here--I can't promise to solve it, but I or some other reader may be able to.)

    Good luck and let me know if you find this useful.

    Sunday, January 9, 2011

    Adventures on Android NDK, Part II: Some useful tools

    Finally got some more time to play around with the NDK. And yes, I'm still stubbornly in Visual Studio. Here's what I'm using to make my life easier:

    • Visual Assist (of course!)
    • NShader for GLSL syntax highlighting
    • GnuWin32 so I can take advantage of tools like "which" and "sed" without having to go full Cygwin. (I never go full Cygwin if I can help it.)
    • WinGDB which almost, any day now, will support native Android debugging.... I'm counting the days!
    I also upgraded to VS2010. Jury's still out on whether that's an improvement. They moved some stuff around and made it purple. 

    Thursday, January 6, 2011

    Quick Tip: Per-Project Environment Scripts

    Setting up build scripts for my Android exploration reminded me of something that I learned a while back on the Xbox 360 XDK: scripts that set up a project environment for you. I don't know if this is common knowledge generally--it's a pretty simple idea--but I don't see a lot of game developers doing it, so it seems like it's worth sharing.

    The idea behind per-project environment scripts is to create a set of environment variables that describe all of the paths and settings and such for one project. Why would you want to do this? Well, for a simple project it's probably not important. For a project the size of the XDK, though, it was a lifesaver. The determining factor isn't really project size, though. It's the number of permutations you have to deal with. On the XDK, I usually had to deal with four or five completely different build environments:
    1. My top-of-tree development branch
    2. The current release candidate branch
    3. The last released branch
    4. A retail install of the XDK toolchain
    5. A build of DirectX, where we shared some code from the XDK (and thus had to sanity-check every change we made to make sure we hadn't broken DX).
    6. A (possibly modified) copy of the XDK that the game I was currently debugging had built against
    7. All of the above, replicated on my laptop and on my machine at home.
    To that set of permutations, add all of the different per-user and machine-specific settings you have in a large engineering team. It gets out of hand pretty quickly. 

    IDEs like Visual Studio have some features to deal with build and environment permutations, but in my experience they're fragile and hard to discover. In any case, the solution I'm going to talk about here works fine with Visual Studio, as long as you launch devenv.exe from a command line after running the per-project script (or, more likely, invoke devenv.exe from the script itself).

    The solution that the XDK used, which I've rewritten and refined a bit to suit my own needs, was really elegant. At the root of each project tree is a shell script named something short and easy like "go.cmd." The script starts* with the most important magic scripting incantation you will ever need to know: the shell script command that remembers the path to the script itself.


    or, in the much less cryptic bash lingo:

    export PROJECT_ROOT=`dirname $0`

    Either command sets PROJECT_ROOT to the directory where the script lives. It doesn't matter where you called the script from, or where the current directory is--PROJECT_ROOT is dependent only on the location of the script. This is what makes the magic happen. It lets you have multiple copies of the same script on your local filesystem, each of which does a subtly different job. Each copy of the script sets up essentially the same environment, but it sets it up relative to its own project root.

    After that first magic line, the script settles down to setting all of the variables that are dependent on PROJECT_ROOT, like INCLUDE and BINDIR and OUTDIR (or whatever you call the place where object files and executables land in your project). Pretty mundane stuff. This is also a pretty good place to set up values that are shared between all of the solutions in your project--you can use IDE features like Visual Studio solution variables for this, too, but this way seems simpler and easier to me. YMMV.

    After the project setup there's one more bit of cleverness: 

    call %PROJECT_ROOT%\Users\%USERNAME%\setup_environment.cmd

    The script calls a per-user override file. This is a script that lives in the user's directory under the project root, and calling it gives the user the chance to add their own specific environment variables or what have you. (Notice, by the way, that it has a longer and more descriptive name than "go.cmd." That's because nobody calls this directly, so its name can be as unwieldy as you want.)

    Wait, back up a minute: Per-user directories? Yeah, here's the deal with that: Somewhere under the project root, in source control, you make a directory called "users." In that directory there's a directory for each user that works on the project. This might seem a little weird if you haven't worked on a project that's structured like this before, but there are two great reasons for doing this:
    1. It gives everyone a place where they can store version-controlled junk that nobody else wants to see. For instance, I used it to store test data for features that weren't ready yet. Since everyone unmaps all of the users/ directory except for their own subdirectory, it gives you a way to keep things in the repository without pissing off all of your teammates.
    2. It lets you add the per-user configuration scripts that we just talked about. And that lets each user set up their own, possibly temporary, environment: special #defines, filesystem mappings, command aliases, or what have you.
    This idea isn't limited to just users, either. On a couple of projects I worked on I've also called a per-computer script, so that I could have different configurations on my office, laptop, and home machine. On one project I also built in a per-branch config, which seemed helpful at the time. A little crazy, yes, but so what? As long as you check to see whether a script exists before you call it, you can loop in any number of per-whatever config scripts without actually requiring everyone to write the things.

    The last thing you do, if you're so inclined, is to launch whatever IDE you're working with. I don't, personally--I think every project should build from the command line, whether or not there's also an IDE involved. But that's just me.

    And that's it. Very simple idea, you can set it up in an hour, but it's very powerful. It makes it trivial to manage multiple projects, and multiple versions of each project. And it eliminates the need for every new team member to go and set up their machine environment, and their Visual Studio include paths, and everything else that new team members usually need to set up. Instead, the only thing that a new team member needs to configure is their source control client, because everything else is checked in.**

    *Well, after the normal stuff like "echo off" and "pushd" and such.
    **Including, if you're really good at this stuff, your entire toolchain. All builds should be 100% reproducible from source. 

    Monday, January 3, 2011

    Adventures on Android NDK, Part I

    Over Christmas I brought home my shiny new Nexus S and downloaded the Gingerbread SDK. I'm pretty stoked about Gingerbread, since a lot of the changes between Froyo and Gingerbread seem really helpful for game developers. For instance, you can now write a game entirely in C++, without writing anything in Java.* And there's OpenSL ES, so finally some real support for sound.

    So, here's my story: former Xbox 360 developer, longtime DirectX user, Visual Studio pro, C++ devotee and C# aficianado... now trying my hand at Android development for the first time. I'll try and share what I've learned so far. This might be old hat to Android pros, but it's new to me, so maybe it'll come in handy for other developers with my kind of experience.

    I'm going to start this in Visual Studio. Android really wants you to use Eclipse. I'm totally going to do that someday, because I feel like I should know how to use something other than Visual Studio to write code. But not today. So I set up a makefile project and hooked it up to some code I copied out of the Android NDKr5 NativeActivity sample.

    Cygwin Woes
    First unpleasant realization: Cygwin is absolutely required if you want to build an NDK project on Windows. At least, if you want to build an NDK project that works the way it should. Most of the NDK make system's complications seem to stem from its support for backwards compatibility with different versions of the Android OS. Hmmm. I'm only interested in Gingerbread; do I want to break down the makefile and reconstitute it as an MSBuild project? Well... yes, yes I do. But not today. So, I installed Cygwin.

    Next bit of unpleasantness came when I hooked up the ndk-build script to my Visual Studio build action. It took forever to start up. Well, maybe five or six seconds. It felt like forever. Yeah, I've worked on projects that took more than an hour to build, but this is just a couple hundred lines of C code--what gives. It turns out that newer versions of Cygwin start bash-completion on login, and that's slow. Not sure what it's doing (scanning my hard drive for things to complete?), but I'm pretty sure I don't need it. So out it goes.

    Visual Studio
    I added the NDK include directories to my project file under "additional includes." I told Visual Studio to ignore the standard includes because I don't want Windows stuff cropping up in my Intellisense. And yes, Intellisense works! Nice job, Visual Assist. :-)

    To build, I wrote a couple of scripts to launch ndk-build and ant. The first is a DOS batch script to set up the paths and call Cygwin; the second runs everything inside of a Cygwin bash shell. 

    The scripts are pretty simple, although there were a couple of tricky bits that I might post about later. Most of my trouble at was with the regular expression syntax in sed.** It's just slightly different than in perl, which is different from Visual Studio's system (which is, inexplicably, different from the .Net regexp library... and so on). Ah, regexp, you never cease to surprise me. Anyway, after figuring out that in sed you escape the parens that surround a group, rather than *not* escaping them the way you do in perl, I was on my way.

    And... build works! The only thing that still sucks is that linker errors from the ndk toolchain don't contain the word "error," so Visual Studio doesn't list them in the error log. Well, that and the fact that I can't yet launch from Visual Studio. But that's a topic for another day.

    * Although, now that there's a concurrent garbage collector, Java might not be that bad. But when it comes to managed languages, I'm a C# guy. So anything that gets me out of having to learn another language is good, at least for now.
    ** I usually use perl for this, for no really good reason other than I always have a Perl distro on my machine. Sed is actually a much better choice, because it operates on stdin as a stream, rather than buffering it up the way Perl does. For a task like munging compiler output, sed is totally the way to go.

    Saturday, January 1, 2011

    I learned something today

    Remember the old seasons of South Park, back when Kenny was still getting killed every episode? Back then, a lot of the episodes had sort of an "afterschool special" ending to them. Kyle or Stan would turn to the camera and say, "You know, I learned something today..." And then he'd explain whatever the moral of the episode was. Usually something sick and twisted, but that's what you expect from South Park, right?

    I feel like that line--"I learned something today"--pretty much sums up my whole career right now. My job (which I'll post more about later) involves a lot of learning, digesting, and regurgitating different technologies. I go down a lot of dead ends that don't turn into anything, but I still learn something from them. A lot of what I learn isn't groundbreaking, and sometimes it's not  even useful, but if I write it down then maybe it'll help keep others from making the same dumb mistakes I did.

    So, welcome to my blog. Hope you learn something. :-)