Wednesday, January 18, 2012

Darkness on the Internet

Today is Stop SOPA day, with sites like Wikipedia going dark in protest. Google isn't going dark, but it's got a doodle about it, so you know it's got to be at least as important as, say, Doraemon.

Please take a moment to sign a petition or contact your congressperson about SOPA and PIPA. This is a complicated issue and there are two sides to every argument, but as an internet user I feel SOPA is a horrible idea.

To understand why SOPA is a bad idea, consider this analogy. Let's say I call you up on your T-Mobile phone, and leave you a voicemail. (I'll use voicemail as an example because it's stored on T-Mobile's servers.) As part of my voicemail message, I play you a bit off my new Lady Gaga CD. (I have to use Lady Gaga as an example, because I work for Google. Long story.) Let's assume for the sake of argument that this is illegal. Who is at fault?

Am I at fault for playing the song? To the extent that playing it was illegal, I'd say yes. Absolutely, I'm at fault. Punish me.

Are you at fault for listening? You know what, I'll bite. Let's punish you, too--you should have hung up as soon as you realized there was unauthorized Gaga on the line.

Is T-Mobile at fault for storing my voicemeil? Think for a minute about what that means. How exactly was T-Mobile supposed to prevent this infringement? There's only one way: eavesdropping on every single voicemail that is ever left by anyone that uses their system.

Quite aside from the privacy concerns, this is a logistical nightmare. They'd have to have an army of people listening in on every conversation that anyone ever made, ready to flip a kill switch the moment they heard "Poker Face." And that's an easy example. What if the infringement involved someone less famous, like Elvis Hitler or Toad the Wet Sprocket? What if I am Elvis Hitler, and I'm playing you a track from my long-awaited followup to "Disgraceland?" There's going to be a lot of judgment calls, and a lot of eavesdropping. 

Now let's talk about punishment. In this analogy, SOPA would not only require T-Mobile to do the eavesdropping and make the judgment calls. It would also allow any entertainment company to shut off T-Mobile completely just on the allegation that infringement occurred. Until it proved itself innocent, T-Mobile would be unable to connect to the U.S. telephone system or use U.S. airwaves. (In this analogy I'm granting the government miraculous powers over the electromagnetic spectrum--which, if you think about it, is not much more far-fetched than the miraculous powers it would need to have in order to actually enforce SOPA the way Congress apparently thinks it can.)

Think about that for a minute: an entire company put on hold on the basis of an allegation by some media multinational. Millions of employees out of work. Tens of millions of customers who can no longer communicate. All in the name of stopping infringment. 

Replace "T-Mobile" with "YouTube," "Facebook," or "Google" and that's SOPA.

Listen, I don't want to come across as a big leftie here, but here in America we don't shut companies down even when we have solid proof that they are killing people. Now we're talking about shutting down Internet businesses not because they did something wrong, and not even because we have proof of wrongdoing, but because some entertainment company alleged that that one of that company's customers did something that might be wrong.

And don't even get me started on the actual enforcement clauses, which would empower our government to set up its own Great Firewall. I don't want to sound like Glenn Beck, but folks, this is how liberty ends. Ask a Chinese friend what the Great Firewall is for. Unless they're really dialed into politics, chances are they'll tell you it's to keep out porn and terrorists. We in the West think we'd never stifle political dissent, but we'll do it in a heartbeat if instead of calling it "dissent" you call it "terrorism" or "indecency" or "infringement."

Whatever you think about copyright law, SOPA is not the right way to enforce it. Even if the entertainment industry is right (newsflash: they're not) and a lack of SOPA would cause the entertainment industry to wither up and die, would we really censor the entire Internet in order to keep those Jerry Bruckheimer movies coming every summer? Ben Franklin derided those who would trade liberty for safety. What would he think of those who ask us to trade essential liberties for continued access to mindless entertainment?

So please, contact your congressperson. Wikipedia has a search tool to help you. Don't copy a form letter, write something short and succint and from the heart. Even if it's just a few words, write something. If you're an avid Internet user, SOPA is literally an attack on your way of life. Help put a stop to it.

Monday, November 28, 2011

My first "official" Android Blog post...

...went live last week here! It's about my own personal pet peeve, games that play audio when you're not playing them. One of my achievements this year has been to improve the technical standards for featuring on the Android Market; at this point, it's almost impossible to get a new title featured if it has this bug. Which means, of course, that part of my job is to send off emails to promising game developers, asking if they can fix this sort of issue so that we can feature their game. I'm hoping that publishing this sort of blog post will help.

Stay tuned for more; in return for being allowed to title my post "Part I of a series," I had to commit to at least three more episodes. :-)

Monday, September 19, 2011

Bug in LockFreePipe

Dmitry Vyukov, one of my fellow engineers at Google, pointed out a bug in the implementation of LockFreePipe that you can find here and there on the internet, for example here. That reminded me that Bruce Dawson (who coauthored the original with me) had discovered this bug a while ago and tried to explain it to me, but at the time we didn't have the code in front of us and I didn't completely grok what he was trying to tell me.

Anyway, this particular bug is simple: in Read(), after the call to memcpy(), there should be a barrier before the read pointer is updated. Otherwise the read pointer update can become visible before the memcpy is finished, giving the writer an opportunity to overwrite the memory before it's read.

        unsigned long cbTailBytes = min( bytesLeft, c_cbBufferSize - actualReadOffset );
#ifdef _XBOX_VER
        XMemCpy( pbDest, m_pbBuffer + actualReadOffset, cbTailBytes );
        memcpy( pbDest, m_pbBuffer + actualReadOffset, cbTailBytes );
        bytesLeft -= cbTailBytes;

        if( bytesLeft )
#ifdef _XBOX_VER
            XMemCpy( pbDest + cbTailBytes, m_pbBuffer, bytesLeft );
            memcpy( pbDest + cbTailBytes, m_pbBuffer, bytesLeft );

// OOPS: no barrier here. These operations could be reordered
// by the compiler or by the CPU. Should have lwsync or _ReadWriteBarrier here.
        readOffset += cbDest;
        m_readOffset = readOffset;

Dmitry also pointed out that a store-release is meaningless without a load-acquire, so there should also be a barrier in Write() after m_readOffset is read:

   bool __forceinline          Write( const void* pvSrc, unsigned long cbSrc )
        // Reading the read offset here has the same caveats as reading
        // the write offset had in the Read() function above. 
        DWORD readOffset = m_readOffset;
// OOPS: need lwsync or _ReadWriteBarrier here if we want it to work
        DWORD writeOffset = m_writeOffset;

Needless to say, I'm humiliated :-) and curious to know why this worked so well in the past. My first guess was that MSVC did not reorder because the read pointer was marked volatile. Of course, that raises the question of whether any of our barriers were necessary, if volatile is so magic. Well, yes, they were; MSVC volatile semantics at the time prevented compiler reordering, but didn't insert any barriers at the CPU level. The write could still have become visible any time before the read was issued. So why didn't it?

I'd say we got lucky. Just because the CPU can reorder stores before reads, doesn't mean that it does. There's a good reason for the PowerPC to delay stores--it's something that can usually be deferred until later, because in general nobody's really waiting for a store to complete. (If the data is getting reused, it's probably in one of the PPC's copious number of registers. And if it's not... see "load-hit-store.") There's seldom a good reason for a read to get delayed, because the read is very frequently a dependency of the next instruction in the queue. I'm speaking very generally and in a fashion that involves waving my hands quite a lot, but the point is that the window for failure on this is pretty small. Not, however, small enough to be nonexistent, which is what you need if you're doing lock-free work. Bummer.

The sad thing is that I really like this class. It's small and lightweight, doesn't do any allocation, and is basically perfect for things like passing data in and out of audio routines where blocking would cause glitches. I originally wrote it to dump sample data out of custom DSP routines on the Xbox 360, and it worked awesomely well for that purpose. But after talking to Dmitry, I realize that it really only worked  as well as it did because of quirks in the 360's CPU architecture. It's not a generic piece of code.

In any case, if you're using this class, please fix it by inserting memory barriers as described in the code snippet above. And stay tuned--there may be other bugs that I haven't found yet.

Tuesday, September 6, 2011

Checking In...

I haven't been good about posting to this blog lately, probably because I originally started it to talk about the Android NDK, and I haven't been doing much of that lately. I've been jumping around to different parts of Android, trying to learn more about the entire system. But I've made what Tim Bray says is a fundamental blogging mistake, which is to only write about what you think your audience wants to hear. Oops. I guess I should really start writing about what I've been doing, whether or not it's NDK-related (or even Android related).

What I've mostly been working on, besides the inevitable "partner support" email queue, is a Google Tasks app. Yeah, I know, not groundbreaking. But all of the apps I've found so far do it wrong. They don't authenticate with my Google account token--instead, they ask me for my actual Google credentials, which I'm not going to give them. And they don't sync correctly. I want a Tasks app that's as solid as the Gmail app, and since I have the code to the Gmail app, I think I ought to be able to do it. I've got most of the account management stuff worked out, and I have access to the GTasks API (which is, by the way, kind of incomplete), and now I'm writing a ContentProvider so that I can hook it to a SyncAdapter and make background sync work. That's taken me down a path of inventing a lightweight ORM so that I can write a ContentProvider without having to do a whole lot of redundant keyboarding. (I almost said "typing," but that's an overloaded term in this context.)

So that's probably what I'll be writing about for the next little while: my quixotic attempt to do a simple Android app The Right Way. I want it to be a shining example in every possible way. I've already found a few places where that's going to be impossible, because no Right Way exists. But that's a subject for another post...

Monday, August 8, 2011

Ndk-gdb and service processes don't (currently) mix

Got a bug from one of our partners complaining that they had two almost identical Android projects, but one of them was debuggable and the other one wasn't. After checking all the usual culprits, we finally narrowed it down to a single line in the manifest of the undebuggable project (names have been changed, obviously):

<service android:name="com.somemiddleware.SomeUsefulService" android:process=":com.somemiddleware.useful.process" />

It was, of course, very tempting to blame the SomeMiddleware company and its purportedly Useful service, but it wasn't their fault at all. Instead, it turned out to be a very small error in one of the ndk's awk scripts. It turns out that the script that finds the pid of the process you're supposed to be debugging (extract-pid.awk) was, for various reasons, doing a substring match on the process name. This almost always works, because there's almost always only one process running under your package name at any one time. But it doesn't always work.

In this case we were bitten by a quirk of the Android service architecture. The manifest is allowed to specify a process name under which to run the service by using the android:process attribute. Normally this just specifies the exact name of the process that the service should run under. But there's a special case: if the process name so specified begins with a colon, then the service process is considered "private" and its name is concatenated with that of the main process. So instead of running as "com.somemiddleware.useful.process" the service runs as "com.mycompany.mygame:com.somemiddleware.useful.process."

See the problem? Now we have two processes that match our package id: our main game process that runs as "com.mycompany.mygame" and the middleware service that runs as "com.mycompany.mygame:com.somemiddleware.useful.process" -- and guess which one gets matched by the ndk-gdb awk script? 

The fix to extract-pid.awk is simple, and I've already submitted a patch, so hopefully that'll get accepted and shipped in the next NDK. Meanwhile, if you happen to run into this bug, you can either patch the script yourself, or you can just eliminate the colon from the beginning of your service process name while you're debugging.

If you're interested, here's a gnu diff version of the patch:

< # NOTE: For some reason, simply using $9 == PACKAGE does not work
< #       with this script, so use pattern matching instead.
< #
>     RS = "\r\n"
<     gsub("\\.","\\.",PACKAGE)
>     #gsub("\\.","\\.",PACKAGE)
< $9 ~ PACKAGE {
> $9 == PACKAGE {