From Gian-Carlo Rota's Ten Lessons I Wish I Had Been Taught:
Richard Feynman was fond of giving the following advice on how to be a genius. You have to keep a dozen of your favorite problems constantly present in your mind, although by and large they will lay in a dormant state. Every time you hear or read a new trick or a new result, test it against each of your twelve problems to see whether it helps. Every once in a while there will be a hit, and people will say: "How did he do it? He must be a genius!"I was thinking about database abstractions and how often people get in pissing matches over whether it's even possible or desirable to abstract over persistence. In teaching abstraction, we advocate information hiding as a way of building scalable software, but then there's always some detractor who can't wait to snipe at you just for trying: "no, you can't hide that detail, that's simply too prevalent an issue." Sometimes history proves them wrong, sometimes not.
It's a constant source of heartburn to computer scientists that, for example, the space of viable algorithms (say, the ones that run in polynomial time) is one tiny speck in the universe of algorithms we wish we could use. And lots of folks from Dijkstra to AOP advocates have tried vainly to abstract out efficiency as a separate concern. While we're at it, let's imagine some extreme violations of reality -- let's say I want to create a computer program that does my research for me, grows wings, and cooks me dinner in response to picking up on my brain wave patterns that indicate my desire for it to do so, but never mind the details of how it works.
I suppose this means that ultimately someone has to be responsible for the details. Maybe that's the key: you can only set up a reasonable interface to hide an implementation if the implementation can truly deliver on its contract with the client. Which means that deciding on the interface to begin with relies on a balance between hiding enough from the client on the one hand but being plausible for the provider to achieve on the other. Feynman suggests one way to go about solving this: set ridiculously high goals for yourself, rely on being a genius, and be on the constant lookout for possible solutions.
But I was also struck by Joel Webber's description of the technology behind Google Maps. In particular, when I read about their use of PNG alpha-channels to implement drop-shadows for the pushpins that get overlaid on a map, it occurred to me that this is an absolutely unnecessary feature, and yet an incredibly nice touch. Clearly there wasn't some suit at Google demanding, "dammit, drop shadows are a deal breaker. Get me drop shadows!" Rather, I imagine someone thought, hey, if we stick a semi-transparent PNG on top of an image we can get a nice shadow effect. I bet that would make us look cooler than Apple!
Maybe good hackers work from the bottom up, starting with their existing bag of tricks and figuring out what they can solve with them. Or maybe they work like Feynman, always on the lookout for solutions to the Big Problems. Chances are, good hackers work with what they've got, be it a tool set or a problem set.