Expanding Horizon's: Johnny Lee Sees The Wii's Potential
While Johnny Chung Lee’s name might not ring a bell, his work with the Wii’s Remote has been viewed millions of times on YouTube. Lee’s innovative projects use the input device in some mind-bogglingly clever ways, from using it to track users’ fingers, to creating inexpensive multipoint interactive whiteboards (which is much, much cooler than it sounds), and creating a head-tracking system for VR displays. If you’re not familiar with these videos, you need to watch them immediately before reading on.
We spoke with Lee about his projects—which started off as a procrastination tool from his PhD studies—and his thoughts on how his ideas could affect gaming in general. With the attention the Wii has received for its intuitive and interesting game controls, it’s great to see someone push hard and inspire others to see that the system lives up to its potential.
Game Informer: Can you talk a little about your background and what led to your work with the Wii?
Johnny Lee: I’m a PhD graduate student at Carnegie Mellon University studying human-computer interaction. I’m actually just finishing up my PhD. I’ve been doing a lot of work and am in a community that focuses on new ways to interact with computers, either by using innovative software or hardware technology. The Wii Remote stuff actually started off as sort of a procrastination project, because it’s not my primary thesis topic. I was basically playing with it to do something on the side. The Wii Remote has a lot of really cool devices in it, and it’s an easy platform to experiment with. So these are just projects that I thought would be fun to explore personally, and it turns out that a lot of other people find them interesting as well.
GI: Have you found that the Wii stuff has dovetailed into your PhD thesis work at all?
Lee: A little bit. Toward the end of my thesis work I was looking for high-performing tracking technology. It turns out that the camera in the Wii Remote is pretty good and is relatively easy to work with. So, for some of the demos late in my thesis work, I talked about how it could be done with something like the Wii Remote. My work uses a lot of infrared-light-sensing technology, so it’s also relevant to the Wii Remote—not quite in the same way, but it’s relevant.
GI: Did you gravitate toward the Wii because of the system itself, or was it simply a good way to get all of the components in a relatively inexpensive package?
Lee: Probably more of the latter. I do play video games and I like video games, but I was much more excited about the capabilities of the controller than I was the Wii game console itself. It’s amazingly impressive that Nintendo just allowed the ability to connect it to a computer over Bluetooth, which I think has been great for both the hobby community as well as the research community. It’s a pretty amazing piece of engineering, containing a high-speed infrared camera, accelerometer, plus it has Bluetooth communication, a vibration motor, speaker, buttons—and it does this all running off two AA batteries for a long time. If you had talked about this a few months before it was released, I probably would have said it could not have been done, given my exposure to research. But they did it, and I was massively impressed.
GI: Have you worked with other game tech in the past?
Lee: A little bit. There’s sort of the consumer game technology and then there’s the experimental game technology that you sometimes see at conferences like SIGGRAPH [Special Interest Group on GRAPHics and Interactive Techniques] and UIST [User Interface Software and Technology]—these are sort of the big either industry or academic research communities, and sometimes they have prototypes of things you could do in the future but aren’t necessarily mass market yet. There’s things like eye tracking and 3D trackers and face-recognition systems. All of the high-end ones are pretty difficult to work with or expensive. Things that are at a consumer level are typically normal game controllers, with joysticks and buttons, so there wasn’t anything too interesting there to play with. So the Wii Remote was the first consumer product that I thought that was really interesting to play with.
GI: So you never messed around with the NES Power Glove or anything?
Lee: No, not really.
GI: As a video-game player, what do you think of the gaming landscape now? Do you think there’s an acceptable level of innovation going on in the industry?
Lee: I definitely think that, at least among the traditional gaming platforms—maybe not on the Wii, but on the Xbox and the PlayStation—the way they’re going about providing a gaming experience is sort of an asymptotic experience, where they’re doing a very good job at it, but they’re not really evolving very far beyond that. To me, probably the most interesting stuff that’s going to happen to gaming in the future will be with new ways to control or interact with the system, either physically or using input devices. So actually I think games are one of the most interesting places to explore new computer-interaction techniques. I think the Wii is a particularly good step toward that, but I think there’s still a long way to go.
I think developers are really excited about trying to explore these new ideas, but some of the skill sets that I think are good at making joypad games are going to take a while to appreciate and flesh out the capabilities of new controllers. Even on the Wii, you can see that some of these games are essentially joypad games that are retrofitted to the Wii controller. It works, but it’s not necessarily the best use of the controller.
GI: Have you played around with the Sixaxis controller at all?
Lee: Yes, I played with it. The games that I played that do use the accelerometer in it are cute, and they’re pretty well done, but I think the physical aspect of the joypad controller as it is has a certain affordance to the way it wants to be held and used. It’s sort of limited to roll and tilt. I think the remote-control form factor of the Wii lends itself better to more freeform movement.
GI: One of the most interesting things in your videos is how you occasionally put the Wii Remote on a stand and actually manipulate the Wii’s sensor bar as an input device, which is something a lot of people wouldn’t think to do. Was that a matter of it being an obvious way of doing that for you, or did you have a classic “eureka!” moment?
Lee: I think it was a little bit of both. It took a while to sort of get over the concept of what the Wii Remote is pitched as, which is as a controller. Part of what I do in my work is I take a device and try to break it down into its subcomponents. If you just think about it having a high-resolution camera, there’s a lot of computer vision work that’s been done in the research environment, but it hasn’t made it into the consumer domain. The camera that the Wii provides is pretty impressive. Once I thought about just using the camera component and not as a controller, it opened up a whole bunch of ideas that people may have already done but that most consumers haven’t been exposed to. Once I decided to do that, there were probably 10-ish ideas that I could easily do that are all pretty cool.
GI: Are we going to see any of those ideas soon?
Lee: Hopefully. Soon might be hard, because I’m getting a lot of phone calls and I’m also in the middle of my job search, which means that I’m going to be doing interviews. Unfortunately, the next thing may not come out for a few months. We’ll see. This has stimulated a lot of interest among some really smart people who are technically minded, so once you make the leap of using it as a camera, then a lot of other ideas start flying out. Essentially, I’ve inspired a lot of really smart people to start working with the Wii Remote, so hopefully we’ll see a lot of cool projects by a lot of people.
GI: Have you talked with anyone at Nintendo about your work?
Lee: I haven’t officially talked with anyone at Nintendo. To some degree, I think it’s almost in their best interest not to officially endorse or condemn what I’m doing, simply because the game industry is very hype-oriented, so if they say anything about it, people will jump all over it and expectations might get set in the wrong direction. I know a bunch of developers who are working on Wii games are interested in exploring it seriously, so we might see some games, hopefully sooner than later.
GI: If you could get Nintendo to make one significant change to its hardware, what would you ask for?
Lee: That’s a hard question. There’s nothing that immediately jumps out to me, I think mostly because it’s an odd question to ask—it’s a very weird hypothetical, and I think the industry is already trying to catch up to the capability that it has built into it. Granted, it may not be the most powerful graphically, but from an interaction standpoint the accelerometer is being underused in games. Most people are using it for shake recognition. The pointer, the camera, has the capability to track up to four points, but most of the games are only using two points. Also, most of the games are only using one controller per person, but you could also use two controllers per person and get some more control out of that. So there’s a lot of stuff that they’re not doing that it could do. If anything, they already have more capabilities than they need at the moment.
GI: Do you think this is an area that you can see yourself exploring for a long period of time, or is it mostly a lark?
Lee: It kind of depends on how much people want me to continue doing stuff like this. My goal is essentially to have the work I do affect peoples’ lives in consumer products and have people play with it. If there’s a lot of interest in further developing these Wii concepts or working on similar concepts, then I might spend more time on it. There may not be a lot more headroom along the products you’ve already seen, because they’re concepts. It’s up to content developers to make experiences around using these techniques. That gets into skill sets that I don’t personally have. I’ll probably move into another interactive experience that hopefully is accessible to a lot of people.
GI: These next few questions are related to your head-tracking video. Do you think that kind of tracking would be especially difficult to implement in a game? It seems as though it might be taxing once applied to a 3D world.
Lee: From a pure technical standpoint, it’s trivial. My code uses Direct X and there are maybe only 100 lines of code to make that work. Assuming that the game consoles have the capability of doing that type of rendering—because it’s a slightly different type of rendering to make it look like it’s really 3D; you can move the camera around, but that’s not quite what I’m doing—if they’re willing to and they have the software code to do that and they’re willing to bundle some hardware, like IR glasses, then it’s trivial for them to make a game. Whether or not it’s perhaps taxing as a game player to use it for a long time, I’m not entirely sure yet. There seems to be a lot of enthusiasm around making games that get people moving around and getting involved in the experience, so it might be tiring but that might be a good aspect of playing the game.
It’s up to the developer also to determine how to apply this technique. In software, they can apply a scaler, and small movements translate to big movements in a game, so they can choose or leave it up to player preference. So it’s essentially head-motion sensitivity that you would set. From a technical standpoint, it would be easy to make a game, but it’s up to the developers to use it properly.
GI: Does the head tracking only work in first-person experiences, or would it be effective in a StarFox type game or in God of War-like gameplay?
Lee: I think it’s most compelling in a first-person point of view, but it’s still very useful for third person or other games—even in a side-scrolling platforming game. What it does is give you motion parallax, which is the technical term, which is essentially the way objects occlude with each other when you move your head. So it gives you a better sense of space, which any game that has space involved in it could probably benefit from a head-tracking system.
GI: Do you think 3-dimensional technology is the next step in entertainment? It seems like it’s something that people revisit every so often, but it hasn’t been used in any widely successful applications.
Lee: I think a lot of people are really interested in trying to bring realistic 3D experiences into the home, because it’s sort of the next step beyond television. I think where it’s been sort of a difficult or an uphill battle—or even a failure—in the past is because usually it requires specialized hardware that’s not in everyone’s home already. The nice thing about the Wii is that it’s already in 13-million homes, at least, so most of the hardware is there to do this.
I’ve heard from some technical developers that this is re-exciting the 3D-technology community, and even people who are working on 3D shutter glasses are wondering if they can make them for the Wii Remote or the Nintendo Wii console. That’s entirely speculation at the moment, but essentially it’s a chicken and the egg problem. If you can get the hardware in the homes to do this, then you can make content for it. Hopefully, I think it’ll come. It’ll be up to how well people can adopt the technology if it’s successful or not.
GI: Do you have any advice for people who want to learn how to do this kind of work on their own?
Lee: Well, if you want to do human-computer interaction stuff, like I’ve been doing, you need to know how to program first. If you’re in high school, take computer classes or participate in project activities outside of school, because applying the knowledge you have in a useful manner really helps you see how the world works and how technology works. I also personally have an electrical-engineering background, which is probably boring if you think about it, but it gives you the skill sets to really change everything about a computer. It allows you to not take the hardware or the software for granted. So if the hardware’s not right, you can change the hardware. And you can tell it what to do with software. And then you can make any experience you can imagine with those skills.
Maybe the answer you don’t want to hear is going to engineering school and doing really well in engineering, but most importantly it’s probably holding onto your imagination so you can execute your ideas.
ZWAME Jogos: Final Fantasy 7 Rebirth já se encontra disponível para a PlayStation 5. Podem ler a nossa análise no Portal Jogos.
