Silverlight 3, A Basic Multi-Touch Tutorial

One of the most exciting development in multitouch this year is the introduction of multi-touch support in Silverlight 3.

Silverlight wasn't the first platform to bring multi-touch to the web. Apple already has a multitouch Java script API built into the Safari browser on the iPhone as reported in one of my February posts, "JavaScript Multi-touch API for Safari iPhone Browser".

What makes Silverlight 3 so great is that you can use it across browsers on the desktop which will be really valuable when Microsoft releases Windows 7 later this year.

Oddly, Adobe Flash is way behind on multi-touch which is startling to say the least. I don't think this will undo Flash in favor of Silverlight, but certaintly makes Silverlight more attractive to me. Adobe, where are you?

For a basic demonstration of how to program multi-touch applications in Silverlight 3 read this article by Tim Heuer.

Apple's 10 inch Tablet Will Ship in Q1 2010

AppleInsider, which originally broke the story in 2007 about Apple researching the design of a successor to Newton, has hung its hat on a prediction that Apple will release a 10 inch tablet in Q1 of 2010.

Just about everyone covering this story considers Appleinsider to be an extremely reliable source. In addition this aligns with Piper Jaffray's Gene Munster prediction that Apple would release a tablet but not until sometime in 2010. I'm going to go out on a wide, thick, and sturdy limb an agree that Apple will release a 10 inch tablet in Q1 of 2010.

I think Apple will announce this fall its intention to ship the tablet early next year. They will release some details about the device, perhaps a cool demo, and may even provide some direction to developers. I suspect the two development platforms for the Apple tablet will be, at least initially, the iPhone SDK with some tablet specific features and the Safari browser with its built multi-touch JavaScript support (see "JavaScript Multi-Touch API for Safari iPhone Browser").

As I outlined in my April 09' post, "What Industry Will Apple Try To Dominate Next? Video", the tablet is a multimedia play focused on video games, movie and TV video, and web browsing. The Apple tablet is not intended for productivity applications like word processing although you will be able to dock it on a stand to which you can connect a USB mouse and keyboard. The tablet will be capable of running most of the more than 50 thousand applications in the AppStore (see "Video Games, iPhone and Apple Tablet"), but developers will be encouraged to develop new applications specifically for the device.

Like the iPhone you'll be able to download music and, more importantly movie and TV videos, directly to the device from iTunes via Wi-Fi or third-generation mobile data network. The introduction of direct video downloads in iPhone OS 3.0 is further evidence, in my opinion, that Apple is gearing up to take on the TV and Movie industries with this device. Watching videos on an iPhone works well but downloading and watching TV and Movies on a 10 inch tablet is a game changer.

Download and watch any movie or TV show on iTunes directly to the tablet. Play any of the 10 thousand video games already available for the iPhone, use most of the 40 thousand other iPhone applications, or just surf the web. That's what the tablet is good for and its going to change the way we interact with video (games and movies) as well as the Web.

So, as multi-touch developers how should we prepare for the introduction of the Apple Tablet? I recommend getting up to speed on Objective-C and the iPhone SDK. This is going to be a large gadget not a small computer. The Tablet will sport an operating system that has more in common with the iPod Touch and iPhone than the Macs (although they share a common foundation). Also, start thinking about what industries will benefit the most from this device. Creating applications for the general consumer is all well and good, but the real money for developers will be creating applications for Health, Medical, and other industries that have been slowly adopting hand held devices for the past 10 years.

SpaceClaim 3D Modeling Video

One of the really exciting applications of multi-touch is 3D modeling and CAD. The video below from SpaceClaim, a CAD software company, shows the awesome potential of 3D modeling and CAD when combined with multitouch.

The truth is - as I've said before - 3D modeling with multi-touch will need to be paired with a large drafting style form factor and stylus input to be really useful.

SolidSmack has a great interview with the co-founder of SpaceClaim about the multitouch 3D tool. Apparently it will be available in August and will work with any multi-touch Windows 7 computer. It was nice to see that SpaceClaim's co-founder echos my own thoughts about the need for stylus input and drafting style layout.

Multitouch Spheres: What are they good for?

A year ago Endgadget ran a story about a Microsoft multi-touch sphere prototype which got a lot of press (here is the project page). I think it's a pretty cool device but I have no idea what you would use it for other than as a visual model of the earth with various overlays.

There have been a couple other multi-touch sphere projects including one from the Human Media Lab and more reciently a very large multi-touch sphere created by Seeper (see video below).

While the visuals are interesting I still don't see the practical applications for such a device outside of showing different mappings of the Earth, Moon or some other celestial body. For example Lynn Marentette has post showing a video of Global Imagination's The Magic Planet which is not multi-touch but is a large sphere used as a projection surface. That's cool and it would be even better if you could spin the planet with your fingers but what else can you do with a sphere as a projection and touch surface?

In Progress... Interactive MultiTouch Sphere. from seeper on Vimeo.

Augmented ID by TAT

The Amazing Tribe (TAT) has developed a really cool augmented reality application that allows you to tag yourself with social identifiers (e.g. facebook, youtube, twitter, etc.) and make them available to people through their camera phone.

I think the demo is done for the Android G1 but I'm not sure. Regardless its a very cool idea and would be very handy at any conference or in other situations. Nice work guys! Can we see this on the iPhone please!

Tactical Command using Microsoft Surface

One of the best applications of Microsoft Surface is tactical command: controlling and monitoring the activities of ground troupes.

This was illustrated during the Superbowl when Microsoft Surface was used to help secure the game (see this post) and is demonstrated again in two video demos (see below) produced by ESRI.

I really like the police dispatch demo - its very well designed in my opinion.

Synaptics' ClearPad 3000: Up to 10 fingers and an 8" diaganal Screen

Synaptics has announced that it will be manufacturing smallish multi-touch screen, ClearPad 3000, which can accurately track up to 10 simultaneous finger contacts (accuracy of 1mm). See video below.

As far as I know the iPhone can already handle at least 5 and I suspect more but who can fit more than five fingers on the iPhone screen and even if you could why would you want to?

With slightly larger touch screens, support for multiple fingers becomes more important. Synaptics ClearPad 3000 will support up to a 8 inch diagonal screen which is tablet and eBook territory - a market that is going to really take off over the course of the next year. With these larger form factors supporting four or five fingers could be really useful - support for 10 fingers is a bit excessive at tablet size but its nice to know the capability is there.

In a small form factor (e.g. mobile phone) most gestures are preformed with one or two fingers. In a form factor such as a tablet which might range from 6" to 17" diagonal you'll want to support at least four contacts and probably as many as 8.

With form factors intended for one or two people, which is the case with small devices and tablets, more is not always better in terms of how many fingers are supported. There is a question of physical space - once you get past certain number of fingers you can't see the screen clearly.

A rule of thumb that I use is what I call the "half-diagonal". The maximum number of fingers that a multi-touch application should support is half the display's diagonal size (round up) regardless of the aspect ratio. So, for example, the diagonal of Microsoft Surface is 30" which means applications on Surface should never require more than 15 simultaneous contacts (it supports up to 52). The diagonal of the iPhone is 3.5" so you should limit yourself to designing applications that use 2 or less fingers on that size device. The 8" ClearPad 3000 can support applications that use up to 4 simulations contacts (it supports 10). This is not a law but its a design rule-of-thumb I use when considering the design needs of multitouch devices. You really don't want to exceed the "half-diagonal" in terms of interactions because so many fingers deminish the user experience for that screen size.

Flash Mobile to Support Multi-touch and Accelerometer

Adobe produces two different Flash players, one for the desktop (its likely you've experienced that one browsing the web) and one for mobile. The mobile platform is a lighter weight version and doesn't have as many features but as mobile phones become a primary platform for surfing the web mobile Flash is becoming more and more important.

The desktop version of the Flash player is an excellent platform for developing large screen mulit-touch applications and is used in combination with multi-touch open source frameworks like tBeta and Touchlib. I've blogged about the importance of the Flash desktop version supporting multitouch before and I'm still waiting to find out when the desktop version of Flash will include an Adobe supported API for multi-touch.

In the mean time Kevin Lynch, CEO of Adobe, has announced that multi-touch as well as accelerometer support will be released in a beta version of Flash Mobile by the end of 2009. Thanks to Ryan Stewart for the heads up!

Support for multi-touch and accelerometer will do a lot to put other mobile operating system vendors on better footing to complete with Apple's iPhone. The iPhone has never supported Flash and according to Jobs it never will - that might change but for now you cannot view Flash content when browsing the web with your iPhone. That's a major drawback for the iPhone as there is a lot of Flash content out there - especially video content.

Job's reasoning for not supporting Flash is pretty obvious to me. Right now Apple has unilateral control over the development of resident iPhone applications which must be approved by Apple and deployed through the AppStore in order to be distributed. While you can develop some nice iPhone web applications without having to go through the Apple AppStore, the native SDK for the iPhone is much more powerful.

If Apple supported Flash on the iPhone it would be much easier to develop robust applications for the iPhone accessible over the web without the need of the Apple approval process and AppStore. Job's doesn't want that as it could substantially reduce control over iPhone development and Apple's revenues (they get 30% of the revenue of all iPhone applications available from AppStore). So for now, at least, Apple doesn't support Flash on the iPhone which is something that other smart phones will be able to support once multi-touch support for Flash mobile is in final release.

The iPhone Gold Rush

An acquaintance of mine, for whom I have enormous respect, spent the Summer of 2008 researching the iPhone application market. He had been involved in a couple successful start-ups in the past and was interested in the potential of the iPhone as a application platform and foundation for a new company.

We never got into details but in the end this acquaintance of mine decided not to do a start-up company around iPhone applications. The shear number of iPhone applications already available and the low price point made it difficult - at best - to gain customer awareness and make any real money. Instead of producing iPhone applications, he decided to focus on working as a contract developer for other companies investing in iPhone applications. In other words, he has become a freelance iPhone developer.

If you know what your doing and have a good track record you can make some very good money contracting as an iPhone developer as long as you do two things: You charge by the hour (no fixed bids please) and you do not guarantee that the application will pass approval in the AppStore approval process. In other words stick to providing the code and let others take the risks.

With the success of the iPhone and applications built using the SDK there has been a kind of "Gold Rush" to the iPhone application market place. The fact that there are over 50 thousand applications in the iTunes AppStore is a evidence of that. By Comparison, the California Gold Rush of 1849-1855 attracted about 300 thousand people a number that is probably about the same as the total number of people who have tested the waters in The iPhone Gold Rush since its start in 2007.

While some people did make a lot of money panning for gold - especially early on - most people barely broke even or returned home poorer than when they left. The people who did seem to proposer, for the most part, were those who sold services and equipment to the gold prospectors. Whether it was selling gold pans, lodging, whiskey, or housewares, there was money to be made helping the thousands of gold prospectors live to pan another day. This is the stuff of legend and the reality was, as wikipedia put its, more complex, but the California Gold Rush is great metaphor for the iPhone application market.

You'll make more money helping others prospect for iPhone gold then you will prospecting yourself. In other words, let others come up with grand schemes of riches from iPhone applications; you can stick to the coding for money and avoid disappointment and possibly financial ruin that will, in the end, be part of the legacy of the iPhone market place.

Of course just because you choose to code for money doesn't mean you'll get hired or even make money. Just as the California Gold Rush is littered with stories of merchants who traveled to California in the hopes of riches and ended up broke, so to are there stories of developers who set out to establish a business writing iPhone applications for others but end up with no clients and no income. An Interesting story by Peter Wayner of his adventures trying to deploy a simple application to the iTunes AppStore is echoed throughout the Internet. I've even posted on the subject before and have very mixed feelings about the iPhone marketplace. It's far more open than the mobile market place of the past, but its not exactly a free market place.

If, after reading this, you are still interested in freelancing as an iPhone application developer you might want to try iPhoneFreelancer.Net a site dedicated to connecting iPhone developers with those who want to develop iPhone applications. I haven't bid on any projects myself - I'm pretty busy right now - but I do get regular emails announcing new iPhone development opportunities and its fun to see what people are dreaming of developing. I can't vouch for the site but it seems as good as any place to look for work developing iPhone applications.

Augmented Reality: Combining Gloves and Glasses

A week ago I wrote a blog entitled "Glove Controllers and NUI" where I took a first (for me anyway) look at the use of data gloves. I concluded that if data gloves were in the future of NUI its days were already numbered because of developments like Project Natal. Project Natal can track your movements using a video camera rather than having to don a glove.

I kept thinking about that and realized that while Natal works great in a controlled environment such as your living room or in front of a public kiosk, it does not work everywhere. You have to have a video system watching your body movements and mapping that to some type of controls. What do you do when you are just walking aimlessly around - like when you are a tourist? For that you need a device you can carry with you.

There has been some concept illustrations created for augmented reality and even a couple of pretty cool re-world applications based on hand held devices. I can see this working really well and I have more to say about that in another post. The question is, can you use these devices to interact with the information around you or simply observe it?

If you combine the concepts of augmented reality glasses with controller gloves you can not only see information, you can manipulate it too.

Imagine you are looking at a street in Paris with your augmented reality glasses. You might see streets labeled and an arrow pointing to the Louvre. Using your data glove you could drill down on information you see through your glasses and choose other visual options.

While walking around wearing a pair of AR Glasses and waving your gloved hand in the air would look pretty odd today, it could become commonplace. After all, walking around talking to yourself was considered a sign of insanity until the invention of the bluetooth headset.

The applications that could be built around the AR Gloves and Glasses are pretty cool especially in industries such as surveying, utilities repair, laboratory, and other technician type jobs. Toss out the manuals and put on the glasses and glove and you have a computer-augmented-human doing work much faster and with more data at their fingertips - literally.

In an article from geeks.com.uk Nokia is filing a patent (see PDF) for exactly this combination: Glove controller and augmented reality glasses.



I would be very interested in hearing your opinion. Does the augmented reality glasses and controller glove have a future?

Video Games, iPhone and Apple Tablet

The rumors about an Apple tablet computer are starting to fly again. As was the case last time I think the press has got it all wrong. An Apple tablet computer is, as I said before, a video play not a phone, music, netbook, or web browser play (although it will probably do most of those things too).

Silicone Alley Insider published an article called "A Year Later Apple's iPhone AppStore [is the] Hottest Game Platform on Earth" which talks about the AppStore celebrating its first birthday. According to the article there are about 10,000 game applications on AppStore. To give that number some context: that's ten times as many games as have ever been published for the X-Box and Nintendo Entertainment System combined. And Apple accomplished that in exactly 1 year.

There is no doubt in my mind that after conquering the music business and mobile phone business, Apple's next target is the video game business and not just with an iPhone. Imagine the kinds of games that could be done on a 10" tablet. If the tablet supports AppStore applications it will immediately have a huge ecosystem.

I suspect that is what Apple intends to do; create another gaming platform. But not only a gaming platform, Apple will also try to dominate the movie/TV video business. I go into this business model in more detail in the post,"What Industry Will Apple Try To Dominate Next? Video." , which I made in April of this year.

Demand Evolution's "The Kit" a 50" multi-touch kit for $2,000

CruchGear reported on DemandEvolution a large multi-touch screen company that has two products, Gecko a complete 30" multi-touch enclosure for $5,500.00, and The Kit, a 50" multi-touch you-assemble-it-kit for $2,000.00.

The Kit includes software, a 50" LLP multi-touch screen, and inferred camera on a portable swivil stand. The Kit comes ready to assemble but you have to provide your own projector and computer to run the software. The software itself is based on the free and open TouchLib, Community Core Vision, reactTIVision, or PyMT touch processing software (take your pick). This gives you the option of developing multi-touch applications in Java, Flash, Python and other languages.

The cost of adding a computer can be pretty low (~ $500.00), but the cost of a projector can be significant. Really good projectors will run you up to a couple of thousand dollars. DemandEvolution doesn't recommend a specific projector. So unless you have an extra machine and projector assume that full assembly of The Kit will cost an additional $1,500.00 to $2,000.00 making the full investement $3,500.00 to $4,000.00. Not too bad compared to alternatives.

What I like about The Kit is that it's very portable. You can disassemble the 50" screen from the stand and then pack them up separately along with a computer and projector. Carry the computer and projector on the plane and pack stand and the 50" screen (use a mile of bubble wrap) for check in. The Kit will be great for road-shows on multi-touch - I'm very tempted to buy one myself!

F@stCompany Picks of Killer Apps for Microsoft Surface

F@stCompany has published a list of about 20 "Killer Apps for Microsoft Surface" which is interesting but highly subjective.

Personally, I judge multi-touch applications on three criteria Utility, Design, Aesthetics. If it fails any one of these tests (which are also pretty subjective) than its not a very good application. Most multi-touch applications will do well in one or two areas but few succeed in terms of all three.

Utility is the quality of usefulness of the application. Even a leisure application can be useful, but its use must also be sustainable. In other words, if people get board with it after five minutes its has low utility. Another aspect of utility is whether or not the application accomplishes the task its designed for. For example, if the Surface application is a logistics application does it really offer any advantage over using a desktop computer?

Design refers to the layout and interaction model. Specifically, does the application provide an interaction model that encourages use and discovery? Of the three criteria this is perhaps the least subjective. Either you can easily interact with a Surface application or you cannot.

Aesthetics is the beauty of the application. A Surface application should be beautiful as well as useful and well designed. It should draw people in and get them excited and its should maintain a positive experience over time.

Now judging Surface applications that you have not actually interacted with, but have only seen from a video, is a bit like judging the quality of a movie from its trailer. That said, even movie trailers tell us something about the movie in terms of script, direction, acting, cinematography, etc.

In a future post I'll show examples of Surface applications that excel in Utility, Design and Aesthetics.

Glove Controllers and NUI

At E3 2009 a company called Iron Will Technologies announced a video game glove controller that will be released in Q4 of 2009. The Peregrine glove controller enables users to interact with 3D games via 30 predefined gestures. Basically, there are pressure sensors along each of the fingers and when you make contact with a sensor and your thumb its like pushing a button on a normal game controller.

You might remember how glove controllers were used in Minority Report - it was an application that still leaves me scratching my head - but seems to get people excited. What I'm wondering is: Do glove controllers have a future in NUI (outside of gaming, movies, and niche markets)?

A little bit of research turned up a previous attempt to introduce glove controllers for gaming back in 1989. The Power Glove produced by Mattel for Nintendo's game system apparently didn't work very well and was a complete flop.

Today a company called INITION produces and sells Glove Controllers with development kits - prices range from $1,495.00 to $7,175.00 - prices that are out of range for most causal gamers. You can still find the much less expensive, but no longer produced, product by INITION called Essential Reality P5 Data Glove on eBay for around $20.00 - $100.00. It's been discontinued - another failed mass market glove controller.

The glove controller by Iron Will Technologies has not had its price officially announced yet, but I read somewhere that it will probably fall into the $129.00 range. That will make more affordable but won't assure its success. Another option for NUI developers, which was just announced, is the AcceleGlove which is about $500.00 with the Java SDK. There is also ShapeHand by Mesureand Inc. - they don't list prices which always makes me think the product is going to be really expensive. I'm sure there are many other options if you are really interested.

The thing that strikes me, and maybe I'm just missing something, is how unlikely these data gloves or wired-gloves or glove-controllers (take your pick) are to be used by the general public for anything other than a few video games. Today these gloves are primary used for 3D rendering, controlling robotic hands, or learning sign language. Not exactly mainstream applications.

That's not to say that hand gestures, even very subtle ones like waving a finger back and forth, don't have a place in the future of NUI. They do! But, I think that future will be realized through some other mechanism than donning wired gloves. The kind of gesture tracking seen with Project Natal will probably evolve to track the fine-grained moments of fingers, but until that time perhaps glove controllers will have a place in the NUI landscape.

Avoiding iPhone Rejection

I was speaking to a hiring manager the other day (not for my self but just because he's a friend) and he told me that half the people he interviews tell him that they want to do iPhone development. What is really funny about this is that the company he's hiring for has absolutely no need for an iPhone application - something that should be obvious to anyone applying for the job.

Anyway, if you are dreaming about creating iPhone applications for a living you should check out the list of things you should NOT do when submitting iPhone applications to the AppStore. The list of "do not"s are covered in a two part article (Part 1, Part 2), "Avoiding iPhone App Rejection From Apple", by Bloggle With Harish.

In some ways I'm glad to see Apple filtering the applications that can be downloaded from AppStore. If it was an open system without review I would be a lot less likely to download arbitrary applications. The "Big Brother" aspect of Apple's review system does bother me to some degree - but not that much.

The iPhone and AppStore is Apple's platform and if their semi-closed garden is an issue than the market will punish them for it. Besides you should have seen the how difficult it was to develop, get accepted, and distribute mobile applications before Apple introduced the iPhone - now that was a closed garden!

G-Panel, A New Multitouch Table

Polygon Magic, a video game development company from Japan, has just introduced its own multi-touch table, G-Panel, that is very similar to Microsoft Surface. (thanks to Silconera.com for the heads-up).

Like Microsoft Surface, one of its form factors is a table - although its higher than Surface. It uses projection with inferred detection that appears to support at least 10 fingers.

According to the web site there is a strong focus on "dotted cards" which are pretty much the same as Surface byte or identity tags only they use an invisible inferred ink so you can't see the dot pattern and the tags are larger (thus the emphasis on "cards"). You can buy the inferred ink, a "stealth" cartridge, and print your own cards.

There isn't a lot of information on technical specifications of the table or its price. The market is clearly video arcades as well as hospitality market, so the price point is probably pretty high. Here are some video demonstrations.

UPDATE: July 13, 2009

I sent an email to Polygon Magic to find out how much their multi-touch tables cost and I got the following reply.

---

Mr. Richard Monson-Haefel


Thanks you very much for your inquiry regarding Grid Screen Panel.

Currently, G-panel is not in mass-production and all would be order made this time, where as the price would be approximately 21,000 USD which is equivalent to 2,000,000 Japanese yen(according to today’s Yahoo currency converter)

Therefore, the price will vary as per size, shape, usage as well as the procurement time of materials, so that, in case of purchasing please let us know, we would provide detail Quotation.

Feel free to contact with us in case of further queries.

With best regards,

Nilufa Yasmin
Production Section
Polygon Magic, Inc

Laser Light Plane (LLP) Multitouch Table

There are a number of ways to build multi-touch surfaces some of which are more appropriate for very large surfaces than others. For example, the technique used with an iPhone (Projected Capacitance) works great with screens that are smaller (10" is about has big as you can get right now) but has not been successful implemented on a larger surface such as a table or wall. NTrig's DuoSense, which is used in the HP TouchSmart XT2 and Dell Latitude TX2, works well with screens that are about 4 - 12 inches, but not in large surface installation I've seen. Most touch screen phones and smaller devices today use Resistive technology which, again, is not used much in larger installations.

The famous TED 2006 demo of a large multi-touch table given by Jeff Han used a technique called Frustrated Total Internal Reflection (FTIR) which is a favorite technique used for large multi-touch surfaces. Microsoft Surface uses a technique called Diffused Illumination (DI) and there is another, similar technique, called Diffused Surface Illumination (DSI). All of these techniques used in larger multi-touch surfaces use inferred light projected behind a semi-transparent surface (i.e. DI) or directly into the side of an acrylic surface (i.e. FTIR & DSI).

A new technique called Laser Light Plane (LLP) is quickly gaining traction in the DYI community and with professional installations like the really cool multitouch wall developed by Schematic for Cannes. The LLP technique appears to have been first introduced by Microsoft in a project called LaserTouch and subsequently by NUIGroup community member "AlexP" in May of 2008.

There is a really good DYI blog entry by Georg Kaindl on how to build a LLP table with four lasers mounted at the corners of the screen. According to Georg, LLP is easier and less expensive to build than FTIR, DI, or DSI and is more accurate. His raw video footage (see below) showing the accuracy of tracking in full daylight (a condition that hampers tracking in other inferred technique) seems to support this assertion.





One of the advantages of LLP over other techniques is that you don't have to apply any pressure to the screen to register a contact, that makes it behave more like an iPhone and less like the kind of single-touch devices you encounter at the ATM or grocery store checkout. The laser light can be adjusted to project about 1mm above the surface so the lightest of contacts will work.

One of the disadvantages of LLP is that it doesn't seem to support tagging of objects. You've probably seen video where someone puts an object on the table (mobile phone, bottle of wine, etc.) and information about that object pops up around it on the screen. That's a very cool application of what is called Tangible User Interfaces, but it doesn't seem to work with LLP because the inferred light is not reflecting off the bottom of the objects but from their sides.

Except for the loss of tagged objects I really like LLP and I think its going to grow a lot more popular with DYI and professional installations. It appears to be much more accurate and a lot less difficult and expensive to implement. That said, I morn the loss of tagged objects as this is a really nice feature that is leveraged by Microsoft Surface and the Reactable table.