At the very least I hope to provide appropriate ambient music derived from the instrumentals of songs from Abney Park who has contracted with me to provide music for the book. It would also be good to provide ambient sound effects so, for example, a scene that takes place a busy street has street sounds in the background.
The idea first came to me last year when a MMOG I played, EVE Online, incorporated a more sophisticated sound track into the gaming experience. If you have the sound track playing in EVE it varies over time and the music gets more intense and changes if you entering battle, engaged in battle, or simply floating peacefully through space. It's really a cool effect.
My original thought, and one I considered submitting a patent application for, was to use eye tracking to determine where the person is reading and to have the music and ambient sound effects change seamlessly according the content. I gave up on that idea because of complexity. For Steampunk Holmes: Legacy of the Nautilus I plan to base the music on the page you are reading which is technically much simpler.
Yesterday, I discovered an app book developer called Backtrack which uses a sound track for their books, most notably an app/book version of Sherlock Holmes: Adventure of the Speckled Band and The Power of Six a modern young adult novel. Backtrack has a pretty cool approach. They have a very non-invasive carrot (arrow) that moves along the right side of the page at what they predict to be your speed of reading. As the carrot moves to different paragraphs and even different sentences the ambient music and sound effects change to match the content. I think you can train it to read at your own speed pretty easily - it seems to have slowed down when I kept double tapping the carrot back to the sentence I was still on as it was way ahead of me (evidently I'm slow reader). I really enjoyed the experience and I tip my hat to the folks at Booktrack for the engineering and quality of music and sound effects.
Today, I discovered that iOS 5 just might ship with facial recognition APIs in the SDK including eye tracking. If that is true, and it works well, I can see combining eye tracking with context sensitive sound track and ambient sound effects to improve the experience. That's a nice to have.
For now I'm sticking with page level sound track as that's is easier and my budget is not unlimited. I hope, however, to incorporate eye-tracked ambient sound and music in an update providing the APIs exist and I have the money to afford to implement it.
There are so many things I want to do with the app/book but I'm going to have to throttle my ambitions if I want to be able to afford produce the book. However, just because I can't do them now doesn't mean I can't talk about my ideas which is one the main reasons for this blog. I want to be totally open about what I'm thinking and producing.