Augmented Reality

I attended an interesting CommNexus event regarding Augmented Reality last evening. Two companies (WowWee and Qualcomm) presented their current/future products and shared insights about this new area that is taking shape at an amazingly fast pase, primarily due to the processing power in smartphones.

Here are some insights from the event:

WowWee talked about and demonstrated AR games where fixed, pre-defined physical objects (model airplanes or toy figurines) that when paired with the game running on the smartphone (Android and iOS) would present a new 3D world on the phone screen. For example, an airplane model in the view of the smartphone camera would show up as a virtual airplane on the smartphone screen allowing you to play a dogfight with other virtual airplanes but the background, instead of being a virtual world, would be the view from the camera. So, you could visualize flying the airplane around the walls and ceilings of your room. Quite an interesting demo.

Qualcomm, through their internal startup ventures is pushing the technology behind AR and working with companies trying to use this functionality to develop apps on smartphones (again Android and iOS) that enhance user engagement by including AR into the app. They are providing an SDK for AR called Vuforia that can possibly exploit the AR engine in their SnapDragon chipsets that are being used by a large number of smartphones these days. They mentioned three areas where this is picking up interest - gaming, advertising and instructional apps. The first one was similar to that from WowWee. The second one was quite interesting. Brands (for example, Heinz ketchup) can build an app that, when you bring the ketchup bottle in the view of the smartphone camera, you can see recipes overlaid on top of the bottle. Or another example they gave was apparel manufacturers. Scanning a watch ad and then placing the hand in front of the camera caused the watch to be overlaid on top of the hand on the smartphone display. This was pretty cool and I think has interesting applications where you can "try" out how things might look on you without really stepping outside your home. The third application, instructional apps was also quite novel. If, for example, you point the camera at a TV set, its interactive user manual would be overlaid on top of the camera view on the smartphone display, allowing you to view specific aspects of control/setup of the TV without actually opening a printed or online user manual.

I think this field is in its infancy but has huge potential going forward. One of the key questions that was discussed was if holding the smartphone in one hand was cumbersome and clunky and the possibility of moving to heads up displays like those in the Terminator. The panel seemed to indicate that the expectation from users was always a heads up display but the form factor, weight, connectivity etc. will finally decide if users will adopt it. If the form factor is somewhat similar to that of Google glasses, and performs similar to some of the demos and videos then it might get better adoption. There was also mention of some companies working on contact lenses with this technology built straight into the lens. Spooky? Remember the scene from Minority Report with Tom Cruise walking through the mall and the various stores targeting ads directly to his retina?

I like the idea of using AR to code in front of a huge virtual screen, somewhat similar to that scene in Minority Report with Colin Farrel/Tom Cruise. I use dual monitors at work but I still feel that I need more screen space! Ideally, I would like to have a wall of virtual screen space with windows that I could pan, tilt and zoom, using hand gestures to move windows around and a keyboard to type on the active window and my finder as a 3D mouse. The various windows could be the different browser, debugger, emulator etc. sessions allowing me to switch contexts easily without using precious screen real estate. It would be interesting to see if a proof-of-concept like this could be hacked using a Kinect and a projector?

100 photos in 30 days

After seeing Matt Cutts TED talk on trying something new for 30 days, I was motivated to take on a 30 day challenge. The fact that I took more than a couple of months to actually decide and execute on one is another story for another blog post! My initial goal was to click two photos every day, for 30 days. In order to make sure that I complete it without making it feel like a chore or an extra task in my already full daily schedule, I decided to take the pictures from my smartphone (Motorola DroidX) camera and use Google+ to share them instantly. Additionally, I chose to not go out of the way looking for scenic spots but rather take pictures from places that were part of my daily routine. This would serve twofold - reduce the possibility of goal abandonment and help build a photo-collage of places/things that I pass by every day but sometimes too busy to notice.

This turned out into a really fun and interesting exercise, much more than I had expected on the onset. On most days, I would think for a few minutes in the morning about what I would like to click that day. On other days, I would notice something interesting, take a quick detour and shoot impromptu. There were also those days when I was so busy that I would take pictures of some objects at home, to make sure that I did not miss the deadline for the day. Some of these pictures tuned out to be abstract enough to be interesting and introspect-able. Of course, there were some occasions when I took photos just to make sure I met the "quota" for the day, but I liked that aspect of the challenge too. It kept me from giving up before I reached the 30 days. What was inspiring was that midway into this journey, I decided to up the ante and turn this into a "100 photos in 30 day challenge."

Apart from just collecting 100 photos in my Google+ album, I learnt a lot of things and this is a blog post that talks about some observations and inferences. 

• 30 days is actually not too long or not too short. It is just about the right amount of time to make a change in any small way. It is not too long that the ADD kicks in to attract you to another shiny toy and not too short to feel underwhelmed at the end of the exercise. 


• I (finally) "get" Instagram. Coincidentally, it was in the midst of this 30 day challenge when the Instagram acquisition actually happened. My family and friends who were following me on Google+ were actually having a "connection" and "conversation" with me, even if we were in opposite sides of the globe. Even though I was not really uploading photos for this reason, they were hooked on to looking at the photos daily and +1ing the ones they liked, thereby showing interest and expressing anxiety waiting for the batch of photos for the next day. In fact my wife was disappointed on day 31 to not see any photo posts! A friend of mine "caught" me visiting San Francisco by spotting my photo of the Golden Gate bridge. I even played around with some filters from the "creative kit" to make some photos look richer or abstract.




• Stop and wait to smell flowers. Well, literally and figuratively. Some days I was so much stressed at work that it was so relaxing to just go outdoors and look at flowers and walk around the landscape and take photos. The nice thing about flowers is that they are the easiest to find and it is very refreshing and calming to look at pictures of landscapes. If you look close enough, you might find insects and some of them are photogenic enough to add to your collection.




• It is fun to look at everyday objects from a different angle - bookshelves, toy cars, crayons and even a bag of frozen vegetables can all be made to look interesting!




• Are point and shoot cameras on the verge of extinction? I think I was able to capture fairly good photographs. I got very good results in bright sunlight and reasonably good ones in less light conditions. Automatic panoramic stitch up of photos on smartphones is also quite good as you can see from one of the photos I clicked to get a 180 degree panoramic view from the sidewalk in front of Hard Rock Cafe on Pier 39 in San Francisco. The iPhone seems to have already become the most popular camera on Flickr. In fact, a short film shot entirely on the iPhone won the Berlin Film Award and now there are websites that track this category of short films.

• Looking at some of the photos and judging by some of the feedback that I got, makes me really wonder - if I can get good pictures like these using a smartphone in its "Auto" settings and capturing moments while not going really out of the way, wouldn't it be more fun and rewarding if I got out my DSLR and bought a better zoom lens/filters and got out on photo-breaks during lunch hour or mini photo-vacations?



• A blog post. A conversation starter. I've had a couple of interesting conversations about the photos that I was clicking, while I was taking the photos and even after uploading them. I know 10 years later, one day, I am going to look at this album and remember the fun time I had and have another conversation with somebody!


Lastly, like Matt said in the TED talk, the next time somebody asks me what I do for a living, I don't have to sound geeky or stereotypical and say "Programmer", I can sound artistic by saying I am a "Photographer."

If you haven't done a 30 day challenge, what are you waiting for? Find something small and simple and add it into your daily routine. Keep at it for 30 days, then introspect. It is a rewarding and a fun experience.

Yet Another Steve Jobs Biography Book Review

First of all, I am not a hero-worshipper. Never have been. Never will be. Second, I am not an Apple fanboy. Never have been. Not sure, if I can say, "Never will be!" But, over the years, having worked in technology, having used Apple products and having closely followed product launches (Apple product launches, GoogleIO keynotes and sessions, Kindle announcements, CES etc.), I gradually began to have special regard for Steve Jobs and the products that he championed, even though I hated some of the Apple approaches to dumb down user interface and technology. However, after reading the biography, I have a new found respect for Steve Jobs - his creative talents, his visionary mind, his attention to detail and also the unsung heros at Apple behind those amazing products that they make. I think I have a better understanding of how and why they did the things the "Steve Jobs way."


Being a techie, I choose to judge only things that have "computer science" associated to them and hence I have chosen to neither judge nor comment on any of the personal, emotional or inter-personal aspects (relationships with family, friends, monetary dealings with Wozniack and the others etc.) of Steve. I believe that you have to be one of the parties really involved with the situation to be able to make a judgement and even then, I think, without actually peeking inside ones head and putting all the situational elements (circumstances, past history, personal biases etc.) together, it is really hard to judge somebody, unless you are a psychotherapist. One can argue that everything about him and his accomplishments finally boils down to his personality but since my degree is in computers and not human psychology, I would like to only address things that I can understand and glean from the book.


Some of the thoughts here may not have been explicitly mentioned in the book but may be the way I inferred or interpreted the various scenarios. At times, to drive the point I have added my own 2 cents, since this is what I learnt from the book rather than merely regurgitating sections from the book.

1. Be persuasive.
2. Believe in yourself. Never give up. Be obsessive.
3. Work with and around people who share your vision. Without this, it will seem like sitting in a car with each of its wheels going in a direction that it feels like.
4. Build an "A team". "A team" members can only with "A team" members and working with "C team" members is a huge drag on productivity and employee morale. Picture the car engine of a high performance race car - every piece in it is for a reason and every piece is giving its 100% and was hand picked to produce that level of overall performance. Even as small a thing as a loose nut can bring down that race car.
5. Never do a sloppy job. Never. Not even if nobody is looking. Even if this is an internal tool, even if this is for personal use. Redo it how many ever times it takes to get it right. Visualize everything you do as a work of art that somebody is going to look at.
6. Make technology less scary and more human. Nobody needs a user manual to operate most functions of a refrigerator or a washing machine. Why should technology be any different?
7. If you build a classy product with the end user in mind they will buy it. Don't think about profits. Think only about the product and the end user. Profits will follow.
8. Everybody judges a book by its cover. Even though we know not to do it and we are taught in school not to do it, everybody does it. Admit it. You do it at the grocery store - you buy only products that are packaged in an appealing manner. In fact, grocery stores sell you stuff that you don't need, merely by placing it in an appealing manner. It is very essential to make a great impression. Apple products, starting from the Apple store, to the buying experience there, to the packaging of the product,  to the feeling when you hold the product, to the experience when you use the product all culminate to give you the "Apple Experience".
9. Iterate on your product. Do it multiple times. But do it internally. Don't make your end users your beta testers. Treat products like other consumer electronic devices. Imagine how it would be if your washing machine was released in beta and you would have to work through its water leaks, poor washing capabilities, keep taking upgrades hoping that the next release would give you all the features and finally start working perfectly as advertised? Using the washing machine should not entail having to read through a mass of documents or having to search the web for specific problems or having to ask questions on news groups on how to perform its basic operations.
10. Anybody can construct a building but not everybody can build the Taj Mahal. When you build something, make it grand. Make it a work of art. Make it something that people will appreciate and remember.
11. Focus on a few things. Have laser focus. Do a few things and do them well.
12. Rally your troops by being hands on. Unlike leaders of today, who sit in the luxury of their office and send orders to the poor soldiers who fight in the front, be like the kings of historic times - fight the battle, shoulder to shoulder. Motivate your team. Push them to show them that they can excel.
13. Don't build products like an assembly line - each group (chassis, hardware and software) building their piece in isolation and throwing it over the wall once they are done and the next team picking it up and repeating the same process. Rather than that, build it as one cohesive piece. Once built, it will work like that too. Again, analogy with the automobile. The entire car is envisioned as one piece - otherwise you might land up with a strange car having the chassis of a boat, the wheels of a tractor and a joystick instead of a steering.
14. Don't build things that merely replicate functionality that is already out there. Leapfrog your competition by building something orders of magnitude better.
15. Don't compromise on your key beliefs. Even if it will allow you to get an initial not-so-fully-baked version of your product out sooner. Not even if it will allow you to get a foot in the door first. Compromise is a slippery slope.

When I started writing this blog post, I chose not to comment on personality. However, it is hard for me to resist the opportunity. And since this is my blog, I am going to do it anyway. I think Steve was a genius, one of a kind, passionate, "artistic engineer", who strived to build technology that was more human and resembled art rather than merely the embodiment of product features from a marketing list that were slapped together on a piece of hardware. Most successful engineers (and artists) have made one or two brilliant products but he was able to make so many of them. He was a man way ahead of his times - a visionary. Of course, he was not perfect, especially when it came to inter-personal relationships and he may have burnt a lot of bridges by his abrasive personality. But, like I said in the beginning of the blog, I choose to only absorb the aspects that relate to his geeky part and possibly learn from the mistakes of his inter-personal relationships. I think his son, Reed Jobs, sums it up really well in the book:


"... was not a cold profit-seeking businessman but was motivated by a love of what he did and a pride in the products that he was making"


If you survived reading this post all the way up to this point, you are probably thinking that I am actually a hero-worshipper and an Apple fanboy! It is hard for me to convince you otherwise and all that I can tell you is that if you haven't read the book please do so. I always remember what Dr. Mike Bailey, former Professor of Computer Graphics at UCSD, once said that dramatically changed my thinking. Bill Gates was going to give a talk at UCSD and I was in my "Microsoft is Evil" frame of mind and did not want to attend the talk, which happened to be at the same time slot as the computer graphics class. Dr. Bailey, not only canceled his class but also encouraged us to attend the talk and told us that, weather we liked it or not, Bill Gates was certainly a man who had largely influenced the digital revolution and as computer scientists it was essential for us to have an open mind and listen to what he had to say. If you are curious, yes, I stood in line for more than an hour and listened to Bill Gates but I was really underwhelmed by the contents of his talk and his passion. But that is another story ...

Building a Tagged and Revisioned File system

This blog post consists of discussions with vinod regarding the file system in his blog post about os.next and my mindless mumblings during the 12 hour drive to and from Phoenix for the Christmas 2011 holidays!

Version control has been around for a really long time. Tagging, however, is a fairly new concept, that became popular largely due to Flickr and Gmail. Tagging, instead of creating folders in the native file system for the desktop, seems like an extremely useful feature, especially, for documents, photos, music. Usage of tags on the web has demonstrated that it is convenient to tag/label content to categorize them under multiple buckets. Similarly, for content typically found on a desktop PC, we can apply the same concept of tagging/labeling instead of creating folders. This would allow retrieval of files with tags without having to look for them in specific folders. This is again an attribute that we use extensively on Flickr and Gmail and other cloud content storage/retrieval applications and bringing this feature to the desktop might make storage/retrieval of content as easy as that on the web. This approach of tagging in file system vs. using folders, surprisingly is not a new idea and Googling around revealed a couple of posts: tagfs-tag-semantics-for-hierarchical-file-systems, a-tagged-filesystem.

Version control, on the other hand has been used primarily for managing source code on projects and is usually an action that is manually carried out by the user and typically not used for content on the desktop. However, if files on the desktop automatically get revisioned, then it would make this a background activity in the file system and the user would not have to explicitly checkin/checkout their files. This would be an extremely useful in cases where we clicked on "Save" instead of "SaveAs"  and lost the original copy of a document that we were editing. Again, auto-revisioning is not a new idea. It has been around for a long time but surprisingly hasn't become a default feature in all desktop file systems. Mac OS X has good internal backup and versioning support with Time Machine and Lion introduced the auto-save the feature.

Now, combining the flexibility and convenience of tagging with the benefits of auto-revisioning (rather than explicitly placing a file under some version control) would be an interesting combination for the desktop environment, we think. Here are some design and implementation ideas to build a file system that supports both tagging and auto-revisioning:

Goals:

- tagging instead of folders

- auto revisioning of files and tags

- don't break make and other build tools like autoconf, automake, configure, ant etc.

- accomodate duplicate file names (for example, digital cameras auto-number photos and the file system could the potentially have duplicate file names referring to totally different content)

Design Basics:

- prototype implementation on linux using fuse

- when a file or tag is created or revisioned it is associated with an 32 (or x) bit id.

- by default a file without any tags is equivalent to being in the "root" folder.

- tags and files are revisioned objects and by default accessing either will give their latest versions.

- tags and metadata stored as json/bson (hopefully makes the metadata future proof)

- tags can be ordered or unordered

- ordered tags can be used in cases where a fixed folder hierarchy is necessary (source code, for example) and are associated with parent and children tags (similar to the  Nested Labels feature in GMail)

- tags are also be versioned. suffix @@ followed by a version number for a specific tag version. this allows the user to look at what files were tagged on a specific version, similar to what files were in a folder at a specific version

- file are auto-versioned whenever they get modified. suffix @@ followed by a version number for a specific version

- use forward diffs for revisions - so when a file gets modified to version 2, go and update version 0 in the backing store to contain the diff between v0 and v1. similarly when v3 gets created, update v1 as diff between v2 and v1. this will allow the latest version to be viewed without having to traverse the diff hierarchy and old versions can be pruned from the filesystem, if necessary, without having to go and fix the  diff hierarchy. this approach adds an extra overhead of having to update old versions when new ones are created but it can be setup as a background low priority activity as its primary purpose is to reduce the space used by the backing store. this idea of forward diffs can be applied to the scheme of branches also where there could be multiple children to a particular revision. when a branch is created, v0 on that branch is a full copy of the file on the branch point. this allows the files on the branch to get revisioned using the same approach as that on the main branch.

Tag:

- all metadata stored as json/bson

- root.tag file always present and contains the info for the "root" tag

- tag files contain following info:

* name of the tag

* parent tag id (empty string if directly under root. allows "orphaned" files to be tagged to a parent tag)

* ordered child tag ids (allows a directory structure that may be necessary for some source code stored in the file system)

* file ids that are tagged with this tag (basic means of lookup to file_id.dat and file_id.meta)

Files:

- stored in a separate common folder as file_id.dat

- metadata for a file (timestamp etc and associated tags) stored as file_id.meta in json format 

Example:
root.tag
{
  "parent": "",
  "tags": ["1234abcd", "5678abce"],
  "ordered_tags": ""
  "files": ["2345defc", "9876beef"]
}

1234abcd.tag
{
  "parent": "root",
  "tags": "",
  "ordered_tags": ""
  "files": ["2574dddd", "aeee5621"]
}

2574dddd.meta
{
  "name": "somefile.txt",
  "tags": "1234abcd",
  "last_modified": "01-01-2012 1300 PST"
  ...
  ...
  ...
}

2574dddd.dat
<contents of somefile.txt>

Versioning with Tags:

When versioning is added to the above scheme of Tag and Files, the following adjustments need to be made:

- tag files split into two. first one is versioned_tag_id.tag. this contains the information as before but is a file that gets stored as diffs. so, when a new file gets added to this tag, this versioned tag file moves to a newer version with an updated entry for "files". similar thing happens when a file gets removed from this tag.  tag_id.tag is the higher level metadata that contains the mapping between the tag_id and all its corresponding versioned_tag_id.tag files. it is basically a json file containing a list of version numbers and corresponding versioned_tag_id files. this also allows branches and "release" labels to be added to tags.

- nothing really changes with regard to files. they continue to be tracked as file_id.dat except that they are stored as diffs described in the basics section. the specific version of a file in a branch is listed in the entry of the tag and lookups occur based on that.

- to speed up lookup of the latest version of a file special tags (such as CURRENT or LATEST) could be used in the tag file.

Same example, now with versioning:
root.tag
{
  "parent": "",
  "tags": ["1234abcd", "5678abce"],
  "ordered_tags": ""
  "files": ["2345defc", "9876beef"]
}

1234abcd.tag
{
  "parent": "root",
  "tags": "",
  "ordered_tags": [
      {"0": "1235abcd"},
      {"1": "1236abcd"},
    ]
}

1235abcd.tag
{
  "parent": "root",
  "tags": "",
  "ordered_tags": ""
  "files": ["2574dddd"]
}

1236abcd.tag
{
  "parent": "root",
  "tags": "",
  "ordered_tags": ""
  "files": ["2574dddd", "aeee5621"]
}

2574dddd.meta
{
  "name": "somefile.txt",
  "tags": "1234abcd",
  "last_modified": "01-01-2012 1300 PST"
  ...
  ...
  ...
}

2574dddd.dat
<contents of somefile.txt>

Stretch:

- Add a mini webserver with REST api's for all the tag operations and file accesses (GET for operations and POST for data). This will allow a file explorer to be build possibly using javascript. Additional benefit is that access to files on the file system from anywhere on the network can be accomplished without any samba/nfs style mechanisms. They just need to open a browser and connect to the webserver of the machine they want to view the files on. There is an issue with permissions but this can be resolved in a scheme similar to htaccess file used in Apache via a scheme of login's.

Performance and Scalability:
Hard to comment on these without actually building it and carrying out measurements but it seems like the flexibility gained may outweigh the additional overhead of using json. It will also largely depend on the content that is stored in the file system. It seems like the data on a personal laptop/desktop would be an ideal candidate for a file system like this. It would certainly perform horribly if you use it as a file system for storing your database archive! I think medium sized (a wezel word) source code dev tree may perform well in this file system but mixing it other vcs like subversion may cause strange thrashing behavior between the auto-revisioning and the versioning mechanisms of that vcs.

The approach of adding versioning on top of the tagging approach would possibly allow for versioning to be disabled for certain "mount points" or file types like build objects.

In conclusion, the concept of tagged and auto-revisioned file system can be implemented as a user space file system on Linux. The next step would be to actually prototype this design, add varieties of files and tags to the files and see if the added behavior has value and the performance overhead is not unbearable.

[Update in light of vinod's discovery of fossil dvcs]:
This entire design can be re-adjusted using one of the approaches below:

- use fossil and discard the schemes proposed here

- use sqlite as backing datastore instead of pile-of-files (.meta, .tag and .dat) and possibly the scheme of hash instead of tag/file_id

- modify fossil to retrofit this tag concept instead of folders

- add tagging/versioning to a libsqlfs - a fuse file system based on sqlite