Weekend software experiment — Facebook’s Deepmask

Facebook open-sourced its Deepmask and Sharpmask codebases last week for object segmentation in a scene. In the past I’ve tried traditional CV methods to do object masking, and it is kind of a disaster, you end up with a very finicky codebase full of heuristics and hacks, which fails as soon as it sees a new kind of image. Seems like an archetypical use case for neural nets. So seemed worth giving it a try.

Deepmask and Sharpmask come with pretrained models based on the COCO dataset, it looks laborious to label a new dataset without benefit of some mechanical turk-like process, so I decided to stick with the pretrained models.

To avoid polluting my system, and to share with the rest of the team, I decided to bring this all up in a docker container. Because our build environment has some unique characteristics, I needed to author my own container, but these were excellent dockerfile guides: Torch with CUDA and it’s dependencies. Also Torch install is a good reference.

29-Aug-2016-04-13-54

First results in the picture. Did a nice job on this image for the large objects. The biggest problem I had was running out of GPU memory. The machine I was using only had 4G of video ram and this constrains how large an image you can feed in. We had a bunch of 4K x 3K street images, I found that scaling them down to 756×567 allowed me to get them thru the classifier. My modified classifier that implements a size limit is here

The classifier is slow, seconds to tag an image. It would be interesting to keep it resident as a demon, and to just emit the metadata instead of a modified image, and see if I could get a higher framerate. Also I want to play around with both a faster video card, as well as maybe an opencl implementation to get to non-cuda platforms. This is also my first lua code ever, I have no idea which part of this code is fast or slow.

This was a fun little project. It points out that, if software is eating the world, then ML is eating software. For a certain class of problems, a fairly generic neural net plus a dataset is going to be competitive with the best laboriously hand-coded algorithm.

Weekend gadget review — the Cocoon — not recommended

This morning I set up the Cocoon I received this week — “Protect your whole home with Cocoon. Simple setup, HD video and audio, built-in siren, all controlled from your phone. Feel safe with Cocoon.”

Unboxing and setup — cocoon boxpretty standard box quality, nothing special about. Inside the box you get the Cocoon, usb cable and wall wart, some clunky international adapters, a stand, and a very small manual. While the manual was small, it was complete and clear.

cocoon elementsFor wifi setup the Cocoon uses an audio scheme — the Cocoon app on your phone blurts a song out which the Cocoon listens to for embedded wifi credentials. It was pleasant and fast, and way better than a bluetooth dance or a private wifi hotspot dance. Up and running in minutes, this promise was met.

Apparently the Cocoon will use location services and when it knows you are out of the house (ie when your phone is out of the house), will “arm” itself and start watching for movement and listening for sounds. It will then alert you to anything, and you can connect to see the video snippets, and set off the alarm if you want (or call the police or whatever). I didn’t exhaustively check all this out, I mostly just played with the video.

And, well, video as a data type is hard. So many bits. So opaque. The promised “HD video” was not really HD, was very laggy (my guess is they are using something like HLS or DASH which guarantees lag), was corrupted at times, had a low framerate, sometimes didn’t show up over the LTE connection. I’d say that the Canary I’ve used in the past had better video.
cocoon subsound
The Cocoon also has some feature called Subsound. I could not tell from the app screen what this was supposed to be doing. The website says it uses an infrasound microphone, geolocation, and machine learning. You can only see the app screen if you are at home. I guess this is to detect a broader array of movement or sound.

So…this is fairly expensive device. But with poor video. And a bag of other features which seem ok but not amazing. The economics on this thing are challenging to justify, if you want to see what’s going on in your home, a raspberry pi based system would be FAR cheaper tho of course you would have to futz with it yourself. The big downfall of this device is the poor video experience, at $400 I expect something pretty freaking amazing.

INTC’s ARM play and the ARM ecosystem

My best two investments this year have been NVDA and ARMH. NVDA’s been on fire, it was obvious when I attended their GPU Tech Conference that they were going to do very well this year. And Softbank’s play for ARMH has been rewarding.

All the growth in compute usage and capability has moved to non-Intel processor cores — GPU and ARM, both of which are better in different ways for the high growth compute issues of the day. Hard to imagine this changing, I’d expect to see even more GPU migration (or even on to FPGAs).

INTC’s ARM licensing move is a sign that intel is fully accepting the reality here. I am sure Intel still has a lot of internal religion about x86 instruction set — but they shouldn’t hold back now. They should fully cast aside any religion about instruction sets, and just make sure everyone is opting to put an Intel part everywhere. Put an ARM core on every x86 processor — they certainly have the transistors, it would be great as a developer to have an ARM core at my disposal on my laptop. Put an x86 core on every ARM chip, it would be great to be able to use x86 code in more places.

Softbank meanwhile may have just enabled their biggest competitor. It is great to have Intel as a licensee, but there is no way Intel is going to accept a secondary position in the processor market, they are going to use their design skills and fab skills to try to coopt the ARM market. Softbank will need to ramp up their own ARM ecosystem efforts, probably with a deeper software and services play, or they are going to find themselves with less and less influence.

Comcast, your Olympic coverage is making me feel like a dupe

Comcast must love our household. We have to be in the top tier of cable subscribers — multiple TVs, all DVRs, the most premium and sports channels we can get. Partly inertia, partly because we have been economically fortunate, partly because I want to get all the live HD football I can get. Comcast marketers must sit around salivating about us and wondering how to create more customers like us (or at least hang on to us).

So why is the Comcast/NBC Olympics coverage the worst possible option for viewing live Olympics? The 100M dash was what finally killed me. CBUT had it live. I could have found it on streaming live. NBC and Comcast in their infinite wisdom showed it at who knows what time last night. NBC had far more important things to show like prelim heats of some 800M race, also probably on tape delay. We found 5 minutes of worthwhile content in the first two hours of NBC Olympic coverage, thank goodness we could fast forward through it all.

By any measure the traditional cable NBC Olympics viewing is the worst possible option, which is so strange given the pile of money I give Comcast each month. The clear message is “If you want to watch live Olympics, you should not choose NBC or a cable subscription”. Comcast why would you treat your best customer this way?

The ARM IOT software stack ecosystem is kind of a jumbled mess

Google has announced Brillo which leverages Android and Linux. But no, wait, now there is Google Fuchsia because apparently Linux isn’t good enough. On the pi there are a jillion variants of linux to use, but with glaring holes (general purpose gpu, docker for instance). ARM has mbed tho it is really targetted at low end devices. You can try Windows 10. And all of these require a lot of laborious tinkering.

To say nothing of the cloud, the services for managing ARM devices are even more nascent or non-existent.

You have to wonder what Softbank is going to do about this. A $32B hardware bet needs a corresponding software and services bet.

UPDATE: Nat reminds me that I didn’t even mention toolchains! Cross-compilers, QEMU, etc — there is so much fun in here!

Nat nails one of the core premises of Surround.io

Nat, sharp as ever. One of the reasons we started Surround.io was to take advantage of the Moore’s Law driven wave of sensing technology. Since sensors are just carved out of silicon now, using the same process technology as digital electronics, sensors (cameras, accelerometers, mics, etc) are increasingly ubiquitous, cheap, and powerful. The challenge is software to process the flood of data.

Seattle’s role in the tech ecosystem

Lots of blather about this. Two salient observations by Charles and Eric below. As Eric notes, it is impossible to think about launching a startup today without technology from Seattle.

“Hey John, how do you decide what books to read?”

I was asked this recently. I draw from:

  • The NYTimes Sunday Book Reviews. I especially enjoy the “By The Book” column, in which a living author talks about what they have read and are reading. Always some things off the beaten path there.
  • The LA Times book reviews are useful as well.
  • Various prize lists — Hugo, Nebula, Man Booker, Pulitzer, Nobel, Edgar. Tho I have found the Nobel prize winners to be rough rough going.
  • Goodreads reviews and Amazon reviews. If a book doesn’t get great consensus reviews (or at least a strong positive rating from a subset of readers), I probably won’t read.
  • I have a couple coworkers/friends who are big readers and I listen to their recos, and my college roommate is a voracious and critical reader, and I listen to him. And then anything that a family member recommends, I will consider.

I will read just about any genre with a couple limitations. I find most popular business books to be horrible — 1 or 2 good ideas that are worthy of an article, but nothing worthy of a book. And I do not enjoy farce in any genre, just not my thing.

Recent books — Unruly Places, The Rook, Dispatches from Pluto, Dog Stars, Pamuk, Niceville

  • Unruly Places by plutoAlastair Bonnett. The stories behind some of the most unusual and unlikely places on the globe — places belonging to no nation, to many nations, floating, underground, in the sky — and an understanding of the meaning and importance of place.  Enjoyable.  
  • The Rook and Stiletto by Daniel O’Malley. Great suspenseful adventure in a world filled with magic —  kind of a grown-up Harry Potter milieu.  The first page grabs you — the protagonist awakens with no memory, in a park surrounded by bodies, and in her pocket is a note that says “Read Me”. Stiletto isn’t quite as strong but still a fund read.  
  • Dispatches From Pluto by Richard Grant. An English adventure writer decides to settle in the Mississippi Delta. Interesting look at the culture and issues of a part of the country that seems almost foreign to me.  
  • The Dog Stars by Peter Heller. A different kind of post-apocalyptic novel, the world has largely been emptied out, a very few people try to find their way and remain human.  
  • A Strangeness In My Mind by Orhan Pamuk. Tried to lift my literary sights, but as always, “Nobel Prize Winning” does not equal “interesting”.  
  • Niceville by Carsten Stroud. Not a nice place. Solid horror/mystery.  

Recent Books

  • The Fifth Season by N. K. Jemison. A small set of related characters finding their way on a far-future(?) Earth peopled by humans and human derivatives, littered with the artifacts of past civilizations, undergoing yet another seismic/volcanic reshaping. Of course there is some conspiracy at work. Good tale, characters dealing with painful issues.connectography
  • Connectography by Parag Khanna. How the world is moving beyond nation-states to clusters driven by interconnectivity. Well written and a lot of maps, and I am a sucker for maps.
  • The Ruins of Gorlan by John A. Flanagan. Pretty standard YA fantasy, but fun nevertheless. I’d read more.
  • The Call by Yannick Murphy. Eh, what started out as a clever structural gadget became tiring. I am a little curious to know what happens to the family but I gave up.
  • Life After Life by Kate Atkinson. I like Atkinson, but this book also became tiresome, tho I managed to finish. But I’d recommend a different Atkinson.
  • Farthing by Jo Walton. How did I miss this? Conspiracy, murder in the English countryside in an alternative history where Britain made peace with the Nazis. Excellent.
  • Those Who Wish Me Dead by Michael Koryta. A boy witnesses a murder, and, well, shit happens. Fun but not memorable.

Recent Books — Uprooted, Big Little Lies, A Burglar’s Guide…

  • Spill Simmer Falter Wither by Sarauprooted Baume. Highly praised, but it felt to me like the author was trying very hard to be literary. I stuck with it, but it didn’t really move me.
  • Uprooted by Naomi Novik. A captivating tale of a young witch coming of age and getting drawn into an existential threat for her homeland. Really loved it, deserves the accolades.
  • Big Little Lies by Liane Moriarty. Completely fun story of conflicts between parents at a preschool, hilarious but serious. Apparently coming to HBO, this will be fun
  • A Burglar’s Guide To The City by Geoff Manaugh. A look at how burglars use urban design and building design to achieve their goals. Would have been even better with more anecdotes, but still interesting

Recent Books — High-Rise, Travelers, Song Machine, The Whites, Electric Barracuda

  • High-Rise by J. G. Ballard. ebThe community in a modern high-rise starts to fall apart in “Lord of the Flies” fashion…actually “Lord of the Flies” seems pretty tame after this.
  • The Travelers by Chris Pavone. A travel writer is coopted and stumbles into a job as a spy, and slowly learns that nothing is as it seems in his life. Fun.
  • The Song Machine: Inside the Hit Factory by John Seabrook. No surprise here, pop music is largely a manufactured product.
  • The Whites by Richard Price. Couldn’t finish. Supposedly a gritty police drama but kind of dull.
  • Electric Barracuda by Tim Dorsey. I generally don’t like farce, but there is something endearing about this character — a wise-cracking serial killer with a heart of gold.