robjsoftware.org

A blog about software – researching it, developing it, and contemplating its future.

Stuff Been Happenin’

leave a comment »

[This post was originally written in April 2014, but is only now getting posted in July 2014. Hobby projects are like that!]

Hacking stuff, that is!  Holofunk stuff, to be precise!  Working code, to be sure!

I’ve drifted into posting more on Facebook and on one particular forum of my long acquaintance, so now I get to be a bit lazy and sum up those scraps of news a little more centrally and with more detail. 

In January I got the lead out and ported Holofunk to the new Kinect.  This required first porting it to x64.  The main casualty there was that VST support — sound effect plugin support — stopped working, for no reason I could sort out after a couple of nights.  That’s the maximum length of time I can spend on any blocking issue before I start looking for a workaround.  So now I’m just using the sound effects built into the BASS audio library, which are quite sufficient for the time being.

I did get it working again, and modulo a couple of performance issues that I’m discussing with them, it’s pretty stunning.  I was able to get green-screened color video working in almost no time based on their sample code, for multiple players; this was impossible with the first Kinect.

Hand Pose, At Last

Much more importantly, though, I started getting hand pose data.  And the hand pose data is fast and reliable.  There are only three hand poses supported — open (all fingers spread), pointing (one or two fingers pointing, the rest in a fist), and closed (a plain fist).  It takes a little getting used to, as far as making rapid and clear transitions from one pose to another; but with just a little practice it gets very fluid.

Three hand poses is kind of like having a mouse that has a three-position switch on it… it’s not a whole lot to work with, but it’s enough.  I started brainstorming gestural interfaces, and my wife MIchelle helped me take some notes:

[picture of HF brainstorming notes]

The basic idea is this:

  • You open your hand to get the app’s attention.  (This is “armed” state, internally.)
  • You can then close your hand to start recording yourself; as long as your hand is closed, the recording continues.  When you open your hand, you “drop” the recording at that spot on the screen.

That’s the most basic interaction:  make a fist to record a loop, then let go to play it.

  • You can also point, to enter “pointing mode.”  Basically, each hand has its own “pointing mode” that determines what will happen when that hand points.
  • The default “pointing mode” is “mute/unmute.”  In this mode, you point at a sound or group of sounds, and you make a fist to mute them, or open your hand to unmute them.  If you mute some muted sounds, they get deleted altogether.  This gives you the ability to bring loops in and out.
  • Another “pointing mode” is “sound effects.”  In this mode, you point at a sound or group of sounds, and then you move your hand up/down/left/right to apply one of four sound effects (one per direction).  I prototyped this interface with Holofunk 1.0 and it works OK, so I’m bringing that forwards.
  • There can be multiple “sound effects” modes with different combinations of sound effects.

I’ve implemented the “mute/unmute” behavior and it’s pretty incredible — the hand recognition is fast enough that you really feel like you’re grabbing a bunch of sounds and then shushing them by squeezing them, then opening your hand again to bring them back to life.

So how do you change modes?  My main insight here was that I wanted some kind of “chord” gesture — in a conventional interface you’d have shift-click, or control-click, or something.  So what could be a modifier for the pointing gesture?  I had already implemented radial popup menus, I just needed a way to invoke them.

What I came up with was to use body pose as a modifier.  Specifically, if you put your hand on your hip (akimbo, in other words), then when you point with the other hand, you get a popup menu that lets you pick the pointing mode for that hand.  So you just put your hand on your hip, point your other hand at “effect mode”, grab that menu item, and now that other hand is in effect mode.  It’s natural and feels quite good.  Putting your hand behind your back (rather than elbow-out akimbo) means you’ll get the system popup menu, with commands like “Delete all” and “Change tempo”.

Body pose is modal.  This is your NUI koan for the day.

This combination of hand pose (for pointing and picking), body pose (for modifying that pointing/picking), and per-hand interaction mode means that the interface is truly ambidextrous:  both hands can perform independent gestures simultaneously.  You could have one hand applying reverb/flange/chorus/delay and the other hand applying volume/pan, or one hand muting and unmuting while the other hand tweaks sound effects, or whatever you like.

Right now I have the popup menus coming up, but not interacting properly — some minor issue, I think.  Will be fixing that very soon.

My current code has a nice little hierarchical state machine for the per-hand interaction, so I have two independent state machines, one per hand.  Previously, an event — such as the user pointing — would always cause a transition to a fixed new state.  But in the new interface, pointing while the other hand is on the hip should bring up a popup menu; pointing while in mute/unmute mode should enter mute/unmute state; and so forth.

All I needed to implement this, it turned out, was a “computed transition” that would run some code to determine a target state, rather than using a fixed target state.  This was a very simple thing to add, and wound up perfectly expressing both pointing modes and body-pose modes.

Two-Handed Interaction

Now, having an ambidextrous, bilateral interface is all very well, but not all interactions involve only one hand.  Some interactions want two.  For example, using two hands to drag out a selection rectangle for sound grouping.  Or, dragging out a time distortion envelope for time mapping.

I have two independent state machines, one per hand.  Fine if the hands are independent, but what if they’re not?  Do both hands need to be in the same state, somehow?  How do you coherently use two state machines for one interaction?  It all just felt wrong and ugly and hacky, a sure sign that I needed to sleep on it some more.  When this project gets stalled, it’s either because I don’t have a working brain cell left, or because I haven’t got a clear simple picture of how it should work.  And I don’t have the spare time to write vague code and then have to debug it! 

I plan to build a state machine hierarchy in which a “body” state machine can look at both hands, and if it does not want to consume the state of both hands in a two-handed way, it can delegate the state of each hand to a lower-level, per-hand state machine. State machine delegation, in other words. I think it will work well… once I get there.

Written by robjellinghaus

2014/07/30 at 00:47

Posted in Holofunk

Tagged with ,

To John Carmack: Ship Gamer Goggles First, Then Put Faces And Bodies Into Cyberspace

with 3 comments

John Carmack, CTO of Oculus Rift, tweeted:

Everyone has had some time to digest the FB deal now. I think it is going to be positive, but clearly many disagree. Much of the ranting has been emotional or tribal, but I am interested in reading coherent viewpoints about objective outcomes. What are the hazards? What should be done to guard against them? What are the tests for failure? Blog and I’ll read.

I have already blogged on this but will make this a more focused response to John specifically.

Here are my objective premises:

  • VR goggles, as currently implemented by the Rift, conceal the face and prevent full optical facial / eye capture.
  • VR goggles, ACIBTR, conceal the external environment — in other words, they are VR, not AR.
  • Real-time person-to-person social contact is primarily based on nonverbal (especially facial) expression.
  • Gamer-style “alternate reality” experiences are not primarily social, and are based on ignoring the external environment.

Here are my conclusions:

  • A truly immersive social virtual environment must integrate accurate, low-latency, detailed facial and body capture.
    • Therefore such an environment can’t be fundamentally based on opaque VR goggles, and will require new technologies and sensor integrations.
  • Opaque VR goggles are, however, ideal for gamer-style experiences.
    • Gamer experiences have never had full facial/body capture, and are based on ignoring the external environment.
  • The more immersive such experiences are, the more people will want to participate in them.
    • This means that gratuitous mandatory social features, in otherwise unrelated VR experiences, would fundamentally break that immersion and would damage the platform substantially.
  • Goggle research and development will mostly directly benefit “post-goggle” augmented reality technology.

The hazards for Rift:

  1. If Facebook’s monetization strategy results in mandatory encounters with Facebook as part of all Rift experiences, this could break the primary thing that makes Rift compelling: convincing immersion in another reality that doesn’t much overlap with this one.
  2. If Facebook tries to build an “online social environment” with Rift as we have historically known them (Second Life, Google Lively, PlayStation Home, Worlds Inc., etc., etc., etc.), it will be as niche as all those others. Most importantly, it will radically fail to achieve Facebook’s ubiquity ambitions.
    • This is because true socializing requires full facial and nonverbal bandwidth, and Rift today cannot provide that. Nor can any VR/AR technology yet created, but that’s the research challenge here!
  3. If Facebook and Rift fail to pioneer the innovation necessary to deliver true augmented social reality (including controlled perception of your actual environment, and full facial and body capture of all virtual world participants), some other company will get there first.
    • That other company, and not Facebook, will truly own the future of cyberspace.
  4. If Rift fails to initially deliver a deeply immersive alternate reality platform, it will not get developers to buy in.
    • This risk seems smallest based on Rift’s technical trajectory.

What should be done to guard against them:

  1. Facebook integration should be very easy as part of the Rift platform, but must be 100% completely developer opt-in. Any mandatory Facebook integration will damage your long-term goals (creating the first true social virtual world, requiring fundamentally new technology innovation) and will further lose you mindshare among those skeptical of Facebook.
  2. Facebook should resist the temptation to build a Rift-based virtual world. I know everyone there is itching to get started on Snow Crash, and you could certainly build a fantastic one. But it would still be fundamentally for gamers, because gamers are self-selected to enjoy surreal online places that happen to be inhabited by un-expressive avatars.
    • The world has lots of such places already; they’re called MMOGs, and the MMOG developers can do a better job putting their games into Rift than Facebook can.
  3. Facebook and Rift should immediately begin a long-term research project dedicated to post-goggle technology. Goggles are not the endgame here; in a fully social cyberspace, you’ll be able to see everyone around you (including those physically next to you), faces, bodies, and all. If you really want to put your long-term money where your mouth is, shoot for the post-goggle moon.
    • Retinal projection glasses? LCD projectors inside a pair of glasses? Ubiquitous depth cameras? Facial tracking cameras? Full environment capture? Whatever it takes to really get there, start on it immediately. This may take over a decade to finally pan out, but you have the resources to look ahead that far now. This, and nothing less, is what’s going to make VR/AR as ubiquitous as Facebook itself is today.
  4. Meanwhile, of course, ship a fantastic Rift that provides gamers — and technophiles generally — with a stunning experience they’ve never had before. Sell the hardware at just over cost. Brand it with Facebook if you like, but try to make your money back on some small flat fee of title revenue (5%? as with Unreal now?), so you get paid something reasonable whether the developer wants to integrate with Facebook or not.

Tests for failure:

  1. Mandatory Facebook integration for Rift causes developers to flee Rift platform before it ships.
  2. “FaceRift” virtual world launches; Second Life furries love it, rest of world laughs, yawns, moves on.
  3. Valve and Microsoft team up to launch “Holodeck” in 2020, combining AR glasses with six Kinect 3′s to provide a virtual world in which you can stand next to and see your actual friends; immediately sell billions, leaving Facebook as “that old web site.”
  4. Initial Rift titles make some people queasy and fail to impress the others; Rift fails to sell to every gamer with a PC and a relative lack of motion sickness.

John, you’ve changed the world several times already. You have the resources now to make the biggest impact yet, but it’s got to be both a short-term (Rift) and long-term (true social AR) play. Don’t get the two confused, and you can build the future of cyberspace. Good luck.

(And minor blast from the past:  I interviewed you at the PGL Championships in San Francisco fifteen years ago.  Cheers!)

Written by robjellinghaus

2014/03/28 at 12:25

Posted in Uncategorized

My Take On Zuckey & Luckey: VR Goggles Are (Only) For Gamers

with 2 comments

I am watching the whole “Facebook buys Oculus Rift” situation with great bemusement.

I worked for a cyberspace company — Electric Communities — in the mid-nineties, back in the first heady pre-dot-com nineties wave of Silicon Valley VC fun.

We were building a fully distributed cryptographically based virtual world called Microcosm. In Java 1.0. On the client. In 1995. We had drunk ALL THE KOOL-AID.

Image

(Click that for source.  For a scurrilous and inaccurate — but evocative — take on it all, read this.)

We actually got some significant parts of this working — you could host rooms and/or avatars and/or objects, and you could go from space to space using fully peer-to-peer communication. Because, you see, we envisioned that the only way to make a full cyberspace work was for it to NOT be centralized AT ALL. Instead, everyone would host their own little bits of it and they would all join together into an initially-2D-but-ultimately-3D place, with individual certificates on everything so everyone could take responsibility for their own stuff. Take that, Facebook!!!

(I still remember someone raving at the office during that job, about this new search engine called Google… the concept of “web scale” did not exist yet.)

The whole thing collapsed completely when it became clear that it was too slow, too resource-intensive, and not nearly monetizable enough. I met a few lifelong friends at that job though, quite a few who have gone on to great success elsewhere (Dalvik architect, Google ES6 spec team member, Facebook security guru…).

I also worked at Autodesk circa 1991, in the very very first era of VR goggles, back when they looked like this:

virtual_realityx299

Look familiar?  This was from 1989.  Twenty-five frickin’ years ago.

So I have a pretty large immunity to VR Kool-Aid. I actually think that Facebook is likely to just about get their money back on this deal, but they won’t really change the world. More specifically, VR goggles in general will not really change the world.

VR goggles are a fundamentally bad way to foster interpersonal interaction, because they obscure your entire face, and make it impossible to see your expression. In other words, they block facial capture. This means that they are the exact worst thing possible for Facebook, since they make you faceless to an observer.

This then means that they are best for relatively solitary experiences that transport you away from where you are. This is why they are a great early-adopter technology for the gamer geeks of the world. We are precisely the people who have *already* done all we can to transport ourselves into relatively solitary (in terms of genuine, physical proximity) otherworldly experiences. So VR goggles are perfect for those of us who are already gamers. And they will find a somewhat larger market among people who want to experience this sort of thing (kind of like super-duper 3D TVs).

But in their current form they are never going to be the thing that makes cyberspace ubiquitous. In a full cyberspace, you will have to be able to look directly at someone else *whether they are physically adjacent or not*, and you will have to see them — including their full face, or whatever full facial mapping their avatar is using — directly. This implies some substantially different display technology — see-through AR goggles a la CastAR, or nanotech internally illuminated contact lenses, or retinally scanned holograms, or direct optical neural linkage. But strapping a pair of monitors to your eyeballs? Uh-uh. Always going to be a “let’s go to the movies / let’s hang in the living room playing games” experience; never ever going to be an “inhabit this ubiquitous cyber-world with all your friends” experience.

Maybe Zuckerberg and Luckey together actually have the vision to shepherd Oculus through this goggle period and into the final Really Immersive Cyberworld. But my guess is the pressures of making enough money to justify the deal will lead to various irritating wrongnesses. Still, I expect they will ship a really great Oculus product and I may even buy one if the games are cool enough… but there will be goggle competitors, and it’s best to think of ALL opaque goggle incarnations as gamer devices first and foremost.

So why did Zuckerberg do this deal? I think it’s simple: he has Sergey Brin envy. Google has its moon-shot projects (self-driving cars, humanoid robots, Google Glass). Zuckerberg wants a piece of that. It’s more interesting than the Facebook web site, and he is able to get his company to swing $2 billion on a side project, so why not? Plus he and Luckey are an epic mutual admiration society. That psychology alone is sufficient explanation. It does lead to the amusingly absurd paradox of Facebook spending $2 billion on something that hides users’ faces, but such is our industry, and such has it ever been.

Realistically, the jury is still out on whether Oculus would have been better off going it alone (retaining the love of their community and their pure gaming focus, but needing to raise more and more venture capital to ramp up production), or going with Facebook (no more worries about money, until Facebook’s ad-based business model starts to screw everything up). The former path might have cratered before launch, or succumbed to deeper-pocketed competitors. The latter path has every chance of going wrong — if Facebook handles things as they did their web gaming efforts, it definitely will. We will see whether Zuckerberg can keep Facebook’s hands off of Oculus or not. I am sadly not sanguine… on its face, this is a bad acquisition, since it does not at all play to the technology’s gaming strengths.

It’s worth noting Imogen Heap’s dataglove project on Kickstarter.  I was skeptical that they would get funded, but their AMA on Reddit convinced me they are going about it the best way they can, and they have a clear vision for how the things should work.  So now I say, more power to them!  Go support them!  They are definitely purely community-driven, the way Oculus was until yesterday….

Written by robjellinghaus

2014/03/26 at 08:48

Life Is What Happens When You’re Making Other Plans

leave a comment »

I didn’t exactly expect to take a nine-month blogging hiatus, but the last while has been considerably more exciting than I expected.

Help Wanted

Let me place my main request up front:  I am interested in self-publishing some software, specifically a Windows application that uses the new Kinect sensor for video and audio live looping. 

I have asked people I trust about how to go about doing this (creating an LLC or other corporation, filing taxes, etc.), and I have been told that I need to talk to an accountant and a lawyer, both of whom are licensed in Washington state. 

So: game developers, indie developers, and general friends in the Seattle area, what accountants and/or lawyers would you recommend for someone getting involved in software self-publishing?  All referrals are welcome.  Thanks very much indeed!

Where The Heck I’ve Been

First, last spring, I put sound effects into Holofunk sure enough.  I showed it at the Seattle Mini Maker Faire with some colleagues in the Kinect for Windows booth, which was awesome — managed to spiel it for seven straight hours without driving everyone crazy, so that was good!

Then, in the late spring, I went to a presentation at work about a program to send computer science workers into schools to teach AP computer science.  It’s called TEALS, for Technology Education And Literacy in Schools:  http://tealsk12.org and it is quite an amazing thing. 

During that presentation I saw a picture of Bill Gates looking at an older boy on a Teletype:

As it happens, that is the same device (ASR-33) that I typed my first BASIC programs on at about the same age Bill is in that picture.  I got surprisingly choked up, and realized how big a deal computers were for me back then.  So I knew I had to sign up.  I did so, and committed to teaching at Lindbergh High School in Renton.

Soon thereafter, I re-did the Holofunk sound effects interface, and had some friends over to play with it.  And it was EPIC.  Given that the majority of them had never seen it before, everyone was creating some pretty intense noises quite quickly.  I have some very raw video (dangling cables, the whole bit) here, if you are feeling experimental.

Everyone had so much fun with it that I realized two things:

1) Holofunk can and should be an actual product.

2) Holofunk needs the new Kinect sensor.

Now, realizing this just after having committed to become a half-time one-period high school teacher for the year — on top of intense day job — is not exactly anything like the best timing.  And there were various other complications with the teaching, including my original co-teacher having to drop out of the program after school had already started, an extraordinarily stressful situation.  Fortunately it worked out for the best… but it has left me no time for anything but work, family, and teaching.

Until now!  I would not be writing this if I weren’t finally regaining some energy for hacking.  And I applied for, and got into, the new Kinect for Windows beta program.  I will be ordering the new sensor tomorrow, in fact, and hopefully receiving it sometime late this month, which is extraordinarily motivating!

I’ve already registered holofunk.com and holofunk.org, and it’s time to create Holofunk LLC (or something, subject to legal counsel).  Then it’s time to bring in the new Kinect, video looping, and hand pose detection… no more Wiimotes.  That alone should help with the Microsoft science fair judges!  I hope very much to get all this working, at least at a basic level, by mid-winter sometime (end of February?).

I am ridiculously excited for the potential of this thing.  And I am also very excited for my high school class, some of whom were initially struggling but are really doing better now.  They are actually learning.  I’ll be making a later post (or several) about it all, as I finally get my head above water.

Enjoy the fall!

Written by robjellinghaus

2013/11/13 at 00:38

Posted in Holofunk

Goodbye XNA, helloooo SharpDX

with one comment

Half a year ago I posted about my XNA dilemma, and about how SharpDX looked like the best replacement. XNA = old and busted C# DirectX library from Microsoft, now officially kicked to the curb; SharpDX = amazing generated-from-C++-DirectX-sources open source C# binding to the entire DirectX API, which is waaaaay more than XNA ever did.

Last summer I was a bit daunted by SharpDX, as the API was not very XNA-like. Alex Mutel, the primary author of SharpDX, said he was working on an XNA compatibility layer but it wasn’t ready yet. I then figured out how to get multiple window support (the primary driver at the time) working with XNA, so I set SharpDX aside for a while.

Well, Alex was as good as his word. After a fair amount of very entertaining Holofunking over the holidays (Holidayofunking?), I finally decided that with the new year would come a new 3D library underneath Holofunk. And what do you know, the SharpDX Toolkit library was ready.

Over the course of a couple of weeks, working relatively infrequently and in small chunks, I converted everything off of XNA. If you want to see a reasonably small, but realistic, XNA-to-SharpDX conversion in full, you could do a lot worse than looking at this changelist on CodePlex.

First of all I realized there was no multi-window support in SharpDX. Well, that’s fine, I posted an issue about it and offered to help with the implementation. Alex immediately responded and asked for a short while to build some support pieces. The next thing I knew (three days later), he’d implemented the whole feature. It worked fine for me the first time.

The most surprising changes were:

  • In XNA, the Rectangle constructor takes (int left, int top, int width, int height). In SharpDX, it takes (int left, int top, int right, int bottom). This led to all KINDS of amusing weirdness.
  • The Point type is gone, so I switched to Vector2 with some casting to int. Probably there is a better way but I don’t yet know it.
  • The XNA Color.Transparent is equivalent to Color(0, 0, 0, 0) — e.g. premultiplied black at zero alpha. In SharpDX, it is Color(0, 0, 0, 255) — e.g. black with full alpha. Everywhere I had been using Color.Transparent I switched to “new Color(0)” to get the same effect in SharpDX.
  • In XNA, the Color constructor takes (int r, int g, int b, int a). In SharpDX, it takes either (byte r, byte b, byte g, byte a) or (float r, float g, float b, float a). This caused some of my color math to use the float overload rather than the byte overload, with ensuing hilarity.
  • I ran into a premultiplied alpha problem with PNGs, which aren’t premultiplied. I posted an issue and Alex responded immediately AGAIN. I was able to hack around it with his suggested BlendState.NonPremultiplied workaround.
  • I tried using 32-bit premultiplied BMPs in RGBA format, but ran into ANOTHER issue, which, of course, I posted. We’ll see whether Alex maintains his exemplary responsiveness.

If you’re getting the idea that Alex is incredibly helpful, you’re right. He’s right up there with the excellent authors of the BASS audio library, about whom I’ve similarly gushed. The very best open source projects to use in YOUR open source project are the ones with active and helpful maintainers. And having an open source project of your own really helps get support, since you can show your code and they can point others at your code as an example of something cool done with their work.

So now Holofunk is fully running on SharpDX with all features maxed, multiple window and two-player support, and much faster rendering. It looks pretty much the same, but now all obstacles to video texturing, bizarre postprocessing, Fourier transforms on the GPU, etc., etc., are removed. The sky is now the limit.

holofunk_20130206_for_blog

Next up: sound effects! At last!

Now that I’ve gotten that big move over with, it’s time to start putting sound effects in at last! I bought a copy of Sugarbytes Turnado, the VST plugin Beardyman uses in his live rig, and it is indeed friggin’ amazing. Been having all kinds of fun exploring it.

The tricky part is that it does eat CPU, and I’m not sure how many instances of it I will be able to create with the BASS audio library I’m using. So while part of me wants an interface that lets me apply any random set of effects to each and every individual loop, that would mean one instance of Turnado per loop. In Holofunk it’s easy to create dozens of loops. So I needed some fallback story for how to manage sound effects in some more granular way.

I think what I will try first is splitting the screen into eight wedges (instead of the four quarters in the current version), with each wedge having its own instance of Turnado. So all loops in a given wedge will share a set of effects. It will be possible to select multiple wedges and wave your arms wildly to wiggle the Turnado knobs for all those wedges at once. It will also be possible to record animated knob-wiggling across a set of wedges. That, combined with effect selection, should be more than enough to get some serious madness going on, while being pretty straightforward to implement.

I also have an interesting idea for a graphical interface for scratching, but I’m going to show that one in prototype form rather than blog about it :-)

Meanwhile, Elsewhere In Loopania

djtechtools.com wins the big Looping Journalism Award of the last two months. They have an excellent and long article on Beardyman’s new live performance setup, which he said on Facebook will be coming soon to a TED talk near you. CAN NOT WAIT.

They also have a video of Imogen Heap with the latest iteration of her gestural music gloves. Here’s the video for your convenience — and look, she’s got a Kinect in there!

How ridiculously cool is THAT???

This entire field of musical experimentation is moving along most awesomely, and I’m greatly enjoying my own little part in it.

So: February is the month of Sound Effects Into Holofunk. March I plan another round of beta testing, aka getting together with my Holofunk posse and partying on it. April and May, more work, and then into demo season in June with a public performance at the Maker Faire. Going to try to be ready for a 30-minute gig this time!

Expect another update in March or so. Until then, stay funky!

Written by robjellinghaus

2013/02/07 at 00:27

Posted in Holofunk

A Publication! In Public!

with 7 comments

The team I work on doesn’t get much publicity.  (Yet.)

But recently some guys I work with submitted a paper to OOPSLA, and it was accepted, so it’s a rare chance for me to discuss it:

Uniqueness and Reference Immutability for Safe Parallelism

This is really groundbreaking work, in my opinion.  Introducing “readable”, “writable”, “immutable”, and “isolated” into C# makes it a quite different experience.

When doing language design work like this, it’s hard to know whether the ideas really hold up.  That’s one epic advantage of this particular team I’m on:  we use what we write, at scale.  From section 6 of the paper:

A source-level variant of this system, as an extension to C#, is in use by a large project at Microsoft, as their primary programming language. The group has written several million lines of code, including: core libraries (including collections with polymorphism over element permissions and data-parallel operations when safe), a webserver, a high level optimizing compiler, and an MPEG decoder. These and other applications written in the source language are performance-competitive with established implementations on standard benchmarks; we mention this not because our language design is focused on performance, but merely to point out that heavy use of reference immutability, including removing mutable static/global state, has not come at the cost of performance in the experience of the Microsoft team. In fact, the prototype compiler exploits reference immutability information for a number of otherwise-unavailable compiler optimizations….

Overall, the Microsoft team has been satisfied with the additional safety they gain from not only the general software engineering advantages of reference immutability… but particularly the safe parallelism. Anecdotally, they claim that the further they push reference immutability through their code base, the more bugs they find from spurious mutations. The main classes of bugs found are cases where a developer provided an object intended for read-only access, but a callee incorrectly mutated it; accidental mutations of structures that should be immutable; and data races where data should have been immutable or thread local (i.e. isolated, and one thread kept and used a stale reference).

It’s true, what they say there.

Here’s just a bit more:

The Microsoft team was surprisingly receptive to using explicit destructive reads, as opposed to richer flow-sensitive analyses (which also have non-trivial interaction with exceptions). They value the simplicity and predictability of destructive reads, and like that it makes the transfer of unique references explicit and easy to find. In general, the team preferred explicit source representation for type system interactions (e.g. consume, permission conversion).

The team has also naturally developed their own design patterns for working in this environment. One of the most popular is informally called the “builder pattern” (as in building a collection) to create frozen collections:

isolated List<Foo> list = new List<Foo>(); 
foreach (var cur in someOtherCollection) {
    isolated Foo f = new Foo();
    f.Name = cur.Name; // etc ... 
    list.Add(consume f); 
}
immutable List<Foo> immList = consume list;

This pattern can be further abstracted for elements with a deep clone method returning an isolated reference.

I firmly expect that eventually I will be able to share much more about what we’re doing.  But if you have a high-performance systems background, and if the general concept of no-compromises performance plus managed safety is appealing, our team *is* hiring.  Drop me a line!

Written by robjellinghaus

2012/11/02 at 14:06

Posted in Uncategorized

Holofunk fall update

with 3 comments

August was busy alright!

Despite all the business in my last post about SharpDX etc., I discovered that XNA 4.0 actually supports multiple windows quite straightforwardly, so being a lazy programmer I went with the easy route.  That version of the code is uploaded on CodePlex now.  I’m really happy with how it turned out!

Latency is Evil, Latency is Death

In my last post I also mentioned I met with some talented local loopers.  I didn’t mention that while demoing, a couple of them felt that Holofunk was just… too… laggy.  I was shocked by this as I’d been working for a long time to cut latency and I could only notice a subtle bit of it.  But they insisted it was no good.  So finally I thought to check the ASIO4ALL buffer size.  This was set to 512 samples, which at 48Khz (my current sampling rate) is just over 1/100 of a second.  That’s damn short!  But I shortened it yet further to 192 samples, which is 7/1000 of a second shorter.  And suddenly they loved it.  It felt seamless and right.   And I realized they weren’t kidding — 7/1000 of a second really is very audible!

It turns out that this is well known.  Looping technology is a sonic mirror, and if it doesn’t line up perfectly, it really throws you off.

All of this made such a big impression on me that I made a real point of it at the Microsoft demo I gave in August, which you can now watch.

That went well.  I won a runner-up prize in the “So Fun” category.  My use of a Wiimote kind of disqualified me, since it was a Kinect-for-Windows-sponsored contest and all.  I consider that fair; the applause was reward enough!

Then I did another gig at my sister’s wedding in Boston.  Unfortunately I ran into a bizarre situation I’d never encountered before:  my microphone sounded fine, but the Holofunk loops weren’t playing at all.  This despite the fact that there was only one stereo pair going into the DJ’s mixer!  I could not understand how this could be happening, and it almost blew my whole show in front of 150 people (no pressure!), but finally the DJ yanked one of the wires off and suddenly it worked.  Some bizarre kind of phase problem?  I hadn’t caught it during sound check because I just checked the microphone.  Lesson learned: SOUND CHECK EVERYTHING!  Still, it wasn’t a total disaster and many people told me they enjoyed it, so all’s well that ends well.

Plus, I took it to Cape Cod for our post-wedding family vacation, and my sister (a trained Bulgarian folk singer) turned out to be awesome at it.  She was very impressed and wants to play with it more.  Sooner or later I’ll need to package it for easier distribution.

Sound Weirdness Mega-Party

Then, Tim Thompson of Kinect Space Frame fame came into town for the Decibel Festival.  He very graciously wrote me and asked if we could meet and maybe do some kind of event.  He then contacted a local makerspace, Jigsaw Renaissance, and they got enthusiastically on board.  We wound up hanging out for a few hours, making many weird sounds, and having many interesting brainstorms with plenty of attendees.  It was all kinds of festive!

One highlight was meeting Tarik Barri, who had some cool looping video of various animated headshot clips of himself.  It was exactly the kind of thing I have in mind for Holofunk at some point — clipping out live video of the performer’s head, and looping that in place of the little circles, basically making a live version of Beardyman’s Monkey Jazz piece.  Check out Tarik’s animated face fun (slightly NSFW), and this 3D sound-space, which is also very inspirational for Holofunk’s head-mounted future at some point.

What was especially neat about this evening was that I felt, for the first time, that Holofunk can hold its own in a roomful of weird hacker electronic music projects.  It might not be ready for prime time, but it’s definitely ready for backstage with the big dogs!  Next year I’ll be working to get it into the Decibel Festival in some manner, for sure.

Hiatus, Interrupted

As far as actual coding goes, I’ve been in a low-hacking mode since the end of August, focusing instead on various other types of software, namely Diablo III, Borderlands 2, and XCOM :-)  Fall is the biggest gaming season of the year and I’ve been indulging.  It’s been great.  But the worm is turning and it’s time to get back to Holofunk.

After adding the dual-monitor stuff, which turned out so well, I’m pretty clear that the next great feature is two-player support.  This is technically possible with just the hardware I have now — I can already connect two mikes to my USB audio interface, two Wiimotes via Bluetooth, and Kinect can do two-person skeletal tracking.  All I have to do is to refactor the guts of the code to support two of everything.  Right now I think I am going to let both players “step on each other” — e.g. each person can move and touch anywhere on the screen.  This will introduce some weird boundary cases, but should make for more entertaining play, if you can muck with the sounds the other person made.

The biggie after that is to add VST plugin support, particularly for the Turnado plugin from SugarBytes.  The reason that one is so important (despite its not inconsiderable cost) is that my inspiration Beardyman uses it in his new iPad-based software performance setup.  Bang, that’s the only recommendation I care about in the whooooole world.  Have a listen at this and imagine it in Holofunk.  Hell yes.

Of course, this is going to kick the complexity of the whole thing to another level altogether.  The main conceptual problem I’ve had for a long time is simply what the interface should be.  On the one hand I want it to be very simple and approachable; on the other hand, I want to be able to do ridiculously layered compound effects.  And, as with all Holofunk features, there has to be a smooth ramp from the simple to the sophisticated.

Here’s what I’m thinking:

  • There’s an “effects mode” you can enter.
  • When in “effects mode”, your hands aren’t selection cursors anymore; they’re “knobs.”  Waving the knobs around with the Wiimote changes parameters.  (Up/down = one parameter; left/right = another; forward/back = another.)  This would give six axes of parameter control just with your two hands, which is enough for starters (though really your feet will have to get in on the fun at some point…).
  • There needs to be some menu interface for assigning parameters to axes.  In other words, you should be able to click somewhere and select left/right for pan and up/down for LFO frequency, or whatever.
  • Then you should be able to set all your knob parameters as presets on the Wiimote D-pad, so you push up/down/left/right and instantly get a set of six parameters mapped to your two hands.
  • THEN, holding down the A button activates your effects and lets you immediately mutate the sounds that you were just pointing at.
  • THEN, holding down the A button and the trigger records your parameters as a loop!

So imagine an interaction like this:

  1. Squeeze the trigger, laugh a little “Ho ho ho!”, let go.  Now you have a loop going “Ho ho ho!  Ho ho ho!  Ho ho ho!” forever.
  2. Push the D-pad to pick the “pan = left/right” preset.
  3. Hold the A button over your “Ho ho ho!” loop, and wave your hand left and right.  Now Santa Claus is jumping from the left channel to the right channel and back.  When you stop waving your arm, Santa settles down.
  4. Now hold the A button and squeeze the trigger while you do a full circle with your arm, and then release it.  Santa Claus will now be looping from one speaker to the other.  Basically you recorded an animated envelope of the side-to-side panning.

The A button becomes the “apply effect” button, and the trigger retains its “record loop” behavior… you’re just combining them into a single gesture.  There still needs to be some way to affect the microphone itself (rather than just the selected loops) — I still need to figure that part out a bit better.  Maybe hold A while initially holding the mike and remote close together….

Basically, animated parameters needn’t have the same duration as the loops they apply to, and it should be possible to apply multiple animated parameters to a single loop or set of loops.  This should rapidly compose into brain-meltingly bizarre configurations of sound.  It might also make sense to add some kind of visual feedback for various parameters (e.g. use half-circles for fully panned sounds, etc.).  As with all Holofunk UI ideas, I don’t really know if this will work, but I do think it’s implementable and conceptually reasonably solid.

I also want to add scratching/sampling effects but I have not yet figured out how I want the interface to work, so I’m leaving that on the subconscious back burner until I get some kind of inspiration.  Just getting multiple animated, looped effects working will be plenty of amazeballs on its own!

So, Stay Tuned

It’s pretty mind-blowing that it was only a bit over a year ago that I was in Vancouver demoing all this to Beardyman.  It’s been a fantastic project so far and I expect it to become rapidly more so, as I get more features in and as I start collecting local technopeeps to play with it.  One of the other local loopers I demoed it to in June was Voidnote (and why did I not check out his Soundcloud before now?).  He actually wanted me to add guitar control to Holofunk, so he could use his guitar neck as a cursor, and use pedals in place of Wiimote buttons!  This is of course an awesome idea — how cool would it be to have a guitar/microphone two-player Holofunk jam?!  That’s on the radar as well.

My ultimate goal is to do a Holofunk gig at the annual Friends & Family rave campout.  I spent my twenties raving with that wacky bunch of Bay Area freaks, and I want to return as senior alumnus bearing live techno insane performance gifties.  That’s nine months away, and some of the hairiest features ever between now and then.  But if I can get two-player support by end of November, and animated Turnado support by end of February or March, and a couple more effects and maybe some video by June… then I’ll be ready!!!

Thanks again to everyone who’s enjoyed this project so far.  I’ve learned that one of my deepest satisfactions in life is working on a single project for years.  I’m loving raising my kids for that reason; my job remains excellent after four and a half years with no end in sight; and Holofunk is just getting rolling after 14 months.  Let’s see what the next 14 months bring!

Written by robjellinghaus

2012/10/18 at 22:41

Posted in Holofunk

Follow

Get every new post delivered to your Inbox.