robjsoftware.org

A blog about software – researching it, developing it, and contemplating its future.

Archive for June 2007

Open source update: Maven is not my friend

leave a comment »

Next in the Offline Blogger Editing saga. If anyone mentioned in this post would rather I shut up, please let me know.

Catch-22 with Maven: how can you create a project that has a dependency on another project that hasn’t been stored in some repository somewhere?

I’d love to land my GWT/JSF integration work. But the Ajax4JSF team really likes Maven for some reason. And my code relies on GWT 1.4, which isn’t final yet. So there’s no way for me to check in a Maven build that uses it, because there’s no repository in the world that has it.

Feh. I like what Maven is trying to do, but I would love there to be some middle ground here. Gavin King wants to do lunch next week to talk about the GWT/Seam stuff, though, which is cool.

Meanwhile, Rob Hanson is working on a GWT/Gears ORM implementation, and Ryan Dewsbury (author of the legendary gpokr.com, as well as a book about GWT applications) already has a GWT Blogger client written and working as part of his book. So it looks like gluing those two together would be an offline Blogger client in, like, no time. Ryan’s looking into how soon Addison-Wesley will approve the source for some kind of release, Rob Hanson’s going to have code working by Monday.

Given that the Maven stuff is slowing me down on finalizing the GWT/JSF stuff I did, it may be that Hanson and Dewsbury are going to put this offline Blogger-editor thing to bed before I can get my head out of my butt. Which would be fine — GO GUYS GO! There’s always more code to write, and no sense being selfish :-D

Written by robjellinghaus

2007/06/08 at 06:04

Posted in GWT, Java, offline apps

Blog dilemmas

leave a comment »

  • Many small posts or few large ones?
  • Why does Blogger do such a terrible job of scaling your profile picture?
  • Should I tell everyone on God’s green earth about my blog right away, or should I post a lot and then start spamming about it once I actually have some content?
  • How many people actually read through your back posts anyhow?
  • Given that there’s only so much spare time, how much of it should I spend programming, how much should I spend blogging, how much should I spend reading blogs, and how much should I play one of the dozens of computer games that stare at me longingly from the shelf, wondering where my love went?
  • Is this template just too ugly for people to stand for even one minute?
  • Why is Blogger’s edit window so d**n small?
  • Can I say d**n without the asterisks in this blog?
  • Is editing posts after they’re published bad form, even if you think of lots of things you forgot to say?
  • Am I interested in — and liable to blog about — too wide a variety of topics to attract any readership whatsoever?
  • Who is this blog for anyway, myself or everyone else? Is blogging like art — something that exists only in relationship to an audience?

Written by robjellinghaus

2007/06/08 at 05:45

Posted in blogging

Being Too Ambitious

leave a comment »

Sorry for the logorrheic posting flurry. I’ve been wanting to start a blog for years but for some reason never did. Now that it’s up, the dam has burst.

So this post is about my history as a programmer, and about some of the lessons I’ve learned in ambition and tackling the right problem.

First let me say that I’m hopelessly in love with programming. My basic definition of software engineering is “building machines out of ideas.” And that’s something that essentially has no bottom, and no top. There’s no limit to software.

That flexibility is also a trap, because in reality software has to be built on other software. And overreach is the ambitious engineer’s curse.

Let me give some examples from my own career. My first real post-college job, in 1990, was working on Project Xanadu. I’d read Engines of Creation in school, and it totally changed my life. I wanted to make hypertext happen, now.

So I flew out to California and started working for Autodesk with the Xanadu folks. What followed was one of the most exhilarating, and then one of the most disillusioning, periods of my life.

Xanadu was a glorious triumph of software design, and a consummate failure of software production. Picture it: the entire system was predicated on the assumption that hyperlinks would only work if they were fundamentally coupled to the individual textual characters that made up their anchor, such that any editing operations would exactly track which characters were linked to. The design sought to create a serverless, versioning system for generic hyperdimensional multimedia data, on top of which was an addressing scheme for arbitrary hyperlinks. All this was built on a super secret data structure, essentially a multi-coordinate splay tree (with adaptive optimization for all lookups). An object-oriented data store provided the persistence. A pipelining message protocol provided the networking And all of this was written in Smalltalk, compiled to C++ for efficiency.

IN 1991.

Do you remember how slow computers were back then?

The World Wide Web came along a couple years after the project finally fell apart, and took over everything. Because the fundamental assumption of Xanadu — that the critical problem was tracking all anchors across all content edits — was wrong. That problem wasn’t critical. And Xanadu’s now a footnote.

On the bright side, I have a lot of friends from that era, despite my hastily chosen and immature words in the Wired article that came out in 1993. Dean Tribble and Ravi Pandya now live in Seattle, working on the Singularity OS research project at Microsoft. Mark Miller is at HP now, thinking more deeply about secure mobile code than anyone. Life goes on and ideas flower in their time.

So from that, I decided to reduce the ambition level a lot, and went back to Autodesk to work on Autodesk Animator Studio, a Windows video paint package.

For Windows 3.1.

Anyone remember what watching video in Windows 3.1 was like? On a 1994 Pentium with 8 megs of RAM? Now imagine editing it.

(It looks like the only little video I saved from those days no longer plays in Windows XP. Sigh, bit rot creepeth ever unto the past.)

Yes, this could also be called overly ambitious. We did ship, but the whole experience drove my manager at the time, Jim Kent, out of the software biz for good. Fortunately, six years later, that turned out to be a very good thing for the public version of the Human Genome Project.

But hey, shipping was a big step forward!

A brief detour into charting software, and then I decided I wanted to get into the computer game business. So I worked for a small company named Protozoa in San Francisco. Protozoa was doing performance animation software, and wanted to make a game with their technology. I learned a lot about 3D and realtime programming there. I also learned that if you don’t have a total focus on what makes your game fun, your game will not in fact be fun. I ALSO learned that building a realtime 3D game on DirectX 3, in 1995, is not the best idea. Again, ambition bites, hard.

Years later I learned that one guy I worked with there, Bay Raitt, was destined for glory. Bay was a super genius character modeler. He was agonized by the problems of trying to build characters with only 200 polygons. Exactly how agonized I only realized years later, after seeing the second Lord of the Rings movie. I bought a book about how they created Gollum, and lo and behold, Bay was Gollum’s lead facial animator. In retrospect, seeing him work on a 1995 3D computer game was like seeing Picasso trying to work with a half-dry jar of finger paint.

Alrighty then! Time to move on from computer games, I thought. But the stings of Xanadu had healed somewhat, and I wanted to learn about networking. And a number of the Xanadu folks had re-convened in Cupertino, at a startup named Electric Communities. EC was seeking to take some Xanadu-era ideas about capability security and secure mobile code, and to build from them a distributed, peer-to-peer, cryptographically authenticated, secure-identity-supporting, commerce-capable virtual world.

Using Java version 1.0.

In 1996.

Are you seeing a pattern here?

EC failed for reasons similar to Xanadu: the software was too big, too ambitious, and too slow to make a compelling product. Not only that, but users didn’t want virtual worlds nearly as much as everyone thought. Years later, and with 21st-century hardware, only Second Life has somewhat cracked the nut, and even their infrastructure is suffering pretty deeply right now (word has it).

Again, though, I met a number of super smart people who are now doing great things elsewhere, including Doug Crockford, moderately (ahem) well-known in Web 2.0 circles. Also Arturo Bejar, security ninja at Yahoo; Brian Marick, a major testing god; and Chip Morningstar and Randy Farmer, also now at Yahoo, who have a quite candid assessment of the whole era on THEIR blog. And I got to spend the better part of a year figuring out how to create a secure, capability-based protocol for virtual-world object transfer between untrusting hosts. Getting paid for that? It was really, really fun. Until the money ran out.

EC was really the end of my over-ambitious days. In 1998, after EC, I went on to work for Helium, a now-defunct consulting company founded by an old Protozoa buddy (Gever Tulley) and a number of his oldest friends. We did a bunch of work for Quokka.com (a truly classic dot-com flameout story — they picked the wrong sports to cover and created all their sites by hand, but we had fun doing Internet livecasting of sailing races and the 2000 Olympics!). Then Quokka folded in 2001, Helium reformed, and we went on like that until 2003.

Finally in 2003, I begain working for my current employer, Nimblefish. A number of my old Helium and Quokka friends joined me there, along with other great folks. I’ve been there for three and a half years now, an all-time record for my career, and we’ve got a working production system that just keeps getting better and better. Hibernate, Flex, and other modern tech drives our code; we’ve got real customers, and we’re growing; and we’ve got a clear strategy that we’re executing on at full bandwidth. I think I might have finally kicked the curse of over-ambition…..

…or maybe it’s just dormant! :-)

I’ll close with the word that best captures how I feel about programming:

Onwards!

Written by robjellinghaus

2007/06/05 at 07:00

Posted in Uncategorized

Pluggable types, objects as services, and Google Gears

leave a comment »

Warning: rambling language research post ahead!

I admit it, I am a static typing weenie. Having gotten hooked on autocompletion, real-time code error highlighting, and advanced refactoring support — IntelliJ is my king — I find that it’s hard to imagine doing without.

But that’s not to say that type systems aren’t sometimes a real pain. Particularly ornery is all the machinery around making changes to the type system. Not all of us can be Scott Blum (GWT compiler ninja), and compiler hacking is not a well-known art. But without access to compiler internals, type-system hacking is very hard to come by.

Gilad Bracha, while he was at Sun, had some interesting things to say about pluggable type systems. Now, that paper is somewhat of a gripe fest, in that while it may be clear what he’s complaining about, it’s not entirely clear what to do about it. What does a pluggable type system actually look like?

I occasionally scour the recent proceedings of the ACM for interesting papers. (Any language lawyer who’s not a member of the ACM’s Digital Library is seriously missing out. Also, I’ve got to catch up on Lambda the Ultimate.) Recently in OOPSLA 2006, Andreae, Noble, Markstrum, and Millstein gave some real body to the theory with their paper on JavaCOP, a framework for building pluggable type systems in Java.

One of their examples is a @NotNull annotation to mark variables that can’t have null assigned. Check out their syntax for verifying that no bad assignments happen:

rule checkNonNull (node <<: sym) { 
where(requiresNonNull(sym)) {
require(definitelyNonNull(node)):
error(node, "Possibly null expression "+ node+" considered @NonNull");
}}

Working through this, this defines a constraint on all symbols that have the requiresNonNull property (which they implement with the @NotNull annotation on the symbol). All such symbols further require that the AST node assigned to them definitely has a non-null value. Their system implements all this as a layered extension to javac, that is able to check eight or so extended type properties in no more than double (at worst) the compile time.

Very interesting stuff, which points (to me, anyway) in the direction of generalizing the research to re-express Java's whole type system in rules like this. In other words, they're starting from Java and layering more type rules on top, but they could also work downwards and re-engineer Java's existing rules in their framework.

What would be the ultimate goal here? Well, giving more flexibility to juggle the nominal versus structural issues that Gilad talks about would be one reason. Another reason would be found in a more recent Gilad paper (apparently presented at Google last fall), about implementing offline objects that provide local support for centralized software services.

Now, if you read this paper after the recent shipment of Google Gears, it has all kinds of interesting ramifications. Gilad proposes a generic synchronization system implemented by reflective logging of local object modifications, supporting reconciliation on the server by change log replay. (This is actually very similar to the Dojo Offline model of synchronization, as presented at GDD07 last week. I wish there were a text post describing the model, because all these video posts and podcasts give me a pain. ) I'm not entirely convinced that model is as generic as he would like, and I will need to bang my head against data-based synchronization some more before I go to operation-based synchronization, but it's nonetheless thought-provoking.

Pluggable type systems come in when he discusses upgrading local code in a modular way. It looks to him like a fully static type system such as Java's has inadequate flexibility for changing shared type definitions at runtime. This is getting into the arena of major magic -- I'm personally very queasy about the whole concept of upgrading code modules that are in mid-execution. I'd prefer to treat that problem via an Erlang-like model of strict isolation through message passing -- the model that the E programming language implements. (And by the way, if you haven't bought into capability security yet, read one of Mark Miller's recent papers already and get with the program!)

At GDD07, I ran into Doug Crockford, who mentioned that he and Mark had discussed taking Google Gears' support for isolated worker threads and using it to implement securely encapsulated downloadable code. That squarely jibes with Gilad's recommendations, and is something that could be prototyped Real Soon Now. Of course, Gilad's whole pluggable-type machinery is definitely not on that agenda, but we must walk before we can run.

So where does this leave us? Seems to me there are several directions to pursue:

  • Gears provides a reasonably deployable framework for experimenting, not only with offline applications in general, but with an encapsulated framework for mobile code, and with the overall synchronization / update model Gilad discusses.
  • Pluggable type systems in general, along the lines of the JavaCOP system, provide a rich variety of new data points on the traditionally impoverished "static versus dynamic" axis in programming language design.
  • The right kind of type system to implement extensible, upgradeable local software objects -- particularly local objects with the ability to have their message formats, persistence structure, and inter-module interfaces -- is a huge open question, and pluggable type systems give us a much more fine-grained and flexible toolset for experimentation.

I find myself wondering what a JavaCOP framework built into the GWT compiler would look like....

(Does anyone have any links to work on JIT compilation for Javascript? If Javascript is moving in the direction of being a generic intermediate language, it'd be good to know how fast it can be made.)

Written by robjellinghaus

2007/06/05 at 06:13

Why Blogger Needs Offline Editing Support, and How To Build It With Google Gears

leave a comment »

I’m writing this in Notepad while riding the BART train on my normal morning commute into San Francisco. This commute takes about forty minutes each way for me. Since there’s no Internet, I can’t connect to work at all. Hence this is dedicated time for reading, or writing, or hacking on things that don’t need a full development environment (my laptop is a bit too old to cope with our entire code base from work anymore).

So of course blogging would be a natural. And sure, writing in Notepad is not torture. But it’s also not the right thing. What I want is to be able to write blog posts, edit them, save drafts, etc., etc., and then the minute I get back online, upload ‘em all to Blogger and worry no more about it.

To do that requires some kind of offline application with local storage. As my last post mentioned, I originally wanted to build such an offline app as a standalone Tomcat-based Seam webapp that would run in a local webserver. But Google Gears offers a much more direct route to the same destination.

So this post is a breakdown of how to construct such a client, based on conversations with various Googlers last week at Google’s Mountain View developer day.

The Requirements

Must-have requirements for an offline Blogger editor:

  • be a web application running in the browser (duh!)
  • support rich text editing at least at the level of Blogger’s own online editor
  • support authentication to Blogger
  • support local storage of some or all of the user’s blog posts
  • support creating, viewing, drafting, and editing of posts whether online or offline
  • allow synchronizing any offline edits to Blogger once back online
  • allow synchronizing of any posts edited on Blogger down to local storage
  • allow very minimal reconciliation of conflicting edits to posts on Blogger
  • require minimal installation of plugins or other application-specific software

Nice-to-have requirements for an offline Blogger editor:

  • support archiving / backing up of a user’s entire blog
  • support re-importing some set of content to a new blog that supports the Blogger API
  • support migrating content between blog services
  • support deleting postings ahd handling synchronization of offline deletes
  • provide a nicer browsing/reading UI (arguably Google Reader has this job sewn up already)

(Needless to say, the nice-to-haves don’t get done until all the must-haves are working!)

Explicit NON-requirements:

  • No need for this application to render any static HTML (the blog can presumably still be viewed and searched through Blogger itself; this app can be *just* an editor)

The API Implementation

Clearly this is exactly the kind of application Google Gears was created to support. So let’s assume we’re using Gears. Further, since I like GWT a whole lot, let’s also assume we’re using GWT.

I originally thought that this app could be built as a GWT/Gears webapp connecting directly to the Blogger API. Dan Morrill of the GWT team set me straight on that. The Blogger API is very pure in its REST-fulness, meaning specifically that it uses all the HTTP verbs. However, XMLHTTPRequest, the entire foundation of AJAX, can’t do anything other than GET and POST. So whatever we do is going to have to involve some kind of servlet bridge to the Blogger API.

(Possible workarounds for this: 1) get the Blogger and GData API teams to make GET/POST-only variants of their PUT/DELETE API methods; 2) get the browser vendors to implement PUT/DELETE support in XMLHTTPRequest. Clearly #1 is a lot more doable than #2 given the dreaded words, “legacy support”. You listening, GData team???)

Once we’ve accepted the need for some kind of bridge servlet, we’d better make sure that bridge servlet is fairly scalable and generic. Whoever is hosting this bridge servlet is going to have potentially a lot of traffic. So the bridge servlet needs to be stateless, re-entrant, and as close to a simple pass-through as possible. It shouldn’t have per-session state; it should be just a message relay.

We’ll also need some GWT modules for communicating with the bridge servlet, including beans that map to the data objects defined in the Blogger API, and service interfaces that map to the Blogger API’s methods. And in fact, this suggests that the servlet itself should simply use the Java implementation of the Blogger client API, exporting a GWT-friendly version of it (via GWT RPC) to the GWT client application.

The persistence implementation

Gears has this lovely local SQL database. But how do we get things in and out of it? We could write to the JDBC-like Gears SQL API. But I have tasted Hibernate. And if I’m programming in GWT, then writing JDBC-like code by hand would be like going back to the pre-Hibernate days. And I can’t do it. I CAN’T DO IT, I TELL YOU! I can’t go back. I WON’T go back!

Now when I first started brainstorming this with Ray from timepedia.org at GDD07, we got wild-eyed and started contemplating porting Hibernate to GWT. Saner heads rapidly prevailed. It turns out that Bob Vawter of the GWT team is doing work on a GWT-generator framework for generating a variety of useful artifacts (starting with Javascript-library method exports) from GWT beans.

So let’s use his framework, only instead of exporting to Javascript libs, we’ll make it generate dead simple CRUD operations for our Blogger beans to the Gears data store.

It looks like some other folks on the Gears list have had similar thoughts. Check out this Gears ORM implementation. But since that’s not based on GWT, it’s not directly applicable. Still cool though! But let’s assume we’ll proceed with Bob’s stuff, just for fun. This is all still early days and all experiments still deserve love.

The todo list

So, what do we have?

  1. GWT beans mapping to Blogger API data objects.
  2. GWT service interface mapping to Blogger API methods.
  3. Bridge servlet implementing GWT service interface and using Blogger Java client API implementation to relay messages to Blogger.
  4. Simple persistence code generator (using a GWT compile-time generator implementation) to take the GWT beans and store them in the Gears local database.
  5. Business logic to implement the client-side synchronization to the Blogger server. (That’ll be another post, or maybe lots of them!)
  6. GWT UI code to let the user use all of this infrastructural goodness to meet the must-have requirements!

If we get it right, it hopefully lands in the Google GWT API project. And once Blogger does this, Calendar is next, and then some.

Sounds like a fun project, huh? Interested? Chime in in the comments. Or join this GWT-Contributors thread on the topic.

Onwards!

Written by robjellinghaus

2007/06/05 at 05:33

Why start this blog?

leave a comment »

[ObNewBlogIntentions] Will try to update often. Will try not to spend so much time farting around with templates that I never get around to posting. [/ObNewBlogIntentions]

So, with that out of the way, hello. I’ve been planning a blog for a year and a half now. I realized in early 2006 that I’d spent my whole life programming for companies who owned everything I did, and I wanted to do something open source on my own time. So I thought, what about a blog system? Clearly the world needs more of those :-)

(The nice thing about programming on your own time is you don’t have to justify anything to anybody! Except your wife. But if you cut back on the computer games, so you’re not blowing your personal time budget away, then it’s all good.)

The thing is, I really wanted a blogging system that would support offline editing and offline backup of my blog content. I ride the commuter train, and I don’t entirely trust any single system (even one run by Google) — I like lots of redundant backups that I personally maintain.

So I started on a project to implement a distributed blog application. Which, of course, since I’m a Java weenie, meant that I really needed a Hibernate-based peer-to-peer distributed object version control system.

I built most of that over the bulk of 2006 (or prototyped it, anyway), using a Seam / EJB3 / /Tomcat / JSF stack. Then I thought, how do I build the UI? I started looking into GWT and liked it. I even found a JSF / GWT integration library. But it didn’t handle GWT RPC properly.

Then GWT got open sourced. I started working on fixing the RPC issues in GWT in early 2007, and that led to my submitting what I think was the first major external patch to GWT. I also put together a demo of Seam + JSF + GWT — adding a GWT blog-reading interface to the Seam blog example.

Michael Yuan from JBoss mentioned this at his JavaOne BOF this year, and I got to demo it at the GWT hack session at JavaOne, where I had the pleasure of meeting Bruce, Joel, Kelly, Bob, and the Dans (Peterson and Morrill) from the GWT team. Then I took a couple of weeks off to recuperate — my wife is expecting our second child in late July (our first child is two and a half), so there’s plenty of other distractions keeping me busy!

So that brings us to yesterday. Google Developer Day in Mountain View. And the announcement of Google Gears and GWT for Gears. Now, remember that my original use case was to build a blogging app that could work offline. I was using a technology stack that was Seam + EJB3 + Tomcat + JSF + GWT + Postgres. So to use this app I was building, you’d have to be running a local Postgres install and a local Tomcat install, and you’d have to do a bunch of Java deployment. I might have wound up with a few dozen users, or maybe only myself. (The code could have wider applicability, but direct product usage would’ve been low.)

Along comes Gears. Suddenly there’s a whole new stack option: Gears + GWT + Blogger. Offline functionality usable with any Blogger blog, and a much, much, much easier deployment, with a possibly much larger user base as a result.

So yesterday at GDD07 I brainstormed about this a bunch with Dan Morrill, Miguel Mendez, and Bob Vawter from the GWT team. It looks like a go. My next post will be about the detailed plan for that.

Meanwhile I still have to finish landing the JSF+GWT code, and documenting and submitting the Seam example. This whole project has been one surprise after another, and it’s only getting more fun so far :-)

Onwards!

Written by robjellinghaus

2007/06/02 at 03:32

Posted in GDD07, GWT, Java, Seam

Welcome!

leave a comment »

Hello folks. This blog has been too long in coming, due to many complicated reasons of my own self-inflicted devising. Better late than never. Enjoy!

Written by robjellinghaus

2007/06/01 at 08:10

Posted in Uncategorized

Follow

Get every new post delivered to your Inbox.