A fond farewell.

So I donated a dining room table and chairs today. I know it doesn’t sound like much but it was a bit of a big deal for me. You see I’ve had this table for the last 11 years. It was has seen me through countless chapters in my life during that time.

I purchased it the summer of 1998 when for the first time I lived off campus and shared a tiny flat with my then SO and another couple. It was definitely one of those halcyon day sort of things; to paraphrase Mungo Jerry “It was summer time and the living was fine”. It was really the first piece of real furniture I purchased. Eli and I were driving back from campus through one the Southwest Portland’s darkly slithering roads and we came across a garage sale in its final exhale and in the midst of strewn books and bobs was a fairly decent looking vintage 70′s dining table. Silver legs and faux wood paneling over particle board, including a leaf and six chairs. The owners seemed deflated by the heat of the day and wanting to be done with it they sold it to us for the paltry sum of I believe $35. Eli and I rejoiced at our fortune while we crammed the lot into his tiny 3rd generation Honda Civic affectionately name Jenny.

That year we supped at the table, labored of papers and homework, entertained friends and family. It saw me through the vegetarian portion of my diet (oh what things we do for love). Come the end of the fall semester, my father and I packed it in a 20 foot U-Haul and drug it back to Sisters while I was studying in Ireland. And there it lived until I graduated in the summer of 2000, and once I moved to Beaverton, my parents returned it to me, glad to have reclaimed the corner of their garage again.

At this time I was living the “vida-bachelor loco” with Bryon, a short walk to work and shorter to the grocery store. It didn’t see much use besides the occasional meal when we weren’t crashed on the couches. It stood silently in the corner biding its time.

Then I packed it into another U-Haul and drug it to the North-end of downtown Portland where it stood in our overly trendy digs in the Pearl district amidst massive revitalization. It sat well and served its purpose for Ean and I as we waxed philosophically about our failed love lives and Ean’s 1000 origami cranes.

Then into my studio apartment, the first time living alone. Standing quietly by while I rambled about. I ate breakfast there when I heard about the planes crashing in New York, PA, and DC.

After I moved in with Craig and Sara it served as a desk while I wrote code and watched 3 movies every 2 nights during my binge on the fledgling internet sensation Netflix. I saw more movies supported from the table than I will probably see during the course of my life.

When I moved in with the future Mrs. Kire it returned to its proud place as a dining table. Serving family and friends for a few years, disguised under table cloths during murder mystery dinners, holiday parties, bridal showers and birthdays.

We purchased our current home 4 years ago and it stood in the tiny dining room. Carrie used it for her writing desk and we shared our simple meals there.

With “Wee B” coming soon, and Carrie sick of the site of it, her insistence wore me down and I capitulated into buying a new table. Something more pleasant to sit around, and probably better suited for her desk. I think now that in spite of my protests of honoring frugality and practicality it was really the history that I had with it that made me put up such a tough fight from losing it. It certainly wasn’t a work of art, but it has been with me for what seem like ages. Quietly and faithfully serving.

Are you there, 2? It’s me, Erik: Testing for Existance of Values in a Set

Barry brought an interesting point to me today. He needed to find if there existed any values for a particular set. For example this was his test dataset:


CREATE TABLE test (
id INT UNSIGNED NOT NULL,
foo INT UNSIGNED
);

INSERT INTO test (id, foo)
VALUES
(1, 1),
(1, 2),
(1, NULL),
(3, 4),
(3, 2),
(3, 3),
(4, 3),
(4, NULL),
(4, 5),
(5, 1),
(5, 1),
(5, NULL);

In this example he wanted to return all ids and if that id set contained any foos with a value of 1 or 2.

After bandying about for a bit he discovered that you can actually apply a limiter within an aggregate function.

So to find out which ids had a corresponding foos of 1 or 2 could be found from the following:

mysql> SELECT id, MAX(foo IN (1,2)) FROM test GROUP BY id;
+----+-------------------+
| id | MAX(foo IN (1,2)) |
+----+-------------------+
| 1 | 1 |
| 3 | 1 |
| 4 | 0 |
| 5 | 1 |
+----+-------------------+
4 rows in set (0.00 sec)

You could also apply a SUM function to determine how many results you actually had that matched your criteria.

Learning the hard way.

Database backups are not a difficult thing, but they are tedious and only important when there is a crisis at hand. Recently we ran into a situation where backups failed to protect us from some worst practices and we had to scrape together the binlogs and old dumps to resurrect the table.

What follows is a true story although the names and places have been changed.

At 13:45 a developer execute the following query

DELETE FROM table
return 159 rows effected

He intended to execute a SELECT * FROM table, and just remove the WHERE clause from a previous query but it turned out to be a DELETE FROM table WHERE ID = #.

As soon as the trigger was pulled he realized his mistake. I was brought in to resurrect the data because I previously performed all backups.

Off we go.

  1. Stop replication
  2. Flush logs
  3. if you can do the processing on the server wonderful, other wise copy the binlogs to a recovery system.
  4. mysqlbinlog -d mydb mysql-binlog.[0-9]* | sed '/table/,/;/!d' > recovery_file
  5. start replication

From the recovery_file we had a list of every statement issued against the table and we could replay all transactions to return the table to it’s original state with a little editing.

We are now reviewing our backup and recovery practices. I hope this was educational for you as it was for me.

I found the key point for the sed command in this document, and this tutorial helped greatly.

Beyond REST session

When we talk about modern web APIs we are mostly talking about REST(http://en.wikipedia.org/wiki/REST). In the last couple of years it has easily become more popular than similar layers like XMLRPC and SOAP. While these protocols work in a number of situations many of the popular Web Applications like Flickr have begun to look for methods that afford greater scalability than the those that use HTTP for data exchange.

Since HTTP is a stateless protocol it runs into problems when data changes occur. With HTTP the client must poll for new data, over, and over, and over, and over … Flickr threw up some numbers at the session that nailed home the challenges large web apps face. July alone Flickr had nearly 3 million requests from FriendFeed (a social content aggregator for your online presence), for 6000 users. Clearly this sort of traffic won’t fly as more people climb aboard.

The presenters, Evan Henshaw-Plath and Kellan Elliot proposed repurposing XMPP (typically an IM protocol) for the purposes of creating data streams for more efficient exchange. While REST is a constant request for new data, XMPP really is a subscription model, where a client establishes a session and listens for requested data. I don’t have their metrics handy, but the difference was remarkable, effectively shrinking the whole thing down to about 1% of the original traffic.

Their reason were:

  • persistent connections
  • stateful – less ID on each connection
  • designed to be event streaming protocol
  • natively frederated and asychronous
  • identity, security, and presence built in
  • jabber servers are built to do this

They also used PubSub (http://www.xmpp.org/extensions/xep-0060.html), a protocol extension, to achieve the subscription model, and OAuth for authentication.

While this had little to do with my work, it was fascinating to hear how larger enterprising handle the scaling issues that inevitably plague any popular web app. We have tried to do data streams ourselves in a couple of situations using Flash’s XML Sockets, which work well except 1. it’s in Flash and 2. it doesn’t route through firewalls easily. I don’t necessarily think Flash is bad, but it is difficult to support at times. I thoroughly enjoyed the talk. Evan and Kellan have a great presentation style that makes them easy to relate to.

A couple of miscellaneous notes:
The jid per user oauth authentication handles the identification of users.

XMPP Data service example

  1. hello world
    • jabber:simple
    • xmpphp for php
    • smack javae
  2. jabber account
  3. connect and send a message

Servers

  • ejabberd
  • djabberd
  • openfire/wildfire java server
  • tigase – java server

When setting up a server, turn off lots of features, for example turn off open subscribing. For this sort of data streaming there is little need for the multitude of features that XMPP offers to IM clients.

Build a component example

  • use a compnent XEP-0114
  • persistent talks over local socket
  • YAGNI: rosters. presence
  • Load balance

Starling – no idea what this was about

OAuth over XMPP – delgation of identification

Laika – Better EHR for all of us.

The medical industry has been struggling for years to determine how to exchange patient information in a uniform manner. Every entity has their own method for piecing together the mishmash of data they collect and filing it away for later use. Now, with governement agencies like Medicare/Medicaid (US) pushing hard on providers to fling themselves into the 21st Century and requires all providers to have EHRs by 2009 there has never been a more pressing need for consistent EHRs.

Enter Laika (http://laikaproject.org); a test suite for Electronic Heath Records (EHR). Laika’s purpose is to support the accessibility of data for providers and patients. Publishers of EHRs using the C32 specification must have their data exports certified by the CCHIT (http://en.wikipedia.org/wiki/CCHIT), which uses Laika for that exact purpose. Laika’s architecture consists of a Rails app up front with Java engines behind for the XML parsing. At this time they only support C32, but with other players entering the market they want to support the CCR standard as well and transforming between the two specifications.

The C32 specification is a combination of of both the CDA (Clinical Document Architecture) spec and CCD (Clinical Care Data). These standards are governed by HL7/ATSM (http://en.wikipedia.org/wiki/HL7), the industry gorilla when it comes to EHRs. A more recent cousin of the C32, the CCR (or Continuity of Care Record, http://en.wikipedia.org/wiki/Continuity_of_Care_Record), has struck the fancy of software behemoths Google in their Google Health (https://www.google.com/health/html/about/index.html) system, and Microsoft’s Health Vault (http://en.wikipedia.org/wiki/HealthVault). The main differences between the two are the document size and detail. As one would expect, the software companies prefer the lighter CCR standard while the industry backs the dense but more verbose C32.

This means that for my projects that as we continue to look at the clinical space we will have to eventually either consume or publish resident information in one of these two standards.

Additional Notes:
would be interesting to see what avenues are open to the consumer market. Google’s plan

NIST provides transform tools
HL7 has a style sheet

Drupal login and SSL

After the last article, I discovered that the Drupal framework, while it works perfectly within a SSL context provides little in the way of helping developers direct their users into the secure site. In our application we provide some content to the public at large, but we also require the majority of the site in a private context. This proved challenging because we didn’t want the expense of redirecting the common Joe Browser into the SSL when their casual experience wouldn’t require the overhead. Fortunately, after my last lesson in mod_rewrite I figured a clever approach to pushing authenticated users into the secure context.

RewriteCond %{REQUEST_URI} ^/user
RewriteCond %{SERVER_PORT} ^80$
RewriteCond ^(.*)$ https://example.com/$1 [L,R]

Since Drupal uses the same login section of the site, I was able to push the user under SSL when they went to log in. This doesn’t necessarily prevent them from dropping out of SSL when theyenter the site, but it does mean that they will end up in the right place at the outset.

Great minds, or how I learned to love mod_rewrite.

Today we rushed to push out another site for our primary customer’s new facility opening tomorrow. Since it’s under the same umbrella organization we thought it best to use the subdomain form of the URL, rather than shoehorning it under the primary www site. Of course, within 10 minutes of the new users visiting the site we were already finding users who could not help but add the “www” even though we never mentioned the prefix. So I decided that we should redirect the users to the “correct” form of the subdomain. After a little research, I discovered the joys of mod_rewrite solving this specific issue.

Most sites want users to use the nearly ubiquitous “www” prefix. As Jeff Atwood mentioned in his post on the topic. It has become so pervasive in the web sector that it is nearly a non-entity; if you mention it you are being redundant. As Jeff argues, with the power of mod_rewrite penetrating even the ivory towers of IIS with add-ons like ISAPI Rewrite, we as developers must shoulder the responsibility to conform to our users behavior. It is so simple to gracefully handle the requests that not being flexible for our users is the ultimate crime.

In my experiment I had to tackle removing the “dub-dub-dub” prefix and sending the user the the stripped URL without. In most circumstances, the site maintainer will add the following to either the httpd.conf or to an .htaccess

RewriteCond %{HTTP_HOST} ^www\.(.*)$ [NC]
RewriteRule ^(.*)$ http://%1/$1 [L,R=301]

The RewriteCond is effectively our if statement. If the requested host begins (^) with the world famous “www.”, followed by any number of characters “(.*)”, without regard to character case (“[NC]“), then go to our RewriteRule.

The RewriteRule then changes the whole mess into the previous argument without the “www” mess and informs the browser that the request is permanently redirected.

These two short lines change bizarre “http://www.kire.notneb.net” into the familiar “http://kire.notneb.net” once again righting the wrongs of “www” misuse.

I also found that even great sites like del.icio.us don’t permit the misdirection from “www”, which makes me wonder how far we have to go in our quest for better URL management.

A little productivity.

As much as I love Google’s Reader it is far to tempting a toy to leave just laying around to grab my attention at a weak moment.

To at least give me some sanity while I worked I wrote this little Greasemonkey script that blocks Greader during my productive hours, 9-12 and 13-17. When I try to go to the site I get redirected to a site that reminds me what I’m working on, in this case the Java Language reference. You can easily add sites to black list through the Included Pages in the Greasemonkey management window. And the script is so simple that anyone can modify it to fit their needs.

deathtogreader

iPod or go home?

So after we purchased Carrie her new laptop (hp 2130us), we found out that it came with a $50 gift card. So as a little Merry Christmas to me I thought I would pick up some variety of Digital Audio Player. Everyone knows the iPod is the heavy in making this choice, but I’ve never been a fan of convention, and while the support and standards that have come to surround Apple’s little darling, I’ve always been a bit repulsed by the entire thing. So the question is, where do I go from here. I’ve pretty much concluded that there are really three choices I can make.

First up there is of course the iPod. Everyone loves it, it’s everywhere, hacked and back, and has a great little interface.

Next we have the newest player on the block, Microsofts Zune. With WiFi, FM, and a giant screen this is a likely contender, except for the fact this is MSs first attempt in the DAP market, and they notoriously have problems with first generation hardware.

Then there is the Creative Zen Vision:M, similar to the Zune without the WiFi. Supports tons of formats and supposedly has a great screen.

Outside of the individual features my other requirements are something that will work under linux – preferably ubuntu, something I can use as a UMS drive.

Any thoughts?