PatREST Enhancements & Documentation

A couple of notes regarding the status of PatREST.

First, two new functions have been added to the service. One provides access to tops (or popular items) lists. The other provides access to the new materials lists. I believe these are significant-enough additions to the service that they merit the 1.1 version number.

The top items query is scalable by result size and can be paginated, just like the search results. In addition, it can be scoped by material type: books, cd, dvd, or bocd (books on CD). When applied to the AADL XSLT, it looks something like this:

The new items query is similar to the top query in that it can be scaled by material type, size and be paginated. You can also search new items using their subject headings–useful for querying those new knitting books, Ed… It looks something like this:

Second, and probably more importantly, I’ve finally drafted a specification for PatREST which includes an explanation of it’s XML schema and some documentation for it’s various functions. It’s about time, I know. It can be found in the files section, or downloaded here. (PDF)

I’m not going to go in to too much detail about the two new functions in this post because the new documentation contains everything you need to know to get started with them.

SocialPACs, Community and… Sourdough.

An interesting, but low-key thread unfolded over at Panlibus earlier last week. I found it to be a good starting-point for a larger discussion about how Web 2.0 and Library 2.0 technology and software could come together in a cohesive manner, instead of the traditional ad hoc, piecemeal, vendor-driven method.

In response to Hennepin’s new commenting capability, Talis’s Paul Miller asks the question, “Participation is an important part of moving forward. How much better might shared participation be?” What he’s talking about is allowing other libraries to access Hennepin’s comments in an effort to provide a more enriching search experience beyond Hennepin’s OPAC, say, at Ann Arbor, or wherever.

What Paul goes on to propose in a follow-up post is a shared collectionof user participation much like the UK’s archival project. This would provide a central database and, presumably a set of web tools to access and interact with the data. Libraries anywhere in the world would have access to add and read content. It would be a shared, collaborative clearinghouse of participation.

I’m all for it–but with some caution. Isn’t that what Amazon is now? If you take away the e-commerce, Amazon is a collection of reviews, tags, and ratings on an insanely large amount of material. Interesting? Indeed. Useful? Of course. But I feel the need to point out that libraries are community-based institutions. They are supported by local taxpayers and are run, mostly, by members of the communities they serve. As such, wouldn’t we want any social element that is incorporated into our OPAC to reflect the tastes and opinions and personality of our community? I think so, and so does Ed Vielmetti:

…here in Ann Arbor there are a lot of book readers, and it’d really be rather nice to read comments from people who shared the same town with you. If I want to read random untrusted comments from people all over the world there’s already Amazon.

I mentioned some of this a while ago, though never specifically addressed the local vs. global social data repository idea. All this is not to say that in addition to community-driven social software we can’t access and make use of a shared data store. In response to a comment I made to Paul’s second post on this, Fiona Leslie made two very good points that she has seen come up repeatedly.

1. Libraries have reading groups and staff who create reviews, and there is no mechanism for getting those reviews “available” beyond posting them on notice boards, or perhaps on the library’s web pages. A few of them have an OPAC implementation that allows the reviews to be viewed in their OPAC.

2. Libraries have few or no reviews, but they see the value in having them, and would really appreciate a kickstart by having access to a shared repository of reviews created by other readers and staff in other areas.

Ok, so I’ll address each one separately. First, she is absolutely right, there is no de facto method of integrating participation in to our OPACs. This is compounded by the fact that our OPACs themselves tend to be unalterable beasts and we must rely on vendors themselves to make changes and enhancements to them. Many of you who follow what I write here know that’s a contentious issue for me, but I’ll keep my hackles down for now and simply remind everyone that this is another reason to demand a few basic rights from our vendors.

But even if we did all have unfettered access to our OPACs, or were resourceful and persistent enough to subvert the plain vanilla foisted upon us, what would a socialPAC look like? Fiona specifically mentions reviews, so lets stick with that for the purposes of this post. Actually, let’s not, because it doesn’t matter what the implementation looks like–that’s the fun part you and your development team get to mess about with when you do your redesign. What’s important is what we do with that review data after we get it and the value it adds to the process of searching for material. I’d suggest that the data be made available via two conduits. First would be the end-user interface. That is, the website or application patrons use to both consume and produce the content. How you weave this in to your OPAC is ultimately up to you (or… up to your vendor).

Richard Wallis weighed in ad responded to my comment. He writes:

So what are my assumptions then? Well firstly, the contributions of the citizens of Ann Arbor would be of great use, interest, and value to a far wider audience than just their district. Secondly, contributions to any global pool should be tagged as to their source and type. Thirdly, because of that tagging, selection of results should be able to be via many filters such as library, library authority or institution, library type, country, language etc.

So following through those assumptions in John’s situation, I would hope that contributions for my community would add value to the global pot; be displayable locally in isolation as a coherent set; and optionally could be supplemented by those from other appropriate communities around the country and the rest of the world.

I suppose I was a little unclear in my comment on Panlibus. I’d agree with Paul that, once the data is in, it would be nice to have a way to share it with other libraries. And I also agree with Richard that there is a place for supplementing existing data with a larger pool. In fact, I believe we have an obligation, as libraries, to do some manner of both. I envision Ann Arbor’s system providing a very lean web service on top of this entire system. Using this model, we will be able to share our community-driven social data beyond our borders. Libraries who do not enjoy the same community support that Ann Arbor, has will still benefit from the data. I believe this distributed approach to generating and maintaining socialPAC data will ultimately offer both redundancy and diversity. The thought of hundreds of libraries making their data available is certainly a more appealing alternative than that of the monolithic database. Metadata itself is an archive–it tells a story.

Fiona’s other point was that kick-starting a service may be difficult, especially in communities that are not likely to respond to and prime a service like this. Personally, I think we all might be surprised at the response that socialPACs will have with our constituents. Almost everyone has an opinion, and most people want to share it. That aside, however, Fiona is right. There will be cases where social software is not successful, popular, whatever.

So how do you evaluate your own situation with respect to social social software? Can your community sustain a socialPAC in perpetuity in a manner that will continually reflect a unique attitude and personality? If it can, how do you get it started?

First, you’re going to have to be honest with yourselves about the project itself. Do you want to pursue social software because it’s cool and hip, or do you really want to change the way your users interface with your collection in profound and personal ways while building a legacy at the same time? You can get a good feel for the level of Web 2.0 participation your community engages in by using existing Web 2.0 services which often let you dial in on specific locales. This may give you a good indication of whether a program like this might be a success.

If you’re convinced that your community will support a socialPAC, the next step is to come up with a a design and interface concept that will truly let your existing catalog shine while providing all the Web 2.0 immersion people expect. (easier said than done). This is where the innovators need to step in and start coming up with tangible examples of how this might work. I suspect that most libraries that do this will pursue a variation on a theme, but bear in mind that there are a lot of directions to take this stuff and in the end, it’s the one-of-a-kind feature that will give your OPAC its fingerprint. I suppose that vendors will dial in on the more popular and successful models and run with those. I have no problem with that as long as they adhere to the Web 2.0 spirit.

What about the initial “jump-start”? While I was writing this, my thoughts kept drifting to sourdough and I remembered a good friend of mine who, on occasion, liked to bake sourdough bread. It was this person who introduced me to my first sourdough starter. A shapeless blob that lives in your refrigerator and which, on occasion, you feed. At any rate, he was telling me how some sourdough starters have very rich and colorful histories because they have been passed down, literally, through generations. Some are closely guarded, while others have been disseminated and passed around liberally. It’s pretty fascinating.

But before I digress completely, Fiona’s concern about kick-starting can be addressed if we make our metadata available to systems starting up. Instead of one or two large repositories, however, wouldn’t it be great if we could choose from hundreds and all we had to do was send a request against a web service to get started? The tools are in place to allow this kind of interaction. All it takes is a willingness to communicate and share.

I suppose I may be searching for blue sky here, but Web 2.0 gives us a chance to do things properly from the beginning. Ultimately, the successful system will be rich with good data and useful to your patrons. The deeper significance of a unique repository will not emerge right away, but in time, you’ll see how data, like buildings themselves, can add to the legacy of a place. Make it available to the larger library community and we’ll see some very interesting things, indeed.

III XMLOPAC: findings, promise, and a little relief

Ryan Eby has done what III seems to not be able to do: Create a resource for XMLOPAC users. He's thrown up some wiki pages with the express purpose of documenting III's XMLOPAC. All I can say is, "Thank-you Ryan!" Be sure to participate and help with the documentation--we can all benefit from it.

In addition, Ryan has written up a couple great how-to's on getting item data and featured lists from the XML. I, for one, had no idea it was possible to grab featured lists this way. Ryan Eby has been documenting III's XMLOPAC for quite some time now and he's certainly one of only a handful of authoritative voices on the feature.

David Walker is another, and a very industrious voice at that. Today I had a good chat with him on the Code4Lib IRC channel after he showed me his totally amazing catalog based off his equally cool Shrew project. What he's been doing is exactly what I've been looking for. Even though the Shrew project is currently written in C#, he has plans to port it to PHP5, taking advantage of DomDocument. It's a project I'm completely willing to commit some time to myself, if he asks. The shrew project is "a system that converts the Innovative XML Server into an SRU/SRW and OpenSearch server." He's put a great deal of time into writing XSLT that will translate the III server's output into MARC-XML, Dublin Core, or MODS 3.0. My XMLOPAC class for PHP5 utilizes an older version of his MARC-XML XSLT, but I think when he pulls off this port, the need for my code will go away altogether--his way is preferable.

We also discussed some inherent problems with III's XMLOPAC--of which there are a number, and some potential enhancements. Chief among them would be the ability to conduct business--placing items on hold and such. A lot of work needs to be done, but I'm feeling much more optimistic about my OPAC aspirations now and, thanks to David and Ryan, a renewed sense of enthusiasm. Thanks guys!

[update] Ryan reminded me that Rob Casson is the individual who was kind enough to provide hosting for the wiki [/update]

[tags] OPAC, XMLOPAC, XML, XSLT, Library, Code4Lib, PHP, C#, SRU, MARC, DomDocument, Web Services [/tags]

Conversational Programming

Yes, this is a two-way title, referring to both today's SirsiDynix Institute talk I was lucky enough to be part of and the topic of mashups. Despite the fact that AADL and the surrounding area was under attack and I was disconnected from the data portion of the presentation for the duration, it went extremely well. As usual, I'm humbled by the articulate insights of Stephen Abram, Michael Casey, and Michael Stephens. If you missed it, be sure to catch the archive when it comes out later this week.
The Cathedral
A topic of discussion today was mashups. a mashup, for those who are unfamiliar with the term, is "a website or web application that seamlessly combines content from more than one source into an integrated experience" [wikipedia]. More than likely, you've encountered them already without even knowing they were mashups. These are bits of code that can allow you to either incorporate external data sources into your own site or, conversely, can make data streams available from your site that can be "mashed in" to remote sites. Recently, mashups have become a very vogue topic.

The first ever mashup camp drew to a close yesterday. It was the brain-child of David Berlind and Doug Gold. Essentially, It was a collection of mashup authors from around the country and, ostensibly, the world who gathered to share their creativity and brainpower. Notably, among them was Ann Arbor's Ed Vielmetti. He's reported back on the "camp's" progress--be sure to check out his blog.
The Bazaar
But what does this have to do with libraries, and why should we be paying attention to this? Well, beside the fact that mashups are the new, hot technology and we should be keeping up with all new, hot technology, mashups have enormous potential to redefine he library boundary both in terms of the technology itself and the people creating it.

Immediately, we can see the potential on our own sites by bringing in highly-polished, powerful tools in ways that enhance the information we already have to offer. A good example that Stephen Abram cited, was the ability to use the Google Maps API to provide very specific, very user-friendly directions to library branch locations. What makes mashups so exciting is that creativity and innovation are the key elements at play in the construction of these things.

The fact that new, high-level scripting languages and development engines like Ruby or Ruby on Rails, even, are being developed make the assemblage of Web 2.0 APIs a fairly easy endeavor. As a result, we're starting to see our patrons get into the groove as they begin to spin their own creations. Ed Vielmetti's Amazon mashup is a great example of this. He's written a Greasemonkey plugin that sneaks item availability into an Amazon record. The subversive nature of these things really tickles my fancy--it allows us, as end users, to do things that would mortify any sales team. We need to laud the use of our data wherever our patrons decide it should be.

Mashups are about individual empowerment. As libraries, we need to be able to step right in and lend tools to our users that will allow them to start creating these unintended uses. Again, this requires us to have... that's right, suitable APIs! The PatREST project I'm working on strives to do just that and I'm so grateful that Dave Pattern at Huddersfield has joined me. We've been able to create a bilateral push for this by producing near-identical results using two very different systems.

The mashup also poses some fundamental questions about the library's reach--where it begins (from the patron view) and where it ends (from our insider's view). By enabling users to spin our library tendrils into any place they like, we're creating a very ambiguous border on our OPACs, which, in turn, causes the entire ILS to recede into the background and play a significantly different role. Increasingly, it's just the business logic we want.

And so, as a whole new generation of Frankensteins are born, can you say that your ILS ready? Can you deal your data out under the table? With sleight-of-hand, we're going to make the library insidious.

[tags]Mashups, API, Web 2.0, Library 2.0, Programming, Coders, Superpatron[/tags]

Major enhancements for patron-REST

(Codenamed PatREST in my SVN)

When last I wrote about this, it was little more than a working proof-of-concept, but I've been working on it a lot this week--partly driven by DaveyP's experiments and Ed's tinkering, but mostly because I've been planning to do this for a long time now.

So, to cut to the chase:
PatREST now provides an easy, RESTful URI to do searches on quite a few fields. Essentially, you will plug the search key into the URI, followed by the search term. You can also request optional paging for keyword, author searches by appending hits-per-page and page-# to the URI. It looks like this:

http://www.aadl.org/rest/search/[searchkey]/[searchterm]/[hits-per-page]/[page-#]

The following search keys are available:

title
author
callnum
keyword
subject
gvtdocnum
stdnum (ISBN/ISSN)
titlekey
controlnum
barcode
record (Bib #)
bibnum (Bib # - Same as record)
itemnum

Go ahead and try this keyword search. It'll look something like this:

You'll notice that the XSLT stylesheet presents it nicely, all the data may not be displayed, but the XML is sound. Go ahead and view source to examine the schema. I've departed quite a bit from my original schema in order to provide a little bit of metadata for processing purposes. The XSLT allows you to click on a title which will take you to another RESTful record:

You'll notice that the URI for this record looks like /rest/record/1035670/ ... It's /rest/record/[bibnum]/. The XSLT allows you to click on either the title or cover image to go to the regular OPAC record. Again, view source for the XML schema. This schema has changed little and you'll notice that I'm taking advantage of OCLC's xISBN service.

The real treat in all this, however, is the ability to access your personal records RESTfully. To do this, I make use of the RSS token that is provided to every cardholder who has registered for an online account. This is the token that authenticates RSS readers against our system. It is a 32 character MD5 hash that looks something like "316928e0d260556eaccb6627f2ed657b". Accessing personal data is as easy as using the following URIs:

For checkouts, you would use /rest/checkouts/[token] (ie http://www.aadl.org/rest/checkouts/316928e0d260556eaccb6627f2ed657b). You'll then get output at looks something like:

For holds, the URI is /rest/holds/[token]. Output looks like:

Both of those results allow you to click on the bibnum and access the REST record for that item. Again, check the XML schema with your browser (no point in putting it here).

Please hack away at this and send me your comments/suggestions. You're absolutely correct that it doesn't adhere to any existing standard, but that's because I didn't want it to.

On a slightly less geek-oriented note...

I've gone several rounds with Talis's Richard Wallis before, so I want to call your attention to a post he made the other day in which he suggest that a) DaveyP and I collaborate and b) I/we consider using industry standards. I'd like to respond by saying that Dave and I have been communicating extensively. Ed Vielmetti has been involved as well.

Richard writes:

I encourage John, Dave, and those that follow them to take a look at these standards like we have

I can't speak for Dave, but I'm very familiar with these standards and so is Ed. I think Richard is completely missing the point of this whole project which is to put a friendly, accessible, and useful development interface into the hands of patrons. Ed offers just a small example of the type of features that existing standards don't readily make available:

Already there is a lot more useful function in it than you can get in SRU, e.g. permalinks for card records and a sensible way to get item availability.

Bear in mind, that comment was made prior to my work with RESTful holds and checkouts, which, as far as I know, don't even have a standard XML schema--I don't think it's been done before! (I'd love to be corrected on this) This project will also evolve to the point where holds can be placed RESTfully, items renewed, fines paid. It'll also extend to our other features like checkout history, wifi device management, personal card catalog management. Adding this functionality is not only going to allow the public to develop highly useful applications, but it'll be a framework by which we ourselves can build new services in-house. Vendor's obviously haven't stepped up to this, so we are--and we believe in our patrons. They deserve this kind of access to their public library.