(Codenamed PatREST in my SVN)

When last I wrote about this, it was little more than a working proof-of-concept, but I’ve been working on it a lot this week–partly driven by DaveyP’s experiments and Ed’s tinkering, but mostly because I’ve been planning to do this for a long time now.

So, to cut to the chase:
PatREST now provides an easy, RESTful URI to do searches on quite a few fields. Essentially, you will plug the search key into the URI, followed by the search term. You can also request optional paging for keyword, author searches by appending hits-per-page and page-# to the URI. It looks like this:

http://www.aadl.org/rest/search/[searchkey]/[searchterm]/[hits-per-page]/[page-#]

The following search keys are available:

title
author
callnum
keyword
subject
gvtdocnum
stdnum (ISBN/ISSN)
titlekey
controlnum
barcode
record (Bib #)
bibnum (Bib # - Same as record)
itemnum

Go ahead and try this keyword search. It’ll look something like this:

You’ll notice that the XSLT stylesheet presents it nicely, all the data may not be displayed, but the XML is sound. Go ahead and view source to examine the schema. I’ve departed quite a bit from my original schema in order to provide a little bit of metadata for processing purposes. The XSLT allows you to click on a title which will take you to another RESTful record:

You’ll notice that the URI for this record looks like /rest/record/1035670/ … It’s /rest/record/[bibnum]/. The XSLT allows you to click on either the title or cover image to go to the regular OPAC record. Again, view source for the XML schema. This schema has changed little and you’ll notice that I’m taking advantage of OCLC’s xISBN service.

The real treat in all this, however, is the ability to access your personal records RESTfully. To do this, I make use of the RSS token that is provided to every cardholder who has registered for an online account. This is the token that authenticates RSS readers against our system. It is a 32 character MD5 hash that looks something like “316928e0d260556eaccb6627f2ed657b”. Accessing personal data is as easy as using the following URIs:

For checkouts, you would use /rest/checkouts/[token] (ie http://www.aadl.org/rest/checkouts/316928e0d260556eaccb6627f2ed657b). You’ll then get output at looks something like:

For holds, the URI is /rest/holds/[token]. Output looks like:

Both of those results allow you to click on the bibnum and access the REST record for that item. Again, check the XML schema with your browser (no point in putting it here).

Please hack away at this and send me your comments/suggestions. You’re absolutely correct that it doesn’t adhere to any existing standard, but that’s because I didn’t want it to.

On a slightly less geek-oriented note…

I’ve gone several rounds with Talis’s Richard Wallis before, so I want to call your attention to a post he made the other day in which he suggest that a) DaveyP and I collaborate and b) I/we consider using industry standards. I’d like to respond by saying that Dave and I have been communicating extensively. Ed Vielmetti has been involved as well.

Richard writes:

I encourage John, Dave, and those that follow them to take a look at these standards like we have

I can’t speak for Dave, but I’m very familiar with these standards and so is Ed. I think Richard is completely missing the point of this whole project which is to put a friendly, accessible, and useful development interface into the hands of patrons. Ed offers just a small example of the type of features that existing standards don’t readily make available:

Already there is a lot more useful function in it than you can get in SRU, e.g. permalinks for card records and a sensible way to get item availability.

Bear in mind, that comment was made prior to my work with RESTful holds and checkouts, which, as far as I know, don’t even have a standard XML schema–I don’t think it’s been done before! (I’d love to be corrected on this) This project will also evolve to the point where holds can be placed RESTfully, items renewed, fines paid. It’ll also extend to our other features like checkout history, wifi device management, personal card catalog management. Adding this functionality is not only going to allow the public to develop highly useful applications, but it’ll be a framework by which we ourselves can build new services in-house. Vendor’s obviously haven’t stepped up to this, so we are–and we believe in our patrons. They deserve this kind of access to their public library.

You may have heard that Nintendo recently released the first games for the Nintendo DS that support gaming over a wifi connection. Unfortunately, because of the way our wireless authentication works at AADL, wifi devices require a web browser to be registered. This obviously shuts out the DS.

The solution was to build an advanced wifi management tool into our website that allows patrons to keep track of their wireless devices. This service is available only to registered cardholders and does not replace the existing registration process. It is an additional tool.

The existing process required users to associate the device with our SSID, open up a web browser where they are redirected to an authentication page and prompted to enter their library card number or guest access number (they get that from the staff in our computer labs). If you register or have registered using that method, your device will still show up on the wifi management page which looks like this:

You can then update the device description so you can easily identify which device is which:

The Nintendo DS is a great example of a whole new breed of non-traditional wireless devices that are coming to market. We can expect to see wifi VoIP phones, watches, cell phones, PDAs, other gaming platforms, you name it and they won’t have web browsers or any way to authenticate with our system. They shouldn’t be shut-out from using our wireless service.

If your library has open access without any authentication, these devices probably won’t pose a problem. But if you do have authentication in place, you may run into issues. Ask your IT departments if you’re not sure.

Well, it must be Hack-a-Library month because I was watching in real-time last night while someone set their (rather feeble) botnet to work trying to brute-force their way in to all of our externally-exposed SSH NATs.

It’s amusing to watch something try to brute force its way into a box that only accepts key-auth. This is what it looked like on our firewall:

Nov 16 23:11:00 sauron sshd[19323]: Illegal user sonny from 202.106.41.3
Nov 16 23:11:01 sauron sshd[19327]: Illegal user ronald from 202.106.41.3
Nov 16 23:11:01 sauron sshd[19325]: Illegal user hal from 203.197.97.165
Nov 16 23:11:01 sauron sshd[19326]: Illegal user isabelle from 203.197.97.165
Nov 16 23:11:02 sauron sshd[19331]: Illegal user sonny from 202.106.41.3
Nov 16 23:11:03 sauron sshd[19333]: Illegal user dujoey from 202.106.41.3
Nov 16 23:11:04 sauron sshd[19334]: Illegal user hal from 203.197.97.165
Nov 16 23:11:04 sauron sshd[19336]: Illegal user honey from 203.197.97.165
Nov 16 23:11:04 sauron sshd[19339]: Illegal user sonny from 202.106.41.3
Nov 16 23:11:06 sauron sshd[19341]: Illegal user dujoey from 202.106.41.3
Nov 16 23:11:06 sauron sshd[19343]: Illegal user hal from 203.197.97.165
Nov 16 23:11:07 sauron sshd[19344]: Illegal user honey from 203.197.97.165
Nov 16 23:11:07 sauron sshd[19346]: Illegal user joanna from 202.106.41.3
Nov 16 23:11:08 sauron sshd[19349]: Illegal user dujoey from 202.106.41.3
Nov 16 23:11:09 sauron sshd[19352]: Illegal user joanna from 202.106.41.3
Nov 16 23:11:09 sauron sshd[19351]: Illegal user hacker from 203.197.97.165
Nov 16 23:11:09 sauron sshd[19353]: Illegal user honey from 203.197.97.165
Nov 16 23:11:10 sauron sshd[19357]: Illegal user paul from 202.106.41.3
Nov 16 23:11:11 sauron sshd[19359]: Illegal user joanna from 202.106.41.3
Nov 16 23:11:12 sauron sshd[19361]: Illegal user hacker from 203.197.97.165
Nov 16 23:11:12 sauron sshd[19362]: Illegal user herbert from 203.197.97.165
Nov 16 23:11:13 sauron sshd[19365]: Illegal user paul from 202.106.41.3
Nov 16 23:11:14 sauron sshd[19367]: Illegal user jking from 202.106.41.3
Nov 16 23:11:15 sauron sshd[19369]: Illegal user hacker from 203.197.97.165
Nov 16 23:11:15 sauron sshd[19370]: Illegal user herbert from 203.197.97.165
Nov 16 23:11:15 sauron sshd[19373]: Illegal user paul from 202.106.41.3

and so on, and so on…

Simultaneously, our mail server was making friends:

Nov 16 23:11:02 opus sshd[12682]: Illegal user peaches from 202.106.41.3
Nov 16 23:11:02 opus sshd[12682]: error: Could not get shadow information for NOUSER
Nov 16 23:11:02 opus sshd[12682]: Failed password for illegal user peaches from 202.106.41.3 port 37726 ssh2
Nov 16 23:11:04 opus sshd[12684]: Illegal user merlin from 202.106.41.3
Nov 16 23:11:04 opus sshd[12684]: error: Could not get shadow information for NOUSER
Nov 16 23:11:04 opus sshd[12684]: Failed password for illegal user merlin from 202.106.41.3 port 37770 ssh2
Nov 16 23:11:07 opus sshd[12686]: Illegal user merlin from 202.106.41.3
Nov 16 23:11:07 opus sshd[12686]: error: Could not get shadow information for NOUSER
Nov 16 23:11:07 opus sshd[12686]: Failed password for illegal user merlin from 202.106.41.3 port 37816 ssh2
Nov 16 23:11:09 opus sshd[12688]: Illegal user merlin from 202.106.41.3
Nov 16 23:11:09 opus sshd[12688]: error: Could not get shadow information for NOUSER
Nov 16 23:11:09 opus sshd[12688]: Failed password for illegal user merlin from 202.106.41.3 port 37863 ssh2
Nov 16 23:11:11 opus sshd[12690]: Illegal user kayla from 202.106.41.3
Nov 16 23:11:11 opus sshd[12690]: error: Could not get shadow information for NOUSER
Nov 16 23:11:11 opus sshd[12690]: Failed password for illegal user kayla from 202.106.41.3 port 37907 ssh2
Nov 16 23:11:14 opus sshd[12692]: Illegal user kayla from 202.106.41.3
Nov 16 23:11:14 opus sshd[12692]: error: Could not get shadow information for NOUSER
Nov 16 23:11:14 opus sshd[12692]: Failed password for illegal user kayla from 202.106.41.3 port 37954 ssh2
Nov 16 23:11:16 opus sshd[12694]: Illegal user kayla from 202.106.41.3
Nov 16 23:11:16 opus sshd[12694]: error: Could not get shadow information for NOUSER
Nov 16 23:11:16 opus sshd[12694]: Failed password for illegal user kayla from 202.106.41.3 port 38002 ssh2
Nov 16 23:11:18 opus sshd[12696]: Illegal user russ from 202.106.41.3
Nov 16 23:11:18 opus sshd[12696]: error: Could not get shadow information for NOUSER
Nov 16 23:11:18 opus sshd[12696]: Failed password for illegal user russ from 202.106.41.3 port 38047 ssh2
Nov 16 23:11:21 opus sshd[12698]: Illegal user russ from 202.106.41.3
Nov 16 23:11:21 opus sshd[12698]: error: Could not get shadow information for NOUSER
Nov 16 23:11:21 opus sshd[12698]: Failed password for illegal user russ from 202.106.41.3 port 38093 ssh2
Nov 16 23:11:23 opus sshd[12701]: Illegal user russ from 202.106.41.3

That’s what it was like on every server with an SSH NAT. The attack lasted about two hours before the attacker got discouraged and gave up. Luckily it was only two bots. A significant attack could’ve amounted to a DDoS. As it is, those connections hang around long enough that I had to adjust the TTLs on the firewall just in case the attack was ramped up overnight.

This is very odd. I don’t see this kind of outright aggression very often. I wonder if my blog post had anything to do with it. Bear in mind that TLN’s hack job was against a Unix flavor too.

At any rate, it’s obvious that this is a script kiddie in charge of two bots. These two machines filling our logs are probably desktop machines, one of which is in China, the other in India. I guess China’s censorship firewall is only intended to discourage free thinking, not malicious behavior.

Anyway, be vigilant. Check your logs.

The Library Network (TLN) runs a service called MiLE which handles all of our inter-library loans and does the same for a good part of southeastern Michigan. Recently, they were targeted by a hacker who brought their system down. They were unable to recover from the disaster and restore service. Ed Vielmetti has posted about this with some links to more information.

I want to preface any comments I make by saying that there is no excuse whatsoever for what the hacker did to MiLE. It’s a case of malicious vandalism.

However, that doesn’t change the fact that there are malicious vandals. It also doesn’t change the fact that library systems contain sensitive patron information. TLN is responsible for two major failures.

First, TLN should have made sure that their systems were not vulnerable to an attack of this magnitude. I’m not familiar with their infrastructure, so I can’t comment on this without speculation. There are a good number of simple practical and procedural steps you can take to harden your network and it’s servers. These include using encrypted transmission protocols, segmenting your network behind a firewall, isolating critical servers, intrusion detection systems (an attack like this doesn’t happen without a fair amount of probing beforehand), keeping up-to-date on security patches, doing your own security audits, scans, attacks, etc. I’m fairly certain that there must have been a failure to do much of this on a scheduled or regular basis.

Second, and this is the real shame, is that TLN was unable to recover from this incident because their backup and recovery system either wasn’t in place or failed completely. If you’re operating without a viable disaster recovery plan, you’re essentially operating without a net. Anyone who is responsible for backups knows that part of performing this duty is to change tapes on a regular basis, perform check-sums on the tape data to make sure it’s good, and to backup often enough that your recovery window isn’t so large that it’s useless. At the very least, data that changed every day should have a full backup every week, with a nightly differential backup.

TLN is fully responsible for this failure, but at least they’re picking up and moving on in the best way possible. I give them a lot of credit for being totally transparent about this disaster and making the absolute best from a crumby situation. They will be upgrading to URSA 4.0 and will open with an expanded feature-set. I just hope that they’ve tackled their back-end issues.

My fear is that the problems that plague TLN are systemic in our industry, and I think they probably are. For instance, if you use telnet to access your ILS, your institution is at just as much risk as TLN. There are steps your network administrators must take to ensure the safety of your patrons’ information. When adult patron’s come in to get a new card, most libraries ask for their social security number, and they give it to us, freely and without hesitation. I would certainly think twice given what I know. I’d probably insist that it not be kept on their online system.

So, what are the steps you should take? At the bare minimum, all libraries should do the following:

1) Install a firewall and move every single server, computer, printer, fax machine in the entire organization behind it. If your machines are exposed directly to the internet, they are infected already. Every single windows machine on your network is a “bot”, or soon will be. When you move your machines behind the firewall, they will need to be wiped clean and a fresh operating system installed on them (unless they are macs or Linux boxes).

2) Install virus scanning software on your mail server/gateway. Ideally, you’ll use a mail gateway to vet all incoming mail. Prohibit all attachments with the following extensions: exe com bat vbs mp3 mpg avi pif bas and probably zip and rar.

3) Upgrade all your server software to the latest version. If you can’t do that, at least patch them with the latest patches. Set aside one day a week to check the patch-levels and install new patches where necessary.

4) If you have wireless or patron Ethernet access, separate it from the rest of the network. In addition, create a separate sub-net for all machines the public uses, ie OPAC, internet stations, print stations, etc.

5) Install intrusion detection systems on your firewall. If you have a Cisco router/PIX firewall, forward netflow traffic to Ntop and analyse it daily.

6) Insist on encrypted transmission protocols. Instead of telnet, use SSH. Use SSL on sensitive web pages. I had to insist that our vendors used SSH. This included our ILS vendor who wanted telnet access! Say, “No!” and leave no room for discussion. They can use SSH or nothing. Telnet is the protocol of death and the sooner you acknowledge it, the better.

7) Change your default passwords! You know, after Innovative finished their “implementation”, they were going to walk away and let us continue using the default passwords. Unbelievable. Unbelievable! So as a curiosity, I tried the default passwords on some other III libraries around the country. I tried 5 systems and got into 3. Holy crap. I could have perused patron information to my hearts content. Instead I logged off and emailed those sysadmins immediately.

I’ve heard the remark, “Well, we’re a small library, we only have one IT guy, we don’t have time to do all this”. My response to that is, “Then delete all your patron information now, because you’re not responsible enough to have it in your possession”. Libraries are among the most insecure systems in the country, and that’s shameful.

In my presentation at IL2005, I tried to stress the importance of infrastructure redesign. This is why, right here. All the fun, new things we want to do go right out the window the second a patron’s identity is stolen because we let someone walk right in and take it away.

Glenn Peterson (Hennepin County Library) just sent over a copy of the powerpoint slides from our web panel discussion at IL. The discussion was called “Web Trends and Innovations”. It included myself, David King from Kansas City Public Library (check out their catalog sidebar, by the way… I really want to do something similar.), Sarah Houghton (Librarian in Black) from Marin County Free Library, and, of course, Glenn Peterson.

Some good ideas were discussed. I’d reccomend a quick look through the slides if you’re wondering what you can do at your library to garner a little community interaction.