Wednesday, 28 January 2009

Top-down classification and its detractors

'Hanging together' blog reports on an unpopular move from ERIH:


I agree absolutely with the dissenters, but maybe for different reasons than they do. I have two concerns: firstly, that the three levels are too blunt an instrument, and secondly, that information workers/librarians aren't the right people for the job. 

It's not (or at least, it shouldn't be) the job of any group of information workers to act as arbiters of knowledge value - to tell users what's valuable. It's up to information professionals to facilitate the ways for us to tell them. To tell them, and the rest of... well, us, but that's getting convoluted.

Herein a great paradox of being an information worker: the information worker is never an expert in the information itself. The users are. The information worker manages the system in which the information lives, but the information itself... that's different. Why should a librarian be an expert in the actions of neurotrasmitters? They shouldn't. Why should a neurologist be an expert in taxonomy management? They shouldn't. 

So why should an information worker be in charge of saying what pre-defined group a journal they haven't actually read falls into? They shouldn't. 

An information management system works best when the two sides - information workers and information users - are in collaboration. And what a beautiful age this is that that has finally become possible. 

To demonstrate: instead of three groups, you could assess broader, deeper, and more responsive (the status of journals does change, after all) user-generated information. Journal authors and journal readers are, after all, usually the same people. And no, I'm not talking about out-of-five ratings or 'people who read this journal also read'. How about: 
  1. How often is the journal cited in other journals?
  2. What rank of universities do the authors come from?
  3. How often have articles from this journal been downloaded from the database?
...and so on, and so on. Data like that, generated by users and collected/interpreted by information workers, especially when applied over time, could create a really useful guide to the quality and utility of journals. Dropping a journal into one of three rankings just can't compare.

Monday, 26 January 2009

Sneaky, sneaky, sneaky...

...but such a clever idea.

Google has invented a game to get people to label images with useable metadata. (Worth mentioning: the reason why google image search isn't as good as regular google is that regular google searches text content - and images, well, they don't have any.)

The game is this: you and another player are given a set of images, and you guess tags that you think the other person might use. When you both use the same tag (e.g. it's a picture of a tree, and you both say 'tree'), you get a new image. The agreed tag is added to the google database. Neat, isn't it?

Unfortunately (predictably) some players are using the game to see how many pictures they can get tagged with a racial epithet. Yes, that one. You have been warned. 

Sunday, 25 January 2009

Grep, the best metadata tool ever

Ah, UNIX. So handy, so terrifying.

There's something about a command line that strikes fear into the heart of everyone who grew up in Windows. That black box, that white writing, that awful font... it's like standing outside an abandoned factory with a 'WARNING: DANGER OF DEATH' sign hanging askew from the locked and rusted front gates.

And while I appreciate the efforts of the good people who take the time to write lovely tutorials, most of them are too damn passionate to make sense to bewildered newbies like myself. (I do recommend, however, this one if you're interested/bewildered: http://www.nceas.ucsb.edu/scicomp/Dloads/UnixProg/UnixTutor/index.html)

Getting started in UNIX requires a little brain-breaking, a slight tweaking and rejigging of what you think you know about computers. I'm not going to do that today. So, here is a mechanics-only guide to Doing One Handy Thing. Just one - nothing fancy, just handy. Specifically, extracting information you want.

Don't try this at home. You'll only delete your wedding photos. Also - sys admins know all about this and will usually help in exchange for food.

Using grep

Here's what we want to do: We want to go through a file of book metadata and find all the things that were published in the 1990s.

Step 1: For Goodness sake, if you're doing this for real, make a backup/test directory to do this in. NEVER MESS WITH YOUR ORIGINAL FILES. Seriously.

The scenario is this: Unix works in folders, so we're going to put the files we want to use in the same folder. There are two files we need:

The metadata file (metadata.txt in this example) 
The file that has in it the terms you want to pull out (selections.txt in this example).

The metadata file

Here's a sample to get you started. It's not fancy, but if you copy it and save it as .txt, it will work. (Note: it's tab separated, which is the best way unless you're messing with XML and things like that.)

Title Author YearPublished
The Arrival Sean Tan 2006
Introducing Baudrillard Chris Horrocks 1999
Moab is my Washpot Stephen Fry 1997
The Unbearable Lightness of Being in Aberystwyth 2005

The selection file

Unix works line by line, so the selection terms should be one per line. We want to find records from the 1990s, and we know how that info is expressed in metadata.txt, so, the content of the file looks like this:

1990
1991
1992
1993
1994
1995
1996
1997
1998
1999

Step 3: Logon to the unix box ('box' meaning 'computer' here). Ask your sys admin how.

Step 4: Change directories (directory = folder). In windows, this is the bit where you double-click the folder and go into a sub-folder. In Unix, it's all text-based, so we use 'cd'. 'c'hange 'd'irectory, see?

In Unix, most symbols mean something. Spacebars are crucial, they tell the machine where commands and filenames start and stop (so don't have spacebars in your filenames or directories or this wont' work. I've written [spacebar] where the spaces go in these instructions to make it clear). 

Having logged on, we are faced with a prompt, which is usually a percentage sign (%). We want to get to our directory, so we type:

%cd[spacebar]directory_I_want/subdirectory_I_want/directory_where_my_files_are

(If you get lost, you can go backwards by typing %cd[spacebar].. You can also find out where you are by typing in %pwd. Extra handy: if you start typing the directory and press 'tab', it will autocomplete. If it doesn't, you're probably in the wrong place.)

(It might also help to follow your location in a normal window, so you can see where you are and what's going on in a way you're used to. That's what I do.)

Step 5: OK, technically getting complicated but mechanically pretty simple. Type this:

%grep[spacebar]-f[spacebar]selections.txt[spacebar]metadata.txt>selected_metadata.txt

'grep' is the command (acutally, it's called a program in this environment) that means 'extract'.
'-f' means 'extract the stuff listed in the file I'm about to tell you about'
'selections.txt' is the file that lists the stuff you want extracted
'metadata.txt' is the file that you want to extract the stuff from
'>' means 'when you've got an answer, send it to the file I'm about to tell you about'
'selected_metadata.txt' means 'send the answers to this file please'

So, when you go and open up selected_metadata.txt, you find this:

Introducing Baudrillard Chris Horrocks 1999
Moab is my Washpot Stephen Fry 1997

Ta da!

That's a basic example, but it has some huge applications.

Here's some final points:

1. Notice we lost the header line, which had Title/Author/DatePublished in it. That's because it didn't have a 1990's number in it. 

2. No, you don't have to use the -f option. It's just the handiest way. If you only wanted one term, 1997 for example, you could just use: 

%grep[spacebar]1997[spacebar]selections.txt[spacebar]metadata.txt>selected_metadata.txt

and that would output the Stephen Fry record.

3. No, you don't have to output to a text file - if you want the results to display on screen, you can use this:

%grep[spacebar]-f[spacebar]selections.txt[spacebar]metadata.txt

Bear in mind, though, that you can't do much with the output later. Reading it on screen is about all you can do with it. 

Does it get more complex than this? You betcha. You can search and replace inside files, all sorts of things. But at the starters level, this is still a pretty powerful way to start messing with your data.

Friday, 9 January 2009

Digression: the demon MARC

MARC stands for MAchine Readable Cataloguing. The cheapness of the acronym should tell you something about the quality of the standard.

Many well intentioned people (and many charlatans) try to convince us that you can learn more from failures than from successes. I disagree. There are, however, some lessons that can be learned from the dismal & crumbling institution of MARC.

Lesson one: when it was invented, MARC was excellent. 

That, however, was in the early 1960s. This is not.

Today's theme is obsolescence. Technologies that don't die at the appropriate time are frustrating, dangerous, and impede development of new and better technologies.

This is what a MARC record looks like: 

001 4520371

005 19990823210448.0

008 990108s1999 cou b 001 0 eng

035 $a(DLC) 99011493

906 $a7$bcbc$corignew$d1$eocip$f19$gy-gencatlg

955 $apc14 to la00 01-08-99; lj11 to subj. 01-11-99; lj07 01-11-99; lk02

01-12-99; CIP ver. lh04 to SL 08-03-99

010 $a 99011493

020 $a1563087723 (hardbound)

020 $a1563087022 (softbound)

040 $aDLC$cDLC$dDLC

043 $an-us---

050 00$aZ675.S3$bW8735 1999

082 00$a025.1/978$221

100 1 $aWoolls, Blanche.

245 14$aThe school library media manager /$cBlanche Woolls.

250 $a2nd ed.

260 $aEnglewood, CO :$bLibraries Unlimited,$c1999.

300 $axiv, 340 p. ;$c26 cm.

490 1 $aLibrary and information science text series

504 $aIncludes bibliographical references and index.

650 0$aSchool libraries$zUnited States$xAdministration.

650 0$aMedia programs (Education)$zUnited States$xAdministration.

830 0$aLibrary science text series.

985 $eGAP

991 $bc-GenColl$hZ675.S3$iW8735 1999$oam$tCopy 1$wBOOKS




I would strongly recommend not trying to understand any of that. The thing to notice is that it's ugly. Ugliness matters in metadata design, as in all design - as a general rule, ugly things don't work properly. (Elegance is no guarantee of functionality, but it's a damn good start.)

It's not MARC's fault. In fact, MARC was amazing for it's time. It was the first ever effort at capturing reusable cataloguing information. (Well done libraries - another first!) But not even a librarian-genius could have forseen the type of acrobatics we ask of our information today, and so, MARC is barely capable of sitting down and standing up again. 

They call it 'the curse of the innovator'. Whoever innovates first is, inevitably, saddled with the oldest & clunkiest system in the long term.

One big problem is that when MARC was developed, disk space was at a real premium - that's why we have '245' as a field and not 'Title Statement' - '245' is shorter and easier to store. Similarly, 245$a is 'Title'. This contraction has a nasty knock-on for human readability, which is irreplacable in metadata management.

The 245 subfields alone carry a lot - to get a feel for it, have a look at the full outline:
 
First Indicator
Title added entry
0 - No added entry 
1 - Added entry 

Second Indicator
Nonfiling characters
0 - No nonfiling characters 
1-9 - Number of nonfiling characters 


Subfield Codes
$a - Title (NR) 
$b - Remainder of title (NR) 
$c - Statement of responsibility, etc. (NR) 
$f - Inclusive dates (NR) 
$g - Bulk dates (NR) 
$h - Medium (NR) 
$k - Form (R) 
$n - Number of part/section of a work (R) 
$p - Name of part/section of a work (R) 
$s - Version (NR) 
$6 - Linkage (NR) 
$8 - Field link and sequence number (R) 


Awful.

For me, the real problem with MARC can be summed up by looking at $c. 'Statement of Responsibility', it says. Not author. 'Statement of Responsibility'. Author? Editor? Contributer? Complier? Composer? Who knows? But with all the detail that is contained in the record, you can't help but assume that all the useful information must be there somewhere. 

I spent a long time with MARC, and I can't remember if it's there. Leading us to a general rule of data: If it is there and you can't find it, it might as well not be there at all. 

Herein the danger of bad cataloguing: systems that look specific, but actually aren't. This is not. The less specific, the less useful, and the more prone to misinterpretation - by humans and machines alike.

(Very brave people are invited to delve into the mysteries of control field 008 - http://www.loc.gov/marc/bibliographic/bd008.html - but don't say I didn't warn you).

So - there's MARC. Clunky, unfriendly to human readers, outdated, inflexible, insufficiently specific. Also: Old. So why bother complaining about it?

Because it's still out there, that's why. That's a problem. Outdated combined with obsolete throws up all sorts of awfulness, on which more later.

Sunday, 21 December 2008

10: Find and Replace

Here is Textpad's 'find and replace' box. 

We open it by using F8.



You'll notice that most of the options are the same as ''find' - see post 7 for an explanation of these. There are some differences: 

No wrapped search. 'Find next' only works downwards to the end of the file.
Selected text. You can highlight some text, and make changes in the highlighted text only. 

What we're interested in here are the Go buttons. Once you've typed what you want to find into the 'Find' box, choose between:

Find next Finds the next occurence of what's in the 'find what' box
Replace Replaces the current selection with what's in the 'replace with' box
Replace next Replaces the next match of the 'find what' box' with what's in the 'replace with' box
Replace all Replaces every single match of the 'find what' box' with what's in the 'replace with' box

You will almost always use Replace all.


Monday, 15 December 2008

9: Using 'Find' - the magic of 'Mark all'

If you scroll back a way, you'll find my unbelievably bad spreadsheet of 'books on my coffee table'. Well, here it is again, this time to show the power of 'Mark all'.

We open the find window with F5. 



So: I've looked for books with the publisher 'Penguin'. (Note that I've used \tPenguin\t, which means 'tab Penguin tab'. Otherwise, I might accidentally mark a book with the title 'I love my penguin', which isn't what I want.) 

And there they are - two books which match \tPenguin\t, marked with a little triangle called a 'bookmark'.

Now I want to do something with these bookmarked lines. Here's what I can do:

Cut the bookmarked lines 
Copy the bookmarked lines 
Delete the bookmarked lines

I want to copy them to a new document to see them more closely. I go here:



I open a new document, and I paste my bookmarked lines:



It turns out that both my Penguin books are by PG Wodehouse. Now I can look over the data, modify it, whatever I want. 

The only thing I can't do is modify the data and put it back in it's original place. Bookmarking only works in one direction. If I wanted to modify the records and then put them back in the original file, I would have to cut them from this document and paste them onto the end of the original.

Thursday, 11 December 2008

8: Find

If you're used to excel, or word, or any normal 'find' system, you're used to the idea that 'find' and 'find and replace' are pretty much the same thing. In textpad, they're not. 

F5 opens the 'find' window
F8 opens the 'find and replace' window

First up: Find. The find window looks like this:



Let's look at the options here:

'Text'. 'Hex' means 'Hexadecimal', and is advanced stuff. Always select 'Text'.

'Match whole words' means exactly that - match sets of characters that end with a space, a full-stop (or other punctuation), or a tab. This is very handy - it means that if you search for 'cat' it will find the exact word 'cat' and not 'concatenate'. Leave this box ticked.

'Match case' means exactly that - it will match capital and lower-case letters. This is handy if you want to find March (the month) but not march (walking in formation). It will mess up your results if you leave it on - a search for 'Animal' will not find 'animal' if this box is ticked. Usually, leave this box unticked.

 3. 'Regular expression'. When you check this box, you turn regexes on and off. If it's ticked, \t means 'tab'. If it's not, \t literally means \t. If your regex doesn't work, it is often because this box is not ticked. Always make sure this box matches what you want to do.

4. 'Wrap searches'. Textpad starts finding from the cursor mark and stops at the bottom of the document, which can be annoying. If you tick this box, it will go to the bottom, then back to the top and start again - which means the whole document gets searched. Leave this box ticked. 


Direction: Up and Down. If you really only want to search in one direction from the cursor point (i.e. not a wrap seach), you can choose the direction here. These are used infrequently.

Extend selection and In all documents. You can search is selected text, or in more than one open document, using these buttons. These are used infrequently.


...and now, the 'Go' buttons:

Find next means 'show me the next match after the cursor' (easy)

Mark all means 'put a marker next to all the matches in the file.' 

Mark all is super handy. Super, super, super handy - I cannot stress enought how handy it is. Read all about it in the next post.