10: Find and Replace

Here is Textpad's 'find and replace' box. 

We open it by using F8.

You'll notice that most of the options are the same as ''find' - see post 7 for an explanation of these. There are some differences: 

No wrapped search. 'Find next' only works downwards to the end of the file.

Selected text. You can highlight some text, and make changes in the highlighted text only. 

What we're interested in here are the Go buttons. Once you've typed what you want to find into the 'Find' box, choose between:

Find next Finds the next occurence of what's in the 'find what' box

Replace Replaces the current selection with what's in the 'replace with' box

Replace next Replaces the next match of the 'find what' box' with what's in the 'replace with' box

Replace all Replaces every single match of the 'find what' box' with what's in the 'replace with' box

You will almost always use Replace all.

4. Regular expressions: introduction

'Regular expressions' is a difficult name for a useful tool. Basically, they're very small computer programs. A single regular expression (or 'regex') is a little sequence of special characters that match a pattern in a file.

What we're dealing with here is 'find' and 'replace'.

Consider this little piece of text:

I have a cat because I like cats. I can never catch my cat.

If you wanted to change that to 'dog', in Excel, or Word, you could. You find 'cat' and replace it with 'dog'.

Find: cat

Replace: dog

Output: I have a dog because I like cats. I can never catch my dog.

Excel and Word will skip the word 'cats'. These programs know that you're unlikely to want to change every sequence of c-a-t, so they assume you mean "find 'cat', but only when it's definitely one word and not part of another word, like 'catch' or 'replicate' or 'subcategory'". Fair enough. Easy, too - after all, you can just do a second find-replace for 'cats'.

Textpad isn't that generous. It assumes that you mean exactly what you say, every time, with no exceptions.

In textpad, this would happen:

Find: cat

Replace: dog

Output: I have a dog because I like dogs. I can never dogch my dog.

The obvious response to that example is: That's really stupid.

Absolutely it's stupid. But here's the thing: Finding 'cat' and replacing it with 'dog' is not a regex. (As a general rule, the less text there is in a regex, the better). Regexes are not about text, they are about patterns, and the patterns are expressed in metacharacters.

A metacharacter is a single keyboard-stroke that refers to a type of character, not the character itself.

Here are some examples:

. means 'any character'

? means 'zero or one of the character before the ?

Don't bother trying to understand the next bit. The point of this example is not to show how it's done, only that it can be done. (The red is just to show which bits are doing the work).

Find: cat\(s?\)\>

Replace: dog\1

Output: I have a dog because I like dogs. I can never catch my dog.

And there we go. All the changes have been made accurately and cleanly in one move.

3. Tab separated values

Metadata is usually stored in Tab Seperated Value (tsv) format.

Don't be worried by that - you've already been using it for years. You may never have known it, but MS Excel is in tsv format.

All tsv means is that the fields of information are seperated by tabs. If you're in an excel spreadsheet, and you press 'tab', you move across by one cell - the software knows that 'tab' means 'the next field'.

There are other ways to seperate units of information: commas, semi-colons, colons, all sorts of things. The handy thing about tabs is that, unlike commas, semi-colons, and colons, they tend not to be used in natural language. You might have a book title like 'My life: the story of my life', but you'll never have 'My life (TAB) the story of my life'. That makes tsv very handy.

('Natural language' is any language that isn't a computer language. English, Spanish, Sign language... anything that people use to understand other people. Computers are terrible with natural language, which causes big headaches for the regex user. We'll cover that in more detail later.)

Look at this excel spreadsheet:

...and this textpad document:

The textpad document looks unintelligible, but it isn't. It is exactly the same document as the excel spreadsheet. In fact, all I did was copy & paste between the two - there are no changes at all. This means two things:

1. The tsv format is easy

2. If you ever get lost in textpad, you can pop your data back into excel to see what's going on.

(Sharp eyes will have spotted that this is a terrible, terrible piece of metadata. All I've done is pull some books off my bookshelf and pop them in a list. It's not standardised, the first name and surname are in the same column, the capitals are all over the shop, and my id column is based on the order I grabbed them. There's a reason for this - we're going to fix it.)

2. Textpad is magic

First things first: computers need software.

We tend to think that complex processes require complex software. That's our general experience: Photoshop will do a better job of manipulating your pictures than the free little Paint application that comes with Windows will.

The rule, however, doesn't always hold true. To use regular expressions, you do not need a whizz-bang piece of software. In fact, the less whizzing and the fewer bangs the better. What you need is something stripped down to bare bones.

The reason is this: control. If you are using regexes, you never, ever want the computer to second guess you. This is an environment where you want hte computer to do exactly as it's told, no more, no less. Enter the text editor.

Text editors do just that: they edit text. They don't format, they don't (usually) do fonts. MS Word is not a text editor. (Anyone who has been annoyed by Word trying to predict their formatting or telling them their grammar is bad will already understand the 'control' advantage of slimmed-down software.) As a rule, programmers don't use fancy software - most of them use text editors.

There are a lot of text editors out there, all of which are good for different things (usually programming related). For metadata manipulation, however, there's only one even worth looking at: Textpad. It's cheap, it's fast, it's reliable, and it handles metadata impeccably. And as you can see, it doesn't even look scary - the layout is just like a slimmer version of Word.

www.textpad.com

Unfortunately, Textpad is only made for PCs - I don't know any viable alternative for Macs, but if you do, please let me know!

This blog is written specifically for textpad users. You'll find following it very difficult if you don't have a copy.

1. What Regular Expressions do

These tutorials are intended as a guide for people who wrangle metadata.

More specifically, they're aimed at people who wrangle tab-separated metadata and have noticed that excel has it's limits.

The name 'regular expressions' (or 'regex') isn't particularly intuitive, and at first sight, regular expressions themselves look a lot like goobledegook.

They do make sense though, and with a little practice, can be very, very powerful.