rename

A small script triggering a configuration question.

GoGraz

2019-09-09 GoGraz Lukas Prokop

About

  • Given a bunch of files of similar name
  • How can I rename them easily?
  • Very common task!

Larry Wall's rename

rename 's/[.]jpeg[0-9]*/.jpeg/' *.jpeg[0-9]*

bash

for n in *.jpeg[0-9]* do
  mv -i "$n" "${n%%.jpeg[0-9]*}.jpeg";
done

zsh

autoload zmv
zmv -n '(*.jpeg)<->' '$1'

Approach

  • 225 lines of Go
  • write renaming rules in Lua
  • run Go program to apply renaming to each file
% go run main.go 
usage: rename [--dry-run] [-l <lua-filepath>:<luafunction>
                              <filepath-to-rename>+]+
  this program takes filepaths and renames them
  acc. to the Lua function's return value
exit status 1

Approach

% cat lib.lua
function titlecase(filename)
    return filename:sub(1,1):upper() .. filename:sub(2)
end

% go run main.go --dry-run -l lib.lua:titlecase file.txt
Calling 'titlecase' with 'file.txt
'titlecase' returned 'File.txt' for 'file.txt'
Would rename 'file.txt' to 'File.txt' now, but I am dry-running

Feature

% go run main.go -l lib2.lua file.txt
lib2.lua seems to define the following functions: 'stripWhitespace' 'uppercase' 'lowercase' 'titlecase'
lua reference 'lib2.lua' must contain colon to separate lua path and function name

Lua syntax is very easy. I parsed it for fun. Function definition maches one of regexes:

function ([^(]+)\(
([^ =]+) *= *function

Implementation

  1. determine lua file matching lib.lua:titlecase
  2. determine file's dirname
  3. Call Lua with old basename ⇒ get new basename
  4. join file's dirname with new basename
  5. apply rename

Lua implementation

with github.com/Shopify/go-lua

luaState := lua.NewState()
lua.OpenLibraries(luaState)
_ := lua.DoFile(luaState, luaScriptFilepath)

luaState.Global(luaFunctionName)
// prechecks
if !luaState.IsFunction(luaState.Top()) {
    log.Fatal("loading function failed")
}
if luaState.PushString(baseName) != baseName ||
  !luaState.IsString(luaState.Top()) {
    log.Fatal("loading string failed")
}

Lua implementation

with github.com/Shopify/go-lua

// call 1 argument, 1 result
luaState.Call(1, 1)
// fetch result
result, ok := luaState.ToString(luaState.Top())
if !ok {
    log.Fatal("Reading lua return value somehow failed")
}
if result == "" {
    log.Fatal(`Function %s returned empty string -
        cannot rename to empty string`, luaFunctionName)
}
log.Printf("'%s' returned '%s' for '%s'",
           luaFunctionName, result, baseName)
luaState.Pop(1)

Lua implementation

github.com/Shopify/go-lua

go-lua is a port of the Lua 5.2 VM to pure Go. It is compatible with binary files dumped by luac, from the Lua reference implementation. The motivation is to enable simple scripting of Go applications. For example, it is used to describe flows in Shopify's load generation tool, Genghis.

Lua implementation

github.com/Shopify/go-lua

  • Which function calls do I need?
  • How to handle the stack?

→ Refer to the well-specified Lua C API …

“Fun” fact

> = string.gsub("Hello banana", "banana", "Lua user")
Hello Lua user  1
> -- capture any occurences of "an" and replace
> = string.gsub("banana", "(an)", "%1-")
ban-an-a        2
> -- brackets around n's which follow a's
> = string.gsub("banana", "a(n)", "a(%1)")
ba(n)a(n)a      2
> -- reverse any "an"s
> = string.gsub("banana", "(a)(n)", "%2%1")
bnanaa  2
> = string.gsub("banana", "(a)(n)",
                function(a,b) return b..a end)
bnanaa  2

Lua string lib tutorial

“Fun” fact

…
// {"gmatch", ...},
// {"gsub", ...},
{"len", func(l *State) int { l.PushInteger(len(CheckString(l, 1))); return 1 }},
{"lower", func(l *State) int { l.PushString(strings.ToLower(CheckString(l, 1))); return 1 }},
// {"match", ...},
…
  • github.com/Shopify/go-lua does not implement string.gsub
  • I extracted all string.gsub tests from the official Lua testsuite → gist
  • I should have done some research beforehand

Fundamental question

We want the user to specify (parts of) the behavior of a program. How?

I identified 3 approaches to the problem:

  1. UNIX philosophy
  2. linking and extensibility
  3. sophisticated configuration files

UNIX philosophy

UNIX specifies a lot of tiny syntaxes. /etc/fstab, /etc/hosts, … but also in tools like sed, grep, awk, …

  • many domain-specific languages
  • sysadmins are supposed to know all(?) of them. Lots of learning by heart.
  • lots of syntax - little formal grammar
  • nobody knows whether your file matches the syntax.

Linking and extensibility

  • Write a library, document it, publish it on github
  • import lua "github.com/Shopify/go-lua" and call lua.Whatever()
  • Specifically call the components you want. Omit the ones you replace.
  • Requires clear interfaces. Requires separable components.
  • Available since extensibility by dynamic linking of executables

Sophisticated configuration files

  • JSON, YAML, TOML, etc are fine. But we want program logic.
  • sendmail configuration files are compiled with m4
  • Unlike the UNIX approach, well-defined syntax
  • I picked Lua. No standard way to address an interface (I used a function name)
  • Lots of languages! Which level of power do we need?

Why Lua?

  • Here: because it is easy to learn (for everyone) and easy to embed
  • Well-researched, well-documented and well-tested
  • No batteries included (which is often desirable)
  • Shitty tool support (debugging, package management)

In the end

How and should we make programs more configurable? What about the next generation capable of programming as a common skill?

Thanks!