rename

A small script triggering a configuration question. GoGraz 2019-09-09 [GoGraz](https://gograz.org/) Lukas Prokop --- ## About * Given a bunch of files of similar name * How can I rename them easily? * Very common task! --- ### Larry Wall's rename ``` rename 's/[.]jpeg[0-9]*/.jpeg/' *.jpeg[0-9]* ``` ### bash ``` for n in *.jpeg[0-9]* do mv -i "$n" "${n%%.jpeg[0-9]*}.jpeg"; done ``` ### zsh ``` autoload zmv zmv -n '(*.jpeg)<->' '$1' ``` --- ## Approach * 225 lines of Go * write renaming rules in Lua * run Go program to apply renaming to each file ``` % go run main.go usage: rename [--dry-run] [-l : +]+ this program takes filepaths and renames them acc. to the Lua function's return value exit status 1 ``` --- ## Approach ``` % cat lib.lua function titlecase(filename) return filename:sub(1,1):upper() .. filename:sub(2) end % go run main.go --dry-run -l lib.lua:titlecase file.txt Calling 'titlecase' with 'file.txt 'titlecase' returned 'File.txt' for 'file.txt' Would rename 'file.txt' to 'File.txt' now, but I am dry-running ``` --- ## Feature ``` % go run main.go -l lib2.lua file.txt lib2.lua seems to define the following functions: 'stripWhitespace' 'uppercase' 'lowercase' 'titlecase' lua reference 'lib2.lua' must contain colon to separate lua path and function name ``` Lua syntax is very easy. I parsed it for fun. Function definition maches one of regexes: ``` function ([^(]+)\( ([^ =]+) *= *function ``` --- ## Implementation 1. determine lua file matching `lib.lua:titlecase` 2. determine file's dirname 3. Call Lua with old basename ⇒ get new basename 4. join file's dirname with new basename 5. apply rename --- ## Lua implementation with [github.com/Shopify/go-lua](https://github.com/Shopify/go-lua/) ```go luaState := lua.NewState() lua.OpenLibraries(luaState) _ := lua.DoFile(luaState, luaScriptFilepath) luaState.Global(luaFunctionName) // prechecks if !luaState.IsFunction(luaState.Top()) { log.Fatal("loading function failed") } if luaState.PushString(baseName) != baseName || !luaState.IsString(luaState.Top()) { log.Fatal("loading string failed") } ``` --- ## Lua implementation with [github.com/Shopify/go-lua](https://github.com/Shopify/go-lua/) ```go // call 1 argument, 1 result luaState.Call(1, 1) // fetch result result, ok := luaState.ToString(luaState.Top()) if !ok { log.Fatal("Reading lua return value somehow failed") } if result == "" { log.Fatal(`Function %s returned empty string - cannot rename to empty string`, luaFunctionName) } log.Printf("'%s' returned '%s' for '%s'", luaFunctionName, result, baseName) luaState.Pop(1) ``` --- ## Lua implementation [github.com/Shopify/go-lua](https://github.com/Shopify/go-lua/) > go-lua is a port of the Lua 5.2 VM to pure Go. It is compatible with binary files dumped by luac, from the Lua reference implementation. > The motivation is to enable simple scripting of Go applications. For example, it is used to describe flows in Shopify's load generation tool, Genghis. --- ## Lua implementation [github.com/Shopify/go-lua](https://github.com/Shopify/go-lua/) * Which function calls do I need? * How to handle the stack? → Refer to the well-specified [Lua C API](https://www.lua.org/pil/24.2.html) … --- ## “Fun” fact ```lua > = string.gsub("Hello banana", "banana", "Lua user") Hello Lua user 1 > -- capture any occurences of "an" and replace > = string.gsub("banana", "(an)", "%1-") ban-an-a 2 > -- brackets around n's which follow a's > = string.gsub("banana", "a(n)", "a(%1)") ba(n)a(n)a 2 > -- reverse any "an"s > = string.gsub("banana", "(a)(n)", "%2%1") bnanaa 2 > = string.gsub("banana", "(a)(n)", function(a,b) return b..a end) bnanaa 2 ``` [Lua string lib tutorial](http://lua-users.org/wiki/StringLibraryTutorial) --- ## “Fun” fact ``` … // {"gmatch", ...}, // {"gsub", ...}, {"len", func(l *State) int { l.PushInteger(len(CheckString(l, 1))); return 1 }}, {"lower", func(l *State) int { l.PushString(strings.ToLower(CheckString(l, 1))); return 1 }}, // {"match", ...}, … ``` * github.com/Shopify/go-lua does *[not](https://sourcegraph.com/github.com/Shopify/go-lua/-/blob/string.go#L183)* implement [string.gsub](http://lua-users.org/wiki/StringLibraryTutorial) * I extracted all string.gsub tests from the official Lua testsuite → [gist](https://gist.github.com/meisterluk/9cdf7ad78253ebcb434c19358507690f) * I should have done some research beforehand --- ## Fundamental question We want the user to specify (parts of) the behavior of a program. How? I identified 3 approaches to the problem: 1. UNIX philosophy 2. linking and extensibility 3. sophisticated configuration files --- ## UNIX philosophy UNIX specifies **a lot** of tiny syntaxes. `/etc/fstab`, `/etc/hosts`, … but also in tools like `sed`, `grep`, `awk`, … * many domain-specific languages * sysadmins are supposed to know all(?) of them. Lots of learning by heart. * lots of syntax - little formal grammar * nobody knows whether your file matches the syntax. --- ## Linking and extensibility * Write a library, document it, publish it on github * `import lua "github.com/Shopify/go-lua"` and call `lua.Whatever()` * Specifically call the components you want. Omit the ones you replace. * Requires clear interfaces. Requires separable components. * Available since extensibility by dynamic linking of executables --- ## Sophisticated configuration files * JSON, YAML, TOML, etc are fine. But we want program logic. * sendmail configuration files are compiled with [m4](https://en.wikipedia.org/wiki/M4_(computer_language)) * Unlike the UNIX approach, well-defined syntax * I picked Lua. No standard way to address an interface (I used a function name) * Lots of languages! Which level of power do we need? --- ## Why Lua? * Here: because it is easy to learn (for everyone) and easy to embed * Well-researched, well-documented and well-tested * No batteries included (which is often desirable) * Shitty tool support (debugging, package management) --- ## In the end How and should we make programs more configurable? What about the next generation capable of programming as a common skill? --- **Thanks!**