Unix Scripting with Elixir

Unix Scripting with Elixir

Unix Scripting with Elixir

As a software developer I find myself doing repetitive tasks over and over again, such as normalizing some text or renaming a batch of files. A while back I decided whenever I needed to complete a basic, repetitive task on my computer that I would write a small script to do it for me. Doing this usually takes more time than simply doing the task manually but it helps keep my non-WWW programming skills sharp and I usually learn something about the language I decided to work in.

I decided to start writing new scripts like this in Ruby because Ruby is my main programming language. Now that I'm spending more of my free time learning Elixir I've decided to start writing at least some of these scripts in Elixir. The point of these kinds of scripts is not necessarily to make the best code, but to write it as quickly as possible to the specific case and change it when needed.

The Use Case

I subscribe Death to Stock for stock photos and receive periodic emails from them with a new set of photos ready for download. Because they are zip-archived, I found myself repeatedly extracting the stock photos and renaming the files each time a new set of photos is released. Here's my basic workflow:

  1. Click the link in an email to download the new photo pack
  2. Extract the .zip archive I downloaded
  3. Rename the files based on the photo pack's name and a sequential number

After unzipping the archive the contents of the directory looks like this:

$ ls -la
total 117832
drwx------@ 12 cmoel  staff       408 Mar 11 11:12 .
drwx------+ 25 cmoel  staff       850 Mar 11 11:20 ..
-rw-------@  1 cmoel  staff   6273110 Feb 26 19:25 Death_to_stock_photography_Wake_Up_1.jpg
-rw-------@  1 cmoel  staff   3423945 Feb 29 13:39 Death_to_stock_photography_Wake_Up_10.jpg
-rw-------@  1 cmoel  staff   8620554 Feb 26 19:43 Death_to_stock_photography_Wake_Up_2.jpg
-rw-------@  1 cmoel  staff   5702301 Feb 29 13:40 Death_to_stock_photography_Wake_Up_3.jpg
-rw-------@  1 cmoel  staff   3063833 Feb 29 14:28 Death_to_stock_photography_Wake_Up_4.jpg
-rw-------@  1 cmoel  staff   5930966 Feb 29 14:21 Death_to_stock_photography_Wake_Up_5.jpg
-rw-------@  1 cmoel  staff  10794652 Feb 29 13:39 Death_to_stock_photography_Wake_Up_6.jpg
-rw-------@  1 cmoel  staff   3306988 Feb 29 13:39 Death_to_stock_photography_Wake_Up_7.jpg
-rw-------@  1 cmoel  staff   5786387 Feb 29 13:54 Death_to_stock_photography_Wake_Up_8.jpg
-rw-------@  1 cmoel  staff   7407818 Feb 29 14:48 Death_to_stock_photography_Wake_Up_9.jpg

There's a total of 10 images, each one has the same file name except for the number before the file extension. My goal is to write a script that renames all the files to something like wake-up-01.jpg, wake-up-02.jpg, etc., including leading zeros.

In the workflow steps outlined above, steps 1 & 2 are done manually and step 3 is done through a Ruby script. Because of a limitation of handling .zip archives in Ruby I've also recreated this script in Elixir, which also gave me the ability to extract the zip archive. I'll first show the Ruby version of this script and then the Elixir implementation.

Using Ruby

Everything needed to handle step 3 of my workflow is included in Ruby Core.

The following script is the basic template I use for this, including some generalizations I've made over time such as using Math.log10 to calculate the number of leading zeros required.

pack = "wake-up"
ext  = ".jpg"

Dir.chdir(pack) do |dir|
  names = Dir["*#{ext}"]
  pad   = Integer(Math.log10(names.length) + 1)

  hash = names.each_with_object({}) do |name, memo|
    short_name =
      name
      .slice(/\d+/)
      .rjust(pad, "0")
      .prepend("#{pack}-")
      .concat("#{ext}")
    memo[name] = short_name
  end

  hash.each { |old, new| File.rename(old, new) }
end
  1. I change into the directory that contains the images for the photo pack, a directory called wake-up in this case
  2. Build an array of the filenames I want to change and calculate the number of leading zeros needed for each new filename
  3. Build a hash of each file's current name as the key and the new name as the value
  4. Rename the files with File.rename

To use this script, I need to first extract the zip archive and set the pack variable. While the Ruby script isn't as flexible as I would make it for a production environment (needing to change the pack variable each time I want to run it isn't ideal), it has met my need and isn't difficult to maintain. The main thing I don't like about it, though, is that it only meets point 3 of my workflow: renaming the files. It doesn't extract the zip archive, requiring me to still do that manually. While I could call shell commands I want to keep the script in just Ruby. Because Ruby Core and the Ruby Standard Library were meeting this need I started exploring what this would be like in Elixir.

Using Elixir

Here's a script written in Elixir that accomplishes the same result as the Ruby version above:

pack = "wake-up"
ext  = ".jpg"

File.cd!(pack, fn ->
  zeros = File.ls! |> length |> Integer.digits |> length
  File.ls!
  |> Stream.filter(fn(name)    -> String.ends_with?(name, ext) end)
  |> Stream.map(fn(name)       -> {name, name} end)
  |> Stream.map(fn({old, new}) -> {old, Regex.replace(~r/[^\d]/, new, "")} end)
  |> Stream.map(fn({old, new}) -> {old, String.rjust(new, zeros, ?0)} end)
  |> Stream.map(fn({old, new}) -> {old, "#{pack}-#{new}#{ext}"} end)
  |> Enum.map(fn({old, new})   -> File.rename(old, new) end )
end)

This gets the contents of the directory as a List, does some transforms on each element in the list, and finally renames the files. I'm back at the same place as I was with Ruby.

Expanding the Elixir Example

To extract the zip archive, Elixir provides direct access to Erlang, which includes a library for handling zip archives. Here's an updated version of the Elixir script that creates the directory, extracts the zip archive, moves the files into the new directory, and renames each file:

zip  = "Death_to_stock_photography_Wake_Up_free.zip"
pack = "wake-up"
ext  = ".jpg"

File.mkdir!(pack)

zip
|> String.to_char_list
|> :zip.extract
|> (fn({:ok, files}) -> files end).()
|> Stream.map(fn(name) -> List.to_string(name) end)
|> Enum.map(fn(name) -> File.rename(name, "#{pack}/#{name}") end)

File.cd!(pack, fn ->
  zeros = File.ls! |> length |> Integer.digits |> length
  File.ls!
  |> Stream.filter(fn(name)    -> String.ends_with?(name, ext) end)
  |> Stream.map(fn(name)       -> {name, name} end)
  |> Stream.map(fn({old, new}) -> {old, Regex.replace(~r/[^\d]/, new, "")} end)
  |> Stream.map(fn({old, new}) -> {old, String.rjust(new, zeros, ?0)} end)
  |> Stream.map(fn({old, new}) -> {old, "#{pack}-#{new}#{ext}"} end)
  |> Enum.map(fn({old, new})   -> File.rename(old, new) end )
end)

The file renaming code hasn't changed, I've only added code for creating the directory and extracting the archive.

Similar to the Ruby version of this, the above code isn't something I would deploy to production, and like the Ruby version, I think the Elixir version falls in the realm of good enough for now. There are certainly places where it can fail (such as the File.mkdir! call if there's already a directory by that name) but it accomplishes my goal of extracting the archive and renaming the extracted files. Both the Ruby and Elixir versions are easy to change and the more time I spend working with them, the better I'll understand the changes needed to make them "production-quality." This is the right place to be at this point since it doesn't lock me in to a specific implementation that actually be the wrong abstraction and would be hard to change later.

Conclusion

I wouldn't depend on the two scripts in this post to help solve an actual business problem, whether writing code for a client or internally for Grok. Both examples lack error handling logic and require manual modification before running each time. Scripts like this create a starting point and, who knows, after removing the need to manually edit and monitor for errors, it might turn out to be useful in a production application.

Categories: Software Development | Tags: Unix, Ruby, Elixir

Portrait photo for Christopher Moeller Christopher Moeller

Christopher is a self-taught developer and has been working with Ruby and Ruby on Rails since 2010. He enjoys building the simple, elegant solution to current problem he's solving and has recently picked up functional programming, mostly with the Elixir programming language.

Comments


LET US HELP YOU!

We provide a free consultation to discover competitive advantages for your business. Contact us today to schedule an appointment.