ssledz blog

Everything should be made as simple as possible, but no simpler.

Static Linking With Ghc

In order to compile a given source code using static linking

1
2
-- Main.hs
main = putStrLn "Hello World!"

there is a need to pass two additional flags to ghc like

  • -optl-static
  • -optl-pthread
1
$ ghc -optl-static -optl-pthread --make Main

then

1
2
3
4
$ file Main
Main: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked,
for GNU/Linux 3.2.0, BuildID[sha1]=62d1682f378af2f994758e737ba9dc0b24fc06aa,
with debug_info, not stripped

Option Monad - From Scratch

In a post About Monads - a gentle introduction I have introduced a concept of monad. Since now we should have a good intuition what monad is and be aware of the situations where it can be used to simplify the code and make it more readable.

In this post we will focus on an Option monad which wraps value in a context aware of whether or not the value exists. We will start with a problem and solution not exactly suiting our needs and by adapting Option monad we will try to fix this.

Problem

There are 3 streams of numbers given as a strings.

1
2
3
val xs = List("0", "9", "9")
val ys = List("3", "3", "1")
val zs = List("2", "3", "2")

After zipping and flattening we got a stream called data containing tuples of 3 strings

1
val data : List[(String, String, String)] = flatten(xs.zip(ys).zip(zs))

Our task is to build a pipeline that generates a stream of Doubles which is a result of division one number by another in a sequence in the context of the same tuple.

It means that the pipeline for a stream described as

1
[(x1,y1,z1), (x2,y2,z2),...(xn,yn,zn)]

should generate a stream of numbers given by formula mentioned below

1
[(x1/y1/z1), (x2/y2/z2),... (xn/yn/zn)]

This problem can be solved using following scala code

1
2
def pipeline = data
    .map((DivModule.div _).tupled(_))

div function is defined in DivModule

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
object DivModule {

  def parse(x: String): Double = {

    if (x == null) {
      throw new IllegalArgumentException("arg can't be null")
    }

    x.toDouble

  }

  def div(x: String, y: String, z: String): Double = {

    if (x == null || y == null || z == null) {
      throw new IllegalArgumentException("(x | y | z) can't be null")
    }

    val xx = parse(x)

    val yy = parse(y)

    val zz = parse(z)

    if (yy == 0 || zz == 0) {
      throw new IllegalArgumentException("y or z can't be 0")
    }

    xx / yy / zz

  }

}

For a streams of numbers defined at the beginning we should get

1
0.0, 1.0, 4.5

The numbers are correct, but take a look on implementation of a parse and div functions. Those functions are not total. In functional world a function which is not total is called partial. A partial function is not defined for all values passed as its arguments.

And for following streams of numbers

1
2
3
val xs = List("11", "22", "0" , "9", "9", null)
val ys = List("11", "0" , "33", "3", "3", "1")
val zs = List("0" , "22", "33", "2", "3", "2")

after running pipeline we get an Exception

1
2
3
4
Exception in thread "main" java.lang.IllegalArgumentException: y or z can't be 0
  at learning.monad.example.DivModule$.div(DivModule.scala:28)
  at learning.monad.example.MonadOption$.$anonfun$pipeline$2(MonadOption.scala:25)
  at learning.monad.example.MonadOption$.$anonfun$pipeline$2$adapted(MonadOption.scala:25)

We can easily fix this by lifting a partial function to a total function. Let’s add lift function to the DivModule

1
2
3
4
5
6
7
8
type Fun3 = (String, String, String) => Double

def lift(f: Fun3, defaultValue: Double): Fun3 = (x, y, z) =>
  try {
    f(x, y, z)
  } catch {
    case e: Throwable => defaultValue
  }

and modify the pipeline accordingly

1
2
def pipeline: List[Double] = data
  .map((DivModule.lift(DivModule.div, -1)).tupled(_))

Now pipeline generates streams of numbers like

1
-1.0, -1.0, 0.0, 1.5, 1.0, -1.0

We can spot that for each undefined value we get -1 because of lift function which maps all undefined values to the default one, in our case -1.

In order to get only a valid numbers let’s apply a filter

1
2
3
def pipeline: List[Double] = data
  .map((DivModule.lift(DivModule.div, -1)).tupled(_))
  .filter(_ != -1)

and our result is

1
0.0, 1.5, 1.0

For a first look this solution would seem to be good, but what about all variations of streams for which pipeline could return -1 as a correct result ?

For example, when we change fifth number in zs from 3 to -3

1
2
3
val xs = List("11", "22", "0" , "9", "9", null)
val ys = List("11", "0" , "33", "3", "3", "1")
val zs = List("0" , "22", "33", "2", "-3", "2")

our pipeline will generate

1
0.0, 1.5

instead of

1
0.0, 1.5, -1.0

This is wrong and changing a default value in lift function doesn’t fix this, because for each such case we can find a counterexample where a correct value is filtered out. It is easy to prove that div function returns numbers from the whole Double space. In category theory there is a name for such functions (morphisms) - epimorphisms.

And here an Option monad comes to the rescue.

Option monad

In order to implement an Option monad let’s start with defining a data type constructor

1
trait Option[+A]

Option is an algebraic data type constructor parametrized with A in a co-variant position. We can think of an Option[_] like about container keeping a value. For now it has no context.

Let’s assigned a context to it by defining a value constructor called Some

1
case class Some[A](get: A) extends Option[A]

This constructor introduced a context of container with a none empty value.

A context meaning no value (empty container) can be defined with None

1
case object None extends Option[Nothing]

None reveals why we need a type parameter A to be in co-variant position - we simply requires None to be cast to any Option[A].

Recalling a definition of Monad we know that it consists of three parts:

  • type constructor M[A]
  • type converter (called unit, pure or return)
  • combinator (called bind, >>= or flatMap)

First part is fulfilled. Now we have to implement type converter - pure

1
2
3
4
5
object Option {

  def pure[A](a: A): Option[A] = if (a == null) None else Some(a)

}

pure is a function which is responsible for putting a given value to the minimal meaningful context. For Option we requires that each non null value should be bind to the Some, in other case it should be None.

The toughest to implement is a combinator function called flatMap. For an Option it is very easy task however.

1
2
3
4
5
6
trait Option[+A] {
  def flatMap[B](f: A => Option[B]): Option[B] = this match {
    case Some(a) => f(a)
    case None => None
  }
}

This higher order function is very powerful and can be used as a primitive to implement for example map and filter in a following way

1
2
3
4
5
6
7
8
9
10
trait Option[+A] {
  def flatMap[B](f: A => Option[B]): Option[B] = this match {
    case Some(a) => f(a)
    case None => None
  }

  def map[B](f: A => B): Option[B] = flatMap(a => Option.pure(f(a)))

  def filter(p : A => Boolean) : Option[A] = flatMap(a => if(p(a)) Option.pure(a) else None)
}

Thanks to the flatMap we are able to get a value from container, abstracting whether or not the value exists or not, and apply a map transformation deciding if we want to put the result again or replace container with an empty one.

Putting all parts together we can define Option monad in scala in the following way

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
sealed trait Option[+A] {

  def flatMap[B](f: A => Option[B]): Option[B] = this match {
    case Some(a) => f(a)
    case None => None
  }

  def map[B](f: A => B): Option[B] = flatMap(a => Option.pure(f(a)))

}

object Option {

  def pure[A](a: A): Option[A] = if (a == null) None else Some(a)

}

case class Some[A](get: A) extends Option[A]

case object None extends Option[Nothing]

Building new version of pipeline

Let’s return to our problem. First we need to reimplement parse

1
2
3
4
5
6
7
def parse(x: Option[String]): Option[Double] = x.flatMap { str =>
  try {
    Option.pure(str.toDouble)
  } catch {
    case e: Throwable => None
  }
}

Argument and return type has been lifted respectively to the Option[String] and Option[Double] type. You can spot that I have used flatMap to have an access to the String value and based on the toDouble operation I returned some Double value or nothing - in case of parse exception. When an argument x is None the function passed to flatMap is not executed - so I am sure that String passed to monadic function is not null and I don’t have to make awkward null checks.

Next we need to take care of div

1
2
3
4
5
6
7
8
9
def div(x: Option[String], y: Option[String], z: Option[String]): Option[Double] = {
  def zeroToNone(n: Double) = if (n == 0) None else Some(n)

  for {
    xx <- parse(x)
    yy <- parse(y).flatMap(zeroToNone)
    zz <- parse(z).flatMap(zeroToNone)
  } yield xx / yy / zz
}

String arguments has been lifted to the Option[String] and return type is now Option[Double].

An x argument is passed to the parse and finally we have Double value in xx variable. If parse returns None then the whole function div is evaluated to None.

For y and z things works almost the same with one difference - we additionally requires that yy an zz must be none zero. This is expressed by calling flatMap with function zeroToNone. For 0 value zeroToNone returns an empty container None which causes that the whole expression parse(y).flatMap(zeroToNone) is evaluated to None what moves div function to return None.

Finally pipeline could look following

1
2
3
def pipeline2 = data
  .map(x => map3(x)(Option.pure))
  .map(z => (DivModuleWithOption.div _).tupled(z))

This pipeline generates

1
List(None, None, Some(0.0), Some(1.5), Some(-1.0), None)

At the end we need only to filter out nones and get the value out of the Option

To do so there is a need to add 3 additional methods to Option

1
2
3
4
5
6
7
8
9
10
sealed trait Option[+A] {

  //skipped for brevity

  def isEmpty: Boolean

  def isNonEmpty: Boolean = !isEmpty

  def get : A
}

and to corresponding subclasses

1
2
3
4
5
6
7
8
9
case class Some[A](get: A) extends Option[A] {
  def isEmpty: Boolean = false
}

case object None extends Option[Nothing] {
  def isEmpty: Boolean = true

  def get : Nothing = ???
}

Finally the pipeline

1
2
3
4
5
def pipeline2 = data
  .map(x => map3(x)(Option.pure))
  .map(z => (DivModuleWithOption.div _).tupled(z))
  .filter(_.isNonEmpty)
  .map(_.get)

generates following stream of numbers

1
List(0.0, 1.5, -1.0)

and this is all we need.

Resources

About Monads - a Gentle Introduction

In functional programming monad is a design pattern which is used to express how states of computations are changing. It can take a form of some abstract data type constructor with two abstract functions.

In scala we can define this contract using Monad type class

1
2
3
4
5
6
7
8
9
trait Monad[M[_]] {

  def pure[A](x: A): M[A]

  def flatMap[A, B](xs: M[A])(f: A => M[B]): M[B]

  def map[A, B](xs: M[A])(f: A => B): M[B] = flatMap(xs)(x => pure(f(x)))

}

Functions pure and flatMap for a given monad M[_] have to follow some laws - I will talk about them later.

Function map can be defined in terms of flatMap and pure and this is a bonus which we get for a free when we provide an instance of a Monad for a type M[_]. Moreover many useful functions can be defined in terms of flatMap and pure like map2, ap, filter and so on. This interface is so powerful then often is treated as a primitive one when goes to implement other functions.

We can think about M[A] like about some smart container for a value (values) of type A. This container abstracts away from how this value is kept. We can have many flavors of them like container:

  • aware of whether or not the value exists
  • with more then one value
  • for which getting the value would trigger some kind of IO operation
  • with value which eventually could appear in future
  • with value or error
  • with value dependent on some kind of state
  • with value and some logging information
  • etc

Monad let us focus on what we want to do with the contained value. Monad is like a context in which the value exists. When we want to do some computation we are abstracting over the context so we aren’t disrupted whether or not the value exists, we have many of them or the value may appear in a future. We want just to get the value out of the container for a moment to make some computation and then put it there again. The context is important only when we want to pull out a value permanently.

Another advantage of the monad is an ability of sequencing the computations. Having let’s say two computations we can very easily make dependence between them saying that the computations of the second depends on a result of the first. Of course this can be scaled to more than two.

At first glance, it may seem to be not so impressive because it is very common to make such things during coding. But be aware that monad frees us from thinking about the context in which the value exists. The context can be for example an asynchronous computation. Dealing with concurrency is challenging - we have to be very careful to not make a hard to spot mistakes. Monad takes care about this complexity, providing a result of the first computation as soon as possible giving us possibility to spawn another computation in asynchronous manner.

Laws

Each monad needs to follow three laws

  • Left identity: return a >>= f ≡ f a
  • Right identity: m >>= return ≡ m
  • Associativity: (m >>= f) >>= g ≡ m >>= (\x -> f x >>= g)

These laws was taken from haskell because expressions there are very compact and easy to follow. Function >>= in scala maps to flatMap, return is just a pure, f x is an application of function f with x and the last one \x -> ... is a lambda expression.

Laws in scala can be written in a following way (using ScalaCheck)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
val monad = implicitly[Monad[M]]

property("Left identity: return a >>= f ≡ f a") = forAll { (a: A, f: A => M[B]) =>
  (`return`(a) >>= f) === f(a)
}

property("Right identity: m >>= return ≡ m") = forAll { m: M[A] =>
  (m >>= `return`) === m
}

property("Associativity: (m >>= f) >>= g ≡ m >>= (\\x -> f x >>= g)") =
  forAll { (m: M[A], f: A => M[B], g: B => M[C]) =>
    ((m >>= f) >>= g) === (m >>= (x => f(x) >>= g))
}

val `return`: A => M[A] = monad.pure _

private implicit class MonadOps[A](m: M[A]) {
  def >>=[B](f: A => M[B]): M[B] = monad.flatMap(m)(f)
}

If you are curious about implementation details take a look on this class

Flavors of monads

This section is a placeholder for a list of posts about monads mentioned in this article. I will try my best to deliver a missing content. Watch my blog for an update.

Monads:

  • Option
  • Either
  • Id
  • Writer
  • Reader
  • State
  • Try
  • IO
  • List

Resources

Monad Transformers - Part 2

In a previous post I introduced monad transformers and since now we should have a good feeling about their usage and how they can be helpful.

Designing a monad transformer we decided to fix inner most monad. This decision was dictated by the fact that we couldn’t replace code dependent on internal representation of that inner most monad. I think that this step could not be as obvious as I expected to be. And now I will try to make it more clear.

Let’s try to bite the problem from different side. Assume that we can write a monad transformer and know nothing about monads internal representation. Let’s call it CMonad (shorthand from ComposedMonad).

Such a class could look like

1
2
3
4
5
6
7
8
case class CMonad[F[_], G[_], A](value: F[G[A]]) {

 def flatMap[B](f: A => CMonad[F, G, B])(implicit F: Monad[F], G: Monad[G]): CMonad[F, G, B] =
   ???

 def map[B](f: A => B): CMonad[F, G, B] = ???

}

Here F[_] and G[_] are higher kinded type representing outer and inner most monad.

Then a problem introduced in a previous post could be solved following

1
2
3
4
5
def findStreetByLogin(login: String): CMonad[Future, Option, String] =
  for {
    user <- CMonad(findUserByLogin(login))
    address <- CMonad(findAddressByUserId(user.id))
  } yield address.street

Of course it doesn’t work because we haven’t yet provided implementation for flatMap and map

Let’s start with flatMap. To make things clear a little I introduced a new method flatMapF and defined flatMap in terms of flatMapF

1
2
3
4
5
6
7
8
9
10
11
case class CMonad[F[_], G[_], A](value: F[G[A]]) {

 def flatMapF[B](f: A => F[G[B]])(implicit F: Monad[F], G: Monad[G]): CMonad[F, G, B] =
   ???

 def flatMap[B](f: A => CMonad[F, G, B])(implicit F: Monad[F], G: Monad[G]): CMonad[F, G, B] =
   flatMapF(a => f(a).value)

 def map[B](f: A => B): CMonad[F, G, B] = ???

}

In order to apply f : A => F[G[B]] we need to extract A from value: F[G[A]]

One attempt could end with following code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
case class CMonad[F[_], G[_], A](value: F[G[A]]) {

 def flatMapF[B](f: A => F[G[B]])(implicit F: Monad[F], G: Monad[G]): CMonad[F, G, B] =
   CMonad[F, G, B] {
     F.flatMap(value) { ga: G[A] =>
       val gb: G[B] = G.flatMap(ga) { a: A =>
         ???
       }
       F.pure(gb)
     }
   }

 def flatMap[B](f: A => CMonad[F, G, B])(implicit F: Monad[F], G: Monad[G]): CMonad[F, G, B] =
   flatMapF(a => f(a).value)

 def map[B](f: A => B): ComposedMonad[F, G, B] = ???

}

Now we can apply f with A and we will get fgb : F[G[B]]

1
2
3
4
5
6
7
8
9
10
 def flatMapF[B](f: A => F[G[B]])(implicit F: Monad[F], G: Monad[G]): CMonad[F, G, B] =
   CMonad[F, G, B] {
     F.flatMap(value) { ga: G[A] =>
       val gb: G[B] = G.flatMap(ga) { a: A =>
         val fgb: F[G[B]] = f(a)
         ???
       }
       F.pure(gb)
     }
   }

In order to make compiler happy we need to take one step more - extract G[B] from F[G[B]] and return that value from inner most flatMap. This of course is not possible knowing only that F and G form a monad.

Another attempt can lead us to the code like

1
2
3
4
5
6
7
8
9
 def flatMapF[B](f: A => F[G[B]])(implicit F: Monad[F], G: Monad[G]): CMonad[F, G, B] =
   CMonad[F, G, B] {
     F.flatMap(value) { ga: G[A] =>
       val gfgb: G[F[G[B]]] = G.map(ga) { a: A =>
         f(a)
       }
       ???
     }
   }

And now we need to extract F[G[B]] from G[F[G[B]]]. This also is not possible if we know nothing about internal representation of G.

All this leads us to the conclusion that we can’t write a monad transformer if we know nothing, about the monads.

Monad Transformers - a Quick Recap

Someone have said that monads are like burrito, if you ever taste one than you can’t imagine live without it.

Monads are a powerful tool. Thanks to them we can abstract over computation. We can make one computation depended on another and if needed fail fast.

But one day the time will come when we have two different monads and we will find out that they don’t compose !

Let’s make some code to visualize the problem. I am going to show two use cases and I will start with the simplest one.

Case 1

We have two entities : User and Address and two functions retrieving data with the respect of a given predicate

1
2
3
4
5
case class User(id: Long, login: String)
case class Address(userId: Long, street: String)

def findUserByLogin(login: String): Future[User] = ???
def findAddressByUserId(userId: Long): Future[Address] = ???

Our goal is to write a function which for a given login returns user’s street name

1
2
3
4
5
def findStreetByLogin(login : String) : Future[String] =
  for {
    user <- findUserByLogin(login)
    address <- findAddressByUserId(user.id)
  } yield address.street

So far so good - quite simple and classic enterprise task :)

However there are two caveats to this solution worth noting. What happened if there is no such user or the user exists but it has no address ? It is obvious that we will see Null Pointer Exception - sick !

Of course we can filter out those nulls and rewrite functions to be aware of them but as you already know this is also not a good solution. Can we do better ? Yes we can, let’s introduce a context aware of whether value exists or not (Option data type).

1
2
def findUserByLogin(login: String): Future[Option[User]] = ???
def findAddressByUserId(userId: Long): Future[Option[Address]] = ???

But wait below function is not compiling…

1
2
3
4
5
6
def findStreetByLogin(login: String): Future[Option[String]] =
  for {
    maybeUser <- findUserByLogin(login)
    user <- maybeUser
    address <- findAddressByUserId(user.id)
  } yield address.map(_.street)

It turns out that Future and Option monads do not compose in such a way. For a first look, composition looks very natural in for comprehension, but if we transform it into series of flatMap and map at the end, we will notice that the puzzles don’t feet. If we start with Future than the function passed to flatMap must return a Future. In our case we want to return Option in the middle and based on it return a next Future being a container fo an user’s possible address.

Equipped with this knowledge we can rewrite our function in the following way

1
2
3
4
5
def findStreetByLogin(login: String): Future[Option[String]] =
  findUserByLogin(login).flatMap {
    case Some(user) => findAddressByUserId(user.id).map(_.map(_.street))
    case None => Future.successful(None)
  }

Now it compiles and return correct results. But it is not as readable as our first naive attempt. Can we do better ? Ideally we would want to have something like

1
2
3
4
5
def findStreetByLogin(login: String): ??? =
  for {
    user <- ???(findUserByLogin(login))
    address <- ???(findAddressByUserId(user.id))
  } yield address.street

We need somehow to fuse Future with Option in a smart way to make the composition possible.

Fusing Future with Option

We already know that for comprehension deals with flatMap, map, withFilter and foreach. In our case compiler needs only flaMap and map to de sugar for. So let’s introduce a new data type OptionFuture, which wraps Future[Option[A]] and in a proper way handles flatMap in order to compose Future with Option.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
case class OptionFuture[A](value: Future[Option[A]]) {

  def flatMap[B](f: A => OptionFuture[B])(implicit ex: ExecutionContext): OptionFuture[B] =
    flatMapF(a => f(a).value)

  def flatMapF[B](f: A => Future[Option[B]])
                 (implicit ex: ExecutionContext): OptionFuture[B] = OptionFuture(
    value.flatMap { as =>
      as match {
        case Some(a) => f(a)
        case _ => Future.successful(None)
      }
    }
  )

  def map[B](f: A => B)(implicit ex: ExecutionContext): OptionFuture[B] =
    OptionFuture(value.map { x => x.map(f) })
}

Take a little time to better look at OptionFuture data type. First question coming to my mind is - can we make it more abstract ? It turns out that we can abstract over Future very easly. In terms of Future we are calling only two kinds of functions:

  • flatMap
  • Future.successful

It means that Future can be swapped with Monad.

What about the Option ? Over the Option we are performing pattern matching - so it means that we need to know something about it structure.

And because of that we can’t to abstract over it.

This leads us to the definition of monad transformer for Option and we call it OptionT

Monad transformer for Option

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
case class OptionT[F[_], A](value: F[Option[A]]) {

  def flatMap[B](f: A => OptionT[F, B])(implicit m: Monad[F]): OptionT[F, B] =
    flatMapF(a => f(a).value)

  def flatMapF[B](f: A => F[Option[B]])(implicit F: Monad[F]): OptionT[F, B] = OptionT(
    F.flatMap(value) { as =>
      as match {
        case Some(a) => f(a)
        case _ => F.pure(None)
      }
    }
  )

  def map[B](f: A => B)(implicit F: Monad[F]): OptionT[F, B] =
    OptionT(F.map(value) { x => x.map(f) })
}

OptionT[F[_], A] abstracts over F and A and it only requires that F is a monad. The name of the monad transformer comes from the fact that, in order to implement this wrapper, we need to know what the inner most monad in the stack is - in this case Option. Without this knowledge we can’t compose any two given monads with itself.

Monad quick recap

A minimal api for monad can be described by following trait

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
trait Monad[M[_]] {

  def pure[A](x: A): M[A]

  def flatMap[A, B](xs: M[A])(f: A => M[B]): M[B]

  def map[A, B](xs: M[A])(f: A => B): M[B] = flatMap(xs)(x => pure(f(x)))

}

object Monad {

  def apply[M[_]](implicit m: Monad[M]): Monad[M] = m

}

And its instance for Future you can find below.

1
2
3
4
5
6
7
8
9
object MonadInstances {
  implicit def futureInstance(implicit ex: ExecutionContext): Monad[Future] =
    new Monad[Future] {

      override def pure[A](x: A): Future[A] = Future.successful(x)

      override def flatMap[A, B](xs: Future[A])(f: A => Future[B]): Future[B] = xs.flatMap(f)(ex)
    }
}

A solution

Putting all pieces together we can finally write

1
2
3
4
5
6
7
8
import scala.concurrent.ExecutionContext.Implicits.global
import MonadInstances._

def findStreetByLogin(login: String): OptionT[Future, String] =
  for {
    user <- OptionT(findUserByLogin(login))
    address <- OptionT(findAddressByUserId(user.id))
  } yield address.street

of course we can return directly Future[Option[String]] just by calling value function on the result like

1
2
3
4
5
def findStreetByLogin(login: String): Future[Option[String]] =
  (for {
    user <- OptionT(findUserByLogin(login))
    address <- OptionT(findAddressByUserId(user.id))
  } yield address.street).value

` and that’s it.

Final word

At the beginning I said that I have two cases to show, but because the post could be to long to go through without a brake I decided to split it into two pieces. The whole code base used in this post can be found in the following link

More in part 2

How to Pass Arrays as Parameters in Bash

Let’s say we have a function in bash which simply iterates through all elements from array and prints them on the standard output.

1
2
3
4
5
printElems() {
  for e in ${arr[@]}; do 
    echo $e
  done
}

We can call this function in the following way

1
2
arr=(el1 el2 el3 el4)
printElems

If we want to call function with other parameters we need to update arr variable accordingly

1
2
3
4
arr=(el1 el2 el3 el4)
printElems
arr=(el5 el6 el7 el8)
printElems

So far so good. But what if we want to make our printElems function generic and put it to the separate file. Is it wise enough to stay with our solution to maintain arr variable in global scope ? The answer is - it depends on the size of the project. It is obvious that maintaining global variables is cumbersome in projects which are getting bigger and bigger during their lifetime. So that is there any smart way to improve our function to not pollute the global scope ?

The answer is yes, and in this task will help us bash feature called ‘indirect variable reference’.

Below is an improved version of printElem function. A new function (printElems2) is not dealing with global variable at all. In fact the function receives a variable name and thx to the indirect reference operator $(!variable_name) the value of the function parameter is set to local variable called my_arr.

1
2
3
4
5
6
printElems2() {
  local my_arr=${!1}
  for e in ${my_arr[@]}; do 
    echo $e
  done
}
1
2
3
4
5
arr1=(el1 el2 el3 el4)
arr2=(el5 el6 el7 el8)

printElems2 arr1[@]
printElems2 arr2[@]

More information about ‘indirect variable reference’ feature you can find here