Functional programming in Bash

Introduction to fun.sh library

L2L Warszawa 2017

About me

Sławomir Śledź

Sławomir Śledź

Senior Java Developer

 ssledz.github.io

The Talk

Bash

Bash is an sh-compatible command language interpreter that executes commands read from the standard input or from a file.

Strong sides

  • easy to learn
  • available out of the box on most linux distributions
  • piping
  • flexible i/o redirection
  • painless concurrency
  • simple type system (everything is a string)
  • local variables, read only variables (but who is using them ?)

Weak sides

  • arrays suck
  • associative arrays (dictionaries) suck
  • syntax for if, for, while, case,... statements is hard to remember
  • many traps for beginners
  • is slow
  • global variables
  • debugging is painful
  • functions can only return numbers (status code)

Common mistakes

What does echo print ?


                cnt=0; cat /proc/meminfo | while read line; do

                    cnt=$((cnt+1))

                done

                echo $cnt
        

Common mistakes

Now echo prints correctly the number of lines of the meminfo pseudo file.


                cnt=0; while read line; do

                    cnt=$((cnt+1))

                done < /proc/meminfo

                echo $cnt
        

Common mistakes

Does it print 'xxx' ?


                echo -n 'xxx' | while read line; do

                    echo $line;

                done
        

Common mistakes

Now it prints 'xxx' !


                 echo -n 'xxx' | while read line || [ -n "$line" ]; do

                    echo $line;

                 done
        

What can we do to make programming in bash more pleasant ?

What makes haskel programmers so happy ?

What is functional programming ?

Can we adapt some parts of it in bash ?

Functional programming

In computer science, functional programming is a programming paradigm—a style of building the structure and elements of computer programs—that treats computation as the evaluation of mathematical functions and avoids changing-state and mutable data.

Good parts

  • easier to understand and predict the behavior of a program
  • plays extremely nice with the rising multi-core technology (concurrency)
  • many algorithms can be defined in recursive manner
  • more concise code (comparing to imperative style)
  • better readable code - encourage decoupling / modular design (comparing to imperative style)

Good parts...

  • employs some really nice abstractions to solve problems in very generic way (monads, option, lazy, memo, pattern matching, currying)
  • make you think about problems in more abstract / mathematical way

Bad parts

  • real-world programs are not stateless
  • with immutable data structures you have cheap thread safety at the cost of memory consumption
  • usually programs are shorter and easier to read except the cases where they are longer and harder to read :) (depends on the experience)
  • productivity goes up in order to hire very expensive programmers who know how to program in a functional style
  • it takes some time to be efficient in functional programming

fun-bash a quick start

Available on GitHub  bash-fun

download fun.sh and


            #!/bin/bash
            . fun.sh

            seq 1 4 | sum
        

        #!/bin/bash
        . <(curl -Ls goo.gl/8JeMbR)

        seq 1 4 | sum
    

            #!/bin/bash
            . <(test -e fun.sh || curl -Ls goo.gl/8JeMbR > fun.sh; cat fun.sh)

            seq 1 4 | sum
        

fun-bash a quick start...

You can use bash-import to load bash-fun


        #!/bin/bash

        BASH_REP=https://raw.githubusercontent.com/ssledz/bash-repo/master

        . <(curl -s $BASH_REP/__import.sh 2>/dev/null)

        import pl.softech/fun/1.0.0 as f

        seq 1 4 | f.sum
    

Available on GitHub   bash-import

Example of usage


            #!/bin/bash
            . fun.sh
            backup_dir=/home/adserver/backup
            max_backups=10 # adserver-active-1080-2017-05-22T134440

            prepare() {
                local a=$1; echo $a
                echo $a | strip adserver-active- | strip 1081- | strip 1080-
            }

            ls -1 $backup_dir | grep adserver-active- \
                | map λ a . 'prepare $a' \
                | buff λ a b . 'date=$(echo $b | cut -dT -f1); tup $a $date' \
                | sort -t ',' -k2 -r | drop $max_backups \
                | map λ a . 'echo -n $backup_dir/; tupl $a' | xargs rm -rf
        

Example of usage - test data


#!/bin/bash
xargs mkdir << EOF
  adserver-active-1080-2017-05-22T134440
  adserver-active-1080-2017-05-23T143705
  adserver-active-1080-2017-05-29T110500
  adserver-active-1080-2017-05-30T110812
  adserver-active-1080-2017-06-20T130354
EOF
        

List of available functions

v1.1.1

add append buff curry div drop
factorial filter foldl foldr head join
lambda last list map mod mul
prepend product ret revers_str revers scanl
splitc strip sub sum tail take
try tupl tupr tup tupx unlist
zip λ

list/unlist


            $ list 1 2 3 4 5
            1
            2
            3
            4
            5
            $ list 1 2 3 4 5 | unlist
            1 2 3 4 5
        

take/drop/tail/head/last


            $ list 1 2 3 4 | drop 2
            3
            4
            $ list 1 2 3 4 5 | head
            1
            $ list 1 2 3 4 | tail
            2
            3
            4
            $ list 1 2 3 4 5 | last
            5
            $ list 1 2 3 4 5 | take 2
            1
            2
        

join


            $ list 1 2 3 4 5 | join ,

            1,2,3,4,5

            $ list 1 2 3 4 5 | join , [ ]

            [1,2,3,4,5]
        

map


        $ seq 1 5 | map lambda a . 'echo $((a + 5))'
        6
        7
        8
        9
        10
        $ list a b s d e | map lambda a . 'echo $a$(echo $a | tr a-z A-Z)'
        aA
        bB
        sS
        dD
        eE
        

flat map


        $ seq 2 3 | map lambda a . 'seq 1 $a' | join , [ ]

        [1,2,1,2,3]

        $ list a b c | map lambda a . \
            'echo $a; echo $a | tr a-z A-z' | join , [ ]

        [a,A,b,B,c,C]
        

filter


            $ seq 1 10 | filter lambda a .\
                '[[ $(mod $a 2) -eq 0 ]] && ret true || ret false'

            2
            4
            6
            8
            10
        

foldl/foldr


            $ list a b c d | foldl lambda acc el . 'echo -n $acc-$el'

            a-b-c-d

            $ list '' a b c d | foldr lambda acc el .\
              'if [[ ! -z $acc ]]; then echo -n $acc-$el; else echo -n $el; fi'

            d-c-b-a

            $ seq 1 4 | foldl lambda acc el . 'echo $(($acc + $el))'

            10
        

foldl/foldr...


            $ seq 1 4 | foldl lambda acc el . 'echo $(mul $(($acc + 1)) $el)'

            64 # 1 + (1 + 1) * 2 + (4 + 1) * 3 + (15 + 1) * 4 = 64

            $ seq 1 4 | foldr lambda acc el . 'echo $(mul $(($acc + 1)) $el)'

            56 # 1 + (1 + 1) * 4 + (8 + 1) * 3 + (27 + 1) * 2 = 56
        

tup/tupl/tupr


            $ tup a 1
            (a,1)
            $ tupl $(tup a 1)
            a
            $ tupr $(tup a 1)
            1
            $ tup a 1 | tupl
            a
            $ tup a 1 | tupr
            1
        

buff


            $ seq 1 10 | buff lambda a b . 'echo $(($a + $b))'
            3
            7
            11
            15
            19
            $ seq 1 10 | buff lambda a b c d e . \
                'echo $(($a + $b + $c + $d + $e))'
            15
            40
        

zip


            $ list a b c d e f | zip $(seq 1 10)
            (a,1)
            (b,2)
            (c,3)
            (d,4)
            (e,5)
            (f,6)
            $ list a b c d e f | zip $(seq 1 10) | last | tupr
            6
        

curry


            $ arg='[key1=value1,key2=value2,key3=value3]'
            $ get() {
                local pidx=$1
                local idx=$2
                local arg=$3
                echo $arg | tr -d '[]' | cut -d',' -f$idx | cut -d'=' -f$pidx
            }
            $ curry get_key get 1
            $ curry get_value get 2
            $ get_key 1 $arg
            $ get_value 1 $arg
            key1
            value1
        

curry...


            $ arg='[key1=value1,key2=value2,key3=value3]'

            seq 1 3 | map lambda a . \
                'tup $(get_key $a $arg) $(get_value $a $arg)'

            (key1,value1)
            (key2,value2)
            (key3,value3)
        

try


            $ echo 'ls /home' | try λ cmd status ret . \
                'echo $cmd [$status]; echo $ret'

            ls /home [0]
            lost+found#sof-tech#ssledz

            $ echo '/home' | try λ cmd status ret . \
                'echo $cmd [$status]; echo $ret'

            /home [126]
            ../src/fun.sh:#linia#173:#/home:#Jest#katalogiem
        

scanl


            $ seq 1 5 | scanl lambda a b . 'echo $(($a + $b))'
            1
            3
            6
            10
            15
            $ seq 1 5 | scanl lambda a b . 'echo $(($a + $b))' | last
            15
        

fun-sh core concept


lambda() {
  lam() {
    local arg
    while [[ $# -gt 0 ]]; do
      arg="$1"; shift
      if [[ $arg = '.' ]]; then
        echo "$@"
        return
      else
        echo "read $arg;"
      fi
    done
  }
  eval $(lam "$@")
}
    

Examples

  • counting lines
  • counting words

Counting lines


            $ curl -Ls https://goo.gl/tE1JUh | wc -l

            254
        

            $ curl -s curl -Ls https://goo.gl/tE1JUh \
                | zip $(seq 1 1000) | last | tupl

            254
            

            $ echo 0 | cat - <(curl -Ls https://goo.gl/tE1JUh) \
                | foldl lambda acc el . 'echo $(($acc + 1))'

            254
            

Counting words


            $ cat <(\
                while read line; do \
                    for word in ${line[@]}; do \
                        echo $word; \
                    done; \
                done \
            < <(curl -Ls https://goo.gl/tE1JUh)) | wc -l

            568
        

                $ curl -Ls https://goo.gl/tE1JUh | wc -w

                571
            

            $ echo 0 | cat - <(curl -Ls https://goo.gl/tE1JUh) \
                | map lambda a . 'list $a' \
                | foldl lambda acc el . 'echo $(($acc + 1))'

            568
        

Resources

Thank you -:)