Slides for Lunch2Learn (L2L) session are available here.

Resources

• bash-fun
• bash-import
• bash-repo

Let’s say we have a function in bash which simply iterates through all elements from array and prints them on the standard output.

1
2
3
4
5

printElems() {
  for e in ${arr[@]}; do 
    echo $e
  done
}

We can call this function in the following way

1
2

arr=(el1 el2 el3 el4)
printElems

If we want to call function with other parameters we need to update arr variable accordingly

1
2
3
4

arr=(el1 el2 el3 el4)
printElems
arr=(el5 el6 el7 el8)
printElems

So far so good. But what if we want to make our printElems function generic and put it to the separate file. Is it wise enough to stay with our solution to maintain arr variable in global scope ? The answer is - it depends on the size of the project. It is obvious that maintaining global variables is cumbersome in projects which are getting bigger and bigger during their lifetime. So that is there any smart way to improve our function to not pollute the global scope ?

The answer is yes, and in this task will help us bash feature called ‘indirect variable reference’.

Below is an improved version of printElem function. A new function (printElems2) is not dealing with global variable at all. In fact the function receives a variable name and thx to the indirect reference operator $(!variable_name) the value of the function parameter is set to local variable called my_arr.

1
2
3
4
5
6

printElems2() {
  local my_arr=${!1}
  for e in ${my_arr[@]}; do 
    echo $e
  done
}

1
2
3
4
5

arr1=(el1 el2 el3 el4)
arr2=(el5 el6 el7 el8)

printElems2 arr1[@]
printElems2 arr2[@]

More information about ‘indirect variable reference’ feature you can find here

Below can be found a simple sbt setup for a spark application in scala.

Directory layout

1
2
3
4
5
6
7
8
9
10
11

find .
.
./build.sbt
./src/main
./src/main/scala
./src/main/scala/pl
./src/main/scala/pl/softech
./src/main/scala/pl/softech/WordCountExample.scala
./src/main/resources
./src/main/resources/log4j.properties
./src/main/resources/words.txt

build.sbt

1
2
3
4
5
6
7
8
9

name := "spark-simple-app"

version := "1.0"

scalaVersion := "2.11.8"

val sparkVersion = "2.1.0"

libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion

log4j.properties

1
2
3
4
5

log4j.rootCategory=ERROR, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

WordCountExample.scala

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

package pl.softech

import org.apache.spark.{SparkConf, SparkContext}


object WordCountExample {

  def main(args: Array[String]) {

    val conf = new SparkConf().setAppName("spark-simple-app").setMaster("local[*]")

    val sc = new SparkContext(conf)

    val textFile = sc.textFile("src/main/resources/words.txt")

    val counts = textFile.flatMap(line => line.split(" "))
      .map(word => (word, 1))
      .reduceByKey(_ + _)
      .sortBy(-_._2)

    printf(counts.collect().mkString("\n"))

    sc.stop()
  }

}

Sources can be found here

It passed almost 5 months since I have been learning Scala programming language. This learning is a part of bigger path which I decided to go through to be more familiar with current trends in big data, machine learning and cloud computing.

When I decided to start learning scala the first question coming into my mind was how to start ?

Because I am rather a fun of books, I bought one - “Programming Scala: Scalability = Functional Programming + Objects” by Dean Wampler and Alex Payne” and started reading…. but this was not a good idea. The book I have chosen was written in not an easy going style and I think was too hard for the beginning. After few hundreds of pages I felt lost. At the same time on coursera ‘Functional Programming Principles in Scala’ course was reactivated. Not thinking too much I signed in and started participating. After few weeks I earn a certificate of accomplishment but this was not an end of my journey. I started participating in another course “Functional Program Design in Scala” from the same author (Martin Odersky) as the previous one. In parallel I have started reading a book “Programming in Scala Second Edition” by Martin Odersky, Lex Spoon, Bill Venners and I found it very helpful. Currently (in the time of writing this post) I have done “Parallel programming” course and already enrolled for a “Big Data Analysis with Scala and Spark” (still waiting for course publishing)

I think the knowledge I acquired, thanks to those courses and book, is sufficient to start writing and reading programs in scala. Of course it takes some time to be as proficient as in java but I am little closer to this target each day.

Resources

The aim of this section is to gather the valuable resources regarding scala.

Online courses

• Functional Programming Principles in Scala
• Functional Program Design in Scala
• Parallel programming

Books

• Programming in Scala by Martin Odersky, Lex Spoon, Bill Venners
• Scala for the Impatient
• Programming Scala: Scalability = Functional Programming + Objects

Cheat sheets

• Scala cheat sheet
• Scala reactive cheat sheet
• Scala cheat sheet from scala-lang.org
• Scala cheat sheet written by me

Tutorials

• Scala By Example
• Scala School
• Tour of scala

Libs

• Scalacheck
• ScalaMeter
• Akka

Other

• scala on stackoverflow

First steps

My recommendation about ordering of going through the resources for beginners:

1. Functional Programming Principles in Scala
2. Programming in Scala by Martin Odersky, Lex Spoon, Bill Venners
3. Functional Program Design in Scala
4. Parallel programming

Few weeks ago since now :) I started participating in a course ‘Functional Programming Principles in Scala’ by Martin Odersky. I have already completed 4 weeks (course consists of 6 weeks) and I can tell honestly that this is the best course I’ve ever been doing.

My knowledge about scala is still increasing !

Below you can find a sample of what I can now do. The problem is to implement function flatten(xs: List[Any]): List[Any] which takes a list of anything an tries to flatten it. For example

1

flatten(List(List(1, 1), 2, List(3, List(5, 8))))

should return a following list

1

List[Any] = List(1, 1, 2, 3, 5, 8)

At the end I would like to say ‘thank you’ to Atlassian company for paying a half for this course !

tmux is a great tool. I have been using it since I think half year. Before I used to use screen, but only to manage my remote shells. Now I am using tmux for local and remote shell management.

In order to make a switch from screen to tmux smooth I decided to remap default binding for prefix from C-b to C-a (like in screen). This caused that one of my favorite shortcuts - C-a for going back to the beginning of the line, stopped working.

Recently I have discovered that making:

1

bind C-a send-prefix

binds the shortcut to this sequence C-a C-a !

Reloading a bytecode in a virtual machine when application is running is very limited. In fact HotSpot(TM) VM allows only changing method bodies. To address this problem some commercial and open source tools were created. Among them is Dynamic Code Evolution Virtual Machine (DCEVM) and HotSwapAgent - very promising open source tool.

I have already some experience in using DCEVM. Some times ago I have been working for an insurance company where I was using this modified vm to develop a code in a gosu language. Gosu is another JVM language. I remember that then hot swapping worked very well.

Let’s try this tool. First we need to patch our current jvm.

Installing Dynamic Code Evolution VM

In order to enhance current Java (JRE/JDK) installations with DCEVM you need to download the latest release of DCEVM installer for a given major java version,

java 7 and java 8 are supported

run the installer

1

java -jar DCEVM-light-8u74-installer.jar

then select a proper java installation directory on your disc and press Install DCEVM as altjvm

That’s all really. Very simple isn’t it ?

To validate the installation run:

1

java -version

and if everything went alright you should see something similar to below output:

1
2
3

java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Dynamic Code Evolution 64-Bit Server VM (build 25.71-b01-dcevmlight-10, mixed mode)

Note that in the third line instead of Java HotSpot(TM) we have now Dynamic Code Evolution.

Kind of installers

Worth noting is the fact that there are two kind of installers

• light
• and full

The latter one supports more features (for example, it supports removal of superclasses), but because of the maintenance issues the full edition is available for a fewer versions of jdk.

Downloading HotswapAgent

HotswapAgent does the work of reloading resources and framework configuration. So in order to have a support for reloading a spring bean definitions just after a change occurs, we need to perform one more step - download latest release of hotswap-agent.jar and put it anywhere. For example here: ~/bin/hotswap/hotswap-agent.jar.

Running application in order to test hot swapping

I will use Main and Main2 classes to play with hot swapping:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

public class Main {

    static class Foo {
        int counter;

        void foo() {
            System.out.printf("foo - %08d", counter);
            counter++;
        }
    }

    static Foo foo = new Foo();

    static int counter = 0;

    static void mainLoop() {
        System.out.printf("tick - %08d\t", counter++);
        foo.foo();
        System.out.println();
    }

    public static void main(String[] args) throws InterruptedException {
        while (true) {
            mainLoop();
            Thread.sleep(2000);
        }
    }
}

And the second one:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

public class Main2 {

    static class Foo {

    }

    static void mainLoop() {

        String fields = Stream.of(Foo.class.getFields())
                .map(f -> f.getName())
                .collect(Collectors.joining(",", "[", "]"));
        String methods = Stream.of(Foo.class.getDeclaredMethods())
                .map(m -> m.getName())
                .collect(Collectors.joining(",", "[", "]"));

        System.out.printf("fields=%s\t methods=%s\n", fields, methods);

    }

    public static void main(String[] args) throws InterruptedException {
        while (true) {
            mainLoop();
            Thread.sleep(2000);
        }
    }
}

I will test following use cases:

Intellij IDEA settings

All tests will be performed using Intellij IDEA. Ensure that following options are set

• enable classes reloading

• pass -XXaltjvm=dcevm vm option to run/debug configuration

Case 1 : change body method

Run debug. In the console you should see following output:

1
2
3
4

tick - 00000000	foo - 00000000
tick - 00000001	foo - 00000001
tick - 00000002	foo - 00000002
tick - 00000003	foo - 00000003

Then change counter++; to counter+=2; in Foo class.

1
2
3
4
5
6
7
8

static class Foo {
    int counter;

    void foo() {
        System.out.printf("foo - %08d", counter);
        counter+=2;
    }
}

Hit <ctr>+<shift>+<F9> to compile and after few seconds you should spot that the classes were reloaded successfully

1
2
3

tick - 00000004	foo - 00000004
tick - 00000005	foo - 00000006
tick - 00000006	foo - 00000008

Case 2 : add method

Revert all changes in Main class and run debug. Add method

1
2
3

void bar() {
    System.out.printf("\tbar - %08d", counter);
}

to the Foo class and call it from the mainLoop

1
2
3
4
5
6

static void mainLoop() {
    System.out.printf("tick - %08d\t", counter++);
    foo.foo();
    foo.bar();
    System.out.println();
}

Hit <ctr>+<shift>+<F9> to compile.

1
2
3
4
5

tick - 00000003	foo - 00000003
tick - 00000004	foo - 00000004
tick - 00000005	foo - 00000005	bar - 00000006
tick - 00000006	foo - 00000006	bar - 00000007
tick - 00000007	foo - 00000007	bar - 00000008

Classes were reloaded successfully.

Case 3 : add field

Revert all changes in Main class and run debug. Add field int counter2 to Foo class and append following two statements to the end of foo method.

1
2

System.out.printf("\tcounter2 - %08d", counter2);
counter2++;

Foo class should look following

1
2
3
4
5
6
7
8
9
10
11
12

static class Foo {
    int counter;

    int counter2;

    void foo() {
        System.out.printf("foo - %08d", counter);
        counter++;
        System.out.printf("\tcounter2 - %08d", counter2);
        counter2++;
    }
}

Hit <ctr>+<shift>+<F9> to compile. And appears

1
2
3
4
5

tick - 00000002	foo - 00000002
tick - 00000003	foo - 00000003
tick - 00000004	foo - 00000004	counter2 - 00000000
tick - 00000005	foo - 00000005	counter2 - 00000001
tick - 00000006	foo - 00000006	counter2 - 00000002

that this kind of change was also reloaded successfully.

Case 4 : remove field

Run debug. In the console you should see following output:

1
2
3

fields=[]	 methods=[]
fields=[]	 methods=[]
fields=[]	 methods=[]

Then add two public fields int number and String name to the Foo class.

1
2
3
4

static class Foo {
    public int number;
    public String name;
}

Hit <ctr>+<shift>+<F9> to compile, and after few seconds…

1
2
3

fields=[]	 methods=[]
fields=[number,name]	 methods=[]
fields=[number,name]	 methods=[]

Then remove number field and hit again <ctr>+<shift>+<F9>.

1
2
3

fields=[number,name]	 methods=[]
fields=[name]	 methods=[]
fields=[name]	 methods=[]

The change was reloaded.

Case 5 : remove method

Revert all changes in Main2 class and run debug. Add

• public field String name
• public method String getName()

to the Foo class. Then hit <ctr>+<shift>+<F9> to compile, and after few seconds…

1
2
3

fields=[]	 methods=[]
fields=[name]	 methods=[getName]
fields=[name]	 methods=[getName]

Then remove getName() method and hit again <ctr>+<shift>+<F9>.

1
2
3

fields=[name]	 methods=[getName]
fields=[name]	 methods=[]
fields=[name]	 methods=[]

Seems that this change was also reloaded successfully.

Notes

During the play with hot swapping in Intellij IDEA you could notice that for some circumstances code would not be reloaded. Intellij IDEA has a following limitation about which you need to be aware:

the old code is still used until the VM exits the obsolete stack frame

About that you can read here

Resources

• DCEVM
• HotswapAgent Quick start
• HotswapProjects
• Hotswap/DCEVM doesn’t work in Intellij IDEA (Community Version)
• Using DCEVM and Hotswap for rapid turnaround

Just after passing Java SE 8 Programmer 1Z0-808 Exam (about that you can read here) and being impressed by OCA: Oracle Certified Associate Java Se 8 Programmer I Study Guide: Exam 1z0-808, I decided to buy the newly released OCP: Oracle Certified Professional Java Se 8 Programmer II Study Guide: Exam 1Z0-809. It was a really good purchase.

The book is awesome. It has a similar style and feel as the OCA: Oracle Certified Associate Java Se 8 Programmer I Study Guide: Exam 1z0-808. It covers many new features, in details, introduced in java 8. I learn a lot about lambda, collection streams, date api and so on. I can tell you that this book was worth its money.

It took me almost 4 months to go through this book. To be honest I read it two times. Same as previously (OCA - review) I was generally reading the book during my way to work. First time I really haven’t paid attention on solving tests. I have just gone through the book to better know java 8. After that I took a few weeks break. When I started solving test questions I noticed that I had a problem with java API. It is really annoying that the test exam requires from developer to remember api. In 2016 we have ides, written mainly for this purpose. Due this fact I started reading a book again. This time however, I have been solving test question just after I read a chapter.

In order to prepare to the final exam I used sybex platform. Due the fact that I bought the book I got an access to this site where I found 411 test questions divided into 3 sets:

• Assessment Test (20) - helps you focus your study to specific objectives
• Chapter Tests (211) - taken from the Review Questions at the end of chapters in the book
• Practice Exam (180) - provided to help you prepare for the certification test

Moreover I had an access to the following resources:

• Flashcards - help to reinforce your learning and provide last-minute test prep before the exam
• Searchable Glossary - instant access to the key terms you’ll need to know for the exam
• Bonus Appendix - provides additional information about JavaScript with Nashorn, and enables you to test your knowledge and understanding of this topic.

The exam lasted 150 minutes. At the end of the exam I still haven’t answered on the two questions. I can say that the exam was much more difficult than the Java SE 8 Programmer 1Z0-808 Exam.

After receiving the email that my exam results were available I browsed to the CertView and it appeared that I scored 83%.

After successfully passed Java SE 8 Programmer 1Z0-808 Exam I would like to share my thoughts about the preparation and the exam.

I passed the exam and earn the badge on Dec 2015 - so almost half year ago (since writing this post). Before preparation I was playing already with java 8 a little. I have written two, maybe three applications which were using some new hot features like lambda, default methods and streams. The whole knowledge how to do this I got mainly from oracle tutorials like

• Lambda Expressions
• Aggregate Operations
• Default Methods

In spite of my experience as a java programmer I decided to spend some time to learn more about java 8 to ensure that I would really pass the exam.

My journey I have started with the book: OCA: Oracle Certified Associate Java Se 8 Programmer I Study Guide: Exam 1z0-808. It took me some time, I think one month, to go through all chapters. The book I was mainly reading during my way to work in metro, bus or tram. I can say that I was enjoying reading it though I have already known most of the knowledge. It is written with a very simple language. I can say that this book could be understandable by peoples who speak english in an intermediate level. It covers all the topics which can appear on an exam. After finishing it I bought the voucher on an exam, taking place a week later. I had one whole week to play around java 8, make some tests and review things read in a book. To practice tests I used sybex platform. In order to get an access to this platform you only need to own a book.

On a site you can find 345 test questions divided into 3 sets

• Assessment Test (20) - helps you focus your study to specific objectives
• Chapter Tests (145) - taken from the Review Questions at the end of chapters in the book
• Practice Exam (180) - provided to help you prepare for the certification test Platform also provides an access to 2 other study tools
• Flashcards - help to reinforce your learning and provide last-minute test prep before the exam
• Searchable Glossary - an instant access to the key terms you’ll need to know for the exam

The day before the exam I decided to go sleep early. The exam lasted 150 minutes, and I had enough time to answer on all the questions. They weren’t difficult. I can say that the questions in a mockup exam from sybex were much harder. For example if the question was a multiple choice question, in a real exam I was informed how many answers I needed to choose, in a mockup exam I weren’t.

One hour after the exam I received an email that my exam results were available. My score was 97%. Regarding the response from Oracle I made mistakes in a questions from the following exam objectives:

• Define the scope of variables
• Manipulate data using the StringBuilder class and methods

It could sound strange that the experienced java developer made mistakes from the such basic topics. But believe me that the real intention of authors of the test is to trick you in an unexpected way. The question could seem to be about let’s say lambda syntax, but in real it tests your knowledge about variable scoping. During taking tests you need to be very careful and stay focused all the time.

This is a first post in a series of how Vim can make programmer’s life easier. Vim strikes a nice compromise between simple editor and monolithic IDEs. I find this tool very helpful during my day to day developers tasks. I don’t treat it as a replacement for my favorite IDE but rather as a supporting tool. In this post I would like to introduce folding - very nice and useful feature.

Folding

Folding lets you define which parts of the file you can see. For example in a method you can hide everything inside curly braces letting only definition of function be visible.

When you use fold command, Vim hides given text and leaves in its place a one-line placeholder. The hidden text now can be managed by this placeholder. In a screenshot above, you can spot four folded methods~(there are four one-line placeholders). The first one - fibonacci, consists of 15 rows, takes an int as a parameter and returns a long. There is no limit on how many folds you can create. You can even create folds within folds~(nested folds).

Vim offers six ways to create folds:

• manual - using Vim commands
• indent - corresponding to the text indentation
• expr - define folds with regular expressions
• syntax - based on the file’s language syntax
• diff - a difference between two files define folds
• marker

The fold commands

Manual folding

Suppose we want to hide 3 lines of the if statement in a fold

1
2
3
4
5
6
7
8
9

public static long fibonacci(int n) {

  if (n == 0 || n == 1) {
    return n;
  }

  return fibonacci(n - 1) + fibonacci(n - 2);

}

To do this just move cursor to the beginning of if statement and execute:

1

3zf

The result should be similar to the one shown below

Let’s try more sofisticated command - fold block of code. Position the cursor over the beginning or ending brace of a block of code and type:

1

zf%

There is a one thing to note. Character following zf command - % is a motion command that moves cursor to the matching brace.

To learn more about folding and generally about vim, I recommend to read Learning the Vi and Vim Editors