Droid@Screen
I have recently start to learn programming Android. It’s Java, but the environment is very, very different from ordinary server and/or desktop Java programming. Mostly because the device is limited in all sorts of resources. Therefore, the API puts some burden on the lone hacker.
Anyway, I needed a simple tool to project the screen of my HTC Hero device on my PC, for showing it to an audience. I found one such application, but wasn’t really satisfied with its functionality and because I didn’t want to step on someones toes by messing around and changing way to much of the code I went for my own solution. My primary objective was to ensure it was operational before next Sunday, when I will need it.
Here is how it looks like
It is a simple Java Swing application, delivered as an all-inclusive executable JAR file. Therefore, one can launch it simply by double-clicking on the JAR file, or run the following command
java -jar droidAtScreen-0.3.jar
The Droid@Screen JAR file can be found here (pls understand it is alpha software)
Source Code
The source code is open source and I have put all of it to my git repo at Codaset.
The project is built using Maven. Before you can compile, you need to install two Android JAR files to your Maven repo. It is the android.jar and the ddmlib.jar files. Use the mvn install-file command. The easiest to get the mvn syntax right is to try compile it and wait for Maven to point out how to run the install-file command.
Screen-Shots
Here are some more screen-shots.
Landscape mode
Scaled image
Multiple windows
Save screenshots
Setting the path to the ADB executable
The application relies upon the Android Debug Bridge (ADB) application, which will be started unless it’s already running. So the very first thing Droid@Screen does is to prompt for that path.
—-
Introscope extensions
Back in January 2005 I joined an existing San Francisco based company named Wily Technology. This was the beginning of a few years of hectic traveling, hacking and trouble-shooting. Meaning I traveled around in Europe, mostly UK and Germany, working at customer sites (some very large banks) helping them with performance and problem analysis of large Java EE systems.
The tool Wily provided was named Introscope and was the first tool in its market segment when it started in 1999 and was the market leader all the time. One of my key missions was developing (hacking) numerous extensions for Introscope based on customer requests or my own observation of what I though the customers (or we professional consultants) needed.
As part of my re-launch of Ribomation as an independent consultant I have partnered with CA-Wily to partly do what I once did for Wily. Wily Technology was later acquired by CA. During this spring I have helped a governmental institution in Northen Sweden with a roll-out of Introscope as a monitoring solution for their Java systems.
One specific task was to enable alert integration with Tivoli. Back at the Wily era, I wrote an alert action extension for OpenView integration. In this case, I decided to make the extension more general and versatile, ant not tied to any specific SMC tool. The end-result is now released as open source and can be found here: SMC AlertAction.
New organization of RiboUtils
When re-launched Ribomation in April this year I hastely re-organized the web site as well, moving this blog into blog.ribomation.com and setting up the new ‘corporate’ site using Drupal. Something I (intentionally) left for a later time was making a new home for RiboUtils, my over the years collection of small helper classes and tools.
Now have I started to take care of its new home. From now on you can find all RiboUtils at lib.ribomation.com. The starting page is extremely lame, but it show the links to the directory of the maven generated docs, the maven repo and the subversion repo.
What remains to do, is regenerating all maven docs updating the page links and import the sources into subversion. This is something I will do gradually, hrm…, some rainy day.
Using TMI instead of Actors
Here in Sweden the spring has finally arrived and one of the duties I have to do is switching from winter to summer tires on my car. I did that this morning and while waiting in the garage I was reading the two articles about Actor concurrency in JavaWorld. The first article described the actor semantics and how it was realized in the Erlang programming language and the second article described various ways to realize actors on top of the JVM, such as Scala, Groovy and Java it self.
I wasn’t aware of the multitude of actors implementations in Java, as the second article describe. The author Alex Miller implemented the same problem (Rock-Paper-Scissors) for each actor realization and compared the code and effort. This was interesting reading indeed.
Thread Method Invocation (TMI)
Many years ago, back in 2002 to be exact, I implemented a library for thread method invocation (TMI) in Java. The idea was to provide various message passing techniques to Java, ranging from simple mailboxing to non-deterministic rendezvous. The library is essentially different implementation of message queues.
Together with the library I wrote a (long) tutorial that started from the very beginning with motivation of why synchronized and wait/notify/notifyAll are needed and then step-by-step moved up in the abstraction level via different forms of message passing, ending in non-determinsitic rendezvous, inspired by the Ada programming language. The article was submitted to JavaWorld, but was declined. After that I was busy with so many other things that the idea and the libray went into limbo. Perhaps it might resurrect nowadays.
After having read the second article with its various Rock-Paper-Scissors implementations, I couldn’t resist the temptation to do my own implementation of Rock-Paper-Scissors using TMI. The rest of this post describes my implementation.
Before I continue, let’s state first that TMI is not an actors model. It is close in spirit, becuase it’s a message passing model, but do not honor the value passing semantics. You can pass what ever object you want as a sender argument or as a response object (rendezvous). Still I consider TMI be a very useful concurrency model and simplifies designing thread based Java programs a lot.
Rock-Paper-Scissors using TMI
The main class is simple and straight forward, let’s show it first. It creates two players and a coordinator and kicks the latter to start the game. The program runs for a specified number of seconds.
public class RockPaperScissorsUsingTMI {
public static final String ROCK = "rock";
public static final String PAPER = "paper";
public static final String SCISSORS = "scissors";
public static void main(String[] args) throws InterruptedException {
int runIntervalInSeconds = 20;
if (args.length > 0) runIntervalInSeconds = Integer.parseInt( args[0] );
Player p1 = new Player(1);
Player p2 = new Player(2);
Coordinator c = new Coordinator(p1, p2);
c.start(); p1.start(); p2.start();
c.startPlay();
Thread.sleep(runIntervalInSeconds * 1000);
}
}
Here is a sample output of running the program.
>java -jar RockPaperScissorsUsingTMI-1.0.jar 15 0) Player-1: scissors -> paper 1) Player-1: scissors -> paper 2) Player-1: scissors -> paper 3) Player-1: rock -> scissors 4) Player-2: scissors -> paper 5) Player-2: rock -> scissors 6) TIE: scissors 7) TIE: scissors 8 ) Player-2: rock -> scissors 9) Player-2: scissors -> paper 10) Player-2: rock -> scissors 11) Player-1: rock -> scissors 12) Player-2: rock -> scissors 13) Player-1: scissors -> paper 14) Player-2: rock -> scissors 15) TIE: paper 16) Player-1: rock -> scissors 17) Player-2: paper -> rock 18) TIE: rock >
The Player
The player class below is simple, but illustrates the TMI realization. In this case I have chosen to be close to the actors model and are using an unbounded message queue. The interesting methods are play() and receive() which shows the ’sender’ side and the ‘receiver’ side of a message queue. The sender invokes play(), which inserts a message into the queue. Because the queue is unbouded, this is not a blocking call. It’s easy to change the queue to a rendezvous queue, which results in the sender are blocked until the receiver puts a reply. Inside the run() method, a player recieves the coordinator. The player is blocked as long as the input queue is empty. Then it waits for a random amout of time and performs a move. It responds back to the coordinator using itself as argument.
public class Player extends Thread {
private static final List<String> moves = Arrays.asList(ROCK, PAPER, SCISSORS);
private static final Random r = new Random();
private MessageQueue playMsgs = new UnboundedMessageQueue();
private String move;
public Player(int id) {
super("Player-"+id);
setDaemon(true);
}
public void run() {
try {
while (true) {
Coordinator c = receive();
delay( r.nextInt(1000) );
move = makeMove();
c.throwResult(this);
}
} catch (InterruptedException e) {System.out.println(getName()+" Interrupted");}
}
public void play(Coordinator master) {
try {
playMsgs.put(master);
} catch (InterruptedException e) { }
}
private Coordinator receive() {
try {
return (Coordinator) playMsgs.get();
} catch (InterruptedException e) {return null;}
}
private String makeMove() {
return moves.get( r.nextInt(moves.size()) );
}
public String getMove() {
return move;
}
protected Player setMove(String move) {
this.move = move;
return this;
}
private void delay(int numMilliSecs) {
try {
sleep(10 + numMilliSecs);
} catch (InterruptedException e) { }
}
}
The Coordinator
The coordinator is slightly more complex, due to its responsibilities. It waits for a start message, then ask each player to play and receives the throw messages in either order. It then evalutes who won this round. The concurrency part is not very complicated. It has two unbouded queues, one for each message type. And for each message type it has a public sender method (startPlay/throwResult) and a private receive method (receiveStart/receiveThrow). The Permutation class is just a helper for computing or winner.
public class Coordinator extends Thread {
private MessageQueue startMsgs = new UnboundedMessageQueue(),
throwMsgs = new UnboundedMessageQueue();
private Player player1, player2;
public Coordinator(Player player1, Player player2) {
super("Coordinator");
this.player1 = player1;
this.player2 = player2;
setDaemon(true);
}
public void run() {
try {
for (int count = 0; true; count++) {
receiveStart();
player1.play(this);
player2.play(this);
Player first = recieveThrow(),
second = recieveThrow();
if (isTie(first, second)) {
System.out.printf("%4d) %8s: %s%n", count, "TIE", first.getMove());
} else if (isWinning(first, second)) {
System.out.printf("%4d) %8s: %s -> %s%n",
count, first.getName(), first.getMove(), second.getMove());
} else {
System.out.printf("%4d) %8s: %s -> %s%n",
count, second.getName(), second.getMove(), first.getMove());
}
this.startPlay();
}
} catch (InterruptedException e) {System.out.println(getName()+" Interrupted");}
}
public void startPlay() throws InterruptedException {
startMsgs.put(this);
}
private void receiveStart() throws InterruptedException {
startMsgs.get();
}
public void throwResult(Player p) throws InterruptedException {
throwMsgs.put(p);
}
private Player recieveThrow() throws InterruptedException {
return (Player) throwMsgs.get();
}
protected boolean isWinning(Player first, Player second) {
Permutation permutation = new Permutation(first, second);
return permutation.dominates(ROCK , SCISSORS) ||
permutation.dominates(PAPER , ROCK) ||
permutation.dominates(SCISSORS, PAPER);
}
protected boolean isTie(Player first, Player second) {
return new Permutation(first, second).tie();
}
private static class Permutation {
private Player first, second;
private Permutation(Player first, Player second) {
this.first = first;
this.second = second;
}
public boolean dominates(String m1, String m2) {
return first.getMove().equals(m1) && second.getMove().equals(m2);
}
public boolean tie() {
return first.getMove().equals( second.getMove() );
}
}
}
Source code
My application is developed using Maven, which means it very easy to download and unpack the sources and then type
mvn package java -jar target\RockPaperScissorsUsingTMI-1.0.jar 60
Links
- Source code
- Compiled JAR file
- Thread MessageQueue library
- Towards Ada Style Thread Communication in Java
- Understanding actor concurrency, Part 1: Actors in Erlang
- Understanding actor concurrency, Part 2: Actors on the JVM
- Cuckoo for Concurrency
Scala seems to be fun
The second day of Jfokus was an ordinary conference day, with separate tracks and many talks to choose between.
JavaFX
The key note talk at the beginning described JavaFX, SUNs new “SilverLight and Flash” killer. The demo showing how to flip through a book with very realistic behaviour, was impressive. On the other hand I’m not convinced SUN is gonna make it. MS is pushing out SilverLight in each Windows Update and SUN simply have not that capability to “pre-load” desktop browsers with the appropriate plugin. Remember the applet war ten years ago. Don’t get me wrong - I like JavaFX, however I’m pessimistic. Anybody still remembering JINI?
Capuchin
The second talk I attended was Ericsson’s new JavaME+Flash/light environment. I’m even more pessimistic about this new tool. It might be a bright piece of technology - I cannot really judge without getting my hands on it. However, will it ever be supported by other vendors? Will it be open source? How many of Ericssons own phone models will have it? There are way too many questions, before one starts to invest time and money on a new (isolated) platform.
Groovy
After lunch it started to be fun. First there was a talk about Groovy support in JetBrains’ IntellijIDEA. I have been using IDEA since the beginning of 2002 and usually uses the latest version all the time. For me personally there wasn’t anything new, becuase I have done some Groovy and Grails hacks using IDEA already. But the presentation was fun and entertaining and hopefully it inspired some in the audience to embark the Groovy way.
JPA 2.0
I realized that EJB3 / JPA still is not my cup of tea. Either I prefer something more high-level, like Grails or something more low-level, like Spring’s JDBC-Template.
Puke
A fun role playing of the importance of thinking and not following, when it comes to productive software development. A standpoint I fully support.
JRockit
Some interesting optimization techniques for the JRockit JVM. However, why did they start to compete with SUN in the first place?
Java++
Ola Bini is always fun to see on stage. He outlined the future of Java as a system implementation language and new dynamic languages on top of the JVM, such as Groovy, Scala, JRuby and others. I fully agree. Java has over the years become a dinosaur and it’s time to move on. Sort of, the same evolution as for 15-20 years ago when C faded away for C++.
Scala
My best experience of the day was Jonas Bonér talking about Scala. I have been curious about Scala a while now, but haven’t really reached above the threshold of actually reading and playing around with it. This has now changed. I’m a Groovy advocate, but will over this year make room for Scala as well. I doubt Scala is ready for prime time yet - not because any technical limitations, rather because of commercial aspects. But the day will come and I will be ready that day.
Scala seems to be a multi-paradigm language; static typing but allowing dynamic type constructs, object oriented programming, functional language programming and actors based concurrent programming. I really like the latter. As far as I understand, the support for message passing is inspired by the Erlang programming language. Back in the beginning of the 90’s, as did some hacks in Erlang, like a simulator for a small phone exchange with support for connecting it to a real PBX exchange (MD110). This was part of my teaching course in real-time systems programming, for Royal Inst of Technology in Stockholm. Besides of Erlang, I used Concurrent C++ as well in the course. Those were the days.
I have never liked the path taken by the current support of concurrency in Java. Doug Lea’s library, which now is the java.util.concurrent package, is simply the wrong way to go. Not because the library is bad - it is not. But, the users are . . . I mean, it is inherently difficult to design concurrent programs right. As soon it starts to be a little more complex, than a bunch of threads doing some asynchronous work. Message passing is the right way to realize concurrent programs. It is a pity, my implementation of thread method invocation from 2002 never got any attention. Well, that’s life. If you are interested, you can find the article here.
Summary
The first presentation about WebSockets and the last presentation about Scala, was my personal highlights of the 2009 JFokus conference.
New job
Next week (1st December) I’m switching to a new job for Connecta, which is a consultancy company in central Stockholm. My role will be to lead and build up the Java knowledge of a team in the area of Enterprise Java.
A first look at Spring-Batch, part 2
In my first post about Spring-Batch, I described in detail a Hello-World application using Spring-Batch and discussing the necessary plumbing wiring needed in the spring-beans configuration file. In this second post I will take it one step further, by introducing the concept of tokenizer and field-set mapper.
I will copy-reuse as much as possible from the previous project. Because the reuse is by copy, you can download and study them independent, without any compile/runtime dependencies between them. (In constrast to the Spring-Batch samples, wich is one big heap of code)
The Application
The application reads a set of person data from a file, creates a person object and prints it out. Simple as that and still a toy application, however it allows you to concentrate on the key concepts of tokenizer and mapper.
The Input File
Let’s start with the input data. It is in CSV (Character Separated Values) format and located at the top of the class path.
Name;Street;PostCode;City Anna Conda;Hacker street 17;12345;Javaville Sham Poo;Reboot lane 5;67890;Perlvillage Sandy Shoes;Desert town street 11;98765;Cobolburgh
The Domain Class
The overall objective is to transform each record in the CSV file into a Person object. Here is the Person class for easy reference. It uses the ToStringBuilder from the Jakarta Commons Lang project.
package com.ribomation.tutorial;
import org.apache.commons.lang.builder.ToStringBuilder;
import org.apache.commons.lang.builder.ToStringStyle;
public class Person {
private String name, street, postCode, city;
public String toString() {
return ToStringBuilder.reflectionToString(this, ToStringStyle.SHORT_PREFIX_STYLE);
}
//. . .getters and setters. . .
}
Tokenizers and Mappers
The file above is a so called flat-file, each row is a record and each record is subdivied into fields, where the fields are separated by a semi-colon character. We will be using a FlatFileItemReader to read from our (class path) resource. The reader will read one line at a time, need a means to break the line into fields. This is the task for tokenizer, in our case we will use a DelimitedLineTokenizer and tell it to split fields around semi-colon.
A tokenizer gets a string and returns a FieldSet. The next step is to convert the field-set into an business object. A FieldSetMapper gets a field-set and returns a fresh new object. This mapper class is something you often have to implement yourself, although in many cases you can get way with a BeanWrapperFieldSetMapper. I don’t not want to use too many magics at once, so here is the very straight-forward mapper class.
package com.ribomation.tutorial;
import org.springframework.batch.item.file.mapping.FieldSetMapper;
import org.springframework.batch.item.file.mapping.FieldSet;
public class PersonMapper implements FieldSetMapper {
public Object mapLine(FieldSet fs) {
Person p = new Person();
int idx = 0;
p.setName ( fs.readString(idx++) );
p.setStreet ( fs.readString(idx++) );
p.setPostCode( fs.readString(idx++) );
p.setCity ( fs.readString(idx++) );
return p;
}
}
The Spring Beans
Now we know sufficient to configure the reader in the Spring beans config. As I said initially, I reuse (by copy) as much as possible from the first hello spring-batch application, so I will only show you the new/changed XML snippet.
<bean id="inputFile" class="org.springframework.core.io.ClassPathResource">
<constructor-arg value="/names.csv"/>
</bean>
<bean id="reader" class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="resource" ref="inputFile"/>
<property name="firstLineIsHeader" value="true"/>
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value=";"/>
</bean>
</property>
<property name="fieldSetMapper">
<bean class="com.ribomation.tutorial.PersonMapper"/>
</property>
</bean>
Running the Application
Now we are ready to compile/build and execute. The new POM aslmost the same as the previous, except the artifact name is ‘HelloSpringBatch2′ (plus a new dependency for commons-lang). Build the application with
mvn package
Finally run the application with the command below. We are reusing the LogWriter from the first application, which just prints out the supplied object using its toString() method.
HelloSpringBatch-2> java -jar target\HelloSpringBatch2-1.0.jar hello-spring-batch.xml helloJob
[SimpleJobLauncher] No TaskExecutor has been set, defaulting to synchronous executor.
[SimpleStepFactoryBean] Setting commit interval to default value (1)
[SimpleJobLauncher] Job: [SimpleJob: [name=helloJob]] launched with the following parameters: [{}{}{}{}]
[LogWriter] Person[name=Anna Conda,street=Hacker street 17,postCode=12345,city=Javaville]
[LogWriter] Person[name=Sham Poo,street=Reboot lane 5,postCode=67890,city=Perlvillage]
[LogWriter] Person[name=Sandy Shoes,street=Desert town street 11,postCode=98765,city=Cobolburgh]
[SimpleJobLauncher] Job: [SimpleJob: [name=helloJob]] completed successfully with the following parameters: [{}{}{}{}]
A Small Variation
I mentioned above we can get away with implementing a mapper class, if it’s easy to map fields to bean properties. So let’s do just that.
A BeanWrapperFieldSetMapper need to know the destination class (or use a protype bean) and know the names of each field. My own mapper class above intentionally used indexing instead of field names. Look at the input field again, the first line contain the field names
Name;Street;PostCode;City Anna Conda;Hacker street 17;12345;Javaville . . .
and the reader configuration contained an instruction to interpret the first line as the field name defintion
<bean id="reader" class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="firstLineIsHeader" value="true"/>
<property name="fieldSetMapper" ref="mapper"/>
. . .
That’s all we need, to provide BeanWrapperFieldSetMapper with sufficient information for it to perform its duties.
<bean id="mapper" class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper"> <property name="targetType" value="com.ribomation.tutorial.Person"/> </bean>
Don’t forget to recompile/package (mvn package). The execution will produce the exact same output as before.
HelloSpringBatch-2>java -jar target\HelloSpringBatch2-1.0.jar hello-spring-batch.xml helloJob
[SimpleJobLauncher] No TaskExecutor has been set, defaulting to synchronous executor.
[SimpleStepFactoryBean] Setting commit interval to default value (1)
[SimpleJobLauncher] Job: [SimpleJob: [name=helloJob]] launched with the following parameters: [{}{}{}{}]
[LogWriter] Person[name=Anna Conda,street=Hacker street 17,postCode=12345,city=Javaville]
[LogWriter] Person[name=Sham Poo,street=Reboot lane 5,postCode=67890,city=Perlvillage]
[LogWriter] Person[name=Sandy Shoes,street=Desert town street 11,postCode=98765,city=Cobolburgh]
[SimpleJobLauncher] Job: [SimpleJob: [name=helloJob]] completed successfully with the following parameters: [{}{}{}{}]
This new version only uses two written classes; the domain class Person and the writer class LogWriter. The rest is pure configuration.
A Second Variation
Let’s twist this application a second time by changing the output format. Instead of using our LogWriter, let Spring-Batch convert the output to XML.
We will use another bean called xmlWriter, which is of type StaxEventItemWriter. This bean needs to have a XML serializer, an output resource and the name of the XML root tag. The serializer uses a marshaller, which is an abstraction around different XML O/X mapping tools. We will use XStream, a light-weight XML tool.
Here are all additions to the beans configuration file. The xmlWriter reference should replace the LogWriter reference in the helloStep configuration.
<bean id="xmlWriter" class="org.springframework.batch.item.xml.StaxEventItemWriter">
<property name="rootTagName" value="persons"/>
<property name="serializer" ref="xmlSerializer"/>
<property name="overwriteOutput" value="true"/>
<property name="resource" ref="xmlOutputFile"/>
</bean>
<bean id="xmlSerializer" class="org.springframework.batch.item.xml.oxm.MarshallingEventWriterSerializer">
<constructor-arg>
<bean class="org.springframework.oxm.xstream.XStreamMarshaller"/>
</constructor-arg>
</bean>
<bean id="xmlOutputFile" class="org.springframework.core.io.FileSystemResource">
<constructor-arg value="persons.xml"/>
</bean>
The output now goes to a file, named ‘persons.xml’ in the current directory. Don’t forget to recompile/rebuild and run it as before. The contents of the produced file is
<?xml version="1.0" encoding="UTF-8" ?>
<persons>
<com.ribomation.tutorial.Person>
<name>Anna Conda</name>
<street>Hacker street 17</street>
<postCode>12345</postCode>
<city>Javaville</city>
</com.ribomation.tutorial.Person>
<com.ribomation.tutorial.Person>
<name>Sham Poo</name>
<street>Reboot lane 5</street>
<postCode>67890</postCode>
<city>Perlvillage</city>
</com.ribomation.tutorial.Person>
<com.ribomation.tutorial.Person>
<name>Sandy Shoes</name>
<street>Desert town street 11</street>
<postCode>98765</postCode>
<city>Cobolburgh</city>
</com.ribomation.tutorial.Person>
</persons>
This concludes my second post of Spring-Batch.
Source Code
A first look at Spring Batch
Spring-Batch is a rather new project within the Spring portfolio. It addresses a large field within computing, although not main stream in Java. A lot of corporate computing is managed by batch processing, many business transactions based on file input picked up from FTP drop zones etc.
Back in 2003, I built a batch-oriented system, that could deal with FLV (Cobol) files, assemble transaction data from a database, generate reports in various formats and push them away over FTP, HTTPS or mail. One offspring of that project is my library for reading and writing FLV files.
It is therefore a dejavu to reconnect to the ideas and principles behind Spring-Batch. As common for Spring projects, it solves more than one design problem and provides a smorgasboard of solutions. The only drawback is the lack of introductory reading material, which makes the introduction steeper than it needs to be. So, let’s fill that gap.
A little bit of theory
Spring-Batch can be subdivided into two areas which you can use separately; the first is item handling and the second batch executions.
Item Handling
Let’s start discuss item handling. This means reading and interpreting file or database contents and write it to file or database. In this area SpringBatch really shines. During interpretation you typically want to create business objects, operate and transform them. SpringBatch comes with support for both flat files, structured file and database access.
A flat file is either a CSV (Character Separated Values) of FLV (Fixed Length Values) file. A structured file is for example XML. Typically you assemble a tokenizer with a mapper that produces business objects and the other way as well. A tokenizer understands the file format and shields the rest of the application, making it easy to swap file formats. An activity more common than expected, because transaction data suppliers often delivers in various obscure file formats.
Item handling is the easy to understand part of SpringBatch. And, as I said above, you can use it as is without touching the other part; batch executions.
Batch Execution
Batch executions in general and within SpringBatch in particular, you design a solution around the concept job, which is a named sequence of steps. A step is a chunk of work, typically processing an input file.
One important execution condition for batch processing is operation performance monitoring and management, which in practice means the ability to track individual step instances and in case of need, restart a step (or job) and continue from from where it left. In order to fulfill this requirement, traditional logging is not sufficient and therefore is batch processing surrounded by lots of tracking logic and the execution progress is tracked and persisted to a database.
With this said as background information, it is easier to understand and approach SpringBatch. You can also easier take a decision do you need both components or is it sufficient with the item handling part. For every non-trivial batch application, you will end up with a loop over the input data anyway, so way not give the batch execution part a chance as well? “Nuf talking, show me the code”
The Job Model
You organize a SpringApplication in one or more jobs. If you have more than one job, they typically reuse/share lots of functionalities when ti comes to step and item processing logic. The required infrastructure is a job repository, a job launcher (runner) and a transaction manager. The latter is needed for committing chunks of work.
This is the intended way of working - the model. On the other hand, at least during investigation and early development (and maybe later as well), you have no need for transaction management and persisted execution tracking. The good news is you can fake it, which is exactly what we will do below.
Hello SpringBatch
The (first) SpringBatch application will be a minimal ‘hello world’ application, just to demonstrate what is required to get something up and running. I will use Maven, because it hides all the tedious tasks of managing all 3rd party libraries SpringBatch depends on.
Writer
The initial version contains only one single very small Java class. It prints out its input using Log4j. Let’s start with this one, so we can move on to the interesting parts.
package com.ribomation.tutorial;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.FlushFailedException;
import org.springframework.batch.item.ClearFailedException;
import org.apache.log4j.Logger;
public class LogWriter implements ItemWriter {
private Logger log = Logger.getLogger(this.getClass());
public void write(Object item) throws Exception {
log.info( item );
}
public void flush() throws FlushFailedException { }
public void clear() throws ClearFailedException { }
}
Reader
This class serves as the end-point of a chain of components, that reads items from a list. Here is the list definition as given in a spring-beans configuration file.
<bean id="reader" class="org.springframework.batch.item.support.ListItemReader">
<constructor-arg>
<list>
<value>Hello</value>
<value>Spring</value>
<value>Batch</value>
</list>
</constructor-arg>
</bean>
<bean id="writer" class="com.ribomation.tutorial.LogWriter"/>
You can see the input side is a Reader and the output side is a Writer. For every non-toy application these abstractions are tied to files and/or databases. But leave that out for the moment. So what happens in between?
Step
A step is a chunk of work, for example reading the items from the list one at a time and sending them to the writer. As I said above, SpringBatch supports a heavy-weight execution model intended to track step instance executions ans support restarts. For this reason, the configuration of a trivial step is more complex than expected. You will need a transaction manager and job repository, in addition to the two more obvious reader and writer. Here is our spring snippet
<bean id="helloStep" class="org.springframework.batch.core.step.item.SimpleStepFactoryBean">
<property name="transactionManager" ref="tm"/>
<property name="jobRepository" ref="jobRepository"/>
<property name="itemReader" ref="reader"/>
<property name="itemWriter" ref="writer"/>
</bean>
You can see it uses a factory bean to create the actual step behind the scenes. There are several intricate ways to create a step, but this will do for the moment.
Job
A job is a sequence of steps, which means each step is run to completion before the next is started. (Support for concurrent execution of steps are around the corner). The easiest way to create a job is to (re-)use a SimpleJob. In our case, it has just one single step.
<bean id="helloJob" class="org.springframework.batch.core.job.SimpleJob">
<property name="jobRepository" ref="jobRepository"/>
<property name="steps">
<list>
<ref bean="helloStep"/>
</list>
</property>
</bean>
The listings above captures all our application logic. What remains is batch and build execution infrastructure.
Repo, Launcher and TM
In this toy application we do not need persistent execution tracking support and will use fake components. The job repository will store its job and steps in a hash map and the transaction manager used will be just empty (view its source code).
<bean id="tm" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
<bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean">
<property name="transactionManager" ref="tm"/>
</bean>
<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository"/>
</bean>
That’s it. This is all spring wiring needed. What remains is the Maven POM.
Maven POM
I will not digress into Maven here, just leave it as is. The POM, lists the required dependencies and adds a few nice to have plugins. For example, the dependency-plugin that assembles all 3rd party JAR files into a sub-directory and the jar-plugin that sets the class-path to this lib directory and points out the main-entry point, so we can run the artifact from the command line. The main class is CommandLineJobRunner, which is a small boot-strapper, that loads a spring beans configuration and kicks the job launcher.
Without more addo, here it is.
<?xml version="1.0" encoding="iso-8859-1"?>
<project
xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<name>HelloSpringBatch</name>
<groupId>com.ribomation.tutorial</groupId>
<artifactId>${project.name}</artifactId>
<packaging>jar</packaging>
<version>1.0</version>
<properties>
<javaVersion>1.5</javaVersion>
<springBatchVersion>1.1.1.RELEASE</springBatchVersion>
<springDaoVersion>2.0.8</springDaoVersion>
<springVersion>2.5.5</springVersion>
<log4jVersion>1.2.14</log4jVersion>
<appClass>org.springframework.batch.core.launch.support.CommandLineJobRunner</appClass>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>${springVersion}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-beans</artifactId>
<version>${springVersion}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-dao</artifactId>
<version>${springDaoVersion}</version>
</dependency>
<dependency>
<groupId>org.springframework.batch</groupId>
<artifactId>spring-batch-core</artifactId>
<version>${springBatchVersion}</version>
</dependency>
<dependency>
<groupId>org.springframework.batch</groupId>
<artifactId>spring-batch-infrastructure</artifactId>
<version>${springBatchVersion}</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>${log4jVersion}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-idea-plugin</artifactId>
<configuration>
<jdkLevel>${javaVersion}</jdkLevel>
<downloadSources>true</downloadSources>
<downloadJavadocs>true</downloadJavadocs>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>${javaVersion}</source>
<target>${javaVersion}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<index>true</index>
<manifest>
<mainClass>${appClass}</mainClass>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Log4j
We will also need a minimal log4j configuration file (log4j-properties), which resides together with the hello-spring-batch.xml configuration file, in the src/main/resources directory of our maven project.
log4j.rootLogger=info, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdoutTarget=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{HH:mm:ss} %5p [%c{1}] %m%n
log4j.logger.org.springframework=warn
log4j.logger.org.springframework.batch=info
log4j.logger.com.ribomation.tutorial=debug
Compilation and Execution
Build the maven application using
mvn package
Run it using the command below
java -jar target\HelloSpringBatch-1.0.jar hello-spring-batch.xml helloJob
As you can see, we run the executable JAR file, with two required command line parameters. The first points to the spring beans file in the class path, and the second parameter is the name of the job to run. The output of the execution looks like this
13:17:26 INFO [SimpleJobLauncher] No TaskExecutor has been set, defaulting to synchronous executor.
13:17:26 INFO [SimpleStepFactoryBean] Setting commit interval to default value (1)
13:17:26 INFO [SimpleJobLauncher] Job: [SimpleJob: [name=helloJob]] launched with the following parameters: [{}{}{}{}]
13:17:26 INFO [LogWriter] Hello
13:17:26 INFO [LogWriter] Spring
13:17:26 INFO [LogWriter] Batch
13:17:26 INFO [SimpleJobLauncher] Job: [SimpleJob: [name=helloJob]] completed successfully with the following parameters: [{}{}{}{}]
Let’s take one step back and review our (toy) application. SpringBatch has taken care of all plumbing code to iterate over an input source and invoke various components leaving us to concentrate on the core business - in our case just to print it out to the console.
A minor variation
Before I close this posting, let’s add one small variation. In the initial version of our hello application, we just let the data (items) pass on to the output writer. This is clearly not realistic - if we for the moment ignore the list input and stuff. Typically, the items need to be processed and/or transformed in some way. One possibility is to attach an item transformer to the writer.
Item Transformer
The class below takes care of transforming the input item (a string) into a another item (upper case string).
package com.ribomation.tutorial;
import org.springframework.batch.item.transform.ItemTransformer;
public class UpperCaseTransformer implements ItemTransformer {
public Object transform(Object item) throws Exception {
return item.toString().toUpperCase();
}
}
The next step is to attach the transformer to the Writer, using an ItemTransformerItemWriter, which is a delegating writer combined with a transformer invoker.
<bean id="transformingWriter" class="org.springframework.batch.item.transform.ItemTransformerItemWriter">
<property name="itemTransformer">
<bean class="com.ribomation.tutorial.UpperCaseTransformer"/>
</property>
<property name="delegate" ref="writer"/>
</bean>
Complicated? No, not really. It first invokes the transformer object, and then the writer sending it the transformed item. The only remaining task is to update the step definition, to use the transformingWriter. If you allow me, I leave that as an exercise for you. The only difference in the output is the item strings are now in upper case.
16:16:36 INFO [LogWriter] HELLO 16:16:36 INFO [LogWriter] SPRING 16:16:36 INFO [LogWriter] BATCH
Source Code
The Camel Distribution
Long time ago, before the internet era, I was working on my PhD in the field of distributed event-driven simulation. One of the key questions was how to manage the event time line. The pitch in this case is that the time line is distributed.
To be more precise; a simulation event time line is a priority queue, where event notices are inserted during the processing of one event. By definition, events are posted to a future point in simulated time. Distributed event-driven simulation, means that several processors are processing events, where each event emits new future events. Whenever an event is processed the simulated time is incremented monotonically. If several events are processed in parallel, it means the simulated time is blurred between the earliest and the latest time points of the events being processed. The trick is to ensure no new emitted event falls within this blurred time interval. Becuase, time cannot move backwards.
The time line was implemented using varius priority-queue implementations, suitable for execution on a multi-processor. I was especially interested in the performance of my implementation of a distributioned version of the Calendar queue .
During my tests I realized the need to emulate bursty traffic, i.e., burts of insertions of events into the time line priority queue. I couldn’t find a suitable stochastic distribution, that easily could module bursts. Like, “give me five bursts, over the distribution interval”.
I played around with several different stochastic distributions for random number generation, without finding a simple to use distribution with the property stated above. This inspired me to design a distribution myself, which was named the Camel distribution, because of the humps. A Camel distribution is a composition of one or more Dromedary distributions. A Dromedary distribution has one single hump.
The Camel distribution paper was published in 1991 and have been in use in various computer simulation projects ever since.
My first implementation for random number generation based on the Camel distribution was in C, because that was the language I used for my research. The implementation here is in Java. However, it is very easy to implement the algorithm in some other language. Based on the Java code or straight from the source. More information can be found here.
Speaker at JavaForum
I will give a (swedish) speach at JavaForum in Stockholm, 30 September 2008. The title is Rena kartor - GoogleMaps med AJAX and I will talk about how functional programming now is commodity on both the server and the client side. The basis of my talk is a small AJAX application using Google Maps. More info can be found here.
-
Jens Riboe
