Why oh why, then, does GMail still not have a “reply to list” functionality? It’s been requested many times, and if it was offered as a Google Mail Labs feature, it wouldn’t clutter the interface for non-technical people. How do the Google engineers themselves participate in mailing lists, by cutting and pasting between the Cc: and To: fields?

Seriously, it’s about time.

Rant off.

Recently at work, I wanted to execute some code for each member of a List, and I needed to know which iteration step I was at. A straightforward solution is, obviously, the classic for loop:

for (int i = 0; i < keyword.getSynonyms(); i++) {
    String synonym = keyword.getSynonyms().get(i);
    // do something
}

That’s fine in many cases, but it has two problems. The extra line to get at the List element is annoying. More importantly, depending on the List implementation the get(i) operation might be in O(n), requiring another scan of the list each time.

So we could iterate normally and count ourselves:

int order = 0;
for (String synonym : keyword.getSynonyms())
{
    // do something
    order++;
}

It’s better, but I needed several such loops, and I wondered if I couldn’t write all that plumbing just once. I came up with this:

public abstract class Counting<T, E extends Throwable>
{
    public void loop(Iterable<T> things) throws E
    {
        int step = 0;
        for (T t : things)
        {
            iteration(t, step);
            step++;
        }
    }

    public abstract void iteration(T thing, int step) throws E;
}

As the loop body can throw exceptions, and we might want to declare the specific kind of exception, this needs to be an additional generic type. This breaks if you need more than one exception type.

The above loop then becomes

new Counting<String, SQLException>()
{
    @Override public void iteration(String synonym, int step)
        throws SQLException
    {
        // do something
    }
}.loop( keyword.getSynonyms() );

Hmmm. Even after writing an abstract class to extract the repeated parts, and not counting my preference for opening-brace-on-new-line, I still haven’t saved a single line. Can I have map and first-class functions, please? Time for Scala?

Until two weeks ago, I had never heard of Gen2Phen. Then my colleague Livia asked me to join her to go to their 6th general assembly meeting and present something about UniProt in RDF.

Gen2Phen is a big consortium, including SIB, working on genotype-to-phenotype information. They have two years to go in their grant, and are thinking about adopting SemWeb technologies to enhance data exchange and integration, data interpretation, and to impress funding agencies. Therefore, they invited someone—me in the end— from SIB to speak about our experiences.

My presentation consisted of two parts, an introduction to RDF and why we provide it, and a tour of UniProt‘s RDF. I aimed for 15 minutes, and got only five to present it due to the packed schedule. So I explained the very gist of “why RDF”, showed some examples, and talked about the problems we are encountering.

The problems got, predictably, most attention. Semantic Web “believers” spreading the vision are plenty. Hands-on experiences with complex data sets such as UniProt’s are rarer. I need to write about this in depth at some point. Suffice it to say, I think I dampened some enthusiasm. This despite the fact that I repeatedly stressed that I think of RDF and related technologies as valuable building blocks in the bigger picture, and as clear steps forward on some problems. But the Semantic Web seems to be an all-or-nothing affair for most people.

Tony Brooks is right in saying that given there are only two years left to go for Gen2Phen, it might be late to start with SemWeb technology. A large modeling effort and uncertain scalability challenges could delay the benefits until it’s too late. On the other hand, it’s not that much work to start experimenting. Install Virtuoso and D2R, fire up Protege, write some RDF using Jena, and get a feeling for the whole thing. Design some RDF schema that expresses the basics of the information at the heart of Gen2Phen, and see if existing systems can add it as in- and output format. That would be my recommendation, which I might or might not have gotten across — it was a packed event about an unfamiliar project where the SemWeb was only one of many sessions, so communication was somewhat difficult.

The meeting as such was very nice. Good conversations and awesome food — La Maison de la Lozere in Montpellier was brilliant. So was the city itself; I enjoyed wandering around the beautiful old town.

One other presentation I found interesting was Gudmundur Thorisson’s about ORCID. This initiative aims to unambiguously researchers with an ID instead of their name, which might occur many times. ORCID will then map an article’s DOI to the IDs of the authors, when it’s submitted. Also, and perhaps even more important, ORCID aims to do the same for data sets. Science really needs more, larger, better data sets in the open for people to analyze and train their algorithms on, but currently there is very little benefit for researchers to publish them. ORCID is not really functioning yet, but is backed by more than 120 organizations, and so has a decent chance at becoming the de facto norm in academia.

I presented my “Praktisches RDF in Perl” talk that I recycled from the German Perl Workshop, to spread the word some more. This time, I had prepared an English version, but as I only had German speakers in the audience, I presented in German.

Unfortunately my presentation only drew a handful of people this time. Note to self: work on the abstract some more. I had suspected that my FrOSCamp one was wordy and not catchy, but didn’t get around to rewriting it. At least the audience were pretty engaged and asked lots of questions, which I prefer to a larger crowd that’s half asleep.

The presentation was recorded and is now online as slides+audio. This was a first for me. I could forget about it while presenting, but I was pretty nervous listening to it for the first time, not sure what mess of incoherent rambling and half-finished sentences to expect. Fortunately, I found it ok in the end. Of course, I found several things to improve, but I guess that’s expected for someone who doesn’t present often and is just getting started. My list of the main points to improve is:

• The introduction should be much shorter and more focussed. A bit like a sales pitch, not as in being obnoxious and fake, but as in focussed on getting the audience’s attention and appreciation for the topic.
• Too many sentences didn’t flow properly. Simply doing one or two more dry runs should fix that.
• Have some more visualizations such as diagrams on the slides.

On the other hand, I was pleased with a few things about my presentation: the style of having little text on the slides and more verbal explanation worked well, the code samples seemed to be the right size to digest during a talk, and the questions at the end showed that people had gotten the key points.

Before my presentation, I got to see Renee Baecker‘s talk about Perl::Critic. I’m using it on my code and thus knew the basics, but I appreciated the advanced example towards the end, where Renee walked us through writing our own critic rules. This works via PPI, so you can find patterns in the AST that match the constructs you want to check. I also found it interesting to hear Renee’s personal experience with the severity levels: he’s typically on 3, sometimes 2, but 1 is too harsh.

Other than that, I was mainly hanging out at the Perl booth, a first for me! The booth was staffed by Renee and Roman from Winterthur (CH), two really nice guys whom I had a great time with, discussing everything from Perl modules to freelancing.

BTW, remember the blurb from the FrOSCamp website I quoted at the top about creative commons licensed beer? That wasn’t a joke. FreeBeer is an organic beer, produced by an independent brewery near Zurich, and the recipe is online under a CC license. And it tastes great! A cloudy, full blonde just how I like it :-)

This article by symkat showed me a lot more interesting history features.

Here’s my personal list of what a productive developer needs to know about Bash history:

• history: Show the history in case you get lost.
• !<num>: Execute line again, where is the number shown by history.
• Ctrl-p: Browse the history backwards. Also on cursor-up, but doesn’t make you leave touch typing position.
• Ctrl-r: Search backwards. Invaluable. Matches can anchor anywhere. Pressing it repeatedly shows older matches.
• !$: Substitutes for the last word of the previous line. Very useful for doing something else with the same thing. For example, you might first say ls -R very/long/path, then du !$ to show the directory’s size.
• Esc-. or Alt-. (that’s Alt-PERIOD): Like !$, but Bash inserts the word into your current line right away, so you can check or modify it.

Also, setting the HISTCONTROL environment variable to “Ignoreboth” to ignore duplicate history entries and commands starting with a space is a must.

Session management is about open files, buffer positions and the like on the one hand, and about the frame configuration on the other hand. I realized that my actual use case for frame session management was so simple that I could trivially implement it myself.

All I really wanted was that my Emacs would launch in fullscreen, and split it into two or three vertical windows, depending on whether I was using the laptop screen or the larger external screen. If I could accept the two-split as default, and run a function to create the three-window setup when needed, then all I’d need were some small initialization functions.

Like almost everything else, Emacs exposes its window and frame handling in an ELisp API. So here we go:

(defun my-big-screen ()
  "Set up frame for external screen, with three windows."
  (interactive)
  (my-initialize-frame 3))

(defun my-small-screen ()
  "Set up frame for laptop screen, with two windows."
  (interactive)
  (my-initialize-frame 2))

(defun my-initialize-frame (columns)
  "Set current frame to fullscreen and split it into COLUMNS
vertical windows."
  (set-frame-parameter nil :fullscreen t)
  (delete-other-windows)
  (dotimes (not-used (1- columns))
    (split-window-horizontally))
  (balance-windows))

This is not rocket science and I hope it didn’t bore the Planet Emacsen readers, but maybe it encourages people to get creative with their Emacs’ frames and windows. Start with the Emacs Lisp manual, chapters 28 (Windows) and 29 (Frames).

We first look at the most essential Java memory basics, then see how we can track down out-of-memory problems. I’ve thrown in the most useful references.

Java memory basics

This section is over-simplified to keep it simple.

Objects are placed on a memory region called the heap. The size of the heap depends on the Java parameters -Xms for the initial size, and -Xmx for the maximum size; for example, -Xms512m -Xmx768m.

Objects are not needed anymore when there are no more references to them; they are unreachable. The Java virtual machine will clean these up and free the memory they occupied. This happens when the heap is starting to fill up, or otherwise in somewhat regular intervals, but not directly after an object becomes obsolete. Careful: This means that when you look at how much memory Java is using (see below for how to do that), the figure is only valid right after garbage collection. Otherwise, old objects are still using memory.

When the heap is full, but all objects on it are still needed, you run out of memory and the virtual machine gives up and throws a java.lang.OutOfMemoryError. There is also a variant of this exception that says “GC overhead limit exceeded”. This means that there’s a tiny bit of memory left, but garbage collection can free so little of it that it needs to kick in so soon after its last run finished that most of the program’s time is spent in garbage collection.

Memory leaks

You have a memory leak when you create objects that are never garbage collected, even after you wouldn’t need them anymore. You recognize a memory leak when your memory usage goes up and up while the load stays the same.

Usually, this happens when you put objects into a data structure which you keep using, such as a Map, and don’t delete or overwrite objects in that data structure once you don’t need them anymore.. You might not see any reference to your object in your code, but the data structure needs one to track its members. For instance, when using a Map as a cache to look up objects by ID, the cache will grow continuously as the Map will hold a reference to each object ever stored in it, so the garbage collection can never release anything you put in.

References

• The Truth About Garbage Collection
• The Java Language Specification

Tracking down memory problems

There are two principal ways of examining the memory usage of a Java process. You can connect to the running process to observe its behavior and collect some statistics. And you can create a heap dump, which is basically the complete heap at a certain point in time written out to a file, and analyse it with tools. The latter is the only way to know for sure what objects are using how much memory.

Examining a running Java process

We’ll stick to one tool here, jconsole. There are many more out there, but jconsole is easy, versatile and part of the standard Java SDK. It connects to a running Java process via JMX.

Start it and choose your java process. The “Memory” tab shows the heap usage; right-click to save the numbers as CSV. Click “Perform GC” to run garbage collection before you start an operation you want to test, so your tests always start at the same base line. The “VM Summary” and “MBeans” tabs have all kinds of information. You can also create a heap dump, see the section below.

Heap dumps

For analysing heap dumps you can use the Eclipse Memory Analyzer (MAT). Get the stand-alone download as the Eclipse integration is buggy as of 2010-06.

There are several ways to obtain heap dumps. You should always run your Java processes with the -XX:+HeapDumpOnOutOfMemoryError so you can examine what ate all your memory, should this happen.

You can get a heap dump from a running process directly from MAT via “Acquire heap dump”. jconsole can do it, too: in the MBeans tab, select com.sun.management->HotSpotDiagnostic->Operations->dumpHeap, and click the dumpHeap button.

MAT is quite self-explanatory. Run the “Leak suspects” report and examine the candidates.

References

• Monitoring and Managing Java SE 6 Platform Applications
• Troubleshooting Memory Leaks

Optimizing memory usage and performance

This is a complex topic, and lots could (and has been) written on it. For now, here are just some references and notes.

The jstat utility is part of the Java SDK. It prints out statistics on the console. You can use it like this: jstat -gc <PID> 30s to print out garbage collection statistics of the Java process with every 30 seconds.

References

• Java Tuning White Paper
• Java SE 6 HotSpot[tm] Virtual Machine Garbage Collection Tuning
• Java theory and practice: Garbage collection and performance (very good article)

I gave a 20-minute presentation there called “Semantisches Hacking: RDF in Perl‎”. At Swiss-Prot, we do all our RDF work in Java, but I got interested in how things look on the Perl side, and the Biohackathon in February got me started on exploring that.

Executive summary: The RDF-in-Perl community is organized at www.perlrdf.org, and the core of the available modules is RDF::Trine and RDF::Query by Gregory Williams. For example code, have a look at my simple demo scripts.

At the workshop, I had an audience of about 50-100 people, none of whom had ever worked with RDF or seriously looked into it. So I first introduced RDF in the simplest way possible, as there wasn’t much time, then showed off RDF::Trine and RDF::Query with code examples.

The talk was well received, and I had some interesting conversations afterward where people wanted to know more about RDF. Their questions mainly centered around ontologies/vocabularies, the additional time required to do this properly, and how to build an app on top of a triple store. I had talked about integrating RDF into existing apps in my presentation, for instance using Trine’s support for RDFa, as_hashref, JSON and other possibilities.

Here are the links to the slides (in German), the scripts I took the code snippets from, and the workshop page for the talk.

I think I managed to raise some awareness for RDF and perlrdf.org, and an understanding of the core ideas in an audience where almost no one had had any exposure to these topics, and showed some example code in a way the audience seemed to be able to follow, so I’d say it was a success.

Many programmers today consider Perl to be dying. That is because they read too many shiny new Ruby and Javascript blogs, instead of checking out what the Perl community is actually up to. Damian Conway says it way better than I could.

The slogan of the day, of recent years actually, in the Perl world is Modern Perl. Once you’re ready to get started yourself, get brian d foy’s timely new Effective Perl Programming.

And if you miss the good old days of obfuscated Perl, see the nice Hidden features of Perl collection. You will also find some serious, readable and useful tricks.

But for now, this is about one particular thing he said about editing: you need to have a shortcut in your editor that cuts or copies the text between two given characters around cursor, or point in Emacs lingo. This lets you do many frequent editing tasks easily. Say you’re going through a source file and want to move {}-delimited blocks around, or copy the field you’re on in a CSV file, or the current section in a Markdown file (from # to the next #, roughly).

Damian showed this off in Vim. I knew right away that I wanted it in Emacs. There are of course specialized functions for doing things with certain groups of delimiters, such as the ubiquitous sexp-based functions in Emacs, described extensively on the EmacsWiki. But these rely on pre-configured groups of delimiters, and to kill the text, you have to do one extra operation. I liked the idea of having a single function, and thus key binding, to do all of it. And most importantly, it seemed an interesting challenge for my still-developing Emacs and Lisp skills, so I didn’t even do a lot of research before I dug right in and wrote delim-kill.el.

delim-kill.el contains a single convenience function for editing structured data: delim-kill. Given two characters FROM and TO, delim-kill kills the text between the first occurrence of FROM before point and the first occurrence of TO after point. FROM and TO may be identical.

If FROM and TO are not identical, the function preserves the balance between the two characters: For each FROM that is encountered while looking for TO, one additional TO is required; and vice versa. For example, in “{ foo X{bar} baz }”, with X being point and “{” and “}” as delimiters, the text “{ foo {bar} baz }” will be killed, not “{ foo {bar}”.

I had fun writing this. It turned out to be a bit harder than I expected, mainly due to all the corner cases like point being on one of the delimiters—like most programming, really. I wrote plenty of unit tests to be sure to handle these cases. It’s very little code now, but when I first had a working version, it was twice as big. Once I had it working, I could see lots of symmetry between the different cases that I could extract into shared code.

Also, I finally learned the condition-case error handling of Emacs. And being able to pass functions as arguments is just great, coming from my mostly-Java day job.

Try it out if it sounds useful to you, and let me know what you think.

Slightly edited 2010-05-16.