Instead of developing some Yet Another Big Animated Clock, I decided to write something practical. What I’d like to have is a widget which would ask my e-mail client (KMail) for the headers of the latest unread messages to show them on the desktop.

Two buttons in the top bar are for checking new mail from a server and for opening new message window. Clicking on the top bar somewhere else will open main KMail window.

SuperKaramba comes with a Python API, so everyone who is familiar with Pyton basics can write her own widget (aka “theme”) as a Python script. In my case, I wrote two scripts – one is for widget output and another as a simple KMail folder API wrapped around DCOP calls.

Patching KMail

There was a major problem – KMail DCOP interface had no a call to get unread message headers; only getting a number of them is possible. So I have had to dig into the KMail sourcecode (my current version is 1.9.4) for to add new function to FolderIFace interface (folderIface.cpp).

kmail.diff:

diff kmail/folderIface.cpp kmail_patched/folderIface.cpp
147a148,162
> QStringList
> FolderIface::unreadMessageHeaders()
> {
>      QStringList msgs;
>      mFolder->open();
>      for( int i = 0; i < mFolder->count(); i++) {
>          KMMsgBase *msg = mFolder->getMsgBase(i);
> 	     if (msg->isNew() || msg->isUnread()) {
> 	       msgs.append(msg->fromStrip() + "\\t" + msg->subject() + "\\t" + msg->dateStr());
>          }
>      }
>      mFolder->close();
>      return msgs;
> }
>
diff kmail/folderIface.h kmail_patched/folderIface.h
61c61,62
<
---
>     virtual QStringList unreadMessageHeaders();
>

After adding new function header to folderIface.h and rebuilding KMail, new DCOP call “kmail FolderIface unreadMessageHeaders” became available to get the list of message headers in format “From/Subject/Date” divided by tab characters (\t).

Sourcecode

skkmail.py:

import karamba
import kmail

text = []
images = []
clickAreas = []

# Define widget area width here (height is adjusted automatically)
s_width = 350

# If an actual height is above this value, a widget area will be clipped
maxHeight = 600

# Margins inside a widget area
s_x = 10
s_y = 10

def _update(widget):
    global text, images, clickAreas, s_width, s_x, s_y
    x_gap = 10
    y_gap = 4

    s_height = 14
    s_col1 = (s_width-x_gap)*0.66
    s_col2 = s_width
    s_col3 = (s_width-x_gap-16)*0.33

    folderFont = "Franklin Gothic Medium"
    folderFontSize = 14

    for t in text:
        try:
          karamba.deleteText(widget, t)
        except:
          pass
    for i in images:
        try:
          karamba.deleteImage(widget, i)
        except:
          pass
    for c in clickAreas:
        try:
          karamba.removeClickArea(widget, c)
        except:
          pass
    text = []
    images = []
    clickAreas = []
    folders = kmail.getFolders()
    y = s_y + 25;
    for f in folders:
        msgs = f.unreadMessages
        if (len(msgs) > 0):
            y0 = y
            t = karamba.createText(widget, s_x+15, y, s_width, s_height,
                f.name+" ("+str(len(msgs))+")")
            karamba.changeTextFont(widget, t, folderFont)
            karamba.changeTextSize(widget, t, folderFontSize)
            text.append(t)
            images.append(karamba.createImage(widget, s_x-10, y, "icons/folder.png"))
            y = y + s_height + y_gap*2 +10
            for m in msgs:
                t = karamba.createText(widget, s_x+20, y, s_col1, s_height, m[0])
                text.append(t)
                t = karamba.createText(widget, s_x, y+s_height, s_col2, s_height, m[1])
                text.append(t)
                t = karamba.createText(widget, s_x+20+s_col1+x_gap, y, s_col3, s_height, m[2])
                karamba.setTextAlign(widget, t, "RIGHT")
                text.append(t)
                images.append(karamba.createImage(widget, s_x, y, "icons/mail.png"))
                y = y + s_height*2 + y_gap*2
            clickAreas.append(karamba.createClickArea(widget, s_x-10, y0, s_width, y-y0,
                "dcop kmail KMailIface selectFolder "+f.path+" && kmail"))
            y = y + y_gap*2
    if (y < maxHeight - 20):
        karamba.resizeWidget(widget, s_width+s_x*2, y+20)

#this is called when you widget is initialized
def initWidget(widget):
    bgImage = karamba.createBackgroundImage(widget, 0, 5, "icons/bg.png")
    karamba.resizeImage(widget, bgImage, s_width+s_x*2, 20)
    logo = karamba.createImage(widget, 0, 0, "icons/kmail.png")
    t = karamba.createText(widget, 35, 5, 100, 20, "new mail")
    karamba.changeTextFont(widget, t, "Franklin Gothic Medium")
    karamba.changeTextSize(widget, t, 16)
    karamba.createClickArea(widget, 0, 0, 100, 30, "kmail")
    karamba.createImage(widget, s_width+s_x*2-18, 7, "icons/mail_get.png")
    karamba.createClickArea(widget, s_width+s_x*2-18, 7, 16, 16, "dcop kmail KMailIface checkMail")
    karamba.createImage(widget, s_width+s_x*2-40, 7, "icons/mail_new.png")
    karamba.createClickArea(widget, s_width+s_x*2-40, 7, 16, 16,
        "dcop kmail KMailIface openComposer '' '' '' '' '' ''")
    _update(widget)

#this is called everytime your widget is updated
#the update inverval is specified in the .theme file
def widgetUpdated(widget):
    _update(widget)

#This gets called everytime our widget is clicked.
# Middle button click forces to check new mail
def widgetClicked(widget, x, y, button):
    if (button == 2):
        kmail.dcopCall("kmail KMailIface checkMail")

kmail.py:

#!/usr/bin/python

import os, subprocess, string

def dcopCall(call):
  cmd = "dcop "+call
  pipe = subprocess.Popen(cmd, shell=True, bufsize=1024, stdout=subprocess.PIPE).stdout
  res = pipe.read()
  pipe.close()
  return res

class Folder:
  path = ""
  name = ""
  unreadCount = 0
  unreadMessages = []

  def __init__(self, path):
    dcopCall("kmail KMailIface getFolder "+path)
    self.path = path
    self.unreadMessages = self.getUnreadMessages()

  def getUnreadMessages(self):
    r = dcopCall("kmail FolderIface unreadMessageHeaders")
    ss = string.split(r, "\\n")
    msg = []
    for s in ss:
        if (len(s) > 0):
            msg.append(string.split(s, "\\t"))
    unreadCount = len(msg)
    if (unreadCount > 0):
        self.name = string.strip(dcopCall("kmail FolderIface displayName"))
    msg.reverse()
    return msg

def getFolders():
    r = dcopCall("kmail KMailIface folderList")
    folders = []
    for f in string.split(r, "\\n"):
        folders.append(Folder(f))
    return folders

skkmail.theme:

# you can change initial widget position, area size and updatin interval (default = 1min)
karamba x=100 y=100 w=400 h=200 interval=60000
defaultfont font="Franklin Gothic Book" fontsize=12 color=230,230,230

All code above is GPL‘ed.

Major updates

The most important feature which was introduced is the competed mechanism of “hierarchical tasks”. Before, the projects had only a flat list of “todo” tasks but now it is possible to create tree-like task structures where every task can be extended and detalized by any number of subtasks:

The mechanism and data model of hierarchical tasks was implemented by Ryan Ho with assistance of Michael Bernadsky but it had no user interface for a long time. The user interface (new tasks editor) has been developed by Jyrki Velhonoja and me using a wonderful Swing hack called TreeTable – a combination of JTree and JTable JFC/Swing components.

Note that new task mechanism is backward-compatible – that is your existing (flat) to-do lists should be seamlessly integrated into new hierarchical data model. They simply will be interpreted as one-level task sets and you can extend them with subtasks as you wish.

Another innovation is the improved notes editor with customizable fonts for paragraphs, headers and monospaced text (see new “Editor” tab in “Preferences” dialog). Also there is an option to antialias the text of notes.

The sourcecode got some refactoring. I moved the code of WYSIWYG HTML editor into the main Memoranda codebase to avoid version conflicts (before it was a separate “htmleditor” CVS module and JAR-file). So, you should not check out this module anymore – it is obsoleted. I plan to start new separate project for the editor, because it can be used in many applications, not only in Memoranda.

What’s next?

The final release – I believe. During two years we added a lot of new features and fixed the most of the bugs. It is more than enough for the First Final. We also need the user’s documentation – I have a draft of it but it is seriously outdated – it reflexed some ancient beta-version and it needs to be synchronised with the current state of Memoranda.

Pete was at his computer when the phone rang.
— There are representatives of XYZ corp. here, — said Pete’s boss. — They liked our offers and want to sign a contract. Is it ready?
— Yes, it is. — answered Pete confidently. He had prepared the text of contract a week ago.
— Great, print it and bring it here, please.
Pete put the receiver down and opened the folder named “Contracts” in his “My documents” hierarchy. There were about two hundreds of files, but there was nothing with name about “XYZ”. He browsed few old subfolders of forgotten purpose and obscure names but found nothing too. Being a bit surprised, Pete opened the “Recent documents” list but it was full of newer files. The worst thing was that Pete couldn’t recall a name of the file. He even didn’t remember when exactly he had working on this document. Pete sorted the files in “Contracts” by date of modification, but there were a couple of dozens of files edited at the last week. “When it was? Tuesday or Wednesday or..?” He started to open the documents one by one and it took for about ten minutes. Nothing like.
Pete browsed the root of “My documents” folder (turned into a dump of useless trash for ages) and some other folders, where as he thought the document accidentally might be. Nope. He mopped his brow and loosened the tie. “I have to remember where I saved that damned file. Well, I came to the office, as usual read my email, then played Lines until the boss called me and asked to prepare the contract for XYZ. I took a standard corporate template for contracts and started to work. I had been interrupted few times for checking and answering email and once for lunch. I remember also that after lunch I worked around another urgent document, then got back to the contract and completed it before the end of workday…”
“The search! How could I forgot about it!” Pete quickly ran the file search and typed “XYZ contract” in the search field. He leaned back in the chair and started to look at the flitting indicator. On 16 minute of the search, the phone rang again.
— You should not hurry so much anymore. — said boss. — We have lost that customer.

Pete could not find the document and upset a deal.

This sad but edifying story might be a source of many fair conclusions. Someone would say that the Pete’s company needs CRM. Someone else would mind that Pete should spend less time to play Lines but more to keep his harddisk in order and so on. But I told this story not to show the advantages of enterprise automation and not for moralizing on such slovens some users are. This is a typical “use case” story of using a software tool. A tool that is in our way everyday.

This is a filesystem.

Can we ever live without it?

A hierarchical filesystem is a sort of fundamental computer concept which has almost never been seriously revised against utility, user efficiency and usability. Everyone knows that computers store data in files and folders. It’s a matter of course and it seems like a natural phenomenon, like air we breath and like the sun which rises on the East everyday.

Most of the “non-technical” people think that the filesystem is a necessary technological principle of a computer long-term memory. “Ok, if there was no another way, we have to live with it.” And producers of operating systems silently keep this opinion.

It is wrong. It is misleading. You are deceived. Everybody who more or less understands how a computer works knows that the filesystem has no relation to physical organisation of a disk memory. This is merely an invention of early OS developers who cooked that “convenience” once in the 60’s. There are no any “folders” and “files” inside the computer. This is a programmatic imitation, a phantom superstructure above real physical disk sectors and cylinders and there are myriads of alternative ways for it. The filesystem is not a necessary part of a computer.

What’s wrong with it?

My criticism is not against the filesystems as they are. I believe there is a number of excellent, fast and reliable filesystems with advanced security, journaling and many other useful features. They are doing their job and doing it well. My criticism is against the practice of providing filesystem as an end-user tool.

I’d be very surprised if I ever met some developers convenience or raw technology feature which became a good user experience. I’m sure the good things all come from careful studying the human’s tasks, goals and contexts of use. The task of “to store whatever” is too general to be succesfully implemented. It is nobody’s experience. It is a bare abstraction in hope of adaptation to the real world by the users themselves. It is like a software for “doing anything” which makes a user to define “what” and “how” to do. Sort of meccano, but not a tool.

Users usually have no task “to store” their data at all. It seems the only case is when they back their data up to DVD-R or to another archival media. Many of them even have troubles with understanding the “saving” and “loading” operations. I’ve written a document in the Word, isn’t it already in the computer? The knowledge of “saving” and “loading” files and of distinction between long-term and operating memory, between harddisks and RAM, is needless for users. It makes them to learn for unnecessary things and keep in mind a weird two-part model of a computer storage system. It has no relation to their tasks. It is in their way just because there is a bad design which is countlessly reproduced for 40 years without any improvements.

Try to invent an universal and intuitive definition of a “file” using no technical jargon. “This is a document or an image or an audioclip or a program or anything else…” Anything else what? “Anything what we can save to disk” — is only answer. It could work in the times when the disks (I mean the floppies) were an important part of everyday user experience. Everybody knew that data and programs were on the disks and the disks were the real tangible things. But many users nowadays have no idea what the disk is at all, except for CD and DVD (remember an anecdote that many Mac users have seen a hard disk first on new OS X icon) . The concept of a file is particularly technical and it can be defined in the best way simply as “a storage unit”. It might be useful in context of eventual data backup, restoring or transferring, but not for user’s everyday tasks.

Another weakness of filesystem usability is a tree-like hierarchy. The problem with hierarchical structures is that they have extremely poor scalability. They could work well for 720K or 1.4M floppies with few directories and few dozens of files, but now our storage capacities are thousands times greater. It is proven that usability of hierarchical structure falls exponentially with number of levels and number of items in them. In practice, a structure with more than even two levels substantially impedes the search of a specific choice. Creation of a good intuitive structure (e.g. a catalog) is a very complex task of information design which requires special recursive techniques such as prototyping, card-sorting, careful testing etc. The most of us in real life have no time and skills for that, so the structures we create ad-hoc today, become the dreadful puzzles for us tomorrow. And the filesystem provides us with everything to create those puzzles but doesn’t help us to unscramble them.

Outlook instead of Explorer?

New paradygm of personal resource organization should be based on a simple axiom: “Whatever a user does, she does her tasks on her projects.” A task is an activity to achieving specific goals using specific resources and tools. A project is a common context of more than one tasks. For instance, Pete’s task was to write a text of contract, his goal was to have the text completed and approved, the resource was the contract itself and the project was dealing with XYZ corp. This is a natural model of human horme and it should go out of scope of special project-management and scheduling software to be a foundation of entire human-computer interaction framework. These ideas were (in very roughly and initial form) implemented in the Memoranda project. I believe evolution of this kind of software will replace traditional file managers from their position of main user interface to navigate over resource collections.

It should change the way of everyday personal computing and make it more simple and natural. Instead of creating informal and often casual folder structures, a user would have a personal project-management framework for managing information resources as the internals of her projects. Resources are associated with tasks of the projects, so it would be easy to navigate and find them. In fact, it should be enough to go into the specific project context to get all related resources in the way.

It is not too futuristic. Though it seems rather hard to overcome 40 y.o. conventions, there already are the software which is passed through the files and folders and provide high-level, sofisticated and task-centered data structures.

E-mail clients, personal information managers (PIMs) and calendars working this way for decades. Many of the multimedia software provide their own media libraries management without obvious relation to the filesystem. For instance, working with my amaroK mediaplayer, I only care to put my new media files into a known single folder. The program monitors that folder and automatically updates the media collection, so I can easily find any song browsing the collection by artist name, album title, year, genre and so on. All modern Development Environment software (IDE’s) has advanced project-management tools for software development tasks including resources (sourcecode) management. At last, the Pete’s favourite Lines remembers the game score results and even if it doing so with files, a gamer doesn’t care about it.

Conclusion

…
Pete put the receiver down and clicked on “Dealing with XYZ corp” project icon on his desktop. In the list of tasks sorted by date he clicked the “Prepare the contract” item and the first thing that catched his eyes was the icon of the document in the resources area. He selected “Print” in the context menu and there was exactly 10 seconds since he had stopped to talk with the boss.

Somebody would object this is how an accurate user usually does with the folders and files. But why our software requires us to be so accurate and doesn’t help us if we aren’t get its high standards? Nowadays, the computers are smartest of the artificial things. And I’m wondered why they still supply us with information storages not smarter than a table or a bookcase.

I have a cranky idea: “Would we have to pay many thousands of bucks for CRM, ERP and other enterprise software functions, if our personal tools were a bit more sophisticated?”

Memoranda

I began to think about changing jNotes name as far back as I started to work at Version 2. It seemed to be basically new product, rather than just new version of jNotes. Another reason was annoying confusion with another software called JNotes – some Java stuff related with Lotus Notes. So, I posted my thoughts to the mailing list and asked people to suggest new names.

Ten variants of new name were proposed:

• InteliNotes
• Notestopia
• Memoranda
• LineUp
• Jot
• NoteSoup
• MyNotes
• NoteBurn
• Jilofax
• Chronicles (my variant)
• … oh no, “jNotes” forever! (another my variant)

As a result of voting, the “Memoranda” name invented by Iván Ribas was selected. I very liked that name and voted for it too. At the end of September 2003 we registered new project on SourceForge, moved the codebase and made new release under new name. New web-site had been launched at the same time. The old site was remained as a home of jNotes 1.0 (currently inactive).

Since then, among increasing stability, the software got a lot of new features. The most important ones are:

• Summary page (“Agenda”)
  A “today startup page” which summarize information of active tasks of all projects and upcoming events of this day.
• Hierarchical tasks
  Now the tasks of a project have hierarchical structure, so that every task may contain sub-tasks.
• Multiple notes per day
  until recently, a single calendar date was associated with a single note. Now it is possible to have few notes associated with the same date.
• 14 national localizations
  Now Memoranda speaks Catalan, Chinese, Chinese (Taiwan), Dutch (Belgium), Dutch (Netherlands), Finnish, France, German, Hungarian, Italian, Japan, Russian, Serbian and Spain. The language of GUI is selected automatically depending on user’s system locale settings.

The current status of Memoranda is 1.0 Release Candidate 2. The next release (I hope) will be final.

Memoranda 1.0-RC2 screenshot

Who did it

Many people in different time took part in Memoranda development. Above all, these are code commiters:

• Patrick Bielen (pbielen)
• Ryan Ho (rawsushi)
• Yunjie Liu (noregister)
• Iván Ribas (ivanrise)
• Carel-J Rischmuller (carel-j)
• Jeremy Whitlock (jcscoobyrs)
• Jyrki Velhonoja (velhonoja)

And those who were not the developers, though they did the great help for the project:

• Thomas Chuffart
• Willy Dobe
• David Nagy
• Kenneth J. Pouncey
• Sava Rados
• Michael Radtke
• Milena Vitali-Charewicz
• Toru Watanabe
• … and everyone I may have forgotten, sorry.

(all people listed alphabetically by their second names)

I don’t say thanks because this is our software, not my own ;-)

Memoranda 2.0 ?

My thoughts on Memoranda future are coming soon, so the saga continues…

jNotes

As far as I remember, I always worked in multitasking mode and was involved in few projects simultaneously. Once I had an idea that I needed something to help myself to manage that “concurrency” and organize my work – a kind of scratchpad or notebook to quickly make and find text notes and to-do’s. I tried out few software tools but I didn’t find what might exactly fit my needs. Some of them were heavy project-management tools with a lot of features which were useless for me, while others, on the contrary, had no features I expected. So, I decided to write that tool with my own hands.

My requirements were simple — it should be quick lightweight tool to easily keep brief daily notes and to-do lists. I thought it might be a simple text editor with a calendar interface. Instead of keeping batches of “YYYYMMDD” text files (as I did before), I should be able to select a date with a calendar widget and immediately get a text I had written then, or start an empty page if there was no text. No relations with files (no opening and saving operations) and quick navigation were the special points.

Later I decided that to-do lists were better to implement as special checklist widgets, rather than as simple text. In this case, it would be possible to count progress of the projects. So, they became an another concern, separated from text notes. Although I had no strong needs to schedule time events, I thought it would be an useful and popular feature (and later I convinced it was really so).

I wrote this tool in my spare time for one month and called it “jNotes” – “Java + Notes” (I had no long thinking about its name, you see ;-). It was released on SourceForge in July, 2002.

jNotes 1.0 screenshots

jNotes 1.0 had a plain-text editor for diary notes, to-do checklists editor and events scheduler/notificator. The notes had been grouped in the “books” and to-do items in the “projects”. All data stored transparently as XML files, but the notes editor worked with plain-text files, that is it could be used as an usual text editor. There was also an export feature to save a whole notes book as an HTML file.

It was proved to be a simple but really helpful tool, and it got some popularity with time.

jNotes 2

In the end of 2002, when jNotes became rather stable and bugs-free software, I summarized my personal experience of using it together with users’ feedback and started to think about next version. There were some new ideas and it was clear that the software required a much of redesign to those ideas could be implemented.

jNotes was not longer a tool for my personal needs only, and my needs were changed. It was obvious that the ad-hoc design of jNotes 1.0 should be reinvented on more common and consistent basis. After investigating of users activity models and imagined use cases, I invented a new concept of a project. This concept became a foundation for new user experience.

A project in new version considered as a date-oriented collection of text notes, tasks and links to resources. The concept of “books” were discarded, but the collections of notes became integrated into the projects together with tasks. The concept of the tasks were revised too — no longer one-day todo items with boolean “done or not” status, but the processes within the project’s time bounds. They got a start and end date, priority level and percentage of completion. The links to project-related resources were another innovation. Any local file or URL can be included as a “shortcut” into the project’s resources list for quick access to them with suitable external application.

The second version of jNotes was written almost from scratch for three months and the first Alpha release was appeared at May, 2003.

jNotes 2.0-alpha screenshot

An unsignificant amount of code was inherited from version 1, but almost everything had been rewritten – an object model, user interface and so on. The software got a new WYSIWYG styled text editor with HTML export/import, unlimited undo buffer, find&replace functions and many other improvements and new features.

The alpha of the Version 2 quickly became even more popular than jNotes 1. At July, 2003 I got an offer of help from another developer. Patrick Bielen got involved into the project and started to contribute very enthusiastically. Iván Ribas joined us a little later. Besides, we got a big deal of patches, bug reports, translations and other useful stuff from numerous mailing list members. So, at September, 2003 there was a little but real opensource community.

Part II >>