• Programming is fun
• Programming is simple

It was an astonishing “You can do that” idea: you don’t have to be a mature, qualified engineer to start doing things with the computer right now. I still don’t know whether it’s a property of the C language itself, or of the book (in very good Russian translation); but even complex subjects, like pointers etc., were explained in an amusing way, intuitive for an average 13 y.o. schoolboy.

I don’t know another programming language specification that were such fundamental and detailed but simple and inspiring at the same time. I really owe to it in particular and to the C in general for I had not been lost in the jungles of something like BASIC (as it was suggested by the most of the “beginner” books of those days) and got a taste for a good language and for a fun of programming from the very beginning.

Unfortunately, I never remember Mr. Ritchie when I turn on my Linux (UNIX!) computer for doing my everyday work (those curly braces and all that stuff). And it’s sad.

Disclaimer: Like it says on the author page, I work for UoN and Laboranova project, but I cannot speak for it officially in this blog. These are my own views as a project insider and the other’s may differ.

Back story

As far back as in early 2008, I’ve been introduced into PCN proposal and invited for collaboration on a R&D for its prototype implementation. It was very interesting for me and we developed a vision and basic design principles of the PCN solution to start the prototype development at the beginning of 2009.

This work is a part of Laboranova — a large EU project aimed at new ways of collaboration between knowledge workers and sharing ideas and competencies. Apart from PCN, there are lots of other interesting thngs the project partners do there.

PCN basics

The idea is a social networking where the members were connected via shared areas of expertise. These areas are identified by the topics of interests, discovered at user’s content.

Like other social network services connect people via shared professional activities, schools, hobbies or music tastes, PCN approach employs the shared semantics of texts the people read or write. There is an optimistic assumption that such semantics, expressed as sets of the weighted keywords (or, tag clouds), may reflect an area of user’s expertise and constitute a profile of his professional interests.

Multiple user profiles are aggregated to create expertise information and form a socio-semantic network of people and concepts. Comparing profiles of different users, we can evaluate a similarity of their expertises and thus, estimate their social distance in the network. We also can take a specific area of expertise to find who are the best experts relevant to it, and explore the related overlapping areas. With this network analysis, it is possible to generate a variety of individual recommendations to help people to discover new collaboration opportunities and areas of knowledge.

The Platform

There is a client-server architecture including a central PCN server and a number of clients connected to it. The PCN client software is installed locally and analyses content from different locations specified by its owner: file folders, webs, email, RSS, delicious accounts, SharePoint servers and so on.

A user can let the software extract the keywords from the content automatically, edit the results or choose to tag the content manually. Then he chooses what locations should be submitted to the server and assigns them to one or more named contexts which help to organize the concepts within the user profile. Document metadata and tagging information is uploaded to the server where an individual profile of tags is created and published. As a location is submitted to the server, it is monitored for changes to synchronize the profile with the actual state of the content (by sending incremental updates at a specified time interval).

The Client

The  client part is based on SCAN, so there is a zero barrier to start working with it for SCAN users. Actually, from a SCAN user perspective, there is no difference from usual everyday work — you can enjoy a full set of features one can find in native “offline” SCAN, but also use it for populating your public profile of interests at the PCN server.

Also, variety of location types and document formats are supported by the client thanks to the plugins from the SCAN repository.

The Ontology

The server receives metadata about content resources as SIOC RDF, so in theory, it may work with any SIOC provider, apart from the default PCN client. The server augments resource metadata with relationships to the user profile, concepts and contexts, thus forming the quadripartite PCN ontology model:

PCN quadripartite model

To describe the PCN model, we adopted SCOT (Semantic Clouds Of Tags) ontology aimed at conceptualization of the structure and  semantics of tagging data with strong focus on social interoperability. It is an extension and further development of the Tag Ontology project that describes the relationship between an agent, an arbitrary resource, and one or more tags.

The SCOT (Tag) ontology is based on a tripartite (User—Tag—Resource) model. These three core concepts are connected together via a central concept of Tagging representing the tagging activity. Every Tagging instance can be considered as a result of a single tagging action defining a user who performed it, the tagged resource and what tags have been used. It also can carry auxiliary information about the action, such as the time of tagging.

For users and resources, SCOT relies upon concepts from SIOC — specifically, sioc:User and sioc:Item classes. For our PCN ontology, we extended the SCOT model with the notion of  context by adding another class of entities and a property to relate them with the Tagging instances.

Extended SCOT Tagging as a skeleton of the PCN model

Using SCOT Tagging model, it is possible to avoid excessive verbosity in the PCN ontology, as the relationships between core PCN classes (the edges of the tetrahedron) are inferable from their relationships with the central Tagging class (the skeleton). In the diagram below, the implicit relations are shown as dashed.

PCN Ontology

An interesting possibilities comes from the fact that the scot:Tag is actually a subclass of the SKOS Concept, so all kinds of SKOS reasonings about the concepts are possible in the future. Moreover, MOAT features incorporated in SCOT open a way to integrate the tag descriptions with the Linked Data web.

The API

The client talks to the server via a simple RESTful API for adding and modifying the users metadata. As said above, the metadata is described with SIOC, so it is theoretically possible to use that API and integrate the PCN server with any system which can provide SIOC metadata about the content.

An endpoint for SPARQL queries is planned also.

For more information:

• An early description of the work package (Laboranova Profile System) on Laboranova web-site.
• Marc Pallot et al, “Future Workplaces, towards the ‘Collaborative’ Web“
• Peter Mika, “Ontologies are us : A unified model of social networks and semantics“
• Hak-Lae Kim et al, “Review and Alignment of Tag Ontologies for Semantically-Linked Data in Collaborative Tagging Spaces“ , “The State of the Art in Tag Ontologies: A Semantic Model for Tagging and Folksonomies“
• SCOT
• SIOC

An idea is borrowed from e-mail clients, where the similar feature exists for decades as the user-defined filters for processing the messages. This is how it looks in KMail:

In general, a filter checks if a document matches to specific criteria (a rule) and does some action if yes. For instance, if a condition “text contains ‘viagra’ or ‘cialis’” is true, then some action (“move to spam” or “send assassins to the author”) would be executed. What is especially good is that it’s old, popular and intuitive user experience.

In a content aggregator like SCAN, this concept may allow a user to define custom automation rules to set document metadata properties. For instance,

IF (url starts with "http://cyberborean.wordpress.com") SET author = "me"

Another using I have in my mind is to empower an “artificial intelligence” of autotagging with a human intelligence, by user-defined tagging rules:

IF (text contains "latte") ADD TAG "coffee"

I only doubt about terminology — “filter” might be confusing, as it is already used in SCAN vocabulary (URI filters to include/exclude some documents by their URI pattern). Any ideas?

“Foreword”
chapters corrected edition experience glimpses information process readers reference story

“Prologue”
bilbo book copy families history hobbits shire smials thain westmarch

Book I

baggins bilbo bree butterbur frodo goldberry ponies sam strider tom

1. “A Long-expected Party”
   bag baggins bagginses bilbo birthday fireworks joke lobelia presents sackville
2. “The Shadow of the Past”
   bilbo birthday déagol frodo gandalf gollum grandmother hated ring ted
3. “Three is Company”
   bag frodo gildor lane lobelia pippin sackville sam sir sold
4. “A Short Cut to Mushrooms”
   dogs farm farmer ferry frodo lane maggot mushrooms pippin waggon
5. “A Conspiracy Unmasked”
   bath brandy brandybucks bucklanders fatty ferry fredegar frodo maggot merry
6. “The Old Forest”
   bonfire forest hedge lilies merry path ponies tom willow withywindle
7. “In the House of Tom Bombadil”
   barrows bombadil candle chimney creak derry goldberry tom wights willow
8. “Fog on the Barrow-Downs”
   barrow bombadil fog goldberry hoy jogging lumpkin ponies tom wight
9. “At the Sign of The Prancing Pony”
   barliman bree butterbur comer cow frodo inn landlord nob underhill
10. “Strider”
    baggins bree butterbur ferny frodo landlord letter nob strider underhill
11. “A Knife in the Dark”
    beren bob bree dell ferny galad gil strider tinúviel weathertop
12. “Flight to the Ford”
    bone boot ford frodo glorfindel horse strider troll trolls weathertop

Book II

aragorn boats boromir elrond frodo gandalf gimli haldir legolas sam

1. “Many Meetings”
   bilbo dúnadan elrond flood frodo glorfindel glóin mortals rivendell strider
2. “The Council of Elrond”
   boromir elrond erestor galad gil glóin isildur radagast ring saruman
3. “The Ring Goes South”
   aragorn bill boromir caradhras drift elrond frodo gandalf redhorn snow
4. “A Journey in the Dark”
   bill boromir doors durin gandalf gimli howls mines moria wolves
5. “The Bridge of Khazad-dum”
   balin balrog beats boromir chamber doom door drum hall mazarbul
6. “Lothlórien”
   flet frodo galadhrim gimli haldir ladder legolas lothlórien nimrodel silverlode
7. “The Mirror of Galadriel”
   basin celeborn frodo galadhrim galadriel haldir lady magic mirror pedestal
8. “Farewell to Lórien”
   boat boats boromir cakes celeborn galadhrim galadriel gimli lady lórien
9. “The Great River”
   aragorn boat boats frodo gebir log paddle paddles rapids sarn
10. “The Breaking of the Fellowship”
    amon aragorn boromir brandir frodo hen lawn lhaw sam tol

Book III

aragorn ents gimli isengard legolas saruman théoden treebeard uglúk éomer

1. “The Departure of Boromir”
   aragorn boat boromir galen gimli gulls legolas orcs rauros tire
2. “The Riders of Rohan”
   aragorn downs entwash fangorn gimli horses legolas orcs trail éomer
3. “The Uruk-Hai”
   ankles grishnákh isengarders knoll lugbúrz merry orcs pippin uglúk wrists
4. “Treebeard”
   ent entish ents entwives hasty hoom isengard rowan treebeard trees
5. “The White Rider”
   aragorn fangorn gandalf gimli haft legolas man old saruman treebeard
6. “The King of the Golden Hall”
   aragorn gandalf gríma hall háma neighed théoden wormtongue éomer éowyn
7. “Helm’s Deep”
   aragorn coomb deeping dike erkenbrand gamling helm hornburg westfold éomer
8. “The Road to Isengard”
   caverns caves fords gandalf gimli isen isengard legolas saruman théoden
9. “Flotsam and Jetsam”
   barrels ent ents gimli huorns isengard pipe quickbeam saruman treebeard
10. “The Voice of Saruman”
    dismissed ents gandalf injuries orthanc rail saruman spell théoden treebeard
11. “The Palantír”
    ball bracken gandalf globe orthanc palantíri pippin saruman shadowfax wizard

Book IV

faramir frodo gollum nice precious rope sam shagrat shelob sméagol

1. “The Taming of Sméagol”
   cliff fix frodo gollum knot precious rope sam sméagol yess
2. “The Passage of the Marshes”
   fish frodo gollum gully marshes mires nice precious sam sméagol
3. “The Black Gate is Closed”
   frodo gollum helps lithui master nice oliphaunt oliphaunts sam sméagol
4. “Of Herbs and Stewed Rabbit”
   coney fern gollum herbs mablung pans rabbits sam sméagol taters
5. “The Window on the West”
   arts blindfold boromir damrod faramir frodo isildur mablung men mithrandir
6. “The Forbidden Pool”
   anborn faramir fish fissh frodo gollum precious protection sly sméagol
7. “Journey to the Cross-roads”
   brewing decent faramir frodo glare gollum mould sam staves thorns
8. “The Stairs of Cirith Ungol”
   dad frodo gollum morgul sam sméagol sneaking stairway wink wraith
9. “Shelob”
   frodo glass gollum lair phial sam shelob sméagol stench tunnel
10. “The Choices of Master Samwise”
    cords frodo gorbag lads ladyship lugbúrz sam shagrat shelob tunnel

Book V

aragorn beregond city denethor faramir ghân men pippin éomer éowyn

1. “Minas Tirith”
   beregond bergil citadel city denethor forlong ingold mithrandir pippin street
2. “The Passing of the Grey Company”
   aragorn burg elladan erech gimli halbarad king legolas ride théoden
3. “The Muster of Rohan”
   dwimorberg edoras harrowdale hirgon king merry ride théoden éomer éowyn
4. “The Siege of Gondor”
   beregond citadel city denethor faramir gandalf garrison pelennor pippin retreat
5. “The Ride of the Rohirrim”
   buri dernhelm elfhelm ghân men stonewain théoden wild éomer éored
6. “The Battle of the Pelennor Fields”
   dernhelm harlond king knights mûmakil snowmane southrons théoden éomer éowyn
7. “The Pyre of Denethor”
   beregond bier burn city denethor faramir gandalf healing oil thou
8. “The Houses of Healing”
   aragorn athelas bergil faramir healing imrahil ioreth sick éomer éowyn
9. “The Last Debate”
   aragorn gimli gulls imrahil lamedon lebennin legolas pelargir ships thousands
10. “The Black Gate Opens”
    army captains heralds imrahil lieutenant messenger morannon sauron terms thou

Book VI

cotton faramir frodo lotho rosie ruffians sam shagrat sharkey éowyn

1. “The Tower of Cirith Ungol”
   court frodo gorbag ladder orc sam shagrat shelob snaga turret
2. “The Land of Shadow”
   frodo isenmouthe morgai orc range sam shagrat spur tracker whip
3. “Mount Doom”
   cone frodo gaping gollum load masster mountain sam sammath tearing
4. “The Field of Cormallen”
   captains eagles frodo gwaihir isle knights merriment praise pure sam
5. “The Steward and the King”
   aragorn city faramir healer healing ioreth king lady warden éowyn
6. “Many Partings”
   arwen beggar celeborn hoom orthanc théoden treebeard wain éomer éowyn
7. “Homeward Bound”
   barliman bill bree butterbur deadmen ferny gandalf nob pony troubles
8. “The Scouring of the Shire”
   cotton farmer hob lotho ruffians sharkey sheds shirriff shirriffs village
9. “The Grey Havens”
   bywater cotton deputy elanor frodo married mayor rosie ruffians sam

Appendices

• A. “Annals of the kings and rulers”
  arthedain arvedui eldacar eärnil gondor king náin tar thráin witch
• B. “The Tale of Years”
  begins birth guldur king meets reaches samwise sauron sets thranduil
• C. “Family Trees”
  addition appendix birth blanco family guests names party recorded recounted
• D. “Shire Calendar”
  calendar corresponded eldar lithe months names quenya reckoning week yule
• E. “Writing and spelling”
  english languages letters pronounced quenya represented represents series sindarin vowel
• F. “The languages and peoples of the Third Age”
  english languages letters pronounced quenya represented represents series sindarin vowel

Resulting Tags Cloud

When testing my new Java application, I noticed strange freezing sometimes. It happened irregularly and the program did nothing serious at that time – no complex calculations, no heavy data transfers, etc. Nevertheless, I noticed that those freezings where related to my internet connection glitches – unfortunately, it occasionally happens here. But my program didn’t do anything with internet at those moments!

I did some profiling and found the bottleneck – to my surprise, it was get() method of a HashMap instance. Sometimes it took about 20 seconds only for just checking if an object exists in the map. I was shocked. HashMaps! I used it for ages and was sure they are reasonably fast kind of collections.

One way or another, I found the problem. The point was that I used java.net.URL instances as the HashMap keys. URL objects in Java, in comparison with simple strings, are convenient as they store URLs as the sets of separate fields (protocol, host, path etc), guarantee URL well-formedness and enable to create a connection with a single connect() method.

But its implementation is horrible. As it turned out, two generic object methods – equals() and hashCode() needed for internet connection and domain name resolving just to check equality of two URLs or to compute object’s hash code.

Two URL objects are equal if they have the same protocol, reference equivalent hosts, have the same port number on the host, and the same file and fragment of the file.

Two hosts are considered equivalent if both host names can be resolved into the same IP addresses; else if either host name can’t be resolved, the host names must be equal without regard to case; or both host names equal to null.

Since hosts comparison requires name resolution, this operation is a blocking operation.

(java.net.URL javadoc)

I have no idea what were the reasons to do that, but it is bad. Simply bad. Such common operations as equality test (and hash code computation) should rely on an object data only. Making it dependent on external unreliable resources is dangerous. Also, those operations must be fast as possible, because they are called frequently (often inside the loops). When the time of their execution is equal to the connection time-out, your program simply don’t work.

Even if there were some reasons for that, it works wrong.

Note: The defined behavior for equals is known to be inconsistent with virtual hosting in HTTP.

(Ibid.)

Sure. For instance, URLs of all WordPress blogs are equal from point of view of java.net.URL, as long as all of them are resolved to the same IP address. And a whole lot of other web-sites, unless they are hosted on the dedicated servers.

URLs are just the text strings with known syntax. They should work offline, no matter they are “http:” or “file:”. They don’t need to be resolved to the IP addresses, unless a referenced resource is requested.

Do not use java.net.URL. Store URLs as simple String objects. If you need to keep URLs structured and well-formed, you can use java.net.URI class instead, but the strings are faster anyway. You always can parse a string into java.net.URI object when it is needed. You can parse it into java.net.URL as well, but.. better don’t do it.

A common case when you might have an idea of using a custom ClassLoader is to avoid the classpath issues using URLClassLoader for loading classes instead of the default one. Classpath is a list of locations (JAR-files or directories) where default ClassLoader is looking for the classes to load. Declaring Classpath explicitly (either with CLASSPATH environment variable, or as a command-line argument of Java virtual machine) is tedious and error-prone job and it works only in the case of monolithic, statically-linked applications with known set of libraries.

But imagine a complex component-based system with a number of replaceable modules (plug-ins). It is obvious that classpath for that system should be built automatically to reflect the configuration changes before each running. It is often done with system-specific startup scripts which analyzes the modules configuration and generates an appropriate classpath before starting the application. However, this method has its own drawbacks – especially in the case of poor Microsoft systems as their rudimentary command interpreter makes this task hardly possible. And this is a way system-dependent, of course.

URLClassLoader, from a distant point of view, looks like an elegant, simple and “100%-pure-java” solution for the classpath headache. It is instantiated with an array of locations (URL’s) so each class loaded with that ClassLoader will use this array as a classpath. An advantage is that the classpath can be generated on the fly, automatically by an application itself.

So, let’s see how it works. Imagine we have an application with main executable class Foo (in package com.example.foo) and a lot of other classes in different locations. We will develop an invocation wrapper for loading Foo using URLClassLoader with an array of locations, generated automatically:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

/*
 * FooInvoker.java
 */
 
package com.example.foo;
 
import java.net.URL;
import java.net.URLClassLoader;
import java.lang.reflect.Method;
import java.lang.reflect.Modifier;
 
public class FooInvoker {
    public static String FOO_CLASS_NAME = "com.example.foo.Foo";
 
    public static void main(String[] args) {
        URL[] classpathURLs = getClasspathURLs();
        ClassLoader loader = new URLClassLoader(classpathURLs);
        try {
            Class fooClass = loader.loadClass(FOO_CLASS_NAME);
            Method main = fooClass.getMethod("main", new Class[] {
                String[].class
            });
            int modifiers = main.getModifiers();
            if (Modifier.isPublic(modifiers) && Modifier.isStatic(modifiers)) {
                main.invoke(null, new Object[] {
                    new String[] {}
                });
                // In this example, we assume Foo.main() takes no
                // command-line arguments
            } else {
                throw new NoSuchMethodException();
            }
        } catch (Exception e) {
            System.err.println("Error running class " + FOO_CLASS_NAME
                    + e.getMessage());
            e.printStackTrace();
        }
    }
 
    private static URL[] getClasspathURLs() {
        URL[] urls = new URL[] {};
        // Here we're looking for all project-related libraries
        // and constructing the classpath array
        // ...
        return urls;
    }
}

Run it (assuming Foo and FooInvoker are packaged both into foo.jar file, which is the core library of our app):

$ java -classpath ./foo.jar com.example.foo.FooInvoker

Alas! Everything located outside the foo.jar (that is, the classpath set in JVM arguments) is failed to load. For instance, if Foo uses class Bar in it’s own bar.jar library, Foo will throw ClassNotFoundException despite of the fact that bar.jar has been passed to URLClassLoader argument. It seems we have no progress at all – all classes are loaded from default JVM classpath as with the system ClassLoader.

Let’s do some debugging. Add this to Foo.main() code to see which ClassLoader loads Foo:

System.out.println("Foo is loaded with: " +
                           Foo.class.getClassLoader().getClass().getName());

Instead of “java.net.URLClassLoader” in output (as we hoped), we will see:

Foo is loaded with: sun.misc.Launcher$AppClassLoader

This is the default system ClassLoader. We really have no progress – it seems our URLClassLoader simply does not come into play. Our code is right but it doesn’t work. Is it a Java bug? Should we give up? Or report to Sun?

Wait a minute. The problem is not in code but in how Java VM executes it. Let’s try a trick: change FooInvoker.getClasspathURLs() algorythm to include foo.jar path in the resulted array (to be passed to URLClassLoader among other locations) and compile FooInvoker as a separate class file (not inside the foo.jar). Then run it (no classpath is declared!):

$ java com.example.foo.FooInvoker
...
Foo is loaded with: java.net.URLClassLoader

Voila! Our URLClassLoader has been used to load Foo as well as other classes, including those from locations in our dynamic classpath. It works! So what is happened?

In this case we found ourselves in one of the dark and odd Java territories ruled by the Masters of The Black Java Art (also known as the Heavy Java Geeks) who believe the programs should not behave as their author expects, but following the mysterious Rules instead. One of those Rules is called “ClassLoader Delegation Model” and it reads:

The ClassLoader class uses a delegation model to search for classes and resources. Each instance of ClassLoader has an associated parent class loader. When requested to find a class or resource, a ClassLoader instance will delegate the search for the class or resource to its parent class loader before attempting to find the class or resource itself.

A parent of URLClassLoader is the default system ClassLoader (as we used single-argument constructor). When we tried to load Foo with an instance of URLClassLoader, it asked the parent if the latter could load Foo. As Foo was in the default classpath, the system ClassLoader, of course, could. By this way, it took a position of a default ClassLoader in a chain of subsequent class invocations and left our URLClassLoader out of business.

When we moved Foo and everything out the default classpath, the system ClassLoader was not able to load anything (except FooInvoker itself). So, it had no chance to took precedence over our custom ClassLoader.

Well, if you like rules, there is another one:

If you use custom ClassLoader, do not give the system one a chance to come into play.

Happy coding!

The tags are a handy intuitive concept for labeling and describing the pieces of information. They are usually assigned informally, and as for any matter of “a common sense”, it is difficult to find their formal definition. It is even more difficult to reproduce and algorithmize the mental processes of tag choosing.

To avoid that fruitless work, I will consider some IR approaches which may be used for evaluation the document terms as the possible tag candidates. For to make that evaluation possible, two important purposes of the tags may be identified:

• Differentiation
  To name a thing means to separate it from others. So, a good tag should identify a document by showing its unique feature – how the document differs from other documents in a corpus.
• Unification
  A tag should also identify a semantic group to which the document belongs.

These two key properties seem totally different and even mutually opposite and discordant. Nevertheless, they are both important and the best tags perhaps integrates both properties in a dialectical way. Brief analysis of human-assigned tags on popular social web-services confirmed that assumption.

Traditional keyword extraction: TFIDF

The first strategy of autotagging is based on a popular term weighting technique, proposed by classical IR discipline. This technique is called shortly as TFIDF and this abbreviation reflects the basic formula for calculating a measure of relevance (or, “weight”) of a specific term in a given document: Term Frequency x Inverted Document Frequency.
Though there is a number of variations of this formula (differ in various normalization techniques), the basic idea behind them is the same: weight of a term in a document is directly proportional to the number of times the term appears in the document (Term Frequency, TF) and inversely proportional to the total number of documents containing this term (Document Frequency, DF).

This weighting scheme allows to effectively filter out so-called “common words” (with high DF) as well as the rare terms and misspellings (with low TF and DF both). The terms with high TF and relatively low DF got highest “importance” ranks and can be considered as possible candidates to tags.

Thus, TFIDF-based document tagging strategy may be described as follows:

1. Build a list of all unique terms of a given document (term vector)
2. Calculate TFIDF weight for each term in the list
3. Sort the terms by their TFIDF weights
4. Select N topmost terms, where N is a number of tags we want to assign for each document. We can also choose another method of tags number limitation, e.g. to select the terms above specified TFIDF threshold.

Improved autotagging strategies

The problem with reviewed simple TFIDF-based analysis is that it conflicts with the second part of our definition of the tags purpose: unification. Reviewed approach is good when we need to know what terms make a document different from others, but it doesn’t help if we need to select the terms which connect the document with its counterparts.

With this technique, the “tags cloud” tends to grow proportionally to the number of indexed documents with minimization of the number of documents per single tag at the same time. In an ideal case, it leads to the situation when each tag corresponds to an unique document. Such level of granularity is not what we want if we are going to control our taxonomy size and care about grouping function of our tags. The tags which would contribute into documents unification and grouping are overlooked by that algorithm because their DF’s are higher than of those contributing into a document uniqueness. We could smooth that effect by reduction an influence of DF parameter, but it increases the risk of appearing the common words as the document tags. Obviously, tweaking the basic TFIDF formula is not a solution for this problem.

To implement better autotagging strategy (which would consider a documents grouping factor), we need to analyze a document together with its semantic context – that is a cluster of documents including an initial document plus the documents similar to it. TFIDF ranking algorithm applied to that cluster gives us the terms relevant to the whole group, while the common words are still effectively suppressed. There is a high probability that the documents inside the cluster are mutually similar to each other, thus this strategy tends to re-use the same tag set for semantically related documents, as possible.

An easy way to find the document similarities is to build the document term vector (as we did before) and to issue it (or its highest ranked part) to a search system. Some search engines implement “more-like-this” feature that works the same way. The better results may be achieved if the search system allows to set a “boost factor” for separate query terms, proportionally to their weights in the term vector. Then, we just select the search results (above specific threshold) as the documents similar to the given one.

The whole strategy may looks like:

1. Find a set of documents, similar to a given document.
2. Build a composite term vector as an union of term vectors of the given document and its similarities.
3. Calculate TFIDF term weights against the composite term vector.
4. Sort the terms by their TFIDF weights
5. Select topmost terms as the document tags

By tweaking the similarity thresholds and TFIDF normalizing (e.g. considering similarity coefficients in TFIDF calculation), we can adjust the results to get “more specific” or “more general” set of the tags (differentiation/unification balance), depending on our goals.

Another improved autotagging strategy may be implemented as an adaptive technique where TFIDF evaluation takes into account whether a given term is already a tag or not. But this approach strictly depends on an initial tag set and likely, will require some manual tagging before it could be used in auto mode.

P.S. If you found this article useful, please consider to participate in this survey. It will help the author in his further work on this subject. Thanks!

1. When you are looking for a document on your machine, the following navigation hints are important (specify an order, e.g. “a, d, c, b”):

a. File name
b. File name and folder
c. Date of creation/modification
d. Document metadata properties
e. Full-text search results

2. Do you use document titles (or another semantic data) as the file names?

a. Yes, always
b. For important documents only
c. Don’t care about the file names

3. Do you keep a sort of semantic folder structure for storing your documents?

a. Yes, and I keep all my documents in a single folders hierarchy, organized semantically
b. Yes, but use the structure for important documents only
c. Use random folder structures, depending on a context of my work
d. Don’t care about where I’m saving my files

4. Do you use document shortcuts (Windows) or symbolic links (*nix) for improving navigation?

a. Yes, often
b. Rarely
c. No

5. How often do you use the system full-text search for finding a specific document?

a. This is my everyday way of finding the documents
b. Only when I need to find the document quickly
c. Only when I gave up to find the document by other ways
d. Only when I’m not sure the document on this topic exists
e. Never use the search for this purpose

6. Do you use “advanced search” capabilities?

a. Yes, often
b. Rarely
c. No, basic search is enough

7. Do you use the system “Recent documents” list?

a. Yes, often
b. Rarely
c. No

8. Do you enter the metadata (title, author, subject, keywords etc) in the “Document properties” dialog box of your text editor?

a. Yes, always
b. For important documents only
c. Don’t care about it

9. How do you mark an importance of a document?

a. Place it into a special folder
b. Place it on the desktop
c. Bookmark it (place into the “Favorites”)
d. Do nothing

10. How many documents (in percentage against all documents on your machine) are used actively?
11. How do you handle outdated documents?

a. Keep them in place
b. Move to a special folder
c. Move to backup media, then delete
d. Delete them

12. Which formats you use for the text documents (specify in order of importance, e.g. “a, d, c, b”):

a. MS Word
b. OpenDocument
c. PDF
d. HTML
e. XML or SGML (DocBook etc)
f. Plain text
g. Other

13. Do you save online documents on your local hard drive?

a. Yes, often
b. Save important or very large documents only
c. Never

14. Do you keep a sort of a personal electronic library?

a. Yes
b. No

15. Do you use specialized software for photo albums or multimedia collections management?

a. Yes
b. No, standard system tools are enough

16. How do you estimate efforts on supporting your local document collections?

a. It is a burden, it takes a lot of my time and harms for my work
b. It takes some time but it worths it
c. It is not a problem with help of the modern desktops
d. Do not see any problem

17. Do you use del.icio.us?

a. Yes, for every interesting stuff I meet on the Web
b. Yes, for important links only
c. Yes, for links I want to share with somebody else
d. No

18. How many tags are in your del.icio.us profile?
19. How many del.icio.us tags you usually assign per a single link (in average)?
20. When choosing the del.icio.us tags, you prefer

a. Your own tags
b. Other’s tags, suggested by the service

21. When choosing the del.icio.us tags, you prefer to

a. Reuse the existing tags, as possible
b. Create new tags

22. Do you try to avoid tags synonymity?

a. Yes
b. No, I don’t care about synonyms

23. Do you use tag bundles?

a. Yes
b. No

24. Which factors are important for tags selection (specify in order of importance, e.g. “a, d, c, b”)?

a. My own subjective associations
b. My vision of of the implicit topic semantics (tend to be objective)
c. Explicit textual properties of the document (terms frequency, etc)
d. Tags, assigned by other people

25. The best synonym of the”tag” is:

a. Category
b. Term
c. Topic
d. Keyword
e. Label

26. The purpose of the tags is:

a. Distinction
b. Unification

27. A number of tags per document is a measure of:

a. Document importance
b. Information diversity
c. Collection size
d. Selection quality

28. Would automatical tags generation be useful?

a. Yes, and it could completely replace human brains in this area
b. Yes, but it matters as a help for human brains only
c. No, the tags should belong to humans

Thank you!

Instead of developing some Yet Another Big Animated Clock, I decided to write something practical. What I’d like to have is a widget which would ask my e-mail client (KMail) for the headers of the latest unread messages to show them on the desktop.

Two buttons in the top bar are for checking new mail from a server and for opening new message window. Clicking on the top bar somewhere else will open main KMail window.

SuperKaramba comes with a Python API, so everyone who is familiar with Pyton basics can write her own widget (aka “theme”) as a Python script. In my case, I wrote two scripts – one is for widget output and another as a simple KMail folder API wrapped around DCOP calls.

Patching KMail

There was a major problem – KMail DCOP interface had no a call to get unread message headers; only getting a number of them is possible. So I have had to dig into the KMail sourcecode (my current version is 1.9.4) for to add new function to FolderIFace interface (folderIface.cpp).

kmail.diff:

diff kmail/folderIface.cpp kmail_patched/folderIface.cpp
147a148,162
> QStringList
> FolderIface::unreadMessageHeaders()
> {
>      QStringList msgs;
>      mFolder->open();
>      for( int i = 0; i < mFolder->count(); i++) {
>          KMMsgBase *msg = mFolder->getMsgBase(i);
> 	     if (msg->isNew() || msg->isUnread()) {
> 	       msgs.append(msg->fromStrip() + "\\t" + msg->subject() + "\\t" + msg->dateStr());
>          }
>      }
>      mFolder->close();
>      return msgs;
> }
>
diff kmail/folderIface.h kmail_patched/folderIface.h
61c61,62
<
---
>     virtual QStringList unreadMessageHeaders();
>

After adding new function header to folderIface.h and rebuilding KMail, new DCOP call “kmail FolderIface unreadMessageHeaders” became available to get the list of message headers in format “From/Subject/Date” divided by tab characters (\t).

Sourcecode

skkmail.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103

import karamba
import kmail
 
text = []
images = []
clickAreas = []
 
# Define widget area width here (height is adjusted automatically)
s_width = 350
 
# If an actual height is above this value, a widget area will be clipped
maxHeight = 600
 
# Margins inside a widget area
s_x = 10
s_y = 10
 
def _update(widget):
    global text, images, clickAreas, s_width, s_x, s_y
    x_gap = 10
    y_gap = 4
 
    s_height = 14
    s_col1 = (s_width-x_gap)*0.66
    s_col2 = s_width
    s_col3 = (s_width-x_gap-16)*0.33
 
    folderFont = "Franklin Gothic Medium"
    folderFontSize = 14
 
    for t in text:
        try:
          karamba.deleteText(widget, t)
        except:
          pass
    for i in images:
        try:
          karamba.deleteImage(widget, i)
        except:
          pass
    for c in clickAreas:
        try:
          karamba.removeClickArea(widget, c)
        except:
          pass
    text = []
    images = []
    clickAreas = []
    folders = kmail.getFolders()
    y = s_y + 25;
    for f in folders:
        msgs = f.unreadMessages
        if (len(msgs) > 0):
            y0 = y
            t = karamba.createText(widget, s_x+15, y, s_width, s_height,
                f.name+" ("+str(len(msgs))+")")
            karamba.changeTextFont(widget, t, folderFont)
            karamba.changeTextSize(widget, t, folderFontSize)
            text.append(t)
            images.append(karamba.createImage(widget, s_x-10, y, "icons/folder.png"))
            y = y + s_height + y_gap*2 +10
            for m in msgs:
                t = karamba.createText(widget, s_x+20, y, s_col1, s_height, m[0])
                text.append(t)
                t = karamba.createText(widget, s_x, y+s_height, s_col2, s_height, m[1])
                text.append(t)
                t = karamba.createText(widget, s_x+20+s_col1+x_gap, y, s_col3, s_height, m[2])
                karamba.setTextAlign(widget, t, "RIGHT")
                text.append(t)
                images.append(karamba.createImage(widget, s_x, y, "icons/mail.png"))
                y = y + s_height*2 + y_gap*2
            clickAreas.append(karamba.createClickArea(widget, s_x-10, y0, s_width, y-y0,
                "dcop kmail KMailIface selectFolder "+f.path+" && kmail"))
            y = y + y_gap*2
    if (y < maxHeight - 20):
        karamba.resizeWidget(widget, s_width+s_x*2, y+20)
 
#this is called when you widget is initialized
def initWidget(widget):
    bgImage = karamba.createBackgroundImage(widget, 0, 5, "icons/bg.png")
    karamba.resizeImage(widget, bgImage, s_width+s_x*2, 20)
    logo = karamba.createImage(widget, 0, 0, "icons/kmail.png")
    t = karamba.createText(widget, 35, 5, 100, 20, "new mail")
    karamba.changeTextFont(widget, t, "Franklin Gothic Medium")
    karamba.changeTextSize(widget, t, 16)
    karamba.createClickArea(widget, 0, 0, 100, 30, "kmail")
    karamba.createImage(widget, s_width+s_x*2-18, 7, "icons/mail_get.png")
    karamba.createClickArea(widget, s_width+s_x*2-18, 7, 16, 16, "dcop kmail KMailIface checkMail")
    karamba.createImage(widget, s_width+s_x*2-40, 7, "icons/mail_new.png")
    karamba.createClickArea(widget, s_width+s_x*2-40, 7, 16, 16,
        "dcop kmail KMailIface openComposer '' '' '' '' '' ''")
    _update(widget)
 
#this is called everytime your widget is updated
#the update inverval is specified in the .theme file
def widgetUpdated(widget):
    _update(widget)
 
#This gets called everytime our widget is clicked.
# Middle button click forces to check new mail
def widgetClicked(widget, x, y, button):
    if (button == 2):
        kmail.dcopCall("kmail KMailIface checkMail")

kmail.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

#!/usr/bin/python
 
import os, subprocess, string
 
def dcopCall(call):
  cmd = "dcop "+call
  pipe = subprocess.Popen(cmd, shell=True, bufsize=1024, stdout=subprocess.PIPE).stdout
  res = pipe.read()
  pipe.close()
  return res
 
class Folder:
  path = ""
  name = ""
  unreadCount = 0
  unreadMessages = []
 
  def __init__(self, path):
    dcopCall("kmail KMailIface getFolder "+path)
    self.path = path
    self.unreadMessages = self.getUnreadMessages()
 
  def getUnreadMessages(self):
    r = dcopCall("kmail FolderIface unreadMessageHeaders")
    ss = string.split(r, "\\n")
    msg = []
    for s in ss:
        if (len(s) > 0):
            msg.append(string.split(s, "\\t"))
    unreadCount = len(msg)
    if (unreadCount > 0):
        self.name = string.strip(dcopCall("kmail FolderIface displayName"))
    msg.reverse()
    return msg
 
def getFolders():
    r = dcopCall("kmail KMailIface folderList")
    folders = []
    for f in string.split(r, "\\n"):
        folders.append(Folder(f))
    return folders

skkmail.theme:

1
2
3

# you can change initial widget position, area size and updatin interval (default = 1min)
karamba x=100 y=100 w=400 h=200 interval=60000
defaultfont font="Franklin Gothic Book" fontsize=12 color=230,230,230

All code above is GPL‘ed.

There is a case study: our RUSMECO project, among many other things, includes development of so-called “Brokerage Area” – an open-access database of the people, organizations, products and services which are related together by many ways. Our main collaboration portal is based on Drupal so it was obvious that reusing the same platform for BA development advantages in all senses. We have investigating which Drupal features might fit our solution – and found all what we needed.

Flexinode

All content objects in Drupal are treated as an abstraction of the “nodes”. There are various types of nodes (pages, stories, forum posts, blog posts etc) which differ by their fields and provided by the core or additional modules. But there is also a mechanism of creating new custom types of nodes (so-called “flexinode”) with arbitrary sets of the data fields. So, you can define your own structured data types right into the Drupal administrative interface without writing any PHP code and creating new database tables. The user interface (the forms) for nodes will be built automatically and can be adjusted later (for instance, the fields order and grouping). These new types will be fully integrated into the core content framework – in fact, they will be the same content objects as the core types and all Drupal functions (the search, categorization, multilingual translation, workflow, access management and so on) will work with them.

In our case, we needed four content types: Member, Company, Product and Service profiles. It is only pity that the built-in system user profile is not a node (though it have a customizable set of the fields), so we left there only a bare minimum of fields (only username, email and password, in fact) and defined a separate Member profile (CV) node type for those who wants to tell more about herself.

Taxonomies

Taxonomies (or, controlled vocabularies) is a standard Drupal way for nodes classification and organization. They are the sets of categories or terms which can be organized hierarchically, related each other and grouped into the vocabularies. Depending on your IA, they can serve as traditional content sections, the “tags” or “labels” in faceted classification system or as thesauri of interrelated terms. The categories can be also related with the content types, so that the nodes of a specific type can be qualified as the members by specific categories only, or be included in a category by default.

Along with predefined categories, it is possible to define new ones (in selected vocabularies) on-the-fly, together with creation of new nodes. For instance, an user who is filling her product profile out, is able to create the new product category at one time.

I18N

Our solution had to be bilingual (en/ru) – it was a principal requirement. It concerns not only an interface, but also that it should be possible to create the translations of the nodes in alternate language. For instance, an user should be able to create two versions of a profile and when another user coming to see it, she would get a variant for her current language (selected in the top panel).

The features of Drupal i18n module enable to do so. What’s more, it is possible also for categories to be translated, so that the user sees all categories in her current language and when she is in a category, all nodes there are in that language too.

Modules

And after all, if you’re well sure Drupal lacks some functionality, you can remember your own PHP and SQL experience and write some code. For instance, you would be needed for a custom sophisticated query iinterface for your database. But you probably will not have to write a lot – as all infrastructural aspects (such as templating etc) are already implemented in Drupal API, you’ll need to implement only your specific functions.

The specific pieces of Drupal functionality are called “the modules”. In fact, the Drupal core is only the component framework and almost all its functions are implemented as the pluggable modules. So, to extend it with your custom functions there is no a better way than develop your own module. It’s pretty easy – a module is simply a PHP file containing the conventionally named functions (aka “hooks”). These functions may define what should be displayed in a module page, module blocks (in left and right columns), module settings page, how the module will be integrated into the navigation menus and configuration interface and so on.

For to start with modules development, consider to read this tutorial.

Reading on subject

• Is Drupal right for you?
• Rolling your own system vs. using Drupal
• Drupal for Information Architects: an overview of configurability
• The Road to Drupal Hell

… more on the Drupal site.