Pimp my Nepomuk

Being both a really old KDE user and a semantic desktop partisan at the same time, I am, of course, keeping my eye on the progress in Nepomuk project.  It was apparently close to my old dream of a tagging framework supported natively and consistently across the whole desktop environment, so I highly appreciated this effort and it was nice to hear that Nepomuk would be officially included into KDE – my desktop of choice for many years.

Unfortunately, the experience was rather disappointing. It’s turned out to be painfully slow, not only slow by itself, but being a brake for overall desktop navigation. Even hovering the cursor over files and folders in Dolphin made Nepomuk process to eat above 50% of CPU time and caused annoying delays. The simple operations like assigning a tag to a file took seconds, the responsiveness which is obviously inappropriate for a real-world desktop system. It thus was turned off in a hope that the things would be improved in future versions (I was confused a bit by how it might appear in the production release, but, frankly, early KDE 4 was full of much more disastrous things). Since then, I checked it after every KDE version upgrade, but there was no visible progress in performance, alas.

It was really good news – the author of the post above argues that the performance issues are in fact, caused by a storage backend which Soprano, an RDF framework underlying to Nepomuk, uses to keep RDF data. By default, it’s shipped with Redland (aka librdf), an RDF database library written in C. Luckily, the backend is easily replaceable and it’s worth to try to install a faster alternative seeking for a better performance. The author recommends Sesame2 – a 100% pure Java RDF framework which works (surprisingly for many, I think – but not for me!) much faster than it’s native code counterpart.

I tested Nepomuk with Sesame and convinced that now it works really faster – as it should, in fact. There is of course, a room for improvements in Nepomuk to be a real end-user tool – e.g. a tag navigation interface without which the tags are rather useless, but its another story. At least, the performance is not a blocker anymore, so Nepomuk now is enabled in my KDE all the time.

For Kubuntu users: How-to

I tested the Sesame2 backend for Nepomuk on Kubuntu 9.04 Jaunty Jackalope, KDE 4.3.0 and Sun JRE 6 (I have no idea if it works with GNU Java, but you can give it a try).

1. Install soprano-backend-sesame package (sudo apt-get install soprano-backend-sesame)
2. Make a symlink from $JAVA_HOME/jre/lib/i386/client/libjvm.so in the /usr/lib directory
3. Restart Nepomuk server

Check if Sesame2 backend is used now:

qdbus org.kde.NepomukStorage /nepomukstorage usedSopranoBackend

It should answer “sesame2“. If it still answers “redland“, something was wrong. You may need also to replace the value “redland” to “sesame2” in ~/.kde/share/config/nepomukserverrc file manually and restart Nepomuk again.

What’s this

It’s a prototype of a social network service where people are connected automatically via shared topics of interests extracted from their texts. You can find more on the PCN theory in this post.

If you’re interested to play with it, we are glad to invite you to join the testing.

How to join

1. Request a personal invitation code by filling this form. The code will be sent to the provided email address.

2. Register on PCN with your invitation code. Your username must be the same you used in the request above.

3. Download, install and configure the PCN client (Conex) as proposed by the system.

To run the client, an installed Java Runtime Environment is required. You also may need the Java plugin installed into your browser for visualisation of the network in your server profile.

The PCN Client

The PCN client software (Conex, from “CONcept EXtraction”) is a desktop content aggregator collecting pieces of content from different locations (local folders, webs, email, RSS feeds, Del.icio.us, SharePoint servers …). The content is analyzed to extract valuable terms from the texts and assign them as the document tags (manual tagging is possible also). A user can browse her content collections with the tag cloud or  metadata facets, edit document metadata, annotate documents, search in the content collections and do a lot of other things. Those who are interested into details of client functionality and in a underlying technology, can check this site.

To configure PCN connection, select  “PCN Client → Configure PCN Client” menu. At the first time, it will ask your username/password on the server and offer to create one or more contexts for your data. The contexts serves as navigation facets to browse the content collection and are useful to organize the content resources depending on your activities. After the contexts are defined, you can assign existing Conex content locations to them.

Autotagging is applied to individual documents by selecting them and choosing “Autotagging…” in the context menu. This operation can also be automated for all documents in a specific location, if “Apply autotagging for new documents” option is set in the location properties (“Settings” tab of the location dialog box). You can edit the results of autotagging or add new tags for selected documents using “Edit tags” option of the context menu.

As the content is tagged, Conex sends the tagging data and resource descriptions to the server, where a profile of user’s interests is created. All updates in the monitored locations are also sent to the server (in the specified time interval) to keep the user profile up-to-date.

By default, Conex works with the files in your local folders. To enable other types of content locations, you need to install the plugins. A plugin is installed with a single click in the Plugin Management console (“Tools → Manage plugins…” menu) and will be activated after restart.

PDF, MS Office, OpenOffice, HTML, XML and plain text documents are supported out of the box.

For SCAN users

To clarify the things, Conex is neither a proprietary fork, nor a new version of SCAN. It’s basically a rebranded distribution bundled with the common document plugins and integrated into the PCN solution. There is a plugin providing connectivity with the PCN Server and it’s possible to install this plugin into “native” SCAN to use it as a full-featured PCN client. This likely won’t work with the released version, but should be ok with a SVN snapshot. Anyway, new SCAN version enabling the PCN plugin will be released soon, so stay tuned.

The PCN Server

The server receives data from the clients and builds the socio-semantic network of users connected via the tags they have in common. After you’ve submitted some data from Conex, you can check your profile on the server to see your tags and  people appeared in your personal network:

The people listed in your network are the users who have the similar interests. On their profile pages, you can see a detailed information on compatibility between you and a profile owner:

The Network view displays a visualization of your complete network as a map of the overlapping clusters of tags you share with the members of your network:

The map is interactive — double-clicking an element will navigate to a user profile or to a tag page.

The people search is also implemented. You can find users by their names, other profile data or by the tags:

And finally, you can explore the content collections of the network members using faceted navigation with tags and contexts (supplemental metadata facets, such as document authors and  creation dates are also available).

Your privacy

⚠ The full text of the documents is not submitted to the server, however, Conex can use excerpts from the documents to fill the title and description metadata (if it is not defined explicitly). So, if your documents contain some sensitive information, parts of it may appear on the public. Please check and make sure that there are no private documents in the Conex locations you are about to assign to the PCN contexts. Well, you’ve been warned.

Report issues

It’s a beta, so  the bugs are likely to be there. Feel free to report the bugs, either in the Server or the Client,  to the bug tracker (registration is needed). Any other feedback, of course, is also more than welcome.

Disclaimer: Like it says on the author page, I work for UoN and Laboranova project, but I cannot speak for it officially in this blog. These are my own views as a project insider and the other’s may differ.

Back story

As far back as in early 2008, I’ve been introduced into PCN proposal and invited for collaboration on a R&D for its prototype implementation. It was very interesting for me and we developed a vision and basic design principles of the PCN solution to start the prototype development at the beginning of 2009.

This work is a part of Laboranova — a large EU project aimed at new ways of collaboration between knowledge workers and sharing ideas and competencies. Apart from PCN, there are lots of other interesting thngs the project partners do there.

PCN basics

The idea is a social networking where the members were connected via shared areas of expertise. These areas are identified by the topics of interests, discovered at user’s content.

Like other social network services connect people via shared professional activities, schools, hobbies or music tastes, PCN approach employs the shared semantics of texts the people read or write. There is an optimistic assumption that such semantics, expressed as sets of the weighted keywords (or, tag clouds), may reflect an area of user’s expertise and constitute a profile of his professional interests.

Multiple user profiles are aggregated to create expertise information and form a socio-semantic network of people and concepts. Comparing profiles of different users, we can evaluate a similarity of their expertises and thus, estimate their social distance in the network. We also can take a specific area of expertise to find who are the best experts relevant to it, and explore the related overlapping areas. With this network analysis, it is possible to generate a variety of individual recommendations to help people to discover new collaboration opportunities and areas of knowledge.

The Platform

There is a client-server architecture including a central PCN server and a number of clients connected to it. The PCN client software is installed locally and analyses content from different locations specified by its owner: file folders, webs, email, RSS, delicious accounts, SharePoint servers and so on.

A user can let the software extract the keywords from the content automatically, edit the results or choose to tag the content manually. Then he chooses what locations should be submitted to the server and assigns them to one or more named contexts which help to organize the concepts within the user profile. Document metadata and tagging information is uploaded to the server where an individual profile of tags is created and published. As a location is submitted to the server, it is monitored for changes to synchronize the profile with the actual state of the content (by sending incremental updates at a specified time interval).

The Client

The  client part is based on SCAN, so there is a zero barrier to start working with it for SCAN users. Actually, from a SCAN user perspective, there is no difference from usual everyday work — you can enjoy a full set of features one can find in native “offline” SCAN, but also use it for populating your public profile of interests at the PCN server.

Also, variety of location types and document formats are supported by the client thanks to the plugins from the SCAN repository.

The Ontology

The server receives metadata about content resources as SIOC RDF, so in theory, it may work with any SIOC provider, apart from the default PCN client. The server augments resource metadata with relationships to the user profile, concepts and contexts, thus forming the quadripartite PCN ontology model:

PCN quadripartite model

To describe the PCN model, we adopted SCOT (Semantic Clouds Of Tags) ontology aimed at conceptualization of the structure and  semantics of tagging data with strong focus on social interoperability. It is an extension and further development of the Tag Ontology project that describes the relationship between an agent, an arbitrary resource, and one or more tags.

The SCOT (Tag) ontology is based on a tripartite (User—Tag—Resource) model. These three core concepts are connected together via a central concept of Tagging representing the tagging activity. Every Tagging instance can be considered as a result of a single tagging action defining a user who performed it, the tagged resource and what tags have been used. It also can carry auxiliary information about the action, such as the time of tagging.

For users and resources, SCOT relies upon concepts from SIOC — specifically, sioc:User and sioc:Item classes. For our PCN ontology, we extended the SCOT model with the notion of  context by adding another class of entities and a property to relate them with the Tagging instances.

Extended SCOT Tagging as a skeleton of the PCN model

Using SCOT Tagging model, it is possible to avoid excessive verbosity in the PCN ontology, as the relationships between core PCN classes (the edges of the tetrahedron) are inferable from their relationships with the central Tagging class (the skeleton). In the diagram below, the implicit relations are shown as dashed.

PCN Ontology

An interesting possibilities comes from the fact that the scot:Tag is actually a subclass of the SKOS Concept, so all kinds of SKOS reasonings about the concepts are possible in the future. Moreover, MOAT features incorporated in SCOT open a way to integrate the tag descriptions with the Linked Data web.

The API

The client talks to the server via a simple RESTful API for adding and modifying the users metadata. As said above, the metadata is described with SIOC, so it is theoretically possible to use that API and integrate the PCN server with any system which can provide SIOC metadata about the content.

An endpoint for SPARQL queries is planned also.

For more information:

• An early description of the work package (Laboranova Profile System) on Laboranova web-site.
• Marc Pallot et al, “Future Workplaces, towards the ‘Collaborative’ Web“
• Peter Mika, “Ontologies are us : A unified model of social networks and semantics“
• Hak-Lae Kim et al, “Review and Alignment of Tag Ontologies for Semantically-Linked Data in Collaborative Tagging Spaces“ , “The State of the Art in Tag Ontologies: A Semantic Model for Tagging and Folksonomies“
• SCOT
• SIOC

“How does SCAN help me?”, “Why should I use it?”, “Who are the users?”, “Why it is smart?”, “Can it replace a …?”, “What is autotagging?”, technical tips-n-tricks, development questions and a lot of other things you would want to know about Smart Content Aggregation & Navigation technology.

Read SCAN FAQ »

An idea is borrowed from e-mail clients, where the similar feature exists for decades as the user-defined filters for processing the messages. This is how it looks in KMail:

In general, a filter checks if a document matches to specific criteria (a rule) and does some action if yes. For instance, if a condition “text contains ‘viagra’ or ‘cialis’” is true, then some action (“move to spam” or “send assassins to the author”) would be executed. What is especially good is that it’s old, popular and intuitive user experience.

In a content aggregator like SCAN, this concept may allow a user to define custom automation rules to set document metadata properties. For instance,

IF (url starts with "http://cyberborean.wordpress.com") SET author = "me"

Another using I have in my mind is to empower an “artificial intelligence” of autotagging with a human intelligence, by user-defined tagging rules:

IF (text contains "latte") ADD TAG "coffee"

I only doubt about terminology — “filter” might be confusing, as it is already used in SCAN vocabulary (URI filters to include/exclude some documents by their URI pattern). Any ideas?

(Read more on SCAN website)

TagClusters visualize the selected tags as overlapping clusters of documents. With that cluster map, it is easy to see how the tags relate each other via the documents they have in common. The plugin uses Tags Grouping – new core SCAN feature for finding the groups of interrelated tags. In TagClusters, this feature is used for automatic expanding a selected tag to a group of its semantic neighbors. By clicking a single tag, a user would see a map visualizing this tag plus all related tags, so that the whole taxonomy can be explored just with few mouse clicks.

These colorful amoebae are drawn by Aduna Cluster Map library. It’s just an eye-candy – it’s going to be a sexiest SCAN panel, I think.

What’s also new in 1.1? Well, hm, a plugin for scanning Del.icio.us accounts, for instance. It’s coming soon – be on the watch for announcements !

“Foreword”
chapters corrected edition experience glimpses information process readers reference story

“Prologue”
bilbo book copy families history hobbits shire smials thain westmarch

Book I

baggins bilbo bree butterbur frodo goldberry ponies sam strider tom

1. “A Long-expected Party”
   bag baggins bagginses bilbo birthday fireworks joke lobelia presents sackville
2. “The Shadow of the Past”
   bilbo birthday déagol frodo gandalf gollum grandmother hated ring ted
3. “Three is Company”
   bag frodo gildor lane lobelia pippin sackville sam sir sold
4. “A Short Cut to Mushrooms”
   dogs farm farmer ferry frodo lane maggot mushrooms pippin waggon
5. “A Conspiracy Unmasked”
   bath brandy brandybucks bucklanders fatty ferry fredegar frodo maggot merry
6. “The Old Forest”
   bonfire forest hedge lilies merry path ponies tom willow withywindle
7. “In the House of Tom Bombadil”
   barrows bombadil candle chimney creak derry goldberry tom wights willow
8. “Fog on the Barrow-Downs”
   barrow bombadil fog goldberry hoy jogging lumpkin ponies tom wight
9. “At the Sign of The Prancing Pony”
   barliman bree butterbur comer cow frodo inn landlord nob underhill
10. “Strider”
    baggins bree butterbur ferny frodo landlord letter nob strider underhill
11. “A Knife in the Dark”
    beren bob bree dell ferny galad gil strider tinúviel weathertop
12. “Flight to the Ford”
    bone boot ford frodo glorfindel horse strider troll trolls weathertop

Book II

aragorn boats boromir elrond frodo gandalf gimli haldir legolas sam

1. “Many Meetings”
   bilbo dúnadan elrond flood frodo glorfindel glóin mortals rivendell strider
2. “The Council of Elrond”
   boromir elrond erestor galad gil glóin isildur radagast ring saruman
3. “The Ring Goes South”
   aragorn bill boromir caradhras drift elrond frodo gandalf redhorn snow
4. “A Journey in the Dark”
   bill boromir doors durin gandalf gimli howls mines moria wolves
5. “The Bridge of Khazad-dum”
   balin balrog beats boromir chamber doom door drum hall mazarbul
6. “Lothlórien”
   flet frodo galadhrim gimli haldir ladder legolas lothlórien nimrodel silverlode
7. “The Mirror of Galadriel”
   basin celeborn frodo galadhrim galadriel haldir lady magic mirror pedestal
8. “Farewell to Lórien”
   boat boats boromir cakes celeborn galadhrim galadriel gimli lady lórien
9. “The Great River”
   aragorn boat boats frodo gebir log paddle paddles rapids sarn
10. “The Breaking of the Fellowship”
    amon aragorn boromir brandir frodo hen lawn lhaw sam tol

Book III

aragorn ents gimli isengard legolas saruman théoden treebeard uglúk éomer

1. “The Departure of Boromir”
   aragorn boat boromir galen gimli gulls legolas orcs rauros tire
2. “The Riders of Rohan”
   aragorn downs entwash fangorn gimli horses legolas orcs trail éomer
3. “The Uruk-Hai”
   ankles grishnákh isengarders knoll lugbúrz merry orcs pippin uglúk wrists
4. “Treebeard”
   ent entish ents entwives hasty hoom isengard rowan treebeard trees
5. “The White Rider”
   aragorn fangorn gandalf gimli haft legolas man old saruman treebeard
6. “The King of the Golden Hall”
   aragorn gandalf gríma hall háma neighed théoden wormtongue éomer éowyn
7. “Helm’s Deep”
   aragorn coomb deeping dike erkenbrand gamling helm hornburg westfold éomer
8. “The Road to Isengard”
   caverns caves fords gandalf gimli isen isengard legolas saruman théoden
9. “Flotsam and Jetsam”
   barrels ent ents gimli huorns isengard pipe quickbeam saruman treebeard
10. “The Voice of Saruman”
    dismissed ents gandalf injuries orthanc rail saruman spell théoden treebeard
11. “The Palantír”
    ball bracken gandalf globe orthanc palantíri pippin saruman shadowfax wizard

Book IV

faramir frodo gollum nice precious rope sam shagrat shelob sméagol

1. “The Taming of Sméagol”
   cliff fix frodo gollum knot precious rope sam sméagol yess
2. “The Passage of the Marshes”
   fish frodo gollum gully marshes mires nice precious sam sméagol
3. “The Black Gate is Closed”
   frodo gollum helps lithui master nice oliphaunt oliphaunts sam sméagol
4. “Of Herbs and Stewed Rabbit”
   coney fern gollum herbs mablung pans rabbits sam sméagol taters
5. “The Window on the West”
   arts blindfold boromir damrod faramir frodo isildur mablung men mithrandir
6. “The Forbidden Pool”
   anborn faramir fish fissh frodo gollum precious protection sly sméagol
7. “Journey to the Cross-roads”
   brewing decent faramir frodo glare gollum mould sam staves thorns
8. “The Stairs of Cirith Ungol”
   dad frodo gollum morgul sam sméagol sneaking stairway wink wraith
9. “Shelob”
   frodo glass gollum lair phial sam shelob sméagol stench tunnel
10. “The Choices of Master Samwise”
    cords frodo gorbag lads ladyship lugbúrz sam shagrat shelob tunnel

Book V

aragorn beregond city denethor faramir ghân men pippin éomer éowyn

1. “Minas Tirith”
   beregond bergil citadel city denethor forlong ingold mithrandir pippin street
2. “The Passing of the Grey Company”
   aragorn burg elladan erech gimli halbarad king legolas ride théoden
3. “The Muster of Rohan”
   dwimorberg edoras harrowdale hirgon king merry ride théoden éomer éowyn
4. “The Siege of Gondor”
   beregond citadel city denethor faramir gandalf garrison pelennor pippin retreat
5. “The Ride of the Rohirrim”
   buri dernhelm elfhelm ghân men stonewain théoden wild éomer éored
6. “The Battle of the Pelennor Fields”
   dernhelm harlond king knights mûmakil snowmane southrons théoden éomer éowyn
7. “The Pyre of Denethor”
   beregond bier burn city denethor faramir gandalf healing oil thou
8. “The Houses of Healing”
   aragorn athelas bergil faramir healing imrahil ioreth sick éomer éowyn
9. “The Last Debate”
   aragorn gimli gulls imrahil lamedon lebennin legolas pelargir ships thousands
10. “The Black Gate Opens”
    army captains heralds imrahil lieutenant messenger morannon sauron terms thou

Book VI

cotton faramir frodo lotho rosie ruffians sam shagrat sharkey éowyn

1. “The Tower of Cirith Ungol”
   court frodo gorbag ladder orc sam shagrat shelob snaga turret
2. “The Land of Shadow”
   frodo isenmouthe morgai orc range sam shagrat spur tracker whip
3. “Mount Doom”
   cone frodo gaping gollum load masster mountain sam sammath tearing
4. “The Field of Cormallen”
   captains eagles frodo gwaihir isle knights merriment praise pure sam
5. “The Steward and the King”
   aragorn city faramir healer healing ioreth king lady warden éowyn
6. “Many Partings”
   arwen beggar celeborn hoom orthanc théoden treebeard wain éomer éowyn
7. “Homeward Bound”
   barliman bill bree butterbur deadmen ferny gandalf nob pony troubles
8. “The Scouring of the Shire”
   cotton farmer hob lotho ruffians sharkey sheds shirriff shirriffs village
9. “The Grey Havens”
   bywater cotton deputy elanor frodo married mayor rosie ruffians sam

Appendices

• A. “Annals of the kings and rulers”
  arthedain arvedui eldacar eärnil gondor king náin tar thráin witch
• B. “The Tale of Years”
  begins birth guldur king meets reaches samwise sauron sets thranduil
• C. “Family Trees”
  addition appendix birth blanco family guests names party recorded recounted
• D. “Shire Calendar”
  calendar corresponded eldar lithe months names quenya reckoning week yule
• E. “Writing and spelling”
  english languages letters pronounced quenya represented represents series sindarin vowel
• F. “The languages and peoples of the Third Age”
  english languages letters pronounced quenya represented represents series sindarin vowel

Resulting Tags Cloud