“How does SCAN help me?”, “Why should I use it?”, “Who are the users?”, “Why it is smart?”, “Can it replace a …?”, “What is autotagging?”, technical tips-n-tricks, development questions and a lot of other things you would want to know about Smart Content Aggregation & Navigation technology.

Read SCAN FAQ »

An idea is borrowed from e-mail clients, where the similar feature exists for decades as the user-defined filters for processing the messages. This is how it looks in KMail:

In general, a filter checks if a document matches to specific criteria (a rule) and does some action if yes. For instance, if a condition “text contains ‘viagra’ or ‘cialis’” is true, then some action (“move to spam” or “send assassins to the author”) would be executed. What is especially good is that it’s old, popular and intuitive user experience.

In a content aggregator like SCAN, this concept may allow a user to define custom automation rules to set document metadata properties. For instance,

IF (url starts with "http://cyberborean.wordpress.com") SET author = "me"

Another using I have in my mind is to empower an “artificial intelligence” of autotagging with a human intelligence, by user-defined tagging rules:

IF (text contains "latte") ADD TAG "coffee"

I only doubt about terminology — “filter” might be confusing, as it is already used in SCAN vocabulary (URI filters to include/exclude some documents by their URI pattern). Any ideas?

(Read more on SCAN website)

The plugin purpose is to crawl the specified local email folders and aggregate the email messages as the documents in SCAN repository. It also introduces new document type “message/rfc822” for emails and uses the email message headers for setting document metadata. Attached files are extracted from messages and processed as the separate documents with appropriate parsers depending on its content type.

As there are no common convention on how to identify a separate message for opening it with an external application, mail plugin implements its own message viewer UI to open the messages by default. The messages are identified with “mid:” URI scheme which is a standard but unfortunately, seems to not supported by known MUA’s so far. However, it is implemented in the hope of that the future MUA’s will support this standard scheme to open the specific messages with their command line (something like “thunderbird mid:message-id“).

Mail plugin uses GNU JavaMail implementation because it includes JavaMail providers for local mail stores – mboxes and maildirs. In addition, Outlook Express mail is also supported via the separate JavaMail provider made by jmbox project. Thus, the mail plugin supports a wide range of popular MUA’s, including mbox-based (Mozilla family), maildir-based, mixed mbox/maildir (KMail and Evolution) and Outlook Express.

The plugin has been tested on Linux with Mozilla Thunderbird 1.5 and KMail 1.9.6 (in maildir mode) and Windows XP with Mozilla Thunderbird 1.5 and MS Outlook Express 5.0.

Problems?

Yep, they are. The issues with mail plugin go both from GNU JavaMail limitations and from the side of the concrete MUA’s which seem to be easy with interpreting mbox/maildir principles for their convenience. For instance, Thunderbird uses the non-standard “.sbd” directories to keep the mail subfolders, which the mbox provider is unaware of. So, recursive crawling the Thunderbird mail folders does not work. The only solution for those problems is to develop the MUA-specific JavaMail providers which would know how to deal with a concrete mbox or maildir implementation with all its peculiarities.

Outlook Express provider, as an example of such MUA-specific implementation works rather good, however we noticed a bug with localized MSOE versions where the translated folder names (“Inbox”, “Sent”, “Trash” etc) with non-latin letters had not been processed.

The reports on testing the mail plugin with other MUA’s/platforms are more than welcome.

TagClusters visualize the selected tags as overlapping clusters of documents. With that cluster map, it is easy to see how the tags relate each other via the documents they have in common. The plugin uses Tags Grouping – new core SCAN feature for finding the groups of interrelated tags. In TagClusters, this feature is used for automatic expanding a selected tag to a group of its semantic neighbors. By clicking a single tag, a user would see a map visualizing this tag plus all related tags, so that the whole taxonomy can be explored just with few mouse clicks.

These colorful amoebae are drawn by Aduna Cluster Map library. It’s just an eye-candy – it’s going to be a sexiest SCAN panel, I think.

What’s also new in 1.1? Well, hm, a plugin for scanning Del.icio.us accounts, for instance. It’s coming soon – be on the watch for announcements !

“Foreword”
chapters corrected edition experience glimpses information process readers reference story

“Prologue”
bilbo book copy families history hobbits shire smials thain westmarch

Book I

baggins bilbo bree butterbur frodo goldberry ponies sam strider tom

1. “A Long-expected Party”
   bag baggins bagginses bilbo birthday fireworks joke lobelia presents sackville
2. “The Shadow of the Past”
   bilbo birthday déagol frodo gandalf gollum grandmother hated ring ted
3. “Three is Company”
   bag frodo gildor lane lobelia pippin sackville sam sir sold
4. “A Short Cut to Mushrooms”
   dogs farm farmer ferry frodo lane maggot mushrooms pippin waggon
5. “A Conspiracy Unmasked”
   bath brandy brandybucks bucklanders fatty ferry fredegar frodo maggot merry
6. “The Old Forest”
   bonfire forest hedge lilies merry path ponies tom willow withywindle
7. “In the House of Tom Bombadil”
   barrows bombadil candle chimney creak derry goldberry tom wights willow
8. “Fog on the Barrow-Downs”
   barrow bombadil fog goldberry hoy jogging lumpkin ponies tom wight
9. “At the Sign of The Prancing Pony”
   barliman bree butterbur comer cow frodo inn landlord nob underhill
10. “Strider”
    baggins bree butterbur ferny frodo landlord letter nob strider underhill
11. “A Knife in the Dark”
    beren bob bree dell ferny galad gil strider tinúviel weathertop
12. “Flight to the Ford”
    bone boot ford frodo glorfindel horse strider troll trolls weathertop

Book II

aragorn boats boromir elrond frodo gandalf gimli haldir legolas sam

1. “Many Meetings”
   bilbo dúnadan elrond flood frodo glorfindel glóin mortals rivendell strider
2. “The Council of Elrond”
   boromir elrond erestor galad gil glóin isildur radagast ring saruman
3. “The Ring Goes South”
   aragorn bill boromir caradhras drift elrond frodo gandalf redhorn snow
4. “A Journey in the Dark”
   bill boromir doors durin gandalf gimli howls mines moria wolves
5. “The Bridge of Khazad-dum”
   balin balrog beats boromir chamber doom door drum hall mazarbul
6. “Lothlórien”
   flet frodo galadhrim gimli haldir ladder legolas lothlórien nimrodel silverlode
7. “The Mirror of Galadriel”
   basin celeborn frodo galadhrim galadriel haldir lady magic mirror pedestal
8. “Farewell to Lórien”
   boat boats boromir cakes celeborn galadhrim galadriel gimli lady lórien
9. “The Great River”
   aragorn boat boats frodo gebir log paddle paddles rapids sarn
10. “The Breaking of the Fellowship”
    amon aragorn boromir brandir frodo hen lawn lhaw sam tol

Book III

aragorn ents gimli isengard legolas saruman théoden treebeard uglúk éomer

1. “The Departure of Boromir”
   aragorn boat boromir galen gimli gulls legolas orcs rauros tire
2. “The Riders of Rohan”
   aragorn downs entwash fangorn gimli horses legolas orcs trail éomer
3. “The Uruk-Hai”
   ankles grishnákh isengarders knoll lugbúrz merry orcs pippin uglúk wrists
4. “Treebeard”
   ent entish ents entwives hasty hoom isengard rowan treebeard trees
5. “The White Rider”
   aragorn fangorn gandalf gimli haft legolas man old saruman treebeard
6. “The King of the Golden Hall”
   aragorn gandalf gríma hall háma neighed théoden wormtongue éomer éowyn
7. “Helm’s Deep”
   aragorn coomb deeping dike erkenbrand gamling helm hornburg westfold éomer
8. “The Road to Isengard”
   caverns caves fords gandalf gimli isen isengard legolas saruman théoden
9. “Flotsam and Jetsam”
   barrels ent ents gimli huorns isengard pipe quickbeam saruman treebeard
10. “The Voice of Saruman”
    dismissed ents gandalf injuries orthanc rail saruman spell théoden treebeard
11. “The Palantír”
    ball bracken gandalf globe orthanc palantíri pippin saruman shadowfax wizard

Book IV

faramir frodo gollum nice precious rope sam shagrat shelob sméagol

1. “The Taming of Sméagol”
   cliff fix frodo gollum knot precious rope sam sméagol yess
2. “The Passage of the Marshes”
   fish frodo gollum gully marshes mires nice precious sam sméagol
3. “The Black Gate is Closed”
   frodo gollum helps lithui master nice oliphaunt oliphaunts sam sméagol
4. “Of Herbs and Stewed Rabbit”
   coney fern gollum herbs mablung pans rabbits sam sméagol taters
5. “The Window on the West”
   arts blindfold boromir damrod faramir frodo isildur mablung men mithrandir
6. “The Forbidden Pool”
   anborn faramir fish fissh frodo gollum precious protection sly sméagol
7. “Journey to the Cross-roads”
   brewing decent faramir frodo glare gollum mould sam staves thorns
8. “The Stairs of Cirith Ungol”
   dad frodo gollum morgul sam sméagol sneaking stairway wink wraith
9. “Shelob”
   frodo glass gollum lair phial sam shelob sméagol stench tunnel
10. “The Choices of Master Samwise”
    cords frodo gorbag lads ladyship lugbúrz sam shagrat shelob tunnel

Book V

aragorn beregond city denethor faramir ghân men pippin éomer éowyn

1. “Minas Tirith”
   beregond bergil citadel city denethor forlong ingold mithrandir pippin street
2. “The Passing of the Grey Company”
   aragorn burg elladan erech gimli halbarad king legolas ride théoden
3. “The Muster of Rohan”
   dwimorberg edoras harrowdale hirgon king merry ride théoden éomer éowyn
4. “The Siege of Gondor”
   beregond citadel city denethor faramir gandalf garrison pelennor pippin retreat
5. “The Ride of the Rohirrim”
   buri dernhelm elfhelm ghân men stonewain théoden wild éomer éored
6. “The Battle of the Pelennor Fields”
   dernhelm harlond king knights mûmakil snowmane southrons théoden éomer éowyn
7. “The Pyre of Denethor”
   beregond bier burn city denethor faramir gandalf healing oil thou
8. “The Houses of Healing”
   aragorn athelas bergil faramir healing imrahil ioreth sick éomer éowyn
9. “The Last Debate”
   aragorn gimli gulls imrahil lamedon lebennin legolas pelargir ships thousands
10. “The Black Gate Opens”
    army captains heralds imrahil lieutenant messenger morannon sauron terms thou

Book VI

cotton faramir frodo lotho rosie ruffians sam shagrat sharkey éowyn

1. “The Tower of Cirith Ungol”
   court frodo gorbag ladder orc sam shagrat shelob snaga turret
2. “The Land of Shadow”
   frodo isenmouthe morgai orc range sam shagrat spur tracker whip
3. “Mount Doom”
   cone frodo gaping gollum load masster mountain sam sammath tearing
4. “The Field of Cormallen”
   captains eagles frodo gwaihir isle knights merriment praise pure sam
5. “The Steward and the King”
   aragorn city faramir healer healing ioreth king lady warden éowyn
6. “Many Partings”
   arwen beggar celeborn hoom orthanc théoden treebeard wain éomer éowyn
7. “Homeward Bound”
   barliman bill bree butterbur deadmen ferny gandalf nob pony troubles
8. “The Scouring of the Shire”
   cotton farmer hob lotho ruffians sharkey sheds shirriff shirriffs village
9. “The Grey Havens”
   bywater cotton deputy elanor frodo married mayor rosie ruffians sam

Appendices

• A. “Annals of the kings and rulers”
  arthedain arvedui eldacar eärnil gondor king náin tar thráin witch
• B. “The Tale of Years”
  begins birth guldur king meets reaches samwise sauron sets thranduil
• C. “Family Trees”
  addition appendix birth blanco family guests names party recorded recounted
• D. “Shire Calendar”
  calendar corresponded eldar lithe months names quenya reckoning week yule
• E. “Writing and spelling”
  english languages letters pronounced quenya represented represents series sindarin vowel
• F. “The languages and peoples of the Third Age”
  english languages letters pronounced quenya represented represents series sindarin vowel

Resulting Tags Cloud

About SCAN

“… the abundance of information will be such that either you have reached such a level of maturity that you are able to be your own filter, or you will desperately need a filter… some professional filter.”
Umberto Eco: A Conversation on Information
(an interview by Patrick Coppock, February, 1995)

SCAN is aiming for a solution of major problems of content organization and findability in information overload era.

SCAN aggregates content from different sources into a single documents collection. This repository may keep records on thousands of documents independently of their original locations and formats. Every document record contains a number of metadata properties (such as title, description, author, creation date, etc) which can be either set automatically or edited manually.

Adding documents to the repository is an automated operation. A user only need to point SCAN to a location and the application will find and add every document from there. Added document locations will be monitored for changes (new, modified or deleted documents) to keep the repository up-to-date.

The documents content is indexed for search and text analysis. You can search the documents either by simple text queries, or by using special forms to make complex queries for searching on document text and properties. The queries can be saved for repeatable use.

The documents collection is structured with a system of tags, similar to the services like del.icio.us or Flickr. Tags are keywords or labels attached to the items to identify them for quick navigation and finding. All tags together form a taxonomy representing the semantics of the documents collection. The taxonomy can be viewed as a “tags cloud” for navigating through the documents repository.

SCAN text analysis mechanism simplifies the process of tagging. It analyzes a document content and suggests the most relevant words as to-be tags. It makes manual tagging as simple as selecting the tags from the proposed candidates. It also can undertake the whole manual process of tagging, either by automated assigning the tags to the documents, or by finding the documents, relevant to a specific tag. Another text analysis application is searching the documents similar to a specific one (search by pattern).

SCAN is a component-based software using a number of plugins for specific features. The basic SCAN platform can be easily extended with plugins for new document formats, document locations (RSS feeds, web-sites, e-mail, etc) and language analyzers. Whole new areas of functionality can be added with user interface extensions. An example of such extensions is the plugin to browse the repository with a calendar (grouping the documents by their creation dates).

SCAN is a Java application, so it works on any Java-enabled platform. SCAN is a free open source software, distributed under Apache License, Version 2.0

See also:

• List of current features
• Download SCAN
• How to obtain SCAN sources from SVN repository
• User’s Manual

The tags are a handy intuitive concept for labeling and describing the pieces of information. They are usually assigned informally, and as for any matter of “a common sense”, it is difficult to find their formal definition. It is even more difficult to reproduce and algorithmize the mental processes of tag choosing.

To avoid that fruitless work, I will consider some IR approaches which may be used for evaluation the document terms as the possible tag candidates. For to make that evaluation possible, two important purposes of the tags may be identified:

• Differentiation
  To name a thing means to separate it from others. So, a good tag should identify a document by showing its unique feature – how the document differs from other documents in a corpus.
• Unification
  A tag should also identify a semantic group to which the document belongs.

These two key properties seem totally different and even mutually opposite and discordant. Nevertheless, they are both important and the best tags perhaps integrates both properties in a dialectical way. Brief analysis of human-assigned tags on popular social web-services confirmed that assumption.

Traditional keyword extraction: TFIDF

The first strategy of autotagging is based on a popular term weighting technique, proposed by classical IR discipline. This technique is called shortly as TFIDF and this abbreviation reflects the basic formula for calculating a measure of relevance (or, “weight”) of a specific term in a given document: Term Frequency x Inverted Document Frequency.
Though there is a number of variations of this formula (differ in various normalization techniques), the basic idea behind them is the same: weight of a term in a document is directly proportional to the number of times the term appears in the document (Term Frequency, TF) and inversely proportional to the total number of documents containing this term (Document Frequency, DF).

This weighting scheme allows to effectively filter out so-called “common words” (with high DF) as well as the rare terms and misspellings (with low TF and DF both). The terms with high TF and relatively low DF got highest “importance” ranks and can be considered as possible candidates to tags.

Thus, TFIDF-based document tagging strategy may be described as follows:

1. Build a list of all unique terms of a given document (term vector)
2. Calculate TFIDF weight for each term in the list
3. Sort the terms by their TFIDF weights
4. Select N topmost terms, where N is a number of tags we want to assign for each document. We can also choose another method of tags number limitation, e.g. to select the terms above specified TFIDF threshold.

Improved autotagging strategies

The problem with reviewed simple TFIDF-based analysis is that it conflicts with the second part of our definition of the tags purpose: unification. Reviewed approach is good when we need to know what terms make a document different from others, but it doesn’t help if we need to select the terms which connect the document with its counterparts.

With this technique, the “tags cloud” tends to grow proportionally to the number of indexed documents with minimization of the number of documents per single tag at the same time. In an ideal case, it leads to the situation when each tag corresponds to an unique document. Such level of granularity is not what we want if we are going to control our taxonomy size and care about grouping function of our tags. The tags which would contribute into documents unification and grouping are overlooked by that algorithm because their DF’s are higher than of those contributing into a document uniqueness. We could smooth that effect by reduction an influence of DF parameter, but it increases the risk of appearing the common words as the document tags. Obviously, tweaking the basic TFIDF formula is not a solution for this problem.

To implement better autotagging strategy (which would consider a documents grouping factor), we need to analyze a document together with its semantic context – that is a cluster of documents including an initial document plus the documents similar to it. TFIDF ranking algorithm applied to that cluster gives us the terms relevant to the whole group, while the common words are still effectively suppressed. There is a high probability that the documents inside the cluster are mutually similar to each other, thus this strategy tends to re-use the same tag set for semantically related documents, as possible.

An easy way to find the document similarities is to build the document term vector (as we did before) and to issue it (or its highest ranked part) to a search system. Some search engines implement “more-like-this” feature that works the same way. The better results may be achieved if the search system allows to set a “boost factor” for separate query terms, proportionally to their weights in the term vector. Then, we just select the search results (above specific threshold) as the documents similar to the given one.

The whole strategy may looks like:

1. Find a set of documents, similar to a given document.
2. Build a composite term vector as an union of term vectors of the given document and its similarities.
3. Calculate TFIDF term weights against the composite term vector.
4. Sort the terms by their TFIDF weights
5. Select topmost terms as the document tags

By tweaking the similarity thresholds and TFIDF normalizing (e.g. considering similarity coefficients in TFIDF calculation), we can adjust the results to get “more specific” or “more general” set of the tags (differentiation/unification balance), depending on our goals.

Another improved autotagging strategy may be implemented as an adaptive technique where TFIDF evaluation takes into account whether a given term is already a tag or not. But this approach strictly depends on an initial tag set and likely, will require some manual tagging before it could be used in auto mode.

P.S. If you found this article useful, please consider to participate in this survey. It will help the author in his further work on this subject. Thanks!