“How does SCAN help me?”, “Why should I use it?”, “Who are the users?”, “Why it is smart?”, “Can it replace a …?”, “What is autotagging?”, technical tips-n-tricks, development questions and a lot of other things you would want to know about Smart Content Aggregation & Navigation technology.

Read SCAN FAQ »

An idea is borrowed from e-mail clients, where the similar feature exists for decades as the user-defined filters for processing the messages. This is how it looks in KMail:

In general, a filter checks if a document matches to specific criteria (a rule) and does some action if yes. For instance, if a condition “text contains ‘viagra’ or ‘cialis’” is true, then some action (“move to spam” or “send assassins to the author”) would be executed. What is especially good is that it’s old, popular and intuitive user experience.

In a content aggregator like SCAN, this concept may allow a user to define custom automation rules to set document metadata properties. For instance,

IF (url starts with "http://cyberborean.wordpress.com") SET author = "me"

Another using I have in my mind is to empower an “artificial intelligence” of autotagging with a human intelligence, by user-defined tagging rules:

IF (text contains "latte") ADD TAG "coffee"

I only doubt about terminology — “filter” might be confusing, as it is already used in SCAN vocabulary (URI filters to include/exclude some documents by their URI pattern). Any ideas?

(Read more on SCAN website)

The plugin purpose is to crawl the specified local email folders and aggregate the email messages as the documents in SCAN repository. It also introduces new document type “message/rfc822” for emails and uses the email message headers for setting document metadata. Attached files are extracted from messages and processed as the separate documents with appropriate parsers depending on its content type.

As there are no common convention on how to identify a separate message for opening it with an external application, mail plugin implements its own message viewer UI to open the messages by default. The messages are identified with “mid:” URI scheme which is a standard but unfortunately, seems to not supported by known MUA’s so far. However, it is implemented in the hope of that the future MUA’s will support this standard scheme to open the specific messages with their command line (something like “thunderbird mid:message-id“).

Mail plugin uses GNU JavaMail implementation because it includes JavaMail providers for local mail stores – mboxes and maildirs. In addition, Outlook Express mail is also supported via the separate JavaMail provider made by jmbox project. Thus, the mail plugin supports a wide range of popular MUA’s, including mbox-based (Mozilla family), maildir-based, mixed mbox/maildir (KMail and Evolution) and Outlook Express.

The plugin has been tested on Linux with Mozilla Thunderbird 1.5 and KMail 1.9.6 (in maildir mode) and Windows XP with Mozilla Thunderbird 1.5 and MS Outlook Express 5.0.

Problems?

Yep, they are. The issues with mail plugin go both from GNU JavaMail limitations and from the side of the concrete MUA’s which seem to be easy with interpreting mbox/maildir principles for their convenience. For instance, Thunderbird uses the non-standard “.sbd” directories to keep the mail subfolders, which the mbox provider is unaware of. So, recursive crawling the Thunderbird mail folders does not work. The only solution for those problems is to develop the MUA-specific JavaMail providers which would know how to deal with a concrete mbox or maildir implementation with all its peculiarities.

Outlook Express provider, as an example of such MUA-specific implementation works rather good, however we noticed a bug with localized MSOE versions where the translated folder names (“Inbox”, “Sent”, “Trash” etc) with non-latin letters had not been processed.

The reports on testing the mail plugin with other MUA’s/platforms are more than welcome.

About SCAN

“… the abundance of information will be such that either you have reached such a level of maturity that you are able to be your own filter, or you will desperately need a filter… some professional filter.”
Umberto Eco: A Conversation on Information
(an interview by Patrick Coppock, February, 1995)

SCAN is aiming for a solution of major problems of content organization and findability in information overload era.

SCAN aggregates content from different sources into a single documents collection. This repository may keep records on thousands of documents independently of their original locations and formats. Every document record contains a number of metadata properties (such as title, description, author, creation date, etc) which can be either set automatically or edited manually.

Adding documents to the repository is an automated operation. A user only need to point SCAN to a location and the application will find and add every document from there. Added document locations will be monitored for changes (new, modified or deleted documents) to keep the repository up-to-date.

The documents content is indexed for search and text analysis. You can search the documents either by simple text queries, or by using special forms to make complex queries for searching on document text and properties. The queries can be saved for repeatable use.

The documents collection is structured with a system of tags, similar to the services like del.icio.us or Flickr. Tags are keywords or labels attached to the items to identify them for quick navigation and finding. All tags together form a taxonomy representing the semantics of the documents collection. The taxonomy can be viewed as a “tags cloud” for navigating through the documents repository.

SCAN text analysis mechanism simplifies the process of tagging. It analyzes a document content and suggests the most relevant words as to-be tags. It makes manual tagging as simple as selecting the tags from the proposed candidates. It also can undertake the whole manual process of tagging, either by automated assigning the tags to the documents, or by finding the documents, relevant to a specific tag. Another text analysis application is searching the documents similar to a specific one (search by pattern).

SCAN is a component-based software using a number of plugins for specific features. The basic SCAN platform can be easily extended with plugins for new document formats, document locations (RSS feeds, web-sites, e-mail, etc) and language analyzers. Whole new areas of functionality can be added with user interface extensions. An example of such extensions is the plugin to browse the repository with a calendar (grouping the documents by their creation dates).

SCAN is a Java application, so it works on any Java-enabled platform. SCAN is a free open source software, distributed under Apache License, Version 2.0

See also:

• List of current features
• Download SCAN
• How to obtain SCAN sources from SVN repository
• User’s Manual

1. When you are looking for a document on your machine, the following navigation hints are important (specify an order, e.g. “a, d, c, b”):

a. File name
b. File name and folder
c. Date of creation/modification
d. Document metadata properties
e. Full-text search results

2. Do you use document titles (or another semantic data) as the file names?

a. Yes, always
b. For important documents only
c. Don’t care about the file names

3. Do you keep a sort of semantic folder structure for storing your documents?

a. Yes, and I keep all my documents in a single folders hierarchy, organized semantically
b. Yes, but use the structure for important documents only
c. Use random folder structures, depending on a context of my work
d. Don’t care about where I’m saving my files

4. Do you use document shortcuts (Windows) or symbolic links (*nix) for improving navigation?

a. Yes, often
b. Rarely
c. No

5. How often do you use the system full-text search for finding a specific document?

a. This is my everyday way of finding the documents
b. Only when I need to find the document quickly
c. Only when I gave up to find the document by other ways
d. Only when I’m not sure the document on this topic exists
e. Never use the search for this purpose

6. Do you use “advanced search” capabilities?

a. Yes, often
b. Rarely
c. No, basic search is enough

7. Do you use the system “Recent documents” list?

a. Yes, often
b. Rarely
c. No

8. Do you enter the metadata (title, author, subject, keywords etc) in the “Document properties” dialog box of your text editor?

a. Yes, always
b. For important documents only
c. Don’t care about it

9. How do you mark an importance of a document?

a. Place it into a special folder
b. Place it on the desktop
c. Bookmark it (place into the “Favorites”)
d. Do nothing

10. How many documents (in percentage against all documents on your machine) are used actively?
11. How do you handle outdated documents?

a. Keep them in place
b. Move to a special folder
c. Move to backup media, then delete
d. Delete them

12. Which formats you use for the text documents (specify in order of importance, e.g. “a, d, c, b”):

a. MS Word
b. OpenDocument
c. PDF
d. HTML
e. XML or SGML (DocBook etc)
f. Plain text
g. Other

13. Do you save online documents on your local hard drive?

a. Yes, often
b. Save important or very large documents only
c. Never

14. Do you keep a sort of a personal electronic library?

a. Yes
b. No

15. Do you use specialized software for photo albums or multimedia collections management?

a. Yes
b. No, standard system tools are enough

16. How do you estimate efforts on supporting your local document collections?

a. It is a burden, it takes a lot of my time and harms for my work
b. It takes some time but it worths it
c. It is not a problem with help of the modern desktops
d. Do not see any problem

17. Do you use del.icio.us?

a. Yes, for every interesting stuff I meet on the Web
b. Yes, for important links only
c. Yes, for links I want to share with somebody else
d. No

18. How many tags are in your del.icio.us profile?
19. How many del.icio.us tags you usually assign per a single link (in average)?
20. When choosing the del.icio.us tags, you prefer

a. Your own tags
b. Other’s tags, suggested by the service

21. When choosing the del.icio.us tags, you prefer to

a. Reuse the existing tags, as possible
b. Create new tags

22. Do you try to avoid tags synonymity?

a. Yes
b. No, I don’t care about synonyms

23. Do you use tag bundles?

a. Yes
b. No

24. Which factors are important for tags selection (specify in order of importance, e.g. “a, d, c, b”)?

a. My own subjective associations
b. My vision of of the implicit topic semantics (tend to be objective)
c. Explicit textual properties of the document (terms frequency, etc)
d. Tags, assigned by other people

25. The best synonym of the”tag” is:

a. Category
b. Term
c. Topic
d. Keyword
e. Label

26. The purpose of the tags is:

a. Distinction
b. Unification

27. A number of tags per document is a measure of:

a. Document importance
b. Information diversity
c. Collection size
d. Selection quality

28. Would automatical tags generation be useful?

a. Yes, and it could completely replace human brains in this area
b. Yes, but it matters as a help for human brains only
c. No, the tags should belong to humans

Thank you!