Traumwind - more VectorSpace thoughts

here's some timing info for my current setup:


[Thu Apr 24 22:49:50 2003] vs.pl: reading archive from ../html/blog/archive

[Thu Apr 24 22:49:50 2003] vs.pl: found 1257 postings

[Thu Apr 24 22:49:50 2003] vs.pl: setting up VectorSpace

[Thu Apr 24 22:49:50 2003] vs.pl: building Index

[Thu Apr 24 22:49:50 2003] vs.pl: Making word list:

[Thu Apr 24 22:50:01 2003] vs.pl: Finished with word list

[Thu Apr 24 22:50:01 2003] vs.pl: doing queries

[Thu Apr 24 22:50:01 2003] vs.pl: making cosines for doc 0, 
2003-04-24_22-33

[Thu Apr 24 22:50:02 2003] vs.pl: writing ../html/blog/archive/sims/2003-04-24_22-33.sim

[Thu Apr 24 22:50:02 2003] vs.pl:

[Thu Apr 24 22:50:02 2003] vs.pl: making cosines for doc 1, 2003-04-24_22-29

[Thu Apr 24 22:50:03 2003] vs.pl: writing ../html/blog/archive/sims/2003-04-24_22-29.sim

[Thu Apr 24 22:50:03 2003] vs.pl:

[Thu Apr 24 22:50:03 2003] vs.pl: making cosines for doc 2, 2003-04-24_22-26

[Thu Apr 24 22:50:05 2003] vs.pl: writing ../html/blog/archive/sims/2003-04-24_22-26.sim

[Thu Apr 24 22:50:05 2003] vs.pl:

[Thu Apr 24 22:50:05 2003] vs.pl: making cosines for doc 3, 2003-04-24_15-53

[Thu Apr 24 22:50:06 2003] vs.pl: writing ../html/blog/archive/sims/2003-04-24_15-53.sim

[Thu Apr 24 22:50:06 2003] vs.pl:

[Thu Apr 24 22:50:06 2003] vs.pl: making cosines for doc 4, 2003-04-24_10-41

so the first optimisation could be to go semi-incremental. No need to re-do all 1200+ entries in each run, as most of them don't really change.
next step surly would be to pre-generate the word-index phase, which could also be used for the keyword search.
each actual 'query' takes roughly a second to preform, that is rather fast if you consider you are comparing one doc to 1200+ others

And no, keeping the whole VectorSpace in-memory between user-queries is not really an option. The Perl VM size for this is about 125MB on my machine, which is ok for running it every so often, but not for running it contiously.

Another gripe is the fact that it's difficult to incrementally add new words (or docs containing new words) to a 'pre-generated' space.

[ by Martin>] [permalink] [similar entries]

similar entries (vs):

ContextGraph similar entries (# 12%)
I must say, those similar entries (# 11%)
tomorrow (# 9%)
lcom Yahoo vs. Google (# 9%)

similar entries (cg):

perl.com: Installing mod_perl without superuser privileges [Apr. 10, 2002] (# 10%)

relevant words

Big things to come (TM) 30th Dez 2002

What wouldn't you do?
Oblique Strategies, Ed.3 Brian Eno and Peter Schmidt

amazon.de Wunschliste

usefull links:
Google Graph browser
Traumwind 6-Colormatch
UAV News

apr (3.6)
thu (3.6)
vs (3.1)
doc (2.5)
cosines (2.3)
making (2.2)
vectorspace (1.7)
writing (1.5)
html (1.5)
word (1.5)
pre (1.3)
for (1.1)
running (1.1)
each (1.1)
list (1)
words (1)
to (0.9)
is (0.9)
contiously (0.9)
preform (0.9)
incrementally (0.9)
semi (0.9)
optimisation (0.9)
gripe (0.9)
queries (0.9)
vm (0.9)
archive (0.9)
timing (0.9)
surly (0.9)
no (0.8)
could (0.8)
really (0.8)
which (0.8)
new (0.8)
be (0.8)
the (0.8)
consider (0.8)
docs (0.8)
phase (0.8)
actual (0.8)
finished (0.8)
option (0.8)
roughly (0.8)
difficult (0.8)
postings (0.7)
comparing (0.7)
setting (0.7)
space (0.7)
containing (0.7)
query (0.7)
mb (0.7)
keeping (0.7)
so (0.7)
it (0.7)
size (0.7)
current (0.7)
not (0.7)
others (0.7)
keyword (0.7)
setup (0.7)
fact (0.7)
index (0.6)
between (0.6)
you (0.6)
another (0.6)
second (0.6)
whole (0.6)
step (0.6)
building (0.6)
machine (0.6)
fast (0.6)
my (0.6)
entries (0.6)
change (0.6)
every (0.6)
user (0.6)
thoughts (0.6)
info (0.6)
add (0.6)
takes (0.6)
often (0.6)
re (0.6)
reading (0.6)
next (0.6)
run (0.6)
rather (0.6)
ok (0.6)
that (0.6)
used (0.6)
found (0.6)
doing (0.6)
perl (0.6)
most (0.5)
in (0.5)
go (0.5)
first (0.5)
search (0.5)
a (0.5)
them (0.5)
up (0.5)
would (0.5)
need (0.5)
also (0.5)
here (0.5)
don (0.4)
more (0.4)
an (0.4)
about (0.4)
if (0.4)
are (0.4)
all (0.4)
as (0.4)
or (0.4)
from (0.4)
some (0.4)
one (0.4)
but (0.4)
with (0.3)
on (0.3)
this (0.3)
of (0.3)
and (0.3)
martin (0.2)
(0)