1. Test the replication on the server side, For that we have yalaforge => test.
2004-04-08 09934, 2004
ruaok
Once we're happy with that, we roll out replication on zim.
2004-04-08 09935, 2004
ruaok
then
2004-04-08 09949, 2004
ruaok
2. zim => delft, zim => grunt
2004-04-08 09922, 2004
ruaok
with the understanding that the replication packets may get invaladidated at any point during the test.
2004-04-08 09936, 2004
djce
ok, fair enough
2004-04-08 09941, 2004
djce
how close is the code right now?
2004-04-08 09915, 2004
ruaok
the core code that manages the internal replication tables and the code that creates the database dumps is in pretty good shape.
2004-04-08 09934, 2004
djce
Is the make-dump code tied in to the Export code?
2004-04-08 09949, 2004
ruaok
on the replicant side, the code that takes a dumpfile and imports it is also in good shape.
2004-04-08 09951, 2004
djce
so that the exports can tie in with replication packet boundaries?
2004-04-08 09910, 2004
ruaok
the code that sticks the file on the ftp site/grabs it from the ftp side is largely non-existant.
2004-04-08 09927, 2004
ruaok
no, that has not been addressed.
2004-04-08 09928, 2004
djce
well, that sounds trivial. no matter.
2004-04-08 09932, 2004
djce
ok
2004-04-08 09947, 2004
ruaok
However, each of these scripts could use your careful eye to spot problems.
2004-04-08 09919, 2004
ruaok
I plan to go in and look at the replication stuff tonight and make sure its up to snuff.
2004-04-08 09945, 2004
ruaok
verify all my assumptions and write an XML dump script that yalaforge can use to populate the lucene search index.
2004-04-08 09914, 2004
djce
no you lost me there - how does this tie in with lucene?
2004-04-08 09933, 2004
ruaok
after that, if you could examine the server replication portions and give it a your seal of approval for the next server release, that would be awesome.
2004-04-08 09905, 2004
ruaok
yalaforge and I have been thinking about how to update the lucene index from the DB.
2004-04-08 09939, 2004
ruaok
The requirements are to fit in with our replication scheme and hopefully be indenpendent enough so that one could create a lucene index server, without actually requireing a postgres install.
2004-04-08 09953, 2004
ruaok
The latter requirement makes for an excellently scalable system.
2004-04-08 09915, 2004
ruaok
We could add more index servers as we need them. Or as the album tagger userbase grows...
2004-04-08 09928, 2004
ruaok
So, to update the lucene index while people are using the index is tricky.
2004-04-08 09951, 2004
ruaok
Thus yalaforge has come up with a shadow index scheme. We keep two indexed around at any one time.
2004-04-08 09907, 2004
ruaok
One to serve queries and one that is being updated behind the scenes.
2004-04-08 09922, 2004
djce
interesting...
2004-04-08 09935, 2004
ruaok
then when an update has been applied to the shadow index, the two indexes rotate.
2004-04-08 09901, 2004
ruaok
The update packets that need to be applied to the lucene index, are derived from the Replication data we already collect on the server.
2004-04-08 09921, 2004
ruaok
so during the replication phase on the server, we will create a DB dump file for the table and an XML lucence update packet.
2004-04-08 09952, 2004
ruaok
any lucene index grabs that from the FTP site and updates its shadow index and then switches the indexes out.
2004-04-08 09957, 2004
ruaok
Does that make sense?
2004-04-08 09919, 2004
ruaok
It seems overly complicated at first, but the amazing flexiblity we derive from it will make us glad later.
2004-04-08 09914, 2004
djce
sounds interesting. That all sounds good, if lucene makes us do that.
2004-04-08 09955, 2004
ruaok
to avoid various IPC issues that yalaforge and I have discussed, this seems like the best approach.
2004-04-08 09912, 2004
ruaok
In the end the web site will do searches via making corba queries to the index server.
2004-04-08 09940, 2004
ruaok
since the index server runs on java and we run on perl, yalaforge decided that corba is the fastest way to couple the two.
2004-04-08 09954, 2004
ruaok
are we hitting acronym overload yet? :-)
2004-04-08 09946, 2004
djce
How much of this is theory so far?
2004-04-08 09908, 2004
ruaok
the lucene index server is getting there.
2004-04-08 09920, 2004
ruaok
I've seen a test that builds indexes and accepts queries at the same time.
2004-04-08 09926, 2004
ruaok
the replication stuff is theory right now.
2004-04-08 09948, 2004
ruaok
yalaforge has created a DTD/xml example for the lucene replication stuff.
2004-04-08 09901, 2004
ruaok
We had agreed to have this stuff finished in about 7 more weeks time.
2004-04-08 09919, 2004
ruaok
yalaforge has another exam next week, so he won't be doing much.
2004-04-08 09935, 2004
ruaok
But after that we will work on it more -- and so far he is making great progress.
2004-04-08 09937, 2004
djce
fair enough. good luck yalaforge :-)
2004-04-08 09950, 2004
ruaok
I have faith -- he is a very smart cookie.
2004-04-08 09954, 2004
djce
great. You guys have been busy
2004-04-08 09900, 2004
ruaok
indeed.
2004-04-08 09907, 2004
ruaok
have you looked at the tagger?
2004-04-08 09907, 2004
WhiteDog
cannot convert parameter 2 from 'int' to 'int *'
2004-04-08 09910, 2004
WhiteDog
any help? :D
2004-04-08 09928, 2004
ruaok
you need to pass in a pointer to an int, not an int
2004-04-08 09932, 2004
ruaok
int foo;
2004-04-08 09937, 2004
ruaok
NOT: function(foo)
2004-04-08 09943, 2004
ruaok
BUT: function($foo)
2004-04-08 09944, 2004
ruaok
er
2004-04-08 09948, 2004
ruaok
BUT: function(&foo)
2004-04-08 09956, 2004
ruaok
(the $ was a typo, sorry)
2004-04-08 09914, 2004
WhiteDog
oh :P
2004-04-08 09918, 2004
WhiteDog
ok :)
2004-04-08 09917, 2004
djce
ruaok: no luck yet. I haven't worked out how to install the py-mb stuff
2004-04-08 09902, 2004
ruaok
easy.
2004-04-08 09912, 2004
ruaok
download the bindings
2004-04-08 09922, 2004
ruaok
then sudo python setup.py install
2004-04-08 09937, 2004
ruaok
and that installs the libmb python bindings
2004-04-08 09956, 2004
djce
ImportError: No module named distutils.core
2004-04-08 09912, 2004
ruaok
you need to install python2.3-dev
2004-04-08 09920, 2004
djce
ok
2004-04-08 09942, 2004
djce
thought it might be that... that was my next guess
2004-04-08 09914, 2004
djce
ImportError: No module named ctypes
2004-04-08 09959, 2004
ruaok
ohh, you got icepick's new implementation of the bindings.
2004-04-08 09917, 2004
ruaok
either back off one version, or install the ctypes python lib.
