-
ruaok
ok, I'll go investigate it.
2010-11-15 31914, 2010
-
ocharles
at least the basics of that role
2010-11-15 31917, 2010
-
murdos
ocharles: ah, ok...
2010-11-15 31919, 2010
-
ruaok
ocharles: ok, can you make that change available to me?
2010-11-15 31930, 2010
-
ruaok
push a branch?
2010-11-15 31930, 2010
-
ocharles
ruaok: I think it might have been part of my Fey rewrite stuff, so I dunno if it's much use
2010-11-15 31937, 2010
-
ocharles
I'll look into it
2010-11-15 31940, 2010
-
ruaok
ok.
2010-11-15 31956, 2010
-
ruaok
if it doesn't impact too much stuff, then I'll make that change and fix the iso_ change.
2010-11-15 31909, 2010
-
ocharles
And do a new data dump?
2010-11-15 31910, 2010
-
ruaok
then its FINITO. BASTA. SCHLUSS.
2010-11-15 31913, 2010
-
ruaok
yes.
2010-11-15 31915, 2010
-
ruaok
and reload test.
2010-11-15 31918, 2010
-
ocharles
ok, I'll do the isco change tomorrow
2010-11-15 31931, 2010
-
ruaok
ocharles: worry about your bug list.
2010-11-15 31936, 2010
-
ocharles
ok
2010-11-15 31937, 2010
-
ruaok
I'll just take care of that one change myself.
2010-11-15 31943, 2010
-
ocharles
sure, that makes sense
2010-11-15 31947, 2010
-
ruaok
in the interest of keeping the branches simple.
2010-11-15 31951, 2010
-
ocharles nods
2010-11-15 31955, 2010
-
ruaok
its got a little fusterclicky last time.
2010-11-15 31900, 2010
-
ruaok
cliky?
2010-11-15 31901, 2010
-
ocharles
I cleared all my branches for that now that it's merged
2010-11-15 31905, 2010
-
ruaok
ok.
2010-11-15 31910, 2010
-
ruaok
moving on.
2010-11-15 31914, 2010
-
ruaok
the test server....
2010-11-15 31920, 2010
-
ruaok
one drive in the test server has failed.
2010-11-15 31925, 2010
-
warp noticed that.
2010-11-15 31931, 2010
-
ruaok
DO NOT STORE ANYTHING IMPORTANT ON THE TEST SERVER.
2010-11-15 31941, 2010
-
ruaok apologizes for shouting
2010-11-15 31950, 2010
-
djce feels it's an important point,
2010-11-15 31951, 2010
-
ruaok
I am NOT (!) going to fix that drive.
2010-11-15 31954, 2010
-
djce
even when the drives work.
2010-11-15 31900, 2010
-
ruaok
djce: indeed!
2010-11-15 31915, 2010
-
ruaok
test should always be considered dispensable at any point in time.
2010-11-15 31923, 2010
-
ruaok
test is also an older server that is sucking a lot of power.
2010-11-15 31940, 2010
-
ruaok
and power is limited in mountain view (I'm guessing google gets all the power up there)
2010-11-15 31949, 2010
-
ruaok
so, we've been asked to decommision this server.
2010-11-15 31902, 2010
-
djce
timeline?
2010-11-15 31917, 2010
-
ruaok
we're either: purchasing a mac mini to run test, or purchasing a slice of a larger VM server.
2010-11-15 31924, 2010
-
ruaok
djce: unclear at this point in time.
2010-11-15 31932, 2010
-
ruaok
hopefully that will shake out soon.
2010-11-15 31942, 2010
-
ruaok
the mini option is $1k, the VM maybe $500.
2010-11-15 31944, 2010
-
ocharles
I think test is so low traffic a slice probably makes the most sense
2010-11-15 31946, 2010
-
warp
ruaok: if you intend to move to the cloud eventually, you might as well start with test.
2010-11-15 31951, 2010
-
ruaok
I'm not jazzed about either.
2010-11-15 31957, 2010
-
ruaok
ocharles: agreed.
2010-11-15 31901, 2010
-
ruaok
warp: its not cost effective.
2010-11-15 31906, 2010
-
warp
ruaok: agreed.
2010-11-15 31919, 2010
-
ruaok
both of the others are one time costs.
2010-11-15 31931, 2010
-
ruaok
whereas test is a recurring cost for a VERY low utlization server.
2010-11-15 31948, 2010
-
ruaok
I personally like the mini solution, but I dont very much like the cost of it.
2010-11-15 31907, 2010
-
ocharles
You could build the equiv of a mini for much less
2010-11-15 31912, 2010
-
ruaok
regardless, test may be unavailable for some period of time. I'm working to have that be as little as possible.
2010-11-15 31922, 2010
-
ruaok
ocharles: its about space AND power.
2010-11-15 31931, 2010
-
ocharles
and mini's are that efficient?
2010-11-15 31933, 2010
-
ruaok
I can't build something as small as a mini.
2010-11-15 31936, 2010
-
ruaok
ocharles: yes.
2010-11-15 31941, 2010
-
ocharles
interesting
2010-11-15 31947, 2010
-
ruaok
remember the setup of minis in the MB rack above MB?
2010-11-15 31956, 2010
-
ocharles
ya, I guess it has reason :)
2010-11-15 31959, 2010
-
ruaok
ding.
2010-11-15 31903, 2010
-
ruaok
ok, onward.
2010-11-15 31903, 2010
-
pbryan
Heh.
2010-11-15 31922, 2010
-
ruaok
the next issue is how we store edit data in the edit tables.
2010-11-15 31935, 2010
-
ruaok
XML was slow and is very verbose, which causes the DB size to go up.
2010-11-15 31946, 2010
-
ruaok
and makes it MUCH harder to keep the edit table in RAM.
2010-11-15 31953, 2010
-
ruaok
thus we need to move to something else.
2010-11-15 31958, 2010
-
ruaok
I personally don't care what.
2010-11-15 31905, 2010
-
ruaok
JSON and hstore are suggested.
2010-11-15 31910, 2010
-
pbryan
+1 to JSON
2010-11-15 31914, 2010
-
ruaok
hstore can be queried, JSON cannot.
2010-11-15 31917, 2010
-
ocharles
I vote json for the speed of serializing it
2010-11-15 31920, 2010
-
warp
the current xml has a lot of unneccesary bloat
2010-11-15 31925, 2010
-
ruaok
warp: +1
2010-11-15 31932, 2010
-
ocharles
I don't think we should even be considering querying because the entire schema is wrong, in my opinion
2010-11-15 31948, 2010
-
ocharles
We should just use the fastest store possible, which from what I see is JSON, or maybe YAML::XS
2010-11-15 31954, 2010
-
warp
so it's not just switching from xml to something else, it's also making sure only the neccesary bits are stored.
2010-11-15 31957, 2010
-
ocharles
my ยข2
2010-11-15 31902, 2010
-
ruaok
ocharles: murdos or luks pointed out that for reporting a table scan would be ok. just not for live queries.
2010-11-15 31904, 2010
-
pbryan
Depending on where you put JSON, it can be queried.
2010-11-15 31908, 2010
-
pbryan
Example: CouchDB.
2010-11-15 31943, 2010
-
warp
json sounds nice and simple, I'm not familiar with hstore and yaml::xs, so don't have much opinion on them.
2010-11-15 31944, 2010
-
ocharles
We could probably make the json queryable in Postgres with an extension too
2010-11-15 31953, 2010
-
ocharles
yaml::xs is just a c backend to generate yaml
2010-11-15 31904, 2010
-
ruaok
ocharles: hstore already has that extension.
2010-11-15 31914, 2010
-
ruaok
which is the reason why hstore was suggested.
2010-11-15 31923, 2010
-
ocharles
Right, but then we have to write a hstore serializer
2010-11-15 31934, 2010
-
ruaok
with is another dep.
2010-11-15 31936, 2010
-
ocharles
so either way, we write something it looks like
2010-11-15 31951, 2010
-
ruaok
do we have to write anything with json?
2010-11-15 31901, 2010
-
ocharles
Well, if you want to query into edit data, then probably
2010-11-15 31906, 2010
-
ruaok wonders if murdos is present
2010-11-15 31906, 2010
-
ocharles
or we just do that querying perl side
2010-11-15 31908, 2010
-
murdos
ocharles: it's easier to write perl code than C code
2010-11-15 31919, 2010
-
ocharles
murdos: no, perl is too slow here
2010-11-15 31929, 2010
-
ocharles
that's why I chose JSON, because serialization is done in C
2010-11-15 31931, 2010
-
pbryan
It's easier to allow an optimized database to perform a query than perl (or C) code.
2010-11-15 31931, 2010
-
ocharles
and it's blazingly fast
2010-11-15 31900, 2010
-
ocharles
(this is respect to the edit migration)
2010-11-15 31904, 2010
-
warp
if we need to query that edit data... it's probably easier for us to write a serializer than a querying thingy for postgres.
2010-11-15 31930, 2010
-
ruaok
well, we're never going to query that table.
2010-11-15 31939, 2010
-
ocharles
but you'll want indexes if you want reports, right?
2010-11-15 31943, 2010
-
ruaok
nothing that does a seq scan on 10M rows can go live into production.
2010-11-15 31959, 2010
-
ruaok
but, for reporting where the table is scanned once per 24 hours is acceptable.
2010-11-15 31912, 2010
-
ocharles
For that, you could probably get away with doing it in Perl then too
2010-11-15 31923, 2010
-
ocharles
just grab all edits of type x, y, z, call new_from_row and check them there
2010-11-15 31933, 2010
-
ocharles
it's a bit messy, but it's probably the easiest solution that avoids us writing c
2010-11-15 31952, 2010
-
ruaok
ok, I'm hearing that more people prefer JSON.
2010-11-15 31904, 2010
-
ruaok
if JSON really bugs you, speak up now!
2010-11-15 31920, 2010
-
ruaok
the current code doesn't use JSON, does it?
2010-11-15 31920, 2010
-
ocharles
My other thought here...
2010-11-15 31924, 2010
-
murdos
I just find strange to store json in db;..
2010-11-15 31950, 2010
-
ocharles
murdos: it's just serialization. It's not json that,s wrong, it's the fact we have serialized data
2010-11-15 31953, 2010
-
luks
well, it's less strange than the mix we have right now :)
2010-11-15 31901, 2010
-
ruaok
luks: +1
2010-11-15 31918, 2010
-
ocharles
that is my problem. And I'm wondering if now would be the best time to address it, but I guess we don't have enough time..
2010-11-15 31922, 2010
-
ocharles
which is a shame
2010-11-15 31934, 2010
-
ruaok
ocharles: well, we're going to fix the edit system after NGS.
2010-11-15 31938, 2010
-
ruaok
lets deal with this issue then.
2010-11-15 31945, 2010
-
ocharles
alright
2010-11-15 31949, 2010
-
ruaok
so, json in the DB should be a time limited thing.
2010-11-15 31955, 2010
-
ruaok
(famous last words).
2010-11-15 31957, 2010
-
warp
lol
2010-11-15 31904, 2010
-
ruaok
so, JSON it is then.
2010-11-15 31906, 2010
-
ocharles
It'll never go away, I don't think, historic data will be too tricky
2010-11-15 31910, 2010
-
pbryan
\o/
2010-11-15 31926, 2010
-
ruaok
ocharles: is it JSON now? if not, can you make it json very soon?
2010-11-15 31932, 2010
-
ocharles
it is now
2010-11-15 31937, 2010
-
ruaok
ruaok has changed the topic to: agenda: collections/lists, replication packets, [got more?]
2010-11-15 31942, 2010
-
ruaok
ok, case closed.
2010-11-15 31947, 2010
-
ruaok
on to the next sticky topic.
2010-11-15 31952, 2010
-
ruaok
collections vs lists.
2010-11-15 31903, 2010
-
pbryan
Pls add to topic: works ACs.
2010-11-15 31910, 2010
-
ruaok
there were some interesting cases presented in the mailing list.
2010-11-15 31924, 2010
-
ruaok
but not anything that clearly stood out.
2010-11-15 31940, 2010
-
ruaok
murdos: I know you feel strongly about this and are willing to throw some time at this.
2010-11-15 31900, 2010
-
ruaok
I'm ok, with letting murdos drive this process and fix up the nomenclature to be more clear.
2010-11-15 31913, 2010
-
ruaok
murdos: are you still interested in working on that?
2010-11-15 31928, 2010
-
warp
warp has changed the topic to: agenda: collections/lists, replication packets, work ACs, [got more?]
2010-11-15 31942, 2010
-
murdos
all the feedback I get from users were that they understand 'collection' but not 'list'
2010-11-15 31907, 2010
-
ruaok
I think I prefer to change the nomenclature back to collections.
2010-11-15 31917, 2010
-
ocharles
I don't, but I do have more thoughts
2010-11-15 31922, 2010
-
ruaok
then when we actually add "list" support, then we should expose a new list concept.
2010-11-15 31933, 2010
-
ocharles
I think collection should be something that is on top of lists. A collection has a list, but it might have more data
2010-11-15 31948, 2010
-
ocharles
This could let client software do special stuff if it needs to know the user's collection
2010-11-15 31949, 2010
-
ruaok
for me its the concept of ordering.
2010-11-15 31957, 2010
-
ruaok
a list should have an ordering and they don't right now.
2010-11-15 31908, 2010
-
ruaok
thats why they should stay collections, IMHO.
2010-11-15 31910, 2010
-
ocharles
Want me to call it a set then? :)
