you familiar with ram drives and caching in the linux kernel?
2017-12-22 35600, 2017
ruaok
just superficially understanding the concepts is all you need to know.
2017-12-22 35612, 2017
samj1912
yup
2017-12-22 35630, 2017
ruaok
often times there would be things I want to go fast.
2017-12-22 35643, 2017
ruaok
like, load an index and rotate it -- perfect example.
2017-12-22 35658, 2017
ruaok
I know, I'll get a ram disk to do that. that guarantees that everything stays in ram in will be fast.
2017-12-22 35609, 2017
ruaok
I tried it and it never was fast.
2017-12-22 35616, 2017
ruaok
because I could not see the whole picture.
2017-12-22 35639, 2017
ruaok
instead, letting linux do the caching automatically by literally doing nothing special, I always got the best performance.
2017-12-22 35652, 2017
ruaok
because linux knows what it needs and can see the whole picture, instantaneously.
2017-12-22 35648, 2017
samj1912 nods
2017-12-22 35649, 2017
ruaok
the point I am getting to is this: not sure you need to have a "large indexer backup". I think a secondary or tertiary indexer is all you need.
2017-12-22 35659, 2017
samj1912
yeah I guess
2017-12-22 35619, 2017
samj1912
talking to you I realized, we could have 2 instances, 1 with an index limit and one without
2017-12-22 35629, 2017
ruaok
and even then there is a fine line -- this is about the old "premature optimization" problem.
2017-12-22 35640, 2017
samj1912
yes
2017-12-22 35658, 2017
ruaok
and it is tricky. clearly we want to avoid having indexing break down because of bottlenecks.
2017-12-22 35608, 2017
samj1912
let me make an artificial call for indexing VA?
2017-12-22 35614, 2017
ruaok
so we want a certain amount of thinking of optimization, but not too much.
2017-12-22 35628, 2017
ruaok
indexing VA == reindex
2017-12-22 35643, 2017
ruaok
should things like that ever be needed, just reindex.
2017-12-22 35609, 2017
ruaok
"1 with an index limit and one without"
2017-12-22 35613, 2017
ruaok
not sure even that is needed.
2017-12-22 35626, 2017
ruaok
ensure that having more than 1 indexer works.
2017-12-22 35631, 2017
ruaok
and leave it at that.
2017-12-22 35648, 2017
ruaok
when real usecases come, we can throw more indexers at the problem.
2017-12-22 35607, 2017
ruaok
and if those are a problem, we've identified an actual problematic use case that we should optimize for.
2017-12-22 35617, 2017
ruaok
but that day may never come, and that is fine too.
2017-12-22 35621, 2017
ruaok
so, come to a close on what you're doing with the indexer, fixing bugs, but don't plan too many more improvements.
2017-12-22 35630, 2017
ruaok
keep watching the logs and then soon let's test this on beta.
2017-12-22 35645, 2017
ruaok
and perhaps reward yourself by playing with picard for a little while. :)
2017-12-22 35620, 2017
samj1912
Hehe okay :p
2017-12-22 35634, 2017
samj1912
I was thinking of taking a crack at MB instead :p
2017-12-22 35616, 2017
padraic joined the channel
2017-12-22 35644, 2017
Ashish_ joined the channel
2017-12-22 35619, 2017
naiveai joined the channel
2017-12-22 35657, 2017
naiveai has quit
2017-12-22 35601, 2017
drsaunders has quit
2017-12-22 35605, 2017
MusicbrainzB0T joined the channel
2017-12-22 35641, 2017
MusicbrainzB0T1 has quit
2017-12-22 35619, 2017
Ashish_ has quit
2017-12-22 35635, 2017
culinko has left the channel
2017-12-22 35625, 2017
zas
ruaok: current search-server on prince is eating a lot of disk space, i had a look; and it appears there are plenty of /tmp/jetty/jetty-0.0.0.0-8080-ROOT.war-_-any-<somenumber>.dir (12455 atm), each of 30M or more. I'll prolly just rebuild a fresh container, but any idea why ?
2017-12-22 35623, 2017
zas
those are leftover on each index reloading...
2017-12-22 35644, 2017
ruaok
when the container restarts, I think jetty is rebuilding shit and not cleaning up.
2017-12-22 35605, 2017
ruaok
given how close samj1912 is, can we just mitigate it and hope for a migration soon?
2017-12-22 35618, 2017
zas
yes, i just clean leftovers
2017-12-22 35626, 2017
ruaok
great, thanks.
2017-12-22 35607, 2017
zas
and those containers logs are big too, truncating as well
2017-12-22 35640, 2017
samj1912
ruaok: can you take a look at https://github.com/metabrainz/sir/pull/67, although I have been running this PR since last night, there might be some things I might've overlooked
2017-12-22 35614, 2017
ruaok
let me find how I am going to be blocked on deploying LB today, before I do sir stuff.
2017-12-22 35625, 2017
samj1912
okay
2017-12-22 35657, 2017
ruaok
I've been doing 10 minutes of work on LB, get stuck, then submit PR, wait, have attention drawn away, repeat.
2017-12-22 35658, 2017
zas
reclaimed 70% of disk space on prince & boingo ...
we had a situation in AB where migration required 1 sql file, 1 py file, 1 sql file for a single migration
2017-12-22 35620, 2017
alastairp
also, not sure of the case in LB, but in AB we have public and private tables, so theoretically we could have 2 schema numbers?? I had a bit of a chat with bitmap about how MB does that, but I didn't come to a good solution
2017-12-22 35631, 2017
ruaok
iliekcomputers: schema changes applied fine.
2017-12-22 35605, 2017
ruaok
2 schema numbers sounds nothing but trouble.
2017-12-22 35601, 2017
ruaok
iliekcomputers: I should be able to trigger a dump of all the data now, right?
2017-12-22 35605, 2017
ruaok
let me do that before I move on.
2017-12-22 35634, 2017
iliekcomputers
ruaok: `python manage.py dump create` would work
2017-12-22 35639, 2017
iliekcomputers
ideally
2017-12-22 35649, 2017
alastairp
ruaok: yeah, 2 does
2017-12-22 35610, 2017
ruaok
iliekcomputers: do we have a means for copying the dumps out of the containers yet? :)
2017-12-22 35621, 2017
iliekcomputers
ruaok: no, not yet
2017-12-22 35636, 2017
ruaok
that should be a next PR then.
2017-12-22 35647, 2017
alastairp
isn't that "just" a docker volume?
2017-12-22 35607, 2017
ruaok
could be, but it isn't setup yet.
2017-12-22 35610, 2017
alastairp
sure
2017-12-22 35617, 2017
alastairp
I put just in "" for a reason :)
2017-12-22 35638, 2017
naiveai joined the channel
2017-12-22 35651, 2017
ruaok
ok, the dumper is trying to connect to the wrong DB.
2017-12-22 35609, 2017
ruaok
I suspect that we duplicated the config file reading and it is reading the wrong config files.
2017-12-22 35624, 2017
ruaok
let me commit the patch I have now, then we can consolidate how we connect.
2017-12-22 35603, 2017
iliekcomputers
ruaok: if it's importing the configuration files directly (which it probably is) then that would happen, yes.
2017-12-22 35623, 2017
D4RK-PH0ENiX has quit
2017-12-22 35645, 2017
iliekcomputers
Me and alastairp have worked out a good way to manage the configurations using the flask app
