the larger part that's currently missing is porting the analysis step to solr, but I think I've outlined that in an email last year
2015-02-10 04110, 2015
Mineo
it's deployable in that it produces valid xml if you search for stuff and getting data from the db into solr works fine
2015-02-10 04146, 2015
ruaok
depolyable, but ... == not deployable.
2015-02-10 04145, 2015
alastairp
ruaok: let's talk about that further
2015-02-10 04157, 2015
ruaok
mentoring?
2015-02-10 04100, 2015
alastairp
yeah
2015-02-10 04104, 2015
ruaok
k.
2015-02-10 04110, 2015
alastairp
AB, as part of MB? in collaboration with here?
2015-02-10 04153, 2015
ruaok
well, UPF could apply, but that would be a kinda pain, I suspect.
2015-02-10 04106, 2015
ruaok
I'm happy to have AB under the metabrainz umbrella.
2015-02-10 04115, 2015
Mineo
well, I'm not sure what deployable means - if you want to change the production search over to it right now, then no, that wouldn't be a good idea. if you want to set it up on rika for example, so people can use it on their sandboxes, that would certainly work (the only change required in MBS is https://bitbucket.org/mineo/musicbrainz-server/co…)
2015-02-10 04158, 2015
ruaok
the former is what I really care about.
2015-02-10 04121, 2015
ruaok
sounds like the SOLR project needs to have a CB styled continuation project behind it.
2015-02-10 04126, 2015
ruaok
part coding/ part sysadmin
2015-02-10 04136, 2015
kepstin-laptop
could drop a 'devops' keyowrd in there to be all hip and cool and stuff.
Freso: nah, I've updated the JIRA ticket I was going to talk to him about, so no need now ;)
2015-02-10 04121, 2015
ruaok joined the channel
2015-02-10 04123, 2015
rk29 joined the channel
2015-02-10 04113, 2015
yeeeargh joined the channel
2015-02-10 04114, 2015
Freso
LordSputnik: Btw, bb.o is 502'ing.
2015-02-10 04124, 2015
LordSputnik
Freso: yeah I noticed a minute ago
2015-02-10 04130, 2015
LordSputnik
looking into it
2015-02-10 04139, 2015
Freso
"Sorry, that means this server's off right now — check back another time or use the main site at http://musicbrainz.org!"
2015-02-10 04151, 2015
Freso
:)
2015-02-10 04103, 2015
ianmcorvidae
yeah, not writing custom 502 pages for non-MBS sandboxes :P
2015-02-10 04108, 2015
Freso
;)
2015-02-10 04132, 2015
LordSputnik
Freso: hmm, it seems I actually stopped the server manually and forgot to restart it last night...
2015-02-10 04138, 2015
LordSputnik
fixed
2015-02-10 04139, 2015
xram joined the channel
2015-02-10 04126, 2015
chirlu` joined the channel
2015-02-10 04119, 2015
alastairp
Mineo: thanks for the comments on that ticket
2015-02-10 04129, 2015
alastairp
annoying that it’s not “easy"
2015-02-10 04141, 2015
alastairp
maybe this should push us towards requests again :-P
2015-02-10 04123, 2015
chirlu`
ianmcorvidae: The subscriptions emails failed again, this time for an SMTP problem (“Temporary local problem - please try later”).
2015-02-10 04132, 2015
chirlu`
a) Was this really just temporary?
2015-02-10 04144, 2015
ianmcorvidae
they regularly do that
2015-02-10 04154, 2015
chirlu`
b) Should I change the script to continue if sending one message fails?
2015-02-10 04117, 2015
ianmcorvidae
if it doesn't already do that then probably :/ I'd understood it already did that
2015-02-10 04150, 2015
ianmcorvidae
but I've barely touched the subscriptions stuff, I think you have a better idea than I do at this point
2015-02-10 04140, 2015
chirlu`
No, apparently such a problem results in an exception being thrown, and the script doesn’t catch it.
2015-02-10 04146, 2015
ianmcorvidae
it appears it was killed
2015-02-10 04157, 2015
ianmcorvidae
so the memory may still be the issue
2015-02-10 04107, 2015
ianmcorvidae
before the memory stuff was happening it would screw up on multiple things sending emails
2015-02-10 04132, 2015
ianmcorvidae
multiple temporary local problem/unroutable address/etc.
2015-02-10 04151, 2015
ianmcorvidae
so somehow that bit isn't killing the script
2015-02-10 04108, 2015
ianmcorvidae
but yeah, the one from this morning it has the same /home/musicbrainz/musicbrainz-server/admin/cron/daily.sh: line 86: 14133 Killed
2015-02-10 04105, 2015
chirlu`
Hm, I’ll check closer what happens in case of an SMTP error.
2015-02-10 04101, 2015
chirlu`
Could we run it manually with “--dry-run --verbose”?
2015-02-10 04137, 2015
ianmcorvidae
sure
2015-02-10 04144, 2015
ianmcorvidae
starting that now
2015-02-10 04116, 2015
ianmcorvidae
it's already the thing using the most memory on the system, also. watching it to see if it drops at some point
2015-02-10 04120, 2015
ianmcorvidae
(right now 8%)
2015-02-10 04143, 2015
ijabz2 joined the channel
2015-02-10 04120, 2015
ianmcorvidae
ah, no, apparently it finished before dying
2015-02-10 04131, 2015
chirlu`
As far as I understand, Perl never returns memory to the system.
2015-02-10 04148, 2015
chirlu`
It only reuses it internally if possible.
2015-02-10 04113, 2015
ianmcorvidae
right
2015-02-10 04103, 2015
ianmcorvidae
last night wouldn't have been the weekly, anyway
2015-02-10 04116, 2015
ianmcorvidae
shall I send you the log of the run? it seems to have gone through it and finished fine
2015-02-10 04145, 2015
ruaok
huh.
2015-02-10 04100, 2015
ruaok
maybe the invoking script has a resource limit set?
2015-02-10 04141, 2015
ianmcorvidae
hm, or it killed it
2015-02-10 04148, 2015
chirlu`
Or it’s the mailer (--dry-run avoids it).
2015-02-10 04113, 2015
ianmcorvidae
the end of dmesg currently has [8885062.052366] Killed process 27816 (perl) total-vm:1546348kB, anon-rss:1315228kB, file-rss:972kB
2015-02-10 04119, 2015
chirlu`
re: log, the interesting parts would probably just be the “End of batch” and “Completed” lines.
2015-02-10 04132, 2015
chirlu`
“End of batch” for the cache statistics.
2015-02-10 04110, 2015
ianmcorvidae
it didn't finish a batch.
2015-02-10 04115, 2015
chirlu`
1.5 GB total-vm looks rather much.
2015-02-10 04124, 2015
ianmcorvidae
so yeah, I think it got killed
2015-02-10 04127, 2015
ianmcorvidae
memory usage is really close to the limit on that system, I guess
2015-02-10 04146, 2015
chirlu`
Perhaps send me the log anyway, I could look how far it got within the first batch at least.
2015-02-10 04111, 2015
ianmcorvidae
sure. I'm poking around the system a bit to see if I can make the memory usage less nasty
2015-02-10 04120, 2015
ianmcorvidae
I don't think MBS should need 14GB of memory, so
2015-02-10 04115, 2015
chirlu`
But neither should the script need 1.5 GB just for one 1000-editor batch.
2015-02-10 04123, 2015
ianmcorvidae
yeah
2015-02-10 04158, 2015
ianmcorvidae
I think there were some extra starman workers lying around
2015-02-10 04103, 2015
ianmcorvidae
so I killed those, might help
2015-02-10 04137, 2015
ianmcorvidae
chirlu`: what email should I send to? PM if you prefer
2015-02-10 04130, 2015
ianmcorvidae
doing another test run as well with the system less memory-heavy, in case that provides more insight
2015-02-10 04129, 2015
chirlu`
I’ll wait for that, then. :) Otherwise any local part at chirlu de should work.
2015-02-10 04137, 2015
ianmcorvidae
k
2015-02-10 04138, 2015
chirlu`
I normally use ulrich.
2015-02-10 04116, 2015
ianmcorvidae
currently memory use is still climbing, but the system is only using ~5G of memory, not ~14 of its 16. so if it levels out at some point hopefully we'll know what that point is, this run
2015-02-10 04115, 2015
ianmcorvidae
still does seem high though, it's at almost 2G of VM
2015-02-10 04158, 2015
ianmcorvidae
heh, and it's currently on the user it died on last time, interesting :)
2015-02-10 04124, 2015
chirlu`
Does that user have lots of subscriptions or something?
2015-02-10 04136, 2015
ianmcorvidae
I don't know, it hasn't gotten far enough to actually print out the username
2015-02-10 04154, 2015
ianmcorvidae
it's whoever it'll get to after DosX
2015-02-10 04159, 2015
ianmcorvidae
there we go, it did pass it
2015-02-10 04101, 2015
ianmcorvidae sees who it was
2015-02-10 04129, 2015
ianmcorvidae
maybe it was just not flushing or something. odd
2015-02-10 04109, 2015
ianmcorvidae
ojnkpjg with 12440 subscriptions wasn't too far after though, so that might be it
2015-02-10 04102, 2015
chirlu`
Yes, that sounds like much.
2015-02-10 04136, 2015
chirlu`
In particular when subscription mails haven’t been sent for a while, so there may have many edits piled up.
2015-02-10 04154, 2015
ianmcorvidae
sure
2015-02-10 04118, 2015
ianmcorvidae
I notice it goes to the trouble of finding subscriptions before checking if there's a confirmed email, that seems like a possibly minor optimization that could happen
2015-02-10 04155, 2015
chirlu`
Hm, possibly.
2015-02-10 04133, 2015
ianmcorvidae
however, memory usage has leveled off now, also, it seems. 18% memory, heh. 3105M of VM
2015-02-10 04156, 2015
ianmcorvidae
yeah, it finished a batch: End of batch: removing 53338 entities from the cache (out of 54214)
2015-02-10 04130, 2015
ianmcorvidae
I get the impression we may want to make batches an order of magnitude smaller, perhaps
2015-02-10 04144, 2015
chirlu`
That sounds good. So it still caches around 1000 entities that were used multiple times.
2015-02-10 04153, 2015
chirlu`
Yes, perhaps 1000 is too much.
2015-02-10 04118, 2015
ianmcorvidae
will be interesting to see how it does with subsequent batches, obviously I won't change it until our test run is done, but :)
2015-02-10 04119, 2015
chirlu`
Though reducing the size will also reduce the benefit of the cache, because only very popular entities will appear twice in a 100-editor batch (say).
2015-02-10 04155, 2015
ianmcorvidae
true
2015-02-10 04127, 2015
chirlu`
Anyway, the second batch should reuse the memory from the 53338 entities that were removed, so memory usage should stay mostly constant now.
2015-02-10 04139, 2015
ianmcorvidae
yeah
2015-02-10 04157, 2015
chirlu`
Unless there is another editor with zillions of subscriptions.
2015-02-10 04104, 2015
ianmcorvidae
reosarevok :P
2015-02-10 04107, 2015
ianmcorvidae
and probably nikki
2015-02-10 04129, 2015
ianmcorvidae
I doubt ojnkpjg is the worst as far as that
2015-02-10 04158, 2015
chirlu`
Hm, perhaps some per-editor limit of how much to put into the cache?
2015-02-10 04148, 2015
chirlu`
“Cache only the 100 most relevant subscriptions”, for some definition of relevant.
2015-02-10 04108, 2015
ianmcorvidae
hm, maybe
2015-02-10 04115, 2015
ianmcorvidae
btw, for artists at least ojnkpjg is fifth
2015-02-10 04145, 2015
ianmcorvidae
drsaunde has 53751 subscriptions looking only at artists :)
2015-02-10 04106, 2015
chirlu`
I had also thought about asking the database beforehand about the most common subscriptions, and then only cache those.
2015-02-10 04117, 2015
ianmcorvidae
drsaunde is the one who'll break this nice and hard for us, I think, if he does :) 53751 artists and 5283 labels
2015-02-10 04133, 2015
chirlu`
I.e. “give me all entities that have at least 3 subscribers” or something.
2015-02-10 04149, 2015
ianmcorvidae
yeah, could work
2015-02-10 04134, 2015
ianmcorvidae
second batch removed 24795 of 26225 entities
2015-02-10 04139, 2015
ianmcorvidae
so about the same amount of growth I guess
2015-02-10 04106, 2015
ianmcorvidae
(in the cache, I mean. I doubt it even got close to filling that 3G of memory with half as many entities)
2015-02-10 04155, 2015
diana_olhovik joined the channel
2015-02-10 04143, 2015
ariscop joined the channel
2015-02-10 04157, 2015
ianmcorvidae
yeah, memory usage is climbing again :(
2015-02-10 04108, 2015
ianmcorvidae
up to 27.1% with 4588M (virt)
2015-02-10 04129, 2015
ianmcorvidae
(4435 resident as of when I started typing this line :P)
2015-02-10 04153, 2015
chirlu`
Is it at reosarevok now? :)
2015-02-10 04113, 2015
ianmcorvidae
dunno :)
2015-02-10 04148, 2015
chirlu`
Would be editor #326637.
2015-02-10 04152, 2015
Freso
Just make drsaunde his own batch.
2015-02-10 04105, 2015
Freso
>_>
2015-02-10 04110, 2015
ianmcorvidae
Freso: what I'm wondering is if that wouldn't still get killed by itself :P
