#musicbrainz-devel

that seems unlikely at the moment - I'll probably have my last exam in a few weeks, so I could write my master's thesis over the summer

nooooeos!

we really need the SOLR stuff finished! :(

https://wiki.musicbrainz.org/User:Mineo/SolrSea... is the stuff you can currently search for on solr, which should cover everything on https://beta.musicbrainz.org/doc/Development/XM...

the larger part that's currently missing is porting the analysis step to solr, but I think I've outlined that in an email last year

it's deployable in that it produces valid xml if you search for stuff and getting data from the db into solr works fine

depolyable, but ... == not deployable.

ruaok: let's talk about that further

mentoring?

yeah

k.

AB, as part of MB? in collaboration with here?

well, UPF could apply, but that would be a kinda pain, I suspect.

I'm happy to have AB under the metabrainz umbrella.

well, I'm not sure what deployable means - if you want to change the production search over to it right now, then no, that wouldn't be a good idea. if you want to set it up on rika for example, so people can use it on their sandboxes, that would certainly work (the only change required in MBS is https://bitbucket.org/mineo/musicbrainz-server/...)

the former is what I really care about.

sounds like the SOLR project needs to have a CB styled continuation project behind it.

part coding/ part sysadmin

could drop a 'devops' keyowrd in there to be all hip and cool and stuff.

damn, just as I was about to message him...

LordSputnik: rob [at mb] ;)

Freso: nah, I've updated the JIRA ticket I was going to talk to him about, so no need now ;)

LordSputnik: Btw, bb.o is 502'ing.

Freso: yeah I noticed a minute ago

looking into it

"Sorry, that means this server's off right now — check back another time or use the main site at http://musicbrainz.org!";

:)

yeah, not writing custom 502 pages for non-MBS sandboxes :P

;)

Freso: hmm, it seems I actually stopped the server manually and forgot to restart it last night...

fixed

Mineo: thanks for the comments on that ticket

annoying that it’s not “easy"

maybe this should push us towards requests again :-P

ianmcorvidae: The subscriptions emails failed again, this time for an SMTP problem (“Temporary local problem - please try later”).

a) Was this really just temporary?

they regularly do that

b) Should I change the script to continue if sending one message fails?

if it doesn't already do that then probably :/ I'd understood it already did that

but I've barely touched the subscriptions stuff, I think you have a better idea than I do at this point

No, apparently such a problem results in an exception being thrown, and the script doesn’t catch it.

it appears it was killed

so the memory may still be the issue

before the memory stuff was happening it would screw up on multiple things sending emails

multiple temporary local problem/unroutable address/etc.

so somehow that bit isn't killing the script

but yeah, the one from this morning it has the same /home/musicbrainz/musicbrainz-server/admin/cron/daily.sh: line 86: 14133 Killed

Hm, I’ll check closer what happens in case of an SMTP error.

Could we run it manually with “--dry-run --verbose”?

sure

starting that now

it's already the thing using the most memory on the system, also. watching it to see if it drops at some point

(right now 8%)

ah, no, apparently it finished before dying

As far as I understand, Perl never returns memory to the system.

It only reuses it internally if possible.

right

last night wouldn't have been the weekly, anyway

shall I send you the log of the run? it seems to have gone through it and finished fine

huh.

maybe the invoking script has a resource limit set?

hm, or it killed it

Or it’s the mailer (--dry-run avoids it).

the end of dmesg currently has [8885062.052366] Killed process 27816 (perl) total-vm:1546348kB, anon-rss:1315228kB, file-rss:972kB

re: log, the interesting parts would probably just be the “End of batch” and “Completed” lines.

“End of batch” for the cache statistics.

it didn't finish a batch.

1.5 GB total-vm looks rather much.

so yeah, I think it got killed

memory usage is really close to the limit on that system, I guess

Perhaps send me the log anyway, I could look how far it got within the first batch at least.

sure. I'm poking around the system a bit to see if I can make the memory usage less nasty

I don't think MBS should need 14GB of memory, so

But neither should the script need 1.5 GB just for one 1000-editor batch.

yeah

I think there were some extra starman workers lying around

so I killed those, might help

chirlu`: what email should I send to? PM if you prefer

doing another test run as well with the system less memory-heavy, in case that provides more insight

I’ll wait for that, then. :) Otherwise any local part at chirlu de should work.

k

I normally use ulrich.

currently memory use is still climbing, but the system is only using ~5G of memory, not ~14 of its 16. so if it levels out at some point hopefully we'll know what that point is, this run

still does seem high though, it's at almost 2G of VM

heh, and it's currently on the user it died on last time, interesting :)

Does that user have lots of subscriptions or something?

I don't know, it hasn't gotten far enough to actually print out the username

it's whoever it'll get to after DosX

there we go, it did pass it

maybe it was just not flushing or something. odd

ojnkpjg with 12440 subscriptions wasn't too far after though, so that might be it

Yes, that sounds like much.

In particular when subscription mails haven’t been sent for a while, so there may have many edits piled up.

sure

I notice it goes to the trouble of finding subscriptions before checking if there's a confirmed email, that seems like a possibly minor optimization that could happen

Hm, possibly.

however, memory usage has leveled off now, also, it seems. 18% memory, heh. 3105M of VM

yeah, it finished a batch: End of batch: removing 53338 entities from the cache (out of 54214)

I get the impression we may want to make batches an order of magnitude smaller, perhaps

That sounds good. So it still caches around 1000 entities that were used multiple times.

Yes, perhaps 1000 is too much.

will be interesting to see how it does with subsequent batches, obviously I won't change it until our test run is done, but :)

Though reducing the size will also reduce the benefit of the cache, because only very popular entities will appear twice in a 100-editor batch (say).

true

Anyway, the second batch should reuse the memory from the 53338 entities that were removed, so memory usage should stay mostly constant now.

yeah

Unless there is another editor with zillions of subscriptions.

reosarevok :P

and probably nikki

I doubt ojnkpjg is the worst as far as that

Hm, perhaps some per-editor limit of how much to put into the cache?

“Cache only the 100 most relevant subscriptions”, for some definition of relevant.

hm, maybe

btw, for artists at least ojnkpjg is fifth

drsaunde has 53751 subscriptions looking only at artists :)

I had also thought about asking the database beforehand about the most common subscriptions, and then only cache those.

drsaunde is the one who'll break this nice and hard for us, I think, if he does :) 53751 artists and 5283 labels

I.e. “give me all entities that have at least 3 subscribers” or something.

yeah, could work

second batch removed 24795 of 26225 entities

so about the same amount of growth I guess

(in the cache, I mean. I doubt it even got close to filling that 3G of memory with half as many entities)

yeah, memory usage is climbing again :(

up to 27.1% with 4588M (virt)

(4435 resident as of when I started typing this line :P)

Is it at reosarevok now? :)

dunno :)

Would be editor #326637.

Just make drsaunde his own batch.

>_>

Freso: what I'm wondering is if that wouldn't still get killed by itself :P