samj1912: spawning subprocesses to handle read/write (which are mainly done through mutagen) should be simple enough
2017-07-26 20749, 2017
iliekcomputers
looking at the doc
2017-07-26 20700, 2017
Sophist-UK
zas, samj1912: We do that already for most processes. But #604 did more of it.
2017-07-26 20728, 2017
ruaok
iliekcomputers: just follow along. if something comes to mind add it. or add questions.
2017-07-26 20733, 2017
ruaok
if nothing else, sanity check
2017-07-26 20742, 2017
iliekcomputers
hmm, okay
2017-07-26 20709, 2017
Sophist-UK
samj1912: I also thought about running sub-processes (instead of sub-threads) - we could allocate albums to sub-processes to handle the heavy loading. But sending data between threads is by pickling, so we would need to establish what the performance impact would be on sending file objects via pickling.
2017-07-26 20701, 2017
Sophist-UK
Also impact on running plugins.
2017-07-26 20717, 2017
Sophist-UK
BUT, I suspect that multi-processing is only beneficial if Picard uses 100% CPU on one processor.
2017-07-26 20730, 2017
Sophist-UK
I also did a performance profile to see what was using most CPU and it was the code that read the XML stream and turned it into python objects which was by far the heaviest load - and that was running on main thread not a worker thread. But now we use json that measurement is no longer valid.
2017-07-26 20731, 2017
zas
Sophist-UK: i don't think it will cost much, especially compared to the time needed to rewrite a 10 minutes hires flac file due to header resizing
2017-07-26 20714, 2017
Sophist-UK
zas: The question is whether writing a hires flac file has an impact when it it done on a worker thread rather than a separate process.
2017-07-26 20729, 2017
zas
currently it has
2017-07-26 20752, 2017
Sophist-UK
Usually it will be the HDD (or network if on a share) that is the limiting factor.
2017-07-26 20757, 2017
samj1912
lazka said it shouldn't
2017-07-26 20715, 2017
lazka joined the channel
2017-07-26 20733, 2017
ruaok
iliekcomputers: I'll detail step 2 for right before switchover.
2017-07-26 20735, 2017
Sophist-UK
Also, writing files is limited to a single thread to avoid multiple Piucard files being mapped to the same actual file and being written simultaneously.
ruaok: that would just involve changing BETA_URL to 'api.listenbrainz.org', yes?
2017-07-26 20733, 2017
Sophist-UK
I have not been following Picard's recent changes in detail, but if master thread uis stable, I can try to rework the core parts of #604 for Py3.
2017-07-26 20751, 2017
iliekcomputers
ALPHA_URL is already set to `alpha.listenbrainz.org` by default
2017-07-26 20756, 2017
ruaok
and ALPHA_URL, and removing the banner and changing "pre-beta test" to "beta"
2017-07-26 20706, 2017
iliekcomputers
okay
2017-07-26 20706, 2017
samj1912
Sophist-UK: picard 2 is pretty stable :)
2017-07-26 20726, 2017
ruaok
which doesn't resolve.
2017-07-26 20729, 2017
zas
Sophist-UK: i don't think much changed in this field, it would be great to have 604 fixed for Picard 2, so we can actually test
2017-07-26 20719, 2017
Sophist-UK
I don't want to retrofit #604 to P2 in one go - it was too big last time. But I can try to extract the key bits.
2017-07-26 20740, 2017
Sophist-UK
I managed to introduce some impossible-to-track-down bug which casused it to hang - so doing it in smaller stable steps is essential.
2017-07-26 20709, 2017
kavorka has quit
2017-07-26 20738, 2017
ruaok
bigquery configs are correct, won't need any changes. using production setup there already
2017-07-26 20707, 2017
iliekcomputers
cool, i'll remove that line then
2017-07-26 20730, 2017
ruaok
keep it.
2017-07-26 20739, 2017
ruaok
strike it through.
2017-07-26 20745, 2017
kavorka joined the channel
2017-07-26 20750, 2017
iliekcomputers
okay
2017-07-26 20725, 2017
ruaok
bitmap, zas: for moving LB to production: We plan to use the site wide redis, rabbitmq and postgres.
2017-07-26 20733, 2017
ruaok
as previously discussed.
2017-07-26 20712, 2017
ruaok
not sure if rabbitmq or redis need any config changes in order for us to use them. do they?
2017-07-26 20725, 2017
black-perl_ joined the channel
2017-07-26 20708, 2017
arbenina_ has quit
2017-07-26 20738, 2017
lazka
samj1912, no idea, sorry. I can only tell you that in my experience mutagen is IO bound, so it's not much different that open().read()
2017-07-26 20722, 2017
lazka
best to profile and look what takes longer than expected
2017-07-26 20729, 2017
drsaunders has quit
2017-07-26 20730, 2017
bitmap
ruaok: start_rabbitmq in docker-server-configs will require some changes to setup users/vhosts/permissions, and you'll have to insert an entry into the amqp.broker table manually (there are examples in the comment at the end)
2017-07-26 20719, 2017
ruaok
ok. no changes for redis?
2017-07-26 20723, 2017
bitmap
I don't think redis needs any changes, I guess just make sure the keys are in a separate namespace
2017-07-26 20748, 2017
ruaok
what is the namespace pattern that is used now?
2017-07-26 20703, 2017
bitmap
mbs prefixes keys with MB:
2017-07-26 20725, 2017
ruaok
ok, we'll follow suit with LB:
2017-07-26 20720, 2017
iliekcomputers
nice color coding ;)
2017-07-26 20733, 2017
bitmap
though there is not a general redis service at the moment...mbs uses two, musicbrainz-redis-store and musicbrainz-redis-cache, which have different configs because one is used as a permanent store
2017-07-26 20736, 2017
ruaok
any suggestions on what we should do?
2017-07-26 20743, 2017
ruaok
I mean, keeping what we have is super easy.
2017-07-26 20735, 2017
bitmap
keeping redis separate might be easier for now, IMO
2017-07-26 20724, 2017
ruaok
ok
2017-07-26 20725, 2017
bitmap
also easier to wipe the whole cache if you need to, if they're separate instances
2017-07-26 20728, 2017
ruaok
iliekcomputers: ^^
2017-07-26 20734, 2017
ruaok
k
2017-07-26 20726, 2017
ruaok
what sort of caveats apply to moving to production postgres?
2017-07-26 20722, 2017
iliekcomputers
just to be clear, now we want two different redis services in LB too ? one for the cache and another for other stuff.
2017-07-26 20732, 2017
github joined the channel
2017-07-26 20732, 2017
github
[listenbrainz-server] paramsingh opened pull request #237: Change header from pre-beta to beta, remove banner warning and update config.BETA_URL. (master...switchover) https://git.io/v73FQ
2017-07-26 20732, 2017
github has left the channel
2017-07-26 20701, 2017
bitmap
if you want to make sure certain keys aren't evicted, yes, but if you only use it as a cache, no
2017-07-26 20726, 2017
ruaok
one should be fine for us.
2017-07-26 20756, 2017
ruaok
zas: what was the minimum DNS TTL at gandi?
2017-07-26 20714, 2017
zas
it depends ;)
2017-07-26 20731, 2017
zas
some entries have a 10 minutes TTL, some are still on 24 hours
