[listenbrainz-server] paramsingh opened pull request #635 (master…param/revert-username-changes): Revert "Update the musicbrainz ID of users if it has changed" https://github.com/metabrainz/listenbrainz-serv...
I'm working on solr cloud load balancers, small downtimes may occur this morning.
Nyanko-sensei has quit
D4RK-PH0ENiX joined the channel
reosarevok
So *zas* is to blame for the downtime after all!
We need different t-shirts for the rest of us
zas
yup, and for uptimes too
ruaok
Mr_Monkey is actually making two different shirts. we want to own our downtime. :)
zas: do you want your shirt to be for picard or MB?
zas
Picard
but if both are possible... ;)
ruaok
both. noted.
zas
I upgraded haproxy to 1.8 on solr cloud load balancers, not much changes. We have response times variations since the start, and apparently haproxy version didn't change this, after last sequence (stop haproxy on lb3, automatic switch to lb4, reboot lb3, automatic switch to lb3) response times are now back to normal (around 100ms).
I really don't know why they sometimes degrade (>200ms) after switches (I did this exact sequence yesterday, it didn't help).
ruaok: when we first setup this stuff, it was already there, I tried everything but basically reboots/failover switches have a random impact on this, I suspect Hetzner Cloud failover IP management and/or routing, because nothing changes on our side. The increase that happened yesterday was after the switch lb3->lb4->lb3, I tried to redo the sequence lb3->lb4->reboot lb3->lb3 and it didn't work, but it did this
morning...
ruaok
indeed.
I wonder if a private VLAN would insulate us from this.
zas
well, not sure it is worth to investigate further, because once it is on "correct" times it usually stays as is for months (of course, it degrades during peak hours, but that's normal)
and apart this, everything is ok (all requests are handled)
kori has quit
outsidecontext joined the channel
btw, the upgrade to 1.8 is an attempt to fix the issue we had yesterday, haproxy instance works, and solr nodes too, but haproxy just stops to forward queries, and this is something which is very hard to detect (because the process runs, accepts queries, etc...). 1.8 has changed a lot regarding how processes/threads are handled, I expect this (not well defined) issue was perhaps fixed in the process. Time will tell, if
it happens again (it did twice in ~2 years) I'll investigate further.
ruaok
ok, fair.
zas
This issue requires human action which is pretty simple: reboot lb3 (it will switch to lb4, and after lb3 comes back, to lb3 again)
Impact for users is huge though: mb website doesn't handle such failure well, because haproxy actually accepts connections (and therefore I don't think mb actually "sees" the problem, it just expires connections after 10s, and retries)
pristine__
Color of tshirt? I missed the previous chats.
I would say pink. Lol.
outsidecontext
zas: hi, do you have a moment?
zas
outsidecontext: sure :)
outsidecontext
sorry, communication with me is currently a bit difficult again :D what do you think about a picard release, do you think we could handle this at end of this week?
kori joined the channel
kori has quit
kori joined the channel
zas
outsidecontext: not sure, because we have to switch database servers too, I'll discuss that with bitmap this afternoon. In any case, release can be done soon, I think everything is ready for it. And we'll prolly have few bugfix releases just after it anyway
outsidecontext
ok, I see. regarding Picard's state I agree
only two things are troubling me a bit: one are the spurious hangs I sometimes get on my system after "Add files" (which I asked you to try reproduce).
outsidecontext: yup, I saw this one, but it seems to me that's a Qt issue rather than a Picard's one
outsidecontext
zas: yes
zas
we have only one user that reported this issue, right? and you couldn't reproduce it?
outsidecontext
zas: regarding the first issue I think about adding just the same "thread.to_main" workaround as I did for the other hang. I have the "feeling" it helps, but it is hard to say because it is hard to reproduce. I need to try very often to reproduce it once.
MBS-5070: Support submitting to the XML Web Service using XMLHttpRequest
zas
those threading issues are awful, that's definitively a field of research for next versions
outsidecontext
zas: yes, I even installed Ubuntu Studio with Xfce as reported. could be a qt issue, or something sound hardware related. who knows
zas: the threading could also be a qt issue. I can't say whether any change in picard introduced it or update to Qt 5.13. Qt 5.13.1 is out, maybe this even changes anything again
zas
btw, that was what I was fearing when this internal player was added (I shared my views on this), and the original author that was supposed to handle this isn't around anymore of course... (https://github.com/metabrainz/picard/pull/1194#...)
outsidecontext
zas: Yes, I thought of your comment already. we could still roll it back again a bit, maybe hide it behind a command line flag?
zas
anyway, now it's in, and we'll have to handle bug reports ;)
well, I propose to release as is, it will provide us more feedback, and if issues are out of control we'll decide what to do
s/propose/suggest/ (poor me)
outsidecontext
zas: fine for me. as a user I actually like the player very much, so I'll just take it and try to handle the reported issues :D
zas
I think it's a good feature, but I'm always worried about new dependencies...
outsidecontext
at least it is part of Qt, that actually put me in favor of this implementation
zas: anyway, so let me know what you think would be a good time for a release and I'll make sure I'm available that day
zas
ok, I'll tell you after I plan stuff with bitmap (which has higher priority, we are lacking of disk space on main database server, which can be a serious issue...)
alastairp
Mr_Monkey: hi
outsidecontext
zas: yes, of course. make the servers happy first. I can also prepare blog post, tag the release etc., but you need to update FTP and website
zas
but it will be great to have Picard out before 19th (I'm travelling to BCN)
outsidecontext
unfortunately I could not make it again this year for personal reasons :( I would have loved to be on a MB summit again
zas
:(
ruaok
outsidecontext: what email should I invite for an IRC cloud account?
D4RK-PH0ENiX has quit
zas
outsidecontext: ^^ it will help a lot ;)
outsidecontext
ruaok: oh, that would be great. main reason communication with IRC does not work with me is that I miss messages because I can't be online all the time
