[listenbrainz-server] paramsingh opened pull request #635 (master…param/revert-username-changes): Revert "Update the musicbrainz ID of users if it has changed" https://github.com/metabrainz/listenbrainz-server…
I'm working on solr cloud load balancers, small downtimes may occur this morning.
2019-09-10 25355, 2019
Nyanko-sensei has quit
2019-09-10 25329, 2019
D4RK-PH0ENiX joined the channel
2019-09-10 25350, 2019
reosarevok
So *zas* is to blame for the downtime after all!
2019-09-10 25358, 2019
reosarevok
We need different t-shirts for the rest of us
2019-09-10 25303, 2019
zas
yup, and for uptimes too
2019-09-10 25324, 2019
ruaok
Mr_Monkey is actually making two different shirts. we want to own our downtime. :)
2019-09-10 25335, 2019
ruaok
zas: do you want your shirt to be for picard or MB?
2019-09-10 25357, 2019
zas
Picard
2019-09-10 25312, 2019
zas
but if both are possible... ;)
2019-09-10 25309, 2019
ruaok
both. noted.
2019-09-10 25321, 2019
zas
I upgraded haproxy to 1.8 on solr cloud load balancers, not much changes. We have response times variations since the start, and apparently haproxy version didn't change this, after last sequence (stop haproxy on lb3, automatic switch to lb4, reboot lb3, automatic switch to lb3) response times are now back to normal (around 100ms).
2019-09-10 25336, 2019
zas
I really don't know why they sometimes degrade (>200ms) after switches (I did this exact sequence yesterday, it didn't help).
ruaok: when we first setup this stuff, it was already there, I tried everything but basically reboots/failover switches have a random impact on this, I suspect Hetzner Cloud failover IP management and/or routing, because nothing changes on our side. The increase that happened yesterday was after the switch lb3->lb4->lb3, I tried to redo the sequence lb3->lb4->reboot lb3->lb3 and it didn't work, but it did this
2019-09-10 25303, 2019
zas
morning...
2019-09-10 25305, 2019
ruaok
indeed.
2019-09-10 25324, 2019
ruaok
I wonder if a private VLAN would insulate us from this.
2019-09-10 25333, 2019
zas
well, not sure it is worth to investigate further, because once it is on "correct" times it usually stays as is for months (of course, it degrades during peak hours, but that's normal)
2019-09-10 25312, 2019
zas
and apart this, everything is ok (all requests are handled)
2019-09-10 25302, 2019
kori has quit
2019-09-10 25311, 2019
outsidecontext joined the channel
2019-09-10 25335, 2019
zas
btw, the upgrade to 1.8 is an attempt to fix the issue we had yesterday, haproxy instance works, and solr nodes too, but haproxy just stops to forward queries, and this is something which is very hard to detect (because the process runs, accepts queries, etc...). 1.8 has changed a lot regarding how processes/threads are handled, I expect this (not well defined) issue was perhaps fixed in the process. Time will tell, if
2019-09-10 25335, 2019
zas
it happens again (it did twice in ~2 years) I'll investigate further.
2019-09-10 25349, 2019
ruaok
ok, fair.
2019-09-10 25345, 2019
zas
This issue requires human action which is pretty simple: reboot lb3 (it will switch to lb4, and after lb3 comes back, to lb3 again)
2019-09-10 25358, 2019
zas
Impact for users is huge though: mb website doesn't handle such failure well, because haproxy actually accepts connections (and therefore I don't think mb actually "sees" the problem, it just expires connections after 10s, and retries)
2019-09-10 25352, 2019
pristine__
Color of tshirt? I missed the previous chats.
2019-09-10 25309, 2019
pristine__
I would say pink. Lol.
2019-09-10 25334, 2019
outsidecontext
zas: hi, do you have a moment?
2019-09-10 25349, 2019
zas
outsidecontext: sure :)
2019-09-10 25337, 2019
outsidecontext
sorry, communication with me is currently a bit difficult again :D what do you think about a picard release, do you think we could handle this at end of this week?
2019-09-10 25340, 2019
kori joined the channel
2019-09-10 25340, 2019
kori has quit
2019-09-10 25340, 2019
kori joined the channel
2019-09-10 25357, 2019
zas
outsidecontext: not sure, because we have to switch database servers too, I'll discuss that with bitmap this afternoon. In any case, release can be done soon, I think everything is ready for it. And we'll prolly have few bugfix releases just after it anyway
2019-09-10 25319, 2019
outsidecontext
ok, I see. regarding Picard's state I agree
2019-09-10 25327, 2019
outsidecontext
only two things are troubling me a bit: one are the spurious hangs I sometimes get on my system after "Add files" (which I asked you to try reproduce).
outsidecontext: yup, I saw this one, but it seems to me that's a Qt issue rather than a Picard's one
2019-09-10 25355, 2019
outsidecontext
zas: yes
2019-09-10 25313, 2019
zas
we have only one user that reported this issue, right? and you couldn't reproduce it?
2019-09-10 25314, 2019
outsidecontext
zas: regarding the first issue I think about adding just the same "thread.to_main" workaround as I did for the other hang. I have the "feeling" it helps, but it is hard to say because it is hard to reproduce. I need to try very often to reproduce it once.
MBS-5070: Support submitting to the XML Web Service using XMLHttpRequest
2019-09-10 25349, 2019
zas
those threading issues are awful, that's definitively a field of research for next versions
2019-09-10 25300, 2019
outsidecontext
zas: yes, I even installed Ubuntu Studio with Xfce as reported. could be a qt issue, or something sound hardware related. who knows
2019-09-10 25333, 2019
outsidecontext
zas: the threading could also be a qt issue. I can't say whether any change in picard introduced it or update to Qt 5.13. Qt 5.13.1 is out, maybe this even changes anything again
2019-09-10 25338, 2019
zas
btw, that was what I was fearing when this internal player was added (I shared my views on this), and the original author that was supposed to handle this isn't around anymore of course... (https://github.com/metabrainz/picard/pull/1194#is…)
2019-09-10 25328, 2019
outsidecontext
zas: Yes, I thought of your comment already. we could still roll it back again a bit, maybe hide it behind a command line flag?
2019-09-10 25329, 2019
zas
anyway, now it's in, and we'll have to handle bug reports ;)
2019-09-10 25311, 2019
zas
well, I propose to release as is, it will provide us more feedback, and if issues are out of control we'll decide what to do
2019-09-10 25328, 2019
zas
s/propose/suggest/ (poor me)
2019-09-10 25330, 2019
outsidecontext
zas: fine for me. as a user I actually like the player very much, so I'll just take it and try to handle the reported issues :D
2019-09-10 25306, 2019
zas
I think it's a good feature, but I'm always worried about new dependencies...
2019-09-10 25343, 2019
outsidecontext
at least it is part of Qt, that actually put me in favor of this implementation
2019-09-10 25359, 2019
outsidecontext
zas: anyway, so let me know what you think would be a good time for a release and I'll make sure I'm available that day
2019-09-10 25308, 2019
zas
ok, I'll tell you after I plan stuff with bitmap (which has higher priority, we are lacking of disk space on main database server, which can be a serious issue...)
2019-09-10 25346, 2019
alastairp
Mr_Monkey: hi
2019-09-10 25305, 2019
outsidecontext
zas: yes, of course. make the servers happy first. I can also prepare blog post, tag the release etc., but you need to update FTP and website
2019-09-10 25310, 2019
zas
but it will be great to have Picard out before 19th (I'm travelling to BCN)
2019-09-10 25353, 2019
outsidecontext
unfortunately I could not make it again this year for personal reasons :( I would have loved to be on a MB summit again
2019-09-10 25308, 2019
zas
:(
2019-09-10 25323, 2019
ruaok
outsidecontext: what email should I invite for an IRC cloud account?
2019-09-10 25357, 2019
D4RK-PH0ENiX has quit
2019-09-10 25300, 2019
zas
outsidecontext: ^^ it will help a lot ;)
2019-09-10 25314, 2019
outsidecontext
ruaok: oh, that would be great. main reason communication with IRC does not work with me is that I miss messages because I can't be online all the time
