is listenbrainz having issue with the listen queue?
2018-05-11 13137, 2018
Rotab
im missing listens since wednesday
2018-05-11 13117, 2018
UmkaDK joined the channel
2018-05-11 13157, 2018
ruaok
Rotab: looks ok to me at first glance
2018-05-11 13135, 2018
ruaok
do you scrobble via last.fm?
2018-05-11 13142, 2018
Rotab
i scrobble via spotify using the hosts "hack"
2018-05-11 13152, 2018
ruaok
if you go to last.fm's application page you now connect a new method that scrobbles your entire history, directly.
2018-05-11 13146, 2018
Rotab
i dont have last.fm
2018-05-11 13100, 2018
ruaok restarts the api-compat container
2018-05-11 13140, 2018
ruaok
try now.
2018-05-11 13151, 2018
D4RK-PH0ENiX has quit
2018-05-11 13155, 2018
d4rkie joined the channel
2018-05-11 13107, 2018
Rotab
hmm, might be my spotify that is wonky. it seems like it is woking from my phone
2018-05-11 13149, 2018
Darkloke joined the channel
2018-05-11 13115, 2018
Freso
Rotab: Spotify announced a new way of scrobbling that they're rolling out that requires you to link the accounts directly, so I'm guessing (as I think ruaok implied) that they're moving towards Spotify-the-site/company scrobbling rather than you scrobbling via your local Spotify client. The mobile app client will probably move to this soon/eventually too.
«We don't currently support Private Sessions so all listens will appear on your Last.fm profile. If you're using Private Sessions to hide your listening history, then please don't enable the new scrobbler yet.» 🤦
2018-05-11 13142, 2018
Rotab
heh
2018-05-11 13131, 2018
Rotab
looks like the spotify client is now only sending the "now playing" requests
2018-05-11 13108, 2018
D4RK-PH0ENiX joined the channel
2018-05-11 13108, 2018
d4rkie has quit
2018-05-11 13159, 2018
Rotab
so, in order for my listens to show up on listenbrainz, i have to import from last.fm? it wont show up in realtime anymore?
2018-05-11 13103, 2018
ruaok
if you don't use last.fm you don't need to change anything.
2018-05-11 13132, 2018
Rotab
it doesnt work without last.fm though
2018-05-11 13107, 2018
ruaok
I wonder if they just removed that support from the client.
2018-05-11 13110, 2018
ruaok
alastairp: ^^
2018-05-11 13135, 2018
ruaok
is there anything iliekcomputers or I could do to help move your spotify PR forward? now is the perfect time to release it.
2018-05-11 13105, 2018
Freso
Rotab: There will be a tool or a feature on LB itself Soon™ to import listening history from Spotify, so you shouldn't need to use Last.FM as an intermediary.
zas: remember how I was saying that this process will repeat itself until we add more capacity?
2018-05-11 13117, 2018
zas
i found the problem !! bowie's cpu is throttling cause of overheat, cpu fan is prolly dying, it may explain everything, why did i skip this check...
2018-05-11 13140, 2018
ruaok
lol, here begins the next wave. :) :)
2018-05-11 13150, 2018
zas
i'll ask hetzner a check
2018-05-11 13157, 2018
yvanzo
would love to have an issue that explains everything :)
2018-05-11 13101, 2018
zas
ok, asked hetzner about it, we'll prolly have to stop the server if the fan has to be replaced.
2018-05-11 13122, 2018
zas
yvanzo: what do you think about those indexes ?
2018-05-11 13137, 2018
zas
samj1912: around? how is it going with solr cloud deployment ? ask for help if needed
2018-05-11 13158, 2018
ruaok
20 minute downtime. can we go read-only and move everything to queen, zas?
2018-05-11 13129, 2018
zas
hmmm, i think so, but i prefer to wait for bitmap to be around before doing anything
2018-05-11 13103, 2018
zas
wal will prolly fall apart, like last time, and we'll need to resync stuff
2018-05-11 13126, 2018
ruaok
agreed 100%
2018-05-11 13130, 2018
zas
bitmap: ping when you're around, we have to take down bowie for a while
2018-05-11 13137, 2018
Darkloke has quit
2018-05-11 13123, 2018
yvanzo
zas: They might not be used often enough to justify keeping it up.
2018-05-11 13118, 2018
zas
does it make sense to have 635MB of indexes for a table containing 189MB of data ? idx_scan count for this one is only 64289 (in the same time, l_area_area_idx_entity1 was used 48560621856 times)
2018-05-11 13158, 2018
Major_Lurker has quit
2018-05-11 13115, 2018
yvanzo
Probably yes, since the 2nd column of medium_index is a toc that might be split in the index.
2018-05-11 13157, 2018
yvanzo
toc have up to 6 values, thus the index might weigh up to 6 times the table
2018-05-11 13102, 2018
CatQuest
oh man, hardware problems is always the last one you consider
2018-05-11 13135, 2018
ferbncode has quit
2018-05-11 13151, 2018
rsh7 has quit
2018-05-11 13135, 2018
darkhorse has quit
2018-05-11 13114, 2018
iliekcom- has quit
2018-05-11 13151, 2018
akhilesh has quit
2018-05-11 13154, 2018
Major_Lurker joined the channel
2018-05-11 13109, 2018
yvanzo
zas: That little usage, I would agree to drop medium_index_idx.
2018-05-11 13147, 2018
ruaok
also, back in the day when PG was not as refined as today, sometime dropping indexes and re-creating them helped matters.
2018-05-11 13106, 2018
zas
Well, it'd be good to review all indexes and drop unused ones (and perhaps rebuild bloated ones). It would reduce db size and memory freed can be used by something more useful.
2018-05-11 13109, 2018
ruaok
during off peak you can drop and create the index in a transaction and in theory no one should notice.
2018-05-11 13126, 2018
zas
i count 277 indexes with zero usage over the stat gathering period
2018-05-11 13126, 2018
darkhorse joined the channel
2018-05-11 13141, 2018
ruaok
how long is the period?
2018-05-11 13146, 2018
zas
about a week
2018-05-11 13157, 2018
ruaok
ok, anything less would not be representative
2018-05-11 13104, 2018
ruaok
277? wow. quite a lot.
2018-05-11 13114, 2018
zas
but imho we need something over 1 month, to include monthly scripts etc
2018-05-11 13129, 2018
ferbncode joined the channel
2018-05-11 13110, 2018
zas
127.0.0.1 postgres@musicbrainz_db=# SELECT schemaname, relname, indexrelname, idx_scan FROM pg_stat_user_indexes ORDER BY idx_scan;
2018-05-11 13159, 2018
rsh7 joined the channel
2018-05-11 13114, 2018
zas
stats reset was 2018-05-03 08:16:51.133681+00
2018-05-11 13104, 2018
akhilesh joined the channel
2018-05-11 13110, 2018
fred_ joined the channel
2018-05-11 13110, 2018
fred_ has quit
2018-05-11 13109, 2018
ruaok
samj1912: what else needs to be done to get solr into production? are all the deployment bits in place?
2018-05-11 13108, 2018
zas
i started to set up alerts for cpu temps, it is a bit messy, because we have different kinds of machines, and ofc they don't use same lm-sensors config, and for AMD ones it doesn't work at all for now
2018-05-11 13131, 2018
zas
yvanzo: did you read about barman yet (https://www.pgbarman.org/) ? i think we should install it on williams, but it requires a bit of config on pg side too.
2018-05-11 13118, 2018
zas
i think it should be done before any index cleanup job ;)
2018-05-11 13138, 2018
iliekcom- joined the channel
2018-05-11 13126, 2018
samj1912
ruaok: we are figuring out solr cloud but apart from that yup
2018-05-11 13157, 2018
samj1912
I tried setting up a bit of things yesterday, but my internet connectivity is currently limited to mobile data
2018-05-11 13127, 2018
samj1912
I see bowie is acting up again
2018-05-11 13132, 2018
outsidecontext has quit
2018-05-11 13155, 2018
github joined the channel
2018-05-11 13155, 2018
github
[listenbrainz-server] paramsingh opened pull request #406: LB-354: Allow deletion of user accounts (production...allow-user-deletion) https://git.io/vpyJU
2018-05-11 13155, 2018
github has left the channel
2018-05-11 13101, 2018
MajorLurker joined the channel
2018-05-11 13154, 2018
Major_Lurker has quit
2018-05-11 13122, 2018
zas
bitmap: ping
2018-05-11 13158, 2018
bitmap
zas: pong!
2018-05-11 13115, 2018
zas
Hey, good morning ;)
2018-05-11 13159, 2018
zas
the good news: i know what happens to bowie. The bad news: we have to stop it for at least 20 minutes, cpu fan replacement
2018-05-11 13140, 2018
zas
we need to plan a time window for it, and sync with hetzner techs
2018-05-11 13153, 2018
zas
sooner the better ofc
2018-05-11 13124, 2018
zas
can we switch everything to queen for a while ?
2018-05-11 13137, 2018
bitmap
ouch, so perf issues are related to the hardware failure?
2018-05-11 13148, 2018
bitmap
it's possible, will require some careful steps
2018-05-11 13120, 2018
zas
yes, i supposed the fan was dying, hence the intermittent behavior
2018-05-11 13126, 2018
zas
but now i think it died
2018-05-11 13102, 2018
zas
cpu is throttling to keep the temp under 95°
2018-05-11 13106, 2018
zas
(celsius)
2018-05-11 13111, 2018
bitmap
ok, so we should switch to queen asap then
2018-05-11 13141, 2018
zas
can you document steps (or review them if already written) while doing ?
2018-05-11 13128, 2018
bitmap
well, due to the master/slave containers being set up differently, we won't be able to use to slave container on queen for the master. so the easiest way will probably be to stop bowie, copy everything over, and start a new master container on queen
2018-05-11 13136, 2018
zas
i let you prepare steps, then we review, and target a time window, so i can sync with techs
