#metabrainz

is listenbrainz having issue with the listen queue?

im missing listens since wednesday

Rotab: looks ok to me at first glance

do you scrobble via last.fm?

i scrobble via spotify using the hosts "hack"

if you go to last.fm's application page you now connect a new method that scrobbles your entire history, directly.

i dont have last.fm

try now.

hmm, might be my spotify that is wonky. it seems like it is woking from my phone

Rotab: Spotify announced a new way of scrobbling that they're rolling out that requires you to link the accounts directly, so I'm guessing (as I think ruaok implied) that they're moving towards Spotify-the-site/company scrobbling rather than you scrobbling via your local Spotify client. The mobile app client will probably move to this soon/eventually too.

(Or maybe it was a Last.FM announcement.)

Rotab: https://twitter.com/lastfm/status/9945844987246...

«We don't currently support Private Sessions so all listens will appear on your Last.fm profile. If you're using Private Sessions to hide your listening history, then please don't enable the new scrobbler yet.» 🤦

heh

looks like the spotify client is now only sending the "now playing" requests

so, in order for my listens to show up on listenbrainz, i have to import from last.fm? it wont show up in realtime anymore?

if you don't use last.fm you don't need to change anything.

it doesnt work without last.fm though

I wonder if they just removed that support from the client.

alastairp: ^^

is there anything iliekcomputers or I could do to help move your spotify PR forward? now is the perfect time to release it.

Rotab: There will be a tool or a feature on LB itself Soon™ to import listening history from Spotify, so you shouldn't need to use Last.FM as an intermediary.

Freso: alright :)

[musicbrainz-server] yvanzo closed pull request #666: MBS-9704: Skip checking credentials twice for WS (master...mbs-9704-fix-ws2-auth) https://git.io/vp6Yd

yvanzo, bitmap : bowie load is high again (since 18 hours) ... :(

I checked mbdb indexes for bloat, there are few things that may be worth looking at.

medium_index_idx is 635MB for a table of 189MB

btw, i'm vacuuming databases one by one to see if it has any impact, or last time was just "luck".

i had a look at logs found nothing obvious, https://stats.metabrainz.org/d/000000048/hetzne... shows when it started

zas: remember how I was saying that this process will repeat itself until we add more capacity?

i found the problem !! bowie's cpu is throttling cause of overheat, cpu fan is prolly dying, it may explain everything, why did i skip this check...

lol, here begins the next wave. :) :)

i'll ask hetzner a check

would love to have an issue that explains everything :)

ok, asked hetzner about it, we'll prolly have to stop the server if the fan has to be replaced.

yvanzo: what do you think about those indexes ?

samj1912: around? how is it going with solr cloud deployment ? ask for help if needed

20 minute downtime. can we go read-only and move everything to queen, zas?

hmmm, i think so, but i prefer to wait for bitmap to be around before doing anything

wal will prolly fall apart, like last time, and we'll need to resync stuff

agreed 100%

bitmap: ping when you're around, we have to take down bowie for a while

zas: They might not be used often enough to justify keeping it up.

does it make sense to have 635MB of indexes for a table containing 189MB of data ? idx_scan count for this one is only 64289 (in the same time, l_area_area_idx_entity1 was used 48560621856 times)

Probably yes, since the 2nd column of medium_index is a toc that might be split in the index.

toc have up to 6 values, thus the index might weigh up to 6 times the table

oh man, hardware problems is always the last one you consider

zas: That little usage, I would agree to drop medium_index_idx.

also, back in the day when PG was not as refined as today, sometime dropping indexes and re-creating them helped matters.

Well, it'd be good to review all indexes and drop unused ones (and perhaps rebuild bloated ones). It would reduce db size and memory freed can be used by something more useful.

during off peak you can drop and create the index in a transaction and in theory no one should notice.

i count 277 indexes with zero usage over the stat gathering period

how long is the period?

about a week

ok, anything less would not be representative

277? wow. quite a lot.

but imho we need something over 1 month, to include monthly scripts etc

127.0.0.1 postgres@musicbrainz_db=# SELECT schemaname, relname, indexrelname, idx_scan FROM pg_stat_user_indexes ORDER BY idx_scan;

stats reset was 2018-05-03 08:16:51.133681+00

samj1912: what else needs to be done to get solr into production? are all the deployment bits in place?

i started to set up alerts for cpu temps, it is a bit messy, because we have different kinds of machines, and ofc they don't use same lm-sensors config, and for AMD ones it doesn't work at all for now

yvanzo: did you read about barman yet (https://www.pgbarman.org/) ? i think we should install it on williams, but it requires a bit of config on pg side too.

i think it should be done before any index cleanup job ;)

ruaok: we are figuring out solr cloud but apart from that yup

I tried setting up a bit of things yesterday, but my internet connectivity is currently limited to mobile data

I see bowie is acting up again

[listenbrainz-server] paramsingh opened pull request #406: LB-354: Allow deletion of user accounts (production...allow-user-deletion) https://git.io/vpyJU

bitmap: ping

zas: pong!

Hey, good morning ;)

we need to plan a time window for it, and sync with hetzner techs

sooner the better ofc

can we switch everything to queen for a while ?

ouch, so perf issues are related to the hardware failure?

it's possible, will require some careful steps

yes, i supposed the fan was dying, hence the intermittent behavior

but now i think it died

cpu is throttling to keep the temp under 95°

(celsius)

ok, so we should switch to queen asap then

can you document steps (or review them if already written) while doing ?

well, due to the master/slave containers being set up differently, we won't be able to use to slave container on queen for the master. so the easiest way will probably be to stop bowie, copy everything over, and start a new master container on queen

i let you prepare steps, then we review, and target a time window, so i can sync with techs

I'll start a doc